The Context

With container technologies likes Docker and Kubernetes, Pod and Container are ephemeral. All data are lost after restart. The way to maintain application data after restart is to attach a volume to the pod.
This doesn't address 1 thing however: the volume need to reside on a storage backend and the volume will inherit the constraint of the storage backend.
Some of those constraints are:

• High availability and resilience: The data may be deleted or be unavailable if the storage itself is removed or decommissioned. This is the case for local on prem servers for examples.
• Speed and price: some storage are either too slow or too pricey for some given use case. NFS for example is a very common network storage but is not able to sustain high troughput scenarios.

A Possible Approach

The approach described here is to setuup a dynamic backup and restore at directly in the lifecycle of the Pod. The technology supporting this workflow is Restic.

Here is how the workflow works:

• Init Container: The pod is associated to an Init container. On startup, the init container is checking if the data is available locally or not. If the data is not available on the designated location, it will restore the snapshot for the backup storage. (for example a storage like S3)
• Backup Container: A backup container will run in parallel and regularly create a new snapshot data and send the backup storage

Use Cases and Limitation

This patten can be useful for some cases:

• On premise or home server: for application running in an environment where the storage is not by nature fully resilient, this pattern give an extra level of safety
• Dyanamic infrastructure: for infrastructure that are design to be completely decomissioned and re-created frequently, this can be a way to automate the restoration of application data. This is the case for example of GitOps workflow where the infrastructure itself can be re-created.
• Tests: having an ephemeral storage that restores a verison of a snapshots can be useful for a dynamic test environment. Where the environment is dynamically started with the right tests data.
• Separation between runtime storage and backup storage: The runtime storage can be closer to the application whereas the backup storage can be cheaper and slower. This can be the example where you may want the application to store it's data on a local block storage like AWS EBS. But the backup to be on a more resilient AWS S3.

There are some contraints and limitations that have to be taken into accounts for these kind of workflows. If the local application data is completely ephemeral, or in case of loss of the local data infrastructure, the data loss will depend on the backup frequency. This kind of use can may not be fully implemented depending on the update frequency of the data and on the criticity.

Implementation Example

The following tool is an example of implementation of this workflow: https://github.com/devopsplaybook-io/container-utils/

An exemple of Kubernetes definition can be:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-application
  labels:
    app: my-application
spec:
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
    spec:
      containers:
        - image: my-application
          name: my-application
          volumeMounts:
            - mountPath: /data
              name: pod-volume
        - name: backup
          image: restic/restic:latest
          command: ["sh", "-c"]
          args:
            - wget -O /tmp/container-backup.sh https://raw.githubusercontent.com/devopsplaybook-io/container-utils/init/container-backup.sh && chmod +x /tmp/container-backup.sh && /tmp/container-backup.sh
          volumeMounts:
            - mountPath: /data
              name: pod-volume
          env:
            - name: BACKUP_FOLDER
              value: "/data"
            - name: BACKUP_RESTIC_REPO
              value: "... ..."
            - name: RESTIC_PASSWORD
              value: "... ..."
            - name: BACKUP_DO_PROCESS
              value: "Y"
            - name: BACKUP_DO_START_DELAY
              value: "10800"
            - name: BACKUP_DO_LOOP_FREQUENCY
              value: "10800"
      initContainers:
        - name: init
          image: restic/restic:latest
          command: ["sh", "-c"]
          args:
            - "wget -O /tmp/container-backup.sh https://raw.githubusercontent.com/devopsplaybook-io/container-utils/main/container-backup.sh && chmod +x /tmp/container-backup.sh && /tmp/container-backup.sh"
          volumeMounts:
            - mountPath: /data
              name: pod-volume
          env:
            - name: BACKUP_FOLDER
              value: "/data"
            - name: BACKUP_RESTIC_REPO
              value: "... ..."
            - name: RESTIC_PASSWORD
              value: "... ..."
            - name: BACKUP_DO_RESTORE
              value: "Y"
      volumes:
        - name: pod-volume
          emptyDir: {}