Copy PVC files locally without a dedicated pod

We have a PVC that is written to by many k8s cronjobs. We'd like to periodically copy this data locally. Ordinarily one would use kubectl cp to do such tasks, but since there's no actively running pod with the PVC mounted, this is not possible.

We've been using a modified version of this gist script kubectl-run-with-pvc.sh to create a temporary pod (running sleep 300) and then kubectl cp from this temporary pod to get the PVC data. This "works" but seems kludgey.

Is there a more elegant way to achieve this?

May I propose to use NFS instead of PVC?

If you do not have an NFS server, you can run one inside k8s cluster using this image @ https://hub.docker.com/r/itsthenetwork/nfs-server-alpine. The in-cluster NFS server indeed uses PVC for its storage but your pods should mount using NFS instead.

Meaning, from pod --> PVC to pod --> NFS --> PVC.

Here is the script that I quite often use to created dedicated in-cluster NFS servers (just modify the variables at the top of the script accordingly):

export NFS_NAME="nfs-share"
export NFS_SIZE="10Gi"
export NFS_SERVER_IMAGE="itsthenetwork/nfs-server-alpine:latest"
export STORAGE_CLASS="thin-disk"
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: ${NFS_NAME}
labels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
spec:
ports:
- name: tcp-2049
port: 2049
protocol: TCP
- name: udp-111
port: 111
protocol: UDP
selector:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
name: ${NFS_NAME}
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: ${NFS_SIZE}
storageClassName: ${STORAGE_CLASS}
volumeMode: Filesystem
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ${NFS_NAME}
labels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
template:
metadata:
labels:
app.kubernetes.io/name: nfs-server
app.kubernetes.io/instance: ${NFS_NAME}
spec:
containers:
- name: nfs-server
image: ${NFS_SERVER_IMAGE}
ports:
- containerPort: 2049
name: tcp
- containerPort: 111
name: udp
securityContext:
privileged: true
env:
- name: SHARED_DIRECTORY
value: /nfsshare
volumeMounts:
- name: pvc
mountPath: /nfsshare
volumes:
- name: pvc
persistentVolumeClaim:
claimName: ${NFS_NAME}
EOF

To mount the NFS inside your pod, first you need to get its service IP:

export NFS_NAME="nfs-share"
export NFS_IP=$(kubectl get --template={{.spec.clusterIP}} service/$NFS_NAME)

Then:

  containers:
- name: apache
image: apache
volumeMounts:
- mountPath: /var/www/html/
name: nfs-vol
subPath: html
volumes:
- name: nfs-vol
nfs: 
server: $NFS_IP
path: /

This way, not only you have a permanently running pod (which is the NFS server pod) for you to do the kubectl cp, you also have the opportunity to mount the same NFS volume to multiple pods concurrently since NFS does not have the single-mount restriction that most PVC drivers have.

N.B: I have been using this in-cluster NFS server technique for almost 5 years with no issues supporting production-grade traffic volumes.

You can use Restic to backup your filesystem and restore it wherever you need to.

For creating backup, you can write a job like this:
first create the init job:

apiVersion: batch/v1
kind: Job
metadata:
name: restic-init-job
spec:
template:
spec:
containers:
- image: restic/restic:0.14.0
name: restic
env:
- name: RESTIC_PASSWORD
value: "123"
command: ["restic"]
args: ["init", "--repo", "/tmp/backup"]
volumeMounts:
- name: restic-backup
mountPath: /tmp/backup/
restartPolicy: OnFailure
volumes:
- name: restic-backup
persistentVolumeClaim:
claimName: restic-claim

then backup:

apiVersion: batch/v1
kind: Job
metadata:
name: restic-backup
spec:
template:
spec:
containers:
- image: restic/restic:0.14.0
name: restic
env:
- name: RESTIC_PASSWORD
value: "123"
command: ["restic"]
args: ["--repo", "/tmp/backup", "backup", "/bitnami/postgresql"]
volumeMounts:
- name: html
mountPath: /bitnami/postgresql
readOnly: true
- name: restic-backup
mountPath: /tmp/backup/
restartPolicy: OnFailure
volumes:
- name: html
persistentVolumeClaim:
claimName: data-postgres-0
- name: restic-backup
persistentVolumeClaim:
claimName: restic-claim

and for restoring:

apiVersion: batch/v1
kind: Job
metadata:
name: restic-restore
spec:
template:
spec:
containers:
- image: restic/restic:0.14.0
name: restic
env:
- name: RESTIC_PASSWORD
value: "123"
command: ["restic"]
args: ["--repo", "/tmp/backup", "restore", "latest", "--target", "/"]
volumeMounts:
- name: html
mountPath: /bitnami/postgresql
- name: restic-backup
mountPath: /tmp/backup/
restartPolicy: OnFailure
volumes:
- name: html
persistentVolumeClaim:
claimName: data-postgres-0
- name: restic-backup
persistentVolumeClaim:
claimName: restic-claim