Skip to content

Migrating a PVC between StorageClasses (file to block) without losing data

rsync-based runbook to move an in-use PVC from one StorageClass to another — including pre-migration drill, quiesce window, copy, verify, and cut-over.

You inherit a cluster where the application’s data PVC was originally provisioned on a file-mode CSI (NFS, CephFS, or a generic shared filesystem) “because that’s what the installer’s chart picked.” The application has since grown, the file driver’s per-op latency is now visible in p95 tail, and the workload doesn’t actually need ReadWriteMany — a single pod mounts the PVC. You want to move the data to a block-mode StorageClass like Longhorn or a cloud provider’s block CSI. There is no kubectl move-pvc command. The migration is a tightly-scripted runbook with a quiesce window, an rsync copy, a verification pass, and a single pod restart that cuts over.

What’s happening

A PVC is bound to a specific PersistentVolume, which is bound to a specific CSI driver and StorageClass. The binding is not editable — you cannot change spec.storageClassName on an existing PVC and have the underlying storage migrate. The only paths are: a CSI driver that supports live migration (rare and provider-specific), CSI volume snapshots followed by restore into a different class (only works if both classes are on the same CSI driver — a file-to-block change is usually a different driver), or a copy-and-cut-over runbook using rsync.

The runbook works for any file-system-backed migration: NFS to block, CephFS to RBD, file-mode to block-mode on the same provider, even cross-CSI migrations. It does not work for raw block volumes — those need dd or a CSI-native clone. The window of downtime is “long enough for rsync to do a final incremental pass on already-warm data,” which on most workloads is under a minute even for hundreds of GiB.

The risk is that the writer pod modifies data between the bulk rsync and the cut-over. The mitigation is a two-phase rsync: bulk copy while the pod is running (most of the data, accepts that it’s slightly stale), then quiesce, run an incremental rsync against the now-frozen source, swap PVCs, restart the pod. Verification is row-counts and checksums where the application semantics allow.

How to verify

Before starting, audit the source PVC: size used, access mode, mount points, whether the workload actually requires ReadWriteMany.

NS=production
PVC_SRC=app-data
APP_POD=$(kubectl get pod -n ${NS} -l app=app -o jsonpath='{.items[0].metadata.name}')

# Used vs requested
kubectl exec -n ${NS} ${APP_POD} -- df -h | grep -E "/data|app-data"

# Access mode and current class
kubectl get pvc -n ${NS} ${PVC_SRC} \
  -o jsonpath='{.spec.accessModes}{" / "}{.spec.storageClassName}{"\n"}'

# Number of pods currently mounting it
kubectl get pod -n ${NS} -o json | \
  jq -r --arg p "${PVC_SRC}" \
  '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName == $p) | .metadata.name'

If more than one pod mounts the source PVC, the workload truly needs ReadWriteMany and the migration is a different conversation — block-mode CSI drivers don’t offer it. If exactly one pod, you’re clear to proceed.

The fix

The runbook is six phases: prepare the destination, run the bulk rsync online, schedule the maintenance window, quiesce the application, run the final rsync, swap, restart and verify. Each phase has a rollback.

Phase 1 — prepare the destination PVC

Create the destination PVC at the same size or larger, on the target StorageClass:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data-block
  namespace: production
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: gold-nvme-replicated   # target class
  resources:
    requests:
      storage: 200Gi                       # >= source size
kubectl apply -f app-data-block-pvc.yaml
kubectl get pvc -n production app-data-block
# Should show Bound after WaitForFirstConsumer is triggered below

Phase 2 — bulk rsync while the application is still serving

Run a sidecar pod that mounts both PVCs and copies data. Source is ReadWriteOnce so the pod must be scheduled on the same node — handled by affinity or by accepting that one of the two PVCs is shared-mode for the migration window.

apiVersion: v1
kind: Pod
metadata:
  name: migrate-bulk
  namespace: production
spec:
  restartPolicy: Never
  containers:
    - name: rsync
      image: instrumentisto/rsync-ssh:3.3
      command:
        - sh
        - -c
        - |
          set -e
          echo "Starting bulk rsync at $(date -u)"
          rsync -aHX --info=progress2 --numeric-ids /src/ /dst/
          echo "Bulk rsync done at $(date -u)"
          sleep 60
      volumeMounts:
        - mountPath: /src
          name: src
          readOnly: true
        - mountPath: /dst
          name: dst
  volumes:
    - name: src
      persistentVolumeClaim:
        claimName: app-data       # source: file-mode
    - name: dst
      persistentVolumeClaim:
        claimName: app-data-block # destination: block-mode

For source PVCs that are strictly ReadWriteOnce and mounted by an active pod, you cannot run a second pod with the source mounted unless the source CSI supports concurrent read-only mounts (NFS does; most block drivers do not). If the source is block-mode ReadWriteOnce, you must either: do the entire migration during a maintenance window (skip phase 2 and go straight to phase 4), or take a CSI snapshot of the source, restore the snapshot into a temporary read-only PVC, and rsync from there.

kubectl apply -f migrate-bulk.yaml
kubectl logs -n production migrate-bulk -f

This may take hours for terabyte volumes. Monitor disk throughput on the involved nodes — if the source storage is shared (NFS, CephFS), the rsync will compete with the live application.

Phase 3 — schedule and announce the maintenance window

You need a 2-5 minute window where the application is quiesced. Communicate it via your usual change-management channel. For a database, “quiesced” means writes drain — for a generic application it means the deployment is scaled to zero. For a database-backed application, snapshot before quiesce as a safety net.

Phase 4 — quiesce and final rsync

Scale the workload to zero replicas. Wait for pods to terminate:

kubectl scale -n production deploy/app --replicas=0
kubectl wait -n production --for=delete pod -l app=app --timeout=120s

Now the source PVC is unbound from any running pod. Run an incremental rsync to catch deltas since the bulk copy. The --delete flag ensures files removed from source are also removed from destination:

apiVersion: v1
kind: Pod
metadata:
  name: migrate-final
  namespace: production
spec:
  restartPolicy: Never
  containers:
    - name: rsync
      image: instrumentisto/rsync-ssh:3.3
      command:
        - sh
        - -c
        - |
          set -e
          echo "Final rsync at $(date -u)"
          rsync -aHX --delete --info=progress2 --numeric-ids /src/ /dst/
          echo "Computing checksum manifests"
          (cd /src && find . -type f -exec sha256sum {} \; | sort > /tmp/src.sha256)
          (cd /dst && find . -type f -exec sha256sum {} \; | sort > /tmp/dst.sha256)
          if diff /tmp/src.sha256 /tmp/dst.sha256 > /tmp/diff.txt; then
            echo "MATCH: source and destination identical"
          else
            echo "MISMATCH:"
            head -20 /tmp/diff.txt
            exit 1
          fi
      volumeMounts:
        - mountPath: /src
          name: src
          readOnly: true
        - mountPath: /dst
          name: dst
  volumes:
    - name: src
      persistentVolumeClaim:
        claimName: app-data
    - name: dst
      persistentVolumeClaim:
        claimName: app-data-block
kubectl apply -f migrate-final.yaml
kubectl logs -n production migrate-final -f
# Wait for "MATCH: source and destination identical"

If the checksum diff is non-empty, do not cut over. Investigate the diff — it is usually transient files (lock files, sockets) which are safe to ignore via a refined exclude list, but verify each.

Phase 5 — swap the PVC reference and restart

Edit the deployment to reference the new PVC and scale back up:

kubectl patch deploy -n production app --type=json -p='[
  {"op": "replace", "path": "/spec/template/spec/volumes", "value": [
    {"name": "data", "persistentVolumeClaim": {"claimName": "app-data-block"}}
  ]}
]'

kubectl scale -n production deploy/app --replicas=2
kubectl rollout status -n production deploy/app --timeout=180s

Phase 6 — verify the application

Hit the application endpoints and confirm data is intact:

# Application health
kubectl exec -n production deploy/app -- curl -fsS http://localhost:8080/healthz

# Application-specific data spot check (example: a known row count)
kubectl exec -n production deploy/app -- /usr/local/bin/app-data-check

Keep the source PVC for 7 days as a rollback path. To roll back: scale to zero, patch the deployment back to claimName: app-data, scale up. After the 7-day window, delete the source PVC and the underlying volume reclaims per its policy.

# 7 days later, after confidence
kubectl delete pvc -n production app-data
kubectl delete pod -n production migrate-bulk migrate-final

Operational note: the runbook above is for filesystem-mode PVCs. For raw-block PVCs (the kind used for some database engines or for ZFS pools), the equivalent is dd if=/dev/source of=/dev/dest bs=4M status=progress inside a privileged pod that mounts both PVCs as block devices — and you cannot do that hot, the source must be quiesced first.

Stack Harbor runs PVC migrations as part of Managed Cloud Ops — including the pre-flight audit, the maintenance window coordination, and the rollback window monitoring. If a workload is on the wrong storage tier and you need help moving it, scope the migration with our deployment team.