You inherit a cluster where the application’s data PVC was originally provisioned on a file-mode CSI (NFS, CephFS, or a generic shared filesystem) “because that’s what the installer’s chart picked.” The application has since grown, the file driver’s per-op latency is now visible in p95 tail, and the workload doesn’t actually need ReadWriteMany — a single pod mounts the PVC. You want to move the data to a block-mode StorageClass like Longhorn or a cloud provider’s block CSI. There is no kubectl move-pvc command. The migration is a tightly-scripted runbook with a quiesce window, an rsync copy, a verification pass, and a single pod restart that cuts over.
What’s happening
A PVC is bound to a specific PersistentVolume, which is bound to a specific CSI driver and StorageClass. The binding is not editable — you cannot change spec.storageClassName on an existing PVC and have the underlying storage migrate. The only paths are: a CSI driver that supports live migration (rare and provider-specific), CSI volume snapshots followed by restore into a different class (only works if both classes are on the same CSI driver — a file-to-block change is usually a different driver), or a copy-and-cut-over runbook using rsync.
The runbook works for any file-system-backed migration: NFS to block, CephFS to RBD, file-mode to block-mode on the same provider, even cross-CSI migrations. It does not work for raw block volumes — those need dd or a CSI-native clone. The window of downtime is “long enough for rsync to do a final incremental pass on already-warm data,” which on most workloads is under a minute even for hundreds of GiB.
The risk is that the writer pod modifies data between the bulk rsync and the cut-over. The mitigation is a two-phase rsync: bulk copy while the pod is running (most of the data, accepts that it’s slightly stale), then quiesce, run an incremental rsync against the now-frozen source, swap PVCs, restart the pod. Verification is row-counts and checksums where the application semantics allow.
How to verify
Before starting, audit the source PVC: size used, access mode, mount points, whether the workload actually requires ReadWriteMany.
NS=production
PVC_SRC=app-data
APP_POD=$(kubectl get pod -n ${NS} -l app=app -o jsonpath='{.items[0].metadata.name}')
# Used vs requested
kubectl exec -n ${NS} ${APP_POD} -- df -h | grep -E "/data|app-data"
# Access mode and current class
kubectl get pvc -n ${NS} ${PVC_SRC} \
-o jsonpath='{.spec.accessModes}{" / "}{.spec.storageClassName}{"\n"}'
# Number of pods currently mounting it
kubectl get pod -n ${NS} -o json | \
jq -r --arg p "${PVC_SRC}" \
'.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName == $p) | .metadata.name'
If more than one pod mounts the source PVC, the workload truly needs ReadWriteMany and the migration is a different conversation — block-mode CSI drivers don’t offer it. If exactly one pod, you’re clear to proceed.
The fix
The runbook is six phases: prepare the destination, run the bulk rsync online, schedule the maintenance window, quiesce the application, run the final rsync, swap, restart and verify. Each phase has a rollback.
Phase 1 — prepare the destination PVC
Create the destination PVC at the same size or larger, on the target StorageClass:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-data-block
namespace: production
spec:
accessModes: [ReadWriteOnce]
storageClassName: gold-nvme-replicated # target class
resources:
requests:
storage: 200Gi # >= source size
kubectl apply -f app-data-block-pvc.yaml
kubectl get pvc -n production app-data-block
# Should show Bound after WaitForFirstConsumer is triggered below
Phase 2 — bulk rsync while the application is still serving
Run a sidecar pod that mounts both PVCs and copies data. Source is ReadWriteOnce so the pod must be scheduled on the same node — handled by affinity or by accepting that one of the two PVCs is shared-mode for the migration window.
apiVersion: v1
kind: Pod
metadata:
name: migrate-bulk
namespace: production
spec:
restartPolicy: Never
containers:
- name: rsync
image: instrumentisto/rsync-ssh:3.3
command:
- sh
- -c
- |
set -e
echo "Starting bulk rsync at $(date -u)"
rsync -aHX --info=progress2 --numeric-ids /src/ /dst/
echo "Bulk rsync done at $(date -u)"
sleep 60
volumeMounts:
- mountPath: /src
name: src
readOnly: true
- mountPath: /dst
name: dst
volumes:
- name: src
persistentVolumeClaim:
claimName: app-data # source: file-mode
- name: dst
persistentVolumeClaim:
claimName: app-data-block # destination: block-mode
For source PVCs that are strictly ReadWriteOnce and mounted by an active pod, you cannot run a second pod with the source mounted unless the source CSI supports concurrent read-only mounts (NFS does; most block drivers do not). If the source is block-mode ReadWriteOnce, you must either: do the entire migration during a maintenance window (skip phase 2 and go straight to phase 4), or take a CSI snapshot of the source, restore the snapshot into a temporary read-only PVC, and rsync from there.
kubectl apply -f migrate-bulk.yaml
kubectl logs -n production migrate-bulk -f
This may take hours for terabyte volumes. Monitor disk throughput on the involved nodes — if the source storage is shared (NFS, CephFS), the rsync will compete with the live application.
Phase 3 — schedule and announce the maintenance window
You need a 2-5 minute window where the application is quiesced. Communicate it via your usual change-management channel. For a database, “quiesced” means writes drain — for a generic application it means the deployment is scaled to zero. For a database-backed application, snapshot before quiesce as a safety net.
Phase 4 — quiesce and final rsync
Scale the workload to zero replicas. Wait for pods to terminate:
kubectl scale -n production deploy/app --replicas=0
kubectl wait -n production --for=delete pod -l app=app --timeout=120s
Now the source PVC is unbound from any running pod. Run an incremental rsync to catch deltas since the bulk copy. The --delete flag ensures files removed from source are also removed from destination:
apiVersion: v1
kind: Pod
metadata:
name: migrate-final
namespace: production
spec:
restartPolicy: Never
containers:
- name: rsync
image: instrumentisto/rsync-ssh:3.3
command:
- sh
- -c
- |
set -e
echo "Final rsync at $(date -u)"
rsync -aHX --delete --info=progress2 --numeric-ids /src/ /dst/
echo "Computing checksum manifests"
(cd /src && find . -type f -exec sha256sum {} \; | sort > /tmp/src.sha256)
(cd /dst && find . -type f -exec sha256sum {} \; | sort > /tmp/dst.sha256)
if diff /tmp/src.sha256 /tmp/dst.sha256 > /tmp/diff.txt; then
echo "MATCH: source and destination identical"
else
echo "MISMATCH:"
head -20 /tmp/diff.txt
exit 1
fi
volumeMounts:
- mountPath: /src
name: src
readOnly: true
- mountPath: /dst
name: dst
volumes:
- name: src
persistentVolumeClaim:
claimName: app-data
- name: dst
persistentVolumeClaim:
claimName: app-data-block
kubectl apply -f migrate-final.yaml
kubectl logs -n production migrate-final -f
# Wait for "MATCH: source and destination identical"
If the checksum diff is non-empty, do not cut over. Investigate the diff — it is usually transient files (lock files, sockets) which are safe to ignore via a refined exclude list, but verify each.
Phase 5 — swap the PVC reference and restart
Edit the deployment to reference the new PVC and scale back up:
kubectl patch deploy -n production app --type=json -p='[
{"op": "replace", "path": "/spec/template/spec/volumes", "value": [
{"name": "data", "persistentVolumeClaim": {"claimName": "app-data-block"}}
]}
]'
kubectl scale -n production deploy/app --replicas=2
kubectl rollout status -n production deploy/app --timeout=180s
Phase 6 — verify the application
Hit the application endpoints and confirm data is intact:
# Application health
kubectl exec -n production deploy/app -- curl -fsS http://localhost:8080/healthz
# Application-specific data spot check (example: a known row count)
kubectl exec -n production deploy/app -- /usr/local/bin/app-data-check
Keep the source PVC for 7 days as a rollback path. To roll back: scale to zero, patch the deployment back to claimName: app-data, scale up. After the 7-day window, delete the source PVC and the underlying volume reclaims per its policy.
# 7 days later, after confidence
kubectl delete pvc -n production app-data
kubectl delete pod -n production migrate-bulk migrate-final
Operational note: the runbook above is for filesystem-mode PVCs. For raw-block PVCs (the kind used for some database engines or for ZFS pools), the equivalent is dd if=/dev/source of=/dev/dest bs=4M status=progress inside a privileged pod that mounts both PVCs as block devices — and you cannot do that hot, the source must be quiesced first.
Stack Harbor runs PVC migrations as part of Managed Cloud Ops — including the pre-flight audit, the maintenance window coordination, and the rollback window monitoring. If a workload is on the wrong storage tier and you need help moving it, scope the migration with our deployment team.