06 Sep 2025
By Alex
Hello! I'm Alex, the person with a rack in his living room. I'm the "Kubernetes Person" at La Sentinelle, where I work on everything related to Cloud Native and the vast landscape. I like playing with my homelab (running Kubernetes of course) and am an advocate for FOSS adoption in Mauritius. My personal website and a lot of the services running my life are hosted on Kubernetes (shocker) in my homelab (shocker).
I have a homelab of a few micro PCs running K3s. The other day, I had to do some regular maintenance and delete a few persistent volume claims to free up some space. It wasn't anything super complicated or which risked breaking my whole cluster, or so I thought.
The command to delete all the PVCs in a namespace is kubectl delete pvc --all -n namespace. And that's the command I thought I had run, except I had missed a crucial c and ran kubectl delete pv --all -n namespace.
For the uninitiated, PV stands for persistent volumes, the "base" which persistent volume claims sit on. They are also not namespace-scoped. So, in my stroke of genius, I had deleted all the persistent volumes across the entire cluster!
When I saw some 20 deleted persistentvolume <pvname> scroll past, I realised I had indeed messed up. Thankfully, the PVs were still in use, so were stuck in the terminating phase. This meant there was no immediate data loss and my applications were still fine.
At this point, it's relevant to mention that I use Longhorn as storage solution for my home cluster.
My first thought was to see if I could un-delete the PVCs as they hadn't actually vanished yet. However, save for manually editing the etcd database, which is risky and definitely not recommended, there isn't really a kubectl undelete button.
While the PVs were still present on the cluster, I went over to the Longhorn UI to make a fresh backup. I backup all volumes every night anyway, but at least now I could be sure I wouldn't lose any data since the last snapshot.
Once the backups had been created, I "released" the PVs, allowing them to be deleted. Not having data is, unsurprisingly, a problem for my applications.
Thankfully, Longhorn was able to recover the backups very fast and, in less than 30 minutes, most of my applications were back up, as though nothing had changed. The only one which took a while was my Immich volume, with some 50GB of data on it.
Hmmm, double check the commands you run which involve deleting stuff. Sort of like rm -rf or drop table :D
Oh, and make backups.
