I’ve been running Proxmox in my home lab ever since I built my first virtual machine host around three years ago. Over that time, there have been several new versions of Proxmox that have come out and of course, I’ve kept with the times and updated when the new versions have become available. Looking back, upgrading from version 7 or version 8 I seem to remember as being relatively simple. Grab the latest updates for the current version you’re running, update APT to point to the new version, queue the updates and then run a full upgrade. Give that some time to bake and boom, you have a flashy new version of Proxmox.
Now let’s not skip over something important regarding this process. In fact, I’m almost certain, one of the very first things the documentation says before you upgrade is, make sure you have backups. Now, let me share one of those, do as I say not as I do moments. Being in the IT world, that’s something you hear all the time but being young in my IT journey, I didn’t listen to that advice. That was until I suffered a huge data loss event. Quick story time.
When I was in middle school I got a Western Digital MyBook Live NAS. I thought this was the most awesome thing in the world because I could access my files from anywhere on the network. Fast forward many years, I had the device setup so I could access it through WD’s online tools. Everything was great but the device was severely out of date and while I shouldn’t have had it connected to the internet anymore, that simple fact resulted in me losing most of that data when those devices were targeted and remotely wiped. Western Digital did try to make it right and my device was sent off for professional data recovery but still to this day, I have no idea the total impact of what I have lost.
Now, from this experience, I upgraded my hardware to a Synology NAS running two disks in RAID. Before you say it, no, RAID is not a backup. It does however add an extra layer of protection against data loss. But I actually have my NAS setup to store backups of its data to a cloud service every night. Now when it comes to Proxmox, I have all my VM’s that I care about setup to backup to the NAS. I can’t tell you how many times this has saved me.
So getting back to the task at hand, upgrading Proxmox to version 9. I started by getting my current VM hosts all up to date and then had some additional updates to make per the documentation. Once that was all completed, I was finally ready to run the full upgrade. Perfect I thought, this should be a breeze like last time. Or so I thought. Yeah… let’s just say that was the beginning of a huge mess that would take a few days to sort out. While one of my nodes came back up, the other was stuck and at time, the monitor I had wasn’t playing nice so I had no idea what was going on. Meanwhile, on the node that was working, none of my VM’s would start.
After doing some research, it turns out that because I had my two servers in a cluster and the one node was down, it meant that the working node was waiting for the other before it would start the VMs. After running some commands that admittedly, I had no idea what they did, things started working again. Now is a good time to point out that it’s usually not wise to just copy and paste commands and just run them when you don’t know what they do. There may be some foreshadowing here. So what I had in fact done was told the node that it wasn’t connected to the cluster anymore and that’s why stuff started booting again. In the grand scheme of things, this isn’t really a problem so really, no harm no foul.
So after getting at least one of my nodes back up, it was time to troubleshoot my other node. After changing out some hardware, I was finally able to see what was going on with the other node. It was booting into the BIOS. Now I thought that was a bit odd and after verifying that my disks were in fact showing up, it turns out that the boot order of disks got messed up somehow. After making a simple change, that node started booting as expected. After finally getting that node to boot and being able to get into the web UI, I ran into the same problem where none of the VMs would boot. After performing the same fix, I was back in business.
Great, I have both nodes back up and running and they are both on version 9 of Proxmox. One slight problem, all the VMs from the opposite node I was on were showing up as unknown. At the time, I didn’t realize what that command I ran had done. After I figured that out, I set out to start cleaning things up. To ChatGPT I went. After prompting it a bit, I was presented with some commands to run. As one does, I pasted in the command and boom, all the resources from the other node were gone. That’s when I noticed something, so were the VMs from the current node. Yeah, I had nuked everything. While the VMs and their data was still there, I had removed all the associated records in the dashboard so I could no longer see them anymore.
After a bit of fighting back and forth with ChatGPT, I elected to just clear the node and wipe out all the VMs from it so I could start fresh. Since I was blowing everything out, I did take it as an opportunity to rename one of my storage pools so that was a win. Now while clearing everything out was a bit of a pain, at least I had proper backups of everything so I only lost one VM and it was for some development work that I hadn’t touched in years. So at the end of the day, everything that was important to me was backed up and was easily able to be restored to get me back up and running on both nodes. Looking back on this upgrade experience, I learned a few things. First, backups are great and save your bacon when things go south. Second, it’s always good to understand what a command is doing to some degree before you run it. In the end, while it took some time, I’m back up and running and that’s all I really care about.