Hey everyone, time once again for everyone to gather around the campfire for me to wax poetic about what dumb shit broke today while the machinery actually doing the fixing churns and grinds in the background.
First, we need to head off any panic: No data was lost. We had working, accessible backups the entire time going back to May 1st.
So, what happened?
Outages
First, you may have noted occasional error pages popping up here and there throughout the past few days, all screaming about skins not being configured correctly. This was brought to my attention by the VRWiki operator.
Operating on a total of 2 hours of sleep, I decided my best course of action was to run updates on the entire wikifarm and try to force in the latest MediaWiki version with fresh pulls of skins and extensions.
This was a mistake.
The custom farm scripts made an error and pulled in MediaWiki 1.46 – Which is in a release track branch for some fucking reason despite being in beta. MediaWiki 1.46 also changes how configuration files are loaded, which meant my hacky PHP that pulls default values from MediaWiki just exploded, and fed bad data into the wikifarm scripts, which also exploded and spat out configuration files with no skins installed. It then ran a full database upgrade on all wikis. Which all exploded with errors about database structure issues.
So, naturally, I panicked.
I shut down all the wikis, shut down the database server, and pulled a hot backup (copied the database files to another folder rather than wait for MySQLDump to do its thing). I then tried to recover the databases from the offsite backups, but then realized there was a simpler solution:
Just run MediaWiki 1.46-beta. It goes into release in less than a month.
Idiot.
Resolution
So, wikis are all getting updated to MediaWiki 1.46-beta and should be coming back online shortly.
I will then lock the automatic updates to a specific minor version release branch (such as REL1_46) and only switch after we get notified of a new release instead of trying to get cold logic to figure out the correct version to pull from WikiMedia’s Gerrit.
Let me know if anything blows up by either commenting below or contacting me on Discord through the usual channels.
Leave a Reply