[Complete] Website and App maintenance!
-
The weather should have been our first clue. The morning we were scheduled to begin the final segment of our major update was colder than the norm for the season. Gusts of wind whipping every which way were spinning the fall leaves around in miniature twisters up and down the streets as we made our way into the office.
"Good morning Manager-san!"
"Good morning editor-kun, ready for the last push through this update?"
"Of course sir!"
Famous last words as they say, for what would happen later that day would forever change the way we would view the stories we so proudly publish. -
J jpmspeedbbm referenced this topic on
-
@jpmspeedbbm The worst part is TWO scheduled sections of Hookup Hickups for today have been delayed... and I'm going to be gone for almost a week after today. ARGH! Have to set up laptop for reading...
-
Noooooo I need my daily fix ;_;
-
@SomeOldGuy you could always delay your trip? Sorry everyone we will be putting a hold on all outbound flights until we can receive proper updates on our stories from J-novel.
-
If this was server maintenance, couldn't jnovel have updated half of the servers while keeping the other half online. Then once the maintenance was done with the first half and confirmed to be working, switch to upgrading the last half?
Doing it that way should prevent any down time.
-
@SomeOldGuy - Big deal. I went 8.5 days without power, water, internet or cell phone (apps) service in July after hurricane Beryl ravaged the Texas Gulf Coast. That meant no JNC, Crunchy Roll, Hidive, Netflix, Manga Park, Anime News Network, Reddit, Fandom, You Tube, etc. If this old man can do it, so can you so hang in there Bro.
-
@MryNightmare going into withdrawal ðŸ˜ðŸ˜ LET ME INNNN
-
@karasutengu Hey, as an old guy, I avoid places like that. Actually, I'd have lost my job staying off-line that long. Of course, I'm supposed to be retired...
-
@trutherford2388 that is not generally how servers work. It is rare for infrastructure to be entirely mutually interchangeable servers. Even more so if they are deploying a software or configuration refactor that is not backwards compatible, downtime is way better than data corruption caused by running incompatible software.
-
@Flan-Latte we need a support group...
-
@trutherford2388 ideally that would be the best. but you know some stuff just cannot be partially updated like transactional thingy unless it can be disable or something first. I don't know what they are updating tho, however it could be really big backend stuff that it need to be down to simplify what already complicated work.
-
@SomeOldGuy Hey, aren't all we old codgers supposed to be retired. I think somebody forgot to tell us to stop working...
-
Ok... Was purposefully trying npt to step into this; but, yes, there are multiple ways thus could probably been avoided if they had just known 'X'... But they didn't; and honestly I'm still pretty Ok with my experience overall... though my withdrawal symptoms have just not set deep enough... yet :-P
-
Most productive day in years... It was awful.
-
@Thomas-Smith-yjw56xl Codeine and whiskey will help your withdrawal symptoms. Trust me...
-
@shyevsa They're switching to an entirely new (and hopefully better) database backend to support all the things they want to be able to do with tags and scheduling, so it's not something they would have been able to leave the site up for.
-
It would be pretty simple to keep things up if they were only doing something like rebooting systems. System resilience is a thing, after all.
But that's not what they are doing.
It's possible they could have kept some kind of "legacy" version of existing systems up, but that's a whole headache in itself and it makes any kind of fail back super messy. Much better to bite the bullet and shut everything down.
-
@afdsfafa I've only ever worked at 1 company that did not run backup servers and they paid a heavy financial cost and a hit to their reputation when they tried upgrading their servers and caused a crash. Our entire VoIP system and CMS (Customer Management Software) was down. There was no data updates to the DBs (DataBases) and it took 1 week to get all the systems fully operational and about another month for the data from the week long outage to finally be updated.
Every other company I worked at has had backup servers. The company uses server B to upgrade or do maintenance, test all the changes in house before flipping the servers from customers using Server A to Server B and just disconnect Server A from customer usage for about 1-2 days before doing the same to Server A as what was done to Server B.
Also, industry standard for any web based company is to have redundancy servers not only setup, but also physically located several miles apart. (When the east coast lost internet for about a week due to power outages from a hurricane, big tech companies switched to having servers setup something like Server A in Texas and Server B in the east coast somewhere to account for natural disasters.)
Long story short, only one of those starter companies should have a Server setup without any form of redundancy in place to prevent outages.
-
@karasutengu ... might I suggest a combination of Starlink & EcoFlow power solutions.
I'm not sure if I could make it otherwise. -
I don't mind the outage honestly, even the extended no-ETA one. However, some feedback:
- Given it's a long (and apparently, risky) maintenance, more than the one day advanced notice would have been good
- Posting the notice of upcoming maintenance on the main site and in the app would have been ideal (only found out what was going on after things didn't work)
- A better maintenance page on the main site, with status updates (it wasn't immediately obvious to go here for information)
- Show a maintenance page in the app (right now it just returns errors)
- Perhaps roll back if things are not going as expected
- It's usually industry standard that risky things are done off-hours, even though it means nights and weekends
Again, not bothered, just observations from someone that has been doing this for a while.
