Database maintenance

[mark]

Hi all!

Today we started getting some alerts on the database, so I'm going to do some maintenance to verify the health and wellness of the machine and get it back to a non-alerting status.

I should be able to do this without any downtime, but just in case, you might want to make sure to use your favorite text editor to save a copy of any long entries or comments you're working on.

Once I've got things sorted out, I'll update this with more details for the technically curious.

Update [4:50PM PST]: sb-db06 (the slave) has been rebooted and is recovering, I'm doing system updates on it since the problem looks like a kernel bug (it struck both databases at the same time). Next: master failover then recover the other database.

Update [5:05PM PST]: I'm doing what we call a "master failover" now. This means I'm shifting all traffic from the database that was active (sb-db05) to the spare database (sb-db06). I have to shut off "extra" services like imports, feeds, and searches while this happens.

Update [5:30PM PST]: Well, that was unexpectedly bumpy. Sorry for that. There should be no further bumping, as we're now on the spare database so I can take maintenance on the original master.

Update [6:20PM PST]: If you had userpics not loading, they should be back to normal.

Comments

[denise]

Do you mean your icons, or pictures you've emailed into the site to add to entries?

Either way, try clearing your browser cache (see GENERAL: How do I clear my browser's cache or cookies? from the FAQ for instructions) and reload the page, to see if that clears it up. Then, if that doesn't work, if it's pictures in entries you're having trouble with, make an entry with a picture public so I can take a look and see what's going on, and if it's icons, let me know the keyword of the icon(s) that isn't displaying for you.

[idamus]

It a photo, the icons are fine, I've checked further back and it's only the last couple of posts with pics that have issues
I upload pics by copying their code from the LJ mirror post, this has always worked best for me
I'll open the post for a while

I clear my cache/browser history daily, it didn't fix anything

[denise]

Aha! I took a look at the post you unlocked, and it's linking to http://ic.pics.livejournal.com/idamus/16498843/34664/34664_900.jpg (which is hosted in LiveJournal's photo hosting). Trying to load that image gave me an "access denied" message. So, what's happening there is, you uploaded the photo to LJ with some kind of security on it (probably "friends only"), and the people who are commenting about the photo are on your LJ friends list. I can't see it, because I'm not on the friends list of your LJ account. (You probably can't see it because you aren't logged in to your LJ account.)

To fix that, you'll need to make the photo public on your LJ account (and probably change your settings so that photos uploaded aren't automatically friends-locked), or (if you don't want the photo to be public) also upload it to Dreamwidth's image hosting service instead of loading it from LJ. (instructions here.)

[idamus]

Aha, thanks, guess I missed a change over there or something, 'cause this has always worked in the past, since my friends here are also friends over there it shouldn't be a problem.
At least I know what to tell them if they see a blank space now

[ruriair]

m(o_o)m

Driveby comment but I had no idea you guys had an in-site image hosting service already.

And that you can manage the files unlike on tumblr where things exist forever.

[denise]

It was announced in news posts a while back! It's emailed-in-only for now, but we've got things like in-browser upload and gallery display on the To Be Done list (once we finish all the other backend stuff we need to get done auuugh why is everything always Dependency Hell).