Database maintenance

[mark]

Hi all!

Today we started getting some alerts on the database, so I'm going to do some maintenance to verify the health and wellness of the machine and get it back to a non-alerting status.

I should be able to do this without any downtime, but just in case, you might want to make sure to use your favorite text editor to save a copy of any long entries or comments you're working on.

Once I've got things sorted out, I'll update this with more details for the technically curious.

Update [4:50PM PST]: sb-db06 (the slave) has been rebooted and is recovering, I'm doing system updates on it since the problem looks like a kernel bug (it struck both databases at the same time). Next: master failover then recover the other database.

Update [5:05PM PST]: I'm doing what we call a "master failover" now. This means I'm shifting all traffic from the database that was active (sb-db05) to the spare database (sb-db06). I have to shut off "extra" services like imports, feeds, and searches while this happens.

Update [5:30PM PST]: Well, that was unexpectedly bumpy. Sorry for that. There should be no further bumping, as we're now on the spare database so I can take maintenance on the original master.

Update [6:20PM PST]: If you had userpics not loading, they should be back to normal.

Comments

[jjhunter]

*waves pom-poms*

[juliet316]

Thanks for the updates!

ETA: A third of my icons aren't showing up.

[ai]

Just as a heads up, I'm getting strange errors when it comes to uploading user icons on multiple accounts:

Unable to contact storage server. Your picture has not been saved. at /home/dw/current/cgi-bin/LJ/Userpic.pm line 746

Failed to open MogileFS tempfile at /home/dw/production/htdocs/editicons.bml line 924, line 55.

[queenofwinter]

It looks like comment notifications aren't working, by the way?

[mark]

It should be back now. Sorry for the trouble!

[denise]

Should be fixed now!

[mark]

Yeah, I think you hit the page right when I was moving MogileFS (our userpic storage system). Anyway, I cleared the caches and you should see your pictures again. Sorry!

[mark]

I just brought email back up, can you wait 5-10 minutes and let me know if you're still not getting them?

[decepticon_mistress]

Thanks for everything that you do!

[kore]

Thank you for the hard work and the very clear communication!

[juliet316]

Testing, one two three testing...

Edit: Yep working now, thanks.

[queenofwinter]

They are coming through now! I received a bunch at once and now they seem to be back to normal.

[mark]

Yay, thanks for letting me know!

[veritas_poet]

Thanks for all of your work on it.

[denise]

Should be going out properly now. (Sorry, I mentioned the notifs delay on our Twitter status account, but couldn't add it here since Mark was the one who posted this entry and he was busy doing the fixes!)

[ai]

Thank you very much for helping with the issue!

[idamus]

I can't see the pics in my journal, but apparently they are there, I get comments on them.
For me it's just a white space with the text 'photo' in the upper left corner

[denise]

Do you mean your icons, or pictures you've emailed into the site to add to entries?

Either way, try clearing your browser cache (see GENERAL: How do I clear my browser's cache or cookies? from the FAQ for instructions) and reload the page, to see if that clears it up. Then, if that doesn't work, if it's pictures in entries you're having trouble with, make an entry with a picture public so I can take a look and see what's going on, and if it's icons, let me know the keyword of the icon(s) that isn't displaying for you.

[idamus]

It a photo, the icons are fine, I've checked further back and it's only the last couple of posts with pics that have issues
I upload pics by copying their code from the LJ mirror post, this has always worked best for me
I'll open the post for a while

I clear my cache/browser history daily, it didn't fix anything

[denise]

Aha! I took a look at the post you unlocked, and it's linking to http://ic.pics.livejournal.com/idamus/16498843/34664/34664_900.jpg (which is hosted in LiveJournal's photo hosting). Trying to load that image gave me an "access denied" message. So, what's happening there is, you uploaded the photo to LJ with some kind of security on it (probably "friends only"), and the people who are commenting about the photo are on your LJ friends list. I can't see it, because I'm not on the friends list of your LJ account. (You probably can't see it because you aren't logged in to your LJ account.)

To fix that, you'll need to make the photo public on your LJ account (and probably change your settings so that photos uploaded aren't automatically friends-locked), or (if you don't want the photo to be public) also upload it to Dreamwidth's image hosting service instead of loading it from LJ. (instructions here.)

[novembermond]

do bookmarks work? I tried to add two stories yesterday and the count of my bookmarks went up, but they're not there.

[idamus]

Aha, thanks, guess I missed a change over there or something, 'cause this has always worked in the past, since my friends here are also friends over there it shouldn't be a problem.
At least I know what to tell them if they see a blank space now

[denise]

Yesterday was before any of the issues started, so any problems you're having with memories are unrelated to this database maintenance. The memories feature is a bit old and clunky (we have a new version in the works), but I'm not aware of any problems. I do see from looking at your profile, though, that this account doesn't have any memories saved, and you calling them "bookmarks" and mentioning stories makes me think you might be thinking of the AO3, which has been having some problems with bookmarking stories recently! (We don't run the AO3 site -- you'd have to talk to them if that's what you mean.)

If you do mean memories on Dreamwidth, check the Memorable Entries page and see if you see them there. If not, go back to the entry you added to your memories and hit the "Add Memory" link in the entry interaction footer (on the journal view) or the "Add Memory" icon in the row of entry interaction icons, then make sure you give it a keyword to save it to your journal's memories.

[novembermond]

uuugh, sorry, wrong tab, wrong tab! I had the ao3 post open in the other tab. thx for looking into my non-existsting problem so fast, though.
that teaches me, leaving comments before the first coffee.
you're awesome, dw works perfectly.

[denise]

Aw, you're welcome, and don't worry about it! I am all too painfully familiar with the "cannot brain yet, coffee levels too low" problem. :)

[bluiis]

You guys are amazing.

[ruriair]

m(o_o)m

Driveby comment but I had no idea you guys had an in-site image hosting service already.

And that you can manage the files unlike on tumblr where things exist forever.

[denise]

It was announced in news posts a while back! It's emailed-in-only for now, but we've got things like in-browser upload and gallery display on the To Be Done list (once we finish all the other backend stuff we need to get done auuugh why is everything always Dependency Hell).

[phoenixfire12]

I realize that this is probably off-topic but over the last 6 months or so I have noticed that a lot of communities I used to subscribe to before I did some page-cleaning this afternoon have not been updated in 4-5 years. I hate to suggest you add more work to your TO DO LIST, but I would like to know if there any plans for deleting inactive communities and accounts?

[misskat]

Dreamwidth doesn't delete accounts for inactivity. There are several reasons for this, but the primary one is that even if a user isn't posting entries, they may be using their account in other ways (commenting, or using it to access locked content on communities/journals). The FAQ here explains in a little more detail.

It's a fairly common question, so thanks for asking!

[phoenixfire12]

That is helpful. LiveJournal has been doing their purge for a few years now and a couple of journals and communities I have been part of have been purged. Since DW is/was a sister site I thought there might have been some ideas brought over here from there. Thank you for clearing this up.