mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
Mark Smith ([staff profile] mark) wrote in [site community profile] dw_maintenance2014-01-26 16:06

Database maintenance

Hi all!

Today we started getting some alerts on the database, so I'm going to do some maintenance to verify the health and wellness of the machine and get it back to a non-alerting status.

I should be able to do this without any downtime, but just in case, you might want to make sure to use your favorite text editor to save a copy of any long entries or comments you're working on.

Once I've got things sorted out, I'll update this with more details for the technically curious.

Update [4:50PM PST]: sb-db06 (the slave) has been rebooted and is recovering, I'm doing system updates on it since the problem looks like a kernel bug (it struck both databases at the same time). Next: master failover then recover the other database.

Update [5:05PM PST]: I'm doing what we call a "master failover" now. This means I'm shifting all traffic from the database that was active (sb-db05) to the spare database (sb-db06). I have to shut off "extra" services like imports, feeds, and searches while this happens.

Update [5:30PM PST]: Well, that was unexpectedly bumpy. Sorry for that. There should be no further bumping, as we're now on the spare database so I can take maintenance on the original master.

Update [6:20PM PST]: If you had userpics not loading, they should be back to normal.

jjhunter: A cartoon sketch of a small island labeled "DREAM WIDTH" off the coast of larger island labeled LIVEJOURNAL". (xkcd dreamwidth island)

[personal profile] jjhunter 2014-01-27 01:39 (UTC)(link)
*waves pom-poms*
juliet316: (DW: Clara at Trenzalore)

[personal profile] juliet316 2014-01-27 01:52 (UTC)(link)
Thanks for the updates!

ETA: A third of my icons aren't showing up.
Edited 2014-01-27 01:54 (UTC)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-01-27 02:22 (UTC)(link)
Should be fixed now!
juliet316: (WWE: Layla)

[personal profile] juliet316 2014-01-27 02:41 (UTC)(link)
Testing, one two three testing...

Edit: Yep working now, thanks.
Edited 2014-01-27 02:42 (UTC)
tyrande: (pic#6236046)

[personal profile] tyrande 2014-01-27 01:55 (UTC)(link)
Just as a heads up, I'm getting strange errors when it comes to uploading user icons on multiple accounts:

Unable to contact storage server. Your picture has not been saved. at /home/dw/current/cgi-bin/LJ/Userpic.pm line 746

Failed to open MogileFS tempfile at /home/dw/production/htdocs/editicons.bml line 924, line 55.
tyrande: (pic#6860392)

[personal profile] tyrande 2014-01-27 04:56 (UTC)(link)
Thank you very much for helping with the issue!
queenofwinter: (....Did anyone else feel that?)

[personal profile] queenofwinter 2014-01-27 02:15 (UTC)(link)
It looks like comment notifications aren't working, by the way?
queenofwinter: (Behold my power!)

[personal profile] queenofwinter 2014-01-27 03:03 (UTC)(link)
They are coming through now! I received a bunch at once and now they seem to be back to normal.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-01-27 04:46 (UTC)(link)

Should be going out properly now. (Sorry, I mentioned the notifs delay on our Twitter status account, but couldn't add it here since Mark was the one who posted this entry and he was busy doing the fixes!)

decepticon_mistress: (Default)

[personal profile] decepticon_mistress 2014-01-27 02:25 (UTC)(link)
Thanks for everything that you do!
kore: (Default)

[personal profile] kore 2014-01-27 02:30 (UTC)(link)
Thank you for the hard work and the very clear communication!
veritas_poet: (Lifeline)

[personal profile] veritas_poet 2014-01-27 04:21 (UTC)(link)
Thanks for all of your work on it.
idamus: (Default)

[personal profile] idamus 2014-01-27 06:44 (UTC)(link)
I can't see the pics in my journal, but apparently they are there, I get comments on them.
For me it's just a white space with the text 'photo' in the upper left corner
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-01-27 06:49 (UTC)(link)

Do you mean your icons, or pictures you've emailed into the site to add to entries?

Either way, try clearing your browser cache (see GENERAL: How do I clear my browser's cache or cookies? from the FAQ for instructions) and reload the page, to see if that clears it up. Then, if that doesn't work, if it's pictures in entries you're having trouble with, make an entry with a picture public so I can take a look and see what's going on, and if it's icons, let me know the keyword of the icon(s) that isn't displaying for you.

idamus: (Default)

[personal profile] idamus 2014-01-27 07:19 (UTC)(link)
It a photo, the icons are fine, I've checked further back and it's only the last couple of posts with pics that have issues
I upload pics by copying their code from the LJ mirror post, this has always worked best for me
I'll open the post for a while

I clear my cache/browser history daily, it didn't fix anything
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-01-27 07:50 (UTC)(link)
Aha! I took a look at the post you unlocked, and it's linking to http://ic.pics.livejournal.com/idamus/16498843/34664/34664_900.jpg (which is hosted in LiveJournal's photo hosting). Trying to load that image gave me an "access denied" message. So, what's happening there is, you uploaded the photo to LJ with some kind of security on it (probably "friends only"), and the people who are commenting about the photo are on your LJ friends list. I can't see it, because I'm not on the friends list of your LJ account. (You probably can't see it because you aren't logged in to your LJ account.)

To fix that, you'll need to make the photo public on your LJ account (and probably change your settings so that photos uploaded aren't automatically friends-locked), or (if you don't want the photo to be public) also upload it to Dreamwidth's image hosting service instead of loading it from LJ. (instructions here.)
idamus: (Default)

[personal profile] idamus 2014-01-27 08:02 (UTC)(link)
Aha, thanks, guess I missed a change over there or something, 'cause this has always worked in the past, since my friends here are also friends over there it shouldn't be a problem.
At least I know what to tell them if they see a blank space now
ruriair: (Қɧ : boogie panda)

[personal profile] ruriair 2014-01-27 22:40 (UTC)(link)
m(o_o)m

Driveby comment but I had no idea you guys had an in-site image hosting service already.

And that you can manage the files unlike on tumblr where things exist forever.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-01-27 22:47 (UTC)(link)

It was announced in news posts a while back! It's emailed-in-only for now, but we've got things like in-browser upload and gallery display on the To Be Done list (once we finish all the other backend stuff we need to get done auuugh why is everything always Dependency Hell).

novembermond: (thor2)

[personal profile] novembermond 2014-01-27 07:59 (UTC)(link)
do bookmarks work? I tried to add two stories yesterday and the count of my bookmarks went up, but they're not there.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-01-27 08:05 (UTC)(link)

Yesterday was before any of the issues started, so any problems you're having with memories are unrelated to this database maintenance. The memories feature is a bit old and clunky (we have a new version in the works), but I'm not aware of any problems. I do see from looking at your profile, though, that this account doesn't have any memories saved, and you calling them "bookmarks" and mentioning stories makes me think you might be thinking of the AO3, which has been having some problems with bookmarking stories recently! (We don't run the AO3 site -- you'd have to talk to them if that's what you mean.)

If you do mean memories on Dreamwidth, check the Memorable Entries page and see if you see them there. If not, go back to the entry you added to your memories and hit the "Add Memory" link in the entry interaction footer (on the journal view) or the "Add Memory" icon in the row of entry interaction icons, then make sure you give it a keyword to save it to your journal's memories.

novembermond: (Rose)

[personal profile] novembermond 2014-01-27 08:14 (UTC)(link)
uuugh, sorry, wrong tab, wrong tab! I had the ao3 post open in the other tab. thx for looking into my non-existsting problem so fast, though.
that teaches me, leaving comments before the first coffee.
you're awesome, dw works perfectly.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2014-01-27 08:17 (UTC)(link)

Aw, you're welcome, and don't worry about it! I am all too painfully familiar with the "cannot brain yet, coffee levels too low" problem. :)

bluiis: (dw to do by immortalje)

[personal profile] bluiis 2014-01-27 12:43 (UTC)(link)
You guys are amazing.
phoenixfire12: (ChocoCuthulu)

[personal profile] phoenixfire12 2014-03-05 22:58 (UTC)(link)
I realize that this is probably off-topic but over the last 6 months or so I have noticed that a lot of communities I used to subscribe to before I did some page-cleaning this afternoon have not been updated in 4-5 years. I hate to suggest you add more work to your TO DO LIST, but I would like to know if there any plans for deleting inactive communities and accounts?
Edited 2014-03-05 22:59 (UTC)
misskat: Dreamsheep, with MissKat on top, and Staff on the bottom. (Dreamwidth Staff)

[staff profile] misskat 2014-03-06 02:28 (UTC)(link)
Dreamwidth doesn't delete accounts for inactivity. There are several reasons for this, but the primary one is that even if a user isn't posting entries, they may be using their account in other ways (commenting, or using it to access locked content on communities/journals). The FAQ here explains in a little more detail.

It's a fairly common question, so thanks for asking!
phoenixfire12: (ChocoCuthulu)

[personal profile] phoenixfire12 2014-03-07 18:38 (UTC)(link)
That is helpful. LiveJournal has been doing their purge for a few years now and a couple of journals and communities I have been part of have been purged. Since DW is/was a sister site I thought there might have been some ideas brought over here from there. Thank you for clearing this up.