(no subject)

[denise]

We have now cut over to our new hardware! If you can see this, the site should be working for you.

Things you don't need to tell us about:

* The site may look a little weird or naked for a while as your computer/network/ISP/etc catches up with the switch.
* Comment notification emails should be going out properly, but inbound email (post-by-email and comment-by-email) may be delayed for a bit while your outgoing mail server catches up with the switch.
* The site search database is a few days out of date (missing entries from the past two days) and will catch up over the next few days. Entries and comments posted since the site came back up will be added to the index when they're posted.

Known issues:

* Accessing the site will be slow for a while as the caches warm back up, and you may receive Gateway Timeout errors.
* Due to the occasional database connection problem due to high load and site slowness earlier, some comments did not generate emailed notifications and those notifications can't be re-sent. New comments made since we resolved the DB problems are generating email notifications, but slowly (due to the general database slowness).
* Logging in to other sites using your Dreamwidth OpenID is broken (& we'll fix it as soon as we can!)

Let us know if you see anything else weird!

Comments

[niqaeli]

I'm getting intermittent 504 Gateway errors posting/editing comments and actually just a moment ago on reloading this page itself. I suspect this is an overwhelmed database declaring its desire to fuck off to a life of lumberjacking, but I thought I'd note it.

*leaves cookies out for the database*

[nightdog_barks]

I also got a couple of 504 Gateway errors when I was editing a journal page and when I tried to view these comments, but they went away when I refreshed the page. Not worried, just noting. :D

[denise]

You are indeed correct about the life of lumberjacking! (Restarting everything = the caches empty and have to refill = the database gets overloaded because all page loads hit the DB and not the caches = the webserver processes time out waiting for the DB to respond.) You also get the reward for making the whole room crack up.

[denise]

That's related to everything being restarted -- when the caches are cold, every page load has to hit the database, which increases the DB load, which can cause the webservers to time out waiting for a response from the DB. Mark and Robby are doing some more performance tweaks, and things will slowly improve over the next few hours as the caches warm up and more stuff gets put in them!

[mildred_of_midgard]

As a DBA at a company that does its maintenance work at night, I have often made the joke that the life of a DBA is the opposite of that of a lumberjack: "I'm a DBA and I'm okay. I work all night and I sleep all day."

[afuna]

Hooray!!!!

[quartzpebble]

Woohoo, thanks for the work! Looking good here.

[seekingferret]

Let us know if you see anything else weird!

When I reloaded the site, Donald Trump was President-Elect of the United States.

[lilly_the_kid]

I can't seem to log into accounts using my dreamwidth account with open ID. It works with my lj account, so I assume it's not the other site, though I don't know.

[ecks]

I'm also seeing what I'm guessing is a delay in notifications. I'm aware of three notifications I would have received since the site went back up. One made it all the way to my email, one made it only to my on-site inbox, and one hasn't shown up yet at all. Are we just waiting for the caches to fill back up and stop overloading the DB, or is that a separate issue?

[denise]

I read this comment out loud and everyone in the room groaned and said "...we forgot about OpenID." So you are probably right it isn't the other site! We'll look at it, hang on.

[denise]

Two possible causes of missing notifications!

* If the person who commented got the "too many connections" error when they commented, the job for sending the email was unable to make it to the database and can't be re-sent: you won't get an email notification for that comment, sadly.

* We have another issue that we're looking into that may cause a small subset of notifications to fail, and we haven't fixed that yet. (It's the current thing people are working on.) I don't know for sure whether it will be possible to re-send those failed emails once we fix the problem, because we haven't pinpointed the exact problem yet. It's probably not very likely, though.

[illariy]

Error 504 Gateway timeout more often than not.

[illariy]

Ah I See you edited the Page to add that Error...

[yanagi_wa]

I think you did a great job. I didn't notice a thing. Great work.

[denise]

Yup, we're trying to figure out why they're happening so often right now, but they're going to happen on and off for a while over the next few hours.

[ecks]

I just checked again and all the notifications I had been expecting have since come through (they showed up in both my on-site inbox and my email).

[denise]

Awesome! Looks like you lucked out and it was just database load, then.

[denise]

This is why my horrible case of freerunning non-24-hour sleep/wake syndrome comes in handy: if there's a thing that needs to be done at 4AM, chances are good I'll be up then at least once in any given week...

[arethinn]

*snerk*

[illariy]

Hah, I am going to bed then. Thank you for all your hard work!

[denise]

Things should get better pretty quickly now! We hope.

[kore]

Haaaaah. //cries

[mildred_of_midgard]

Whereas every time we do maintenance, I just suffer days of sleep deprivation before my rather fragile sleep-regulating neurons get back to normal. Ah well.

[denise]

I'd offer to lend you some of my sleep disorder, except trust me, you don't want it. :)