(no subject)

[denise]

We have now cut over to our new hardware! If you can see this, the site should be working for you.

Things you don't need to tell us about:

* The site may look a little weird or naked for a while as your computer/network/ISP/etc catches up with the switch.
* Comment notification emails should be going out properly, but inbound email (post-by-email and comment-by-email) may be delayed for a bit while your outgoing mail server catches up with the switch.
* The site search database is a few days out of date (missing entries from the past two days) and will catch up over the next few days. Entries and comments posted since the site came back up will be added to the index when they're posted.

Known issues:

* Accessing the site will be slow for a while as the caches warm back up, and you may receive Gateway Timeout errors.
* Due to the occasional database connection problem due to high load and site slowness earlier, some comments did not generate emailed notifications and those notifications can't be re-sent. New comments made since we resolved the DB problems are generating email notifications, but slowly (due to the general database slowness).
* Logging in to other sites using your Dreamwidth OpenID is broken (& we'll fix it as soon as we can!)

Let us know if you see anything else weird!

Comments

[mark]

Hello, hello! We are here!

[denise]

IT'S ALIVE

[linaelyn]

I am grateful to have such clear communication about all of this process! You all are a breath of fresh air, after dealing with my cable provider!

Thanks for all of your hard work.

[turlough]

♥♥♥

[birguslatro]

Getting a "Your connection is not secure" for embedded YouTube videos when using https. (Or "Down for Maintenance" when not.) Seems to be behaving otherwise.

[alierak]

I think we've figured this out - may take a few minutes for DNS to update though.

[denise]

"...good catch!" says zorkian. Fixing now, thanks!

[secondnature]

Hey there! Out of a precaution that I usually do to check notifications after things like this, notifications are not sending by E-Mail at this time. (I'm on Gmail.)

I did get the test comment in my DW inbox, however!

[denise]

Hmm, that's weird -- we have people in the room who use Gmail and are getting notifications just fine. Can you point me at an entry of yours that I can leave a test comment on?

[badly_knitted]

My update page is stuck on HTML, I can't access the Rich Text option, meaning I can't post anything... Help?

[denise]

This is probably covered under the "the site may look weird or naked for a while" bullet point above! Give it a few hours.

[the]

[niqaeli]

I'm getting intermittent 504 Gateway errors posting/editing comments and actually just a moment ago on reloading this page itself. I suspect this is an overwhelmed database declaring its desire to fuck off to a life of lumberjacking, but I thought I'd note it.

*leaves cookies out for the database*

[denise]

You are indeed correct about the life of lumberjacking! (Restarting everything = the caches empty and have to refill = the database gets overloaded because all page loads hit the DB and not the caches = the webserver processes time out waiting for the DB to respond.) You also get the reward for making the whole room crack up.

[nightdog_barks]

I also got a couple of 504 Gateway errors when I was editing a journal page and when I tried to view these comments, but they went away when I refreshed the page. Not worried, just noting. :D

[denise]

That's related to everything being restarted -- when the caches are cold, every page load has to hit the database, which increases the DB load, which can cause the webservers to time out waiting for a response from the DB. Mark and Robby are doing some more performance tweaks, and things will slowly improve over the next few hours as the caches warm up and more stuff gets put in them!

[afuna]

Hooray!!!!

[quartzpebble]

Woohoo, thanks for the work! Looking good here.

[seekingferret]

Let us know if you see anything else weird!

When I reloaded the site, Donald Trump was President-Elect of the United States.

[arethinn]

*snerk*

[lilly_the_kid]

I can't seem to log into accounts using my dreamwidth account with open ID. It works with my lj account, so I assume it's not the other site, though I don't know.

[denise]

I read this comment out loud and everyone in the room groaned and said "...we forgot about OpenID." So you are probably right it isn't the other site! We'll look at it, hang on.

[ecks]

I'm also seeing what I'm guessing is a delay in notifications. I'm aware of three notifications I would have received since the site went back up. One made it all the way to my email, one made it only to my on-site inbox, and one hasn't shown up yet at all. Are we just waiting for the caches to fill back up and stop overloading the DB, or is that a separate issue?

[denise]

Two possible causes of missing notifications!

* If the person who commented got the "too many connections" error when they commented, the job for sending the email was unable to make it to the database and can't be re-sent: you won't get an email notification for that comment, sadly.

* We have another issue that we're looking into that may cause a small subset of notifications to fail, and we haven't fixed that yet. (It's the current thing people are working on.) I don't know for sure whether it will be possible to re-send those failed emails once we fix the problem, because we haven't pinpointed the exact problem yet. It's probably not very likely, though.

[illariy]

Error 504 Gateway timeout more often than not.

[illariy]

Ah I See you edited the Page to add that Error...

[denise]

Yup, we're trying to figure out why they're happening so often right now, but they're going to happen on and off for a while over the next few hours.

[yanagi_wa]

I think you did a great job. I didn't notice a thing. Great work.

[hellicious]

Only thing I've noticed that's a little different is in posts with IP logging on (where I can view it, at least), it's giving me new via IP?

Usually it just puts the IP of the user up, but in a current thread I'm in, both myself and the party I'm tagging with have the IPs listed as 108.162.###.### via (actual IP). (the last two numbers are different for each of us, but the first two are the same). Is that something that will change or that needs to?

[kareila]

It is an issue that we're looking into - thanks!

[darlas_mom]

Content search isn't working for me. :( I don't know about anyone else, but it's definitely something I've noticed trying to search for entries/comments in communities. everything comes back "found no result," even when I test it with random words that should pull up a trillion results.

[kareila]

There is a search indexing backlog that we are having to rate-limit right now due to database load... I can't give you an estimate on when it might be caught up, but we are keeping an eye on it. Thanks!

[jazzyjj]

All seems to be working great on my end thus far. I posted an entry via email last night, and sure enough when I checked this morning my entry was there. I think I inadvertently created an additional tag, but not to worry. I'll fix that as soon as possible. Thanks again for keeping us users happy!

[scarimonious]

I keep getting "down for maintenance" on anything that has my username in the URL. I can read this page fine, I can read other people's individual entries fine, I can even post fine but I cannot access my journal/profile/reading page etc.

[kareila]

Sounds like scarimonious.dreamwidth.org is still pointing to the old server's IP address on your setup. Hopefully this is something that your network provider will eventually update, but if clearing your browser cache didn't resolve it, rebooting your modem or router may help the process happen more quickly.