dw_maintenance | (no subject)

(no subject)

We have now cut over to our new hardware! If you can see this, the site should be working for you.

Things you don't need to tell us about:

* The site may look a little weird or naked for a while as your computer/network/ISP/etc catches up with the switch.
* Comment notification emails should be going out properly, but inbound email (post-by-email and comment-by-email) may be delayed for a bit while your outgoing mail server catches up with the switch.
* The site search database is a few days out of date (missing entries from the past two days) and will catch up over the next few days. Entries and comments posted since the site came back up will be added to the index when they're posted.

Known issues:

* Accessing the site will be slow for a while as the caches warm back up, and you may receive Gateway Timeout errors.
* Due to the occasional database connection problem due to high load and site slowness earlier, some comments did not generate emailed notifications and those notifications can't be re-sent. New comments made since we resolved the DB problems are generating email notifications, but slowly (due to the general database slowness).
* Logging in to other sites using your Dreamwidth OpenID is broken (& we'll fix it as soon as we can!)

Let us know if you see anything else weird!

Flat | Top-Level Comments Only

Hello, hello! We are here!

IT'S ALIVE

I am grateful to have such clear communication about all of this process! You all are a breath of fresh air, after dealing with my cable provider!

Thanks for all of your hard work.

♥♥♥

Getting a "Your connection is not secure" for embedded YouTube videos when using https. (Or "Down for Maintenance" when not.) Seems to be behaving otherwise.

I think we've figured this out - may take a few minutes for DNS to update though.

"...good catch!" says zorkian. Fixing now, thanks!

aka me!

Working for an http connection now, but still not for https. (I have to go offline now, so won't be able to check again for a few hours...)

Don't worry about not being able to check, we can test on our end! We just ran into A Thing while trying to fix it, so the fix is not working yet...

It's behaving now. Thanks for all the good work!

Hey there! Out of a precaution that I usually do to check notifications after things like this, notifications are not sending by E-Mail at this time. (I'm on Gmail.)

I did get the test comment in my DW inbox, however!

Hmm, that's weird -- we have people in the room who use Gmail and are getting notifications just fine. Can you point me at an entry of yours that I can leave a test comment on?

http://secondnature.dreamwidth.org/936.html - right here! This is where I did my test comment, as well.

Also, for what it's worth, I didn't get your comment as a notification, either.

Edited 2016-12-01 18:53 (UTC)

Figured out the problem -- it's not with email. It's on The List and is up next for Mark and Robby to look into!

Yeah! I did some testing, and it was all of my E-Mails with a + mark in them, so my old RP journals before I used filters got the notification E-Mails fine, but anything that's like myemailaddress+filtersignifier@gmail.com are not sending. I figure you realized this, but I found it satisfying to do some troubleshooting of my own.

Actually, that's probably a coincidence (or a secondary symptom, not the primary symptom), but good pattern spotting. :)

yeah, I did another test and a friend got the plus-sign e-mails fine, so.

GOOD LUCK FIXING IT and thanks for all the hard work.

basically the issue is that you all flocked back to the site the minute we hit the button, so between the caches needing to fill back up and everybody coming to test things, the db keeps hitting max connections. :)

My update page is stuck on HTML, I can't access the Rich Text option, meaning I can't post anything... Help?

This is probably covered under the "the site may look weird or naked for a while" bullet point above! Give it a few hours.

Okay, thanks, I'll probably post without cuts and other embellishments and edit tomorrow when things have had a chance to settle down. No worries.

Try now?

Just thought I'd let you know it's all working fine now!

Great, thanks!

I'm getting intermittent 504 Gateway errors posting/editing comments and actually just a moment ago on reloading this page itself. I suspect this is an overwhelmed database declaring its desire to fuck off to a life of lumberjacking, but I thought I'd note it.

*leaves cookies out for the database*

You are indeed correct about the life of lumberjacking! (Restarting everything = the caches empty and have to refill = the database gets overloaded because all page loads hit the DB and not the caches = the webserver processes time out waiting for the DB to respond.) You also get the reward for making the whole room crack up.

Glad to be of service! XD

As a DBA at a company that does its maintenance work at night, I have often made the joke that the life of a DBA is the opposite of that of a lumberjack: "I'm a DBA and I'm okay. I work all night and I sleep all day."

This is why my horrible case of freerunning non-24-hour sleep/wake syndrome comes in handy: if there's a thing that needs to be done at 4AM, chances are good I'll be up then at least once in any given week...

Whereas every time we do maintenance, I just suffer days of sleep deprivation before my rather fragile sleep-regulating neurons get back to normal. Ah well.

I'd offer to lend you some of my sleep disorder, except trust me, you don't want it. :)

You can keep it. ;)

I also got a couple of 504 Gateway errors when I was editing a journal page and when I tried to view these comments, but they went away when I refreshed the page. Not worried, just noting. :D

That's related to everything being restarted -- when the caches are cold, every page load has to hit the database, which increases the DB load, which can cause the webservers to time out waiting for a response from the DB. Mark and Robby are doing some more performance tweaks, and things will slowly improve over the next few hours as the caches warm up and more stuff gets put in them!

Hooray!!!!

Woohoo, thanks for the work! Looking good here.

Let us know if you see anything else weird!

When I reloaded the site, Donald Trump was President-Elect of the United States.

*snerk*

Haaaaah. //cries

Indeed. Time for a rollback, methinks.

Lol I think that also happened to me and I don't like it one bit. Y'all are fired! *grin*

I can't seem to log into accounts using my dreamwidth account with open ID. It works with my lj account, so I assume it's not the other site, though I don't know.

I read this comment out loud and everyone in the room groaned and said "...we forgot about OpenID." So you are probably right it isn't the other site! We'll look at it, hang on.

It's working fine now. Thank you!!

I'm also seeing what I'm guessing is a delay in notifications. I'm aware of three notifications I would have received since the site went back up. One made it all the way to my email, one made it only to my on-site inbox, and one hasn't shown up yet at all. Are we just waiting for the caches to fill back up and stop overloading the DB, or is that a separate issue?

Two possible causes of missing notifications!

* If the person who commented got the "too many connections" error when they commented, the job for sending the email was unable to make it to the database and can't be re-sent: you won't get an email notification for that comment, sadly.

* We have another issue that we're looking into that may cause a small subset of notifications to fail, and we haven't fixed that yet. (It's the current thing people are working on.) I don't know for sure whether it will be possible to re-send those failed emails once we fix the problem, because we haven't pinpointed the exact problem yet. It's probably not very likely, though.

I just checked again and all the notifications I had been expecting have since come through (they showed up in both my on-site inbox and my email).

Awesome! Looks like you lucked out and it was just database load, then.

Error 504 Gateway timeout more often than not.

Ah I See you edited the Page to add that Error...

Yup, we're trying to figure out why they're happening so often right now, but they're going to happen on and off for a while over the next few hours.

Hah, I am going to bed then. Thank you for all your hard work!

Things should get better pretty quickly now! We hope.

I think you did a great job. I didn't notice a thing. Great work.

Only thing I've noticed that's a little different is in posts with IP logging on (where I can view it, at least), it's giving me new via IP?

Usually it just puts the IP of the user up, but in a current thread I'm in, both myself and the party I'm tagging with have the IPs listed as 108.162.###.### via (actual IP). (the last two numbers are different for each of us, but the first two are the same). Is that something that will change or that needs to?

Edited 2016-12-02 09:53 (UTC)

It is an issue that we're looking into - thanks!

I think this is fixed now, and in the process I found another really interesting bug to fix. Thanks!

It does seem to be fixed as I'm just seeing the singular IP now, but hey, good on finding a random bug =)

Content search isn't working for me. :( I don't know about anyone else, but it's definitely something I've noticed trying to search for entries/comments in communities. everything comes back "found no result," even when I test it with random words that should pull up a trillion results.

There is a search indexing backlog that we are having to rate-limit right now due to database load... I can't give you an estimate on when it might be caught up, but we are keeping an eye on it. Thanks!

All seems to be working great on my end thus far. I posted an entry via email last night, and sure enough when I checked this morning my entry was there. I think I inadvertently created an additional tag, but not to worry. I'll fix that as soon as possible. Thanks again for keeping us users happy!

I keep getting "down for maintenance" on anything that has my username in the URL. I can read this page fine, I can read other people's individual entries fine, I can even post fine but I cannot access my journal/profile/reading page etc.

Sounds like scarimonious.dreamwidth.org is still pointing to the old server's IP address on your setup. Hopefully this is something that your network provider will eventually update, but if clearing your browser cache didn't resolve it, rebooting your modem or router may help the process happen more quickly.

I've cleared my cache and rebooted and that seems to have done the trick - thanks!

Yay, I'm glad that worked!

Flat | Top-Level Comments Only