denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_maintenance2016-12-01 12:14 pm

(no subject)

We have now cut over to our new hardware! If you can see this, the site should be working for you.

Things you don't need to tell us about:

* The site may look a little weird or naked for a while as your computer/network/ISP/etc catches up with the switch.
* Comment notification emails should be going out properly, but inbound email (post-by-email and comment-by-email) may be delayed for a bit while your outgoing mail server catches up with the switch.
* The site search database is a few days out of date (missing entries from the past two days) and will catch up over the next few days. Entries and comments posted since the site came back up will be added to the index when they're posted.

Known issues:

* Accessing the site will be slow for a while as the caches warm back up, and you may receive Gateway Timeout errors.
* Due to the occasional database connection problem due to high load and site slowness earlier, some comments did not generate emailed notifications and those notifications can't be re-sent. New comments made since we resolved the DB problems are generating email notifications, but slowly (due to the general database slowness).
* Logging in to other sites using your Dreamwidth OpenID is broken (& we'll fix it as soon as we can!)

Let us know if you see anything else weird!
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)

[staff profile] mark 2016-12-01 06:21 pm (UTC)(link)
Hello, hello! We are here!
linaelyn: (POTC-Jack by firesignwriter)

[personal profile] linaelyn 2016-12-01 06:26 pm (UTC)(link)
I am grateful to have such clear communication about all of this process! You all are a breath of fresh air, after dealing with my cable provider!

Thanks for all of your hard work.
turlough: art of Party Poison making heart-heands ((mcr art) *heart*)

[personal profile] turlough 2016-12-01 06:29 pm (UTC)(link)
♥♥♥
birguslatro: Birgus Latro III icon (Default)

[personal profile] birguslatro 2016-12-01 06:30 pm (UTC)(link)
Getting a "Your connection is not secure" for embedded YouTube videos when using https. (Or "Down for Maintenance" when not.) Seems to be behaving otherwise.
alierak: (Default)

[personal profile] alierak 2016-12-01 06:40 pm (UTC)(link)
I think we've figured this out - may take a few minutes for DNS to update though.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)

[staff profile] mark 2016-12-01 06:46 pm (UTC)(link)
aka me!
birguslatro: Birgus Latro III icon (Default)

[personal profile] birguslatro 2016-12-01 06:53 pm (UTC)(link)
Working for an http connection now, but still not for https. (I have to go offline now, so won't be able to check again for a few hours...)
birguslatro: Birgus Latro III icon (Default)

[personal profile] birguslatro 2016-12-02 02:46 am (UTC)(link)
It's behaving now. Thanks for all the good work!
secondnature: (Default)

[personal profile] secondnature 2016-12-01 06:43 pm (UTC)(link)
Hey there! Out of a precaution that I usually do to check notifications after things like this, notifications are not sending by E-Mail at this time. (I'm on Gmail.)

I did get the test comment in my DW inbox, however!
secondnature: (Default)

[personal profile] secondnature 2016-12-01 06:52 pm (UTC)(link)
http://secondnature.dreamwidth.org/936.html - right here! This is where I did my test comment, as well.

Also, for what it's worth, I didn't get your comment as a notification, either.
Edited 2016-12-01 18:53 (UTC)
secondnature: (Default)

[personal profile] secondnature 2016-12-01 07:05 pm (UTC)(link)
Yeah! I did some testing, and it was all of my E-Mails with a + mark in them, so my old RP journals before I used filters got the notification E-Mails fine, but anything that's like myemailaddress+filtersignifier@gmail.com are not sending. I figure you realized this, but I found it satisfying to do some troubleshooting of my own.
secondnature: (Default)

[personal profile] secondnature 2016-12-01 07:13 pm (UTC)(link)
yeah, I did another test and a friend got the plus-sign e-mails fine, so.

GOOD LUCK FIXING IT and thanks for all the hard work.
badly_knitted: (JB Weird)

[personal profile] badly_knitted 2016-12-01 06:46 pm (UTC)(link)
My update page is stuck on HTML, I can't access the Rich Text option, meaning I can't post anything... Help?
badly_knitted: (Ianto - I Wonder)

[personal profile] badly_knitted 2016-12-01 06:54 pm (UTC)(link)
Okay, thanks, I'll probably post without cuts and other embellishments and edit tomorrow when things have had a chance to settle down. No worries.
badly_knitted: (Jack - Yay)

[personal profile] badly_knitted 2016-12-01 07:20 pm (UTC)(link)
Just thought I'd let you know it's all working fine now!
the: (nomi  ☛ ( 11 ))

[personal profile] the 2016-12-01 06:53 pm (UTC)(link)
niqaeli: cat with arizona flag in the background (Default)

[personal profile] niqaeli 2016-12-01 07:41 pm (UTC)(link)
I'm getting intermittent 504 Gateway errors posting/editing comments and actually just a moment ago on reloading this page itself. I suspect this is an overwhelmed database declaring its desire to fuck off to a life of lumberjacking, but I thought I'd note it.

*leaves cookies out for the database*
niqaeli: Penelope Garcia of Criminal Minds in her domain (your tech goddess here; speak o mortals)

[personal profile] niqaeli 2016-12-02 03:50 am (UTC)(link)
Glad to be of service! XD
mildred_of_midgard: (Default)

[personal profile] mildred_of_midgard 2016-12-01 08:12 pm (UTC)(link)
As a DBA at a company that does its maintenance work at night, I have often made the joke that the life of a DBA is the opposite of that of a lumberjack: "I'm a DBA and I'm okay. I work all night and I sleep all day."
mildred_of_midgard: (Default)

[personal profile] mildred_of_midgard 2016-12-02 01:43 am (UTC)(link)
Whereas every time we do maintenance, I just suffer days of sleep deprivation before my rather fragile sleep-regulating neurons get back to normal. Ah well.
mildred_of_midgard: (Default)

[personal profile] mildred_of_midgard 2016-12-02 01:45 am (UTC)(link)
You can keep it. ;)
nightdog_barks: Illustrated close-up of a bird's head grasping a red berry in its beak (Bird with Berry)

[personal profile] nightdog_barks 2016-12-01 07:48 pm (UTC)(link)
I also got a couple of 504 Gateway errors when I was editing a journal page and when I tried to view these comments, but they went away when I refreshed the page. Not worried, just noting. :D
afuna: Cat under a blanket. Text: "Cats are just little people with Fur and Fangs" (Default)

[personal profile] afuna 2016-12-01 08:15 pm (UTC)(link)
Hooray!!!!
quartzpebble: (INTERNET)

[personal profile] quartzpebble 2016-12-01 08:21 pm (UTC)(link)
Woohoo, thanks for the work! Looking good here.
seekingferret: Photo of me with my 2012 Purim beard, with stripes shaven into it. (Default)

[personal profile] seekingferret 2016-12-01 08:27 pm (UTC)(link)
Let us know if you see anything else weird!

When I reloaded the site, Donald Trump was President-Elect of the United States.
arethinn: Joel Robinson giggling (amused (joel))

[personal profile] arethinn 2016-12-01 09:59 pm (UTC)(link)
*snerk*
kore: Sam Wilson and Jane foster kiss as Cap and Thor (Default)

[personal profile] kore 2016-12-01 11:01 pm (UTC)(link)
Haaaaah. //cries
moem: A computer drawing that looks like me. (Default)

[personal profile] moem 2016-12-02 01:49 pm (UTC)(link)
Indeed. Time for a rollback, methinks.

[personal profile] jazzyjj 2016-12-02 05:21 pm (UTC)(link)
Lol I think that also happened to me and I don't like it one bit. Y'all are fired! *grin*
lilly_the_kid: (Default)

[personal profile] lilly_the_kid 2016-12-01 08:37 pm (UTC)(link)
I can't seem to log into accounts using my dreamwidth account with open ID. It works with my lj account, so I assume it's not the other site, though I don't know.
lilly_the_kid: (Default)

[personal profile] lilly_the_kid 2016-12-02 06:42 pm (UTC)(link)
It's working fine now. Thank you!!
ecks: (thinkery)

[personal profile] ecks 2016-12-01 08:49 pm (UTC)(link)
I'm also seeing what I'm guessing is a delay in notifications. I'm aware of three notifications I would have received since the site went back up. One made it all the way to my email, one made it only to my on-site inbox, and one hasn't shown up yet at all. Are we just waiting for the caches to fill back up and stop overloading the DB, or is that a separate issue?
ecks: (golem of good cheer)

[personal profile] ecks 2016-12-01 09:46 pm (UTC)(link)
I just checked again and all the notifications I had been expecting have since come through (they showed up in both my on-site inbox and my email).
illariy: entrance into a swimming pool (Default)

[personal profile] illariy 2016-12-01 09:13 pm (UTC)(link)
Error 504 Gateway timeout more often than not.
illariy: entrance into a swimming pool (Default)

[personal profile] illariy 2016-12-01 09:14 pm (UTC)(link)
Ah I See you edited the Page to add that Error...
illariy: entrance into a swimming pool (Default)

[personal profile] illariy 2016-12-01 10:19 pm (UTC)(link)
Hah, I am going to bed then. Thank you for all your hard work!
yanagi_wa: (Default)

[personal profile] yanagi_wa 2016-12-01 09:15 pm (UTC)(link)
I think you did a great job. I didn't notice a thing. Great work.
hellicious: (Default)

[personal profile] hellicious 2016-12-02 09:52 am (UTC)(link)
Only thing I've noticed that's a little different is in posts with IP logging on (where I can view it, at least), it's giving me new via IP?

Usually it just puts the IP of the user up, but in a current thread I'm in, both myself and the party I'm tagging with have the IPs listed as 108.162.###.### via (actual IP). (the last two numbers are different for each of us, but the first two are the same). Is that something that will change or that needs to?
Edited 2016-12-02 09:53 (UTC)
kareila: (Default)

[personal profile] kareila 2016-12-02 02:10 pm (UTC)(link)
It is an issue that we're looking into - thanks!
alierak: (Default)

[personal profile] alierak 2016-12-03 04:07 pm (UTC)(link)
I think this is fixed now, and in the process I found another really interesting bug to fix. Thanks!
hellicious: (Default)

[personal profile] hellicious 2016-12-03 04:30 pm (UTC)(link)
It does seem to be fixed as I'm just seeing the singular IP now, but hey, good on finding a random bug =)
darlas_mom: (blinkingcursor)

[personal profile] darlas_mom 2016-12-02 11:01 am (UTC)(link)
Content search isn't working for me. :( I don't know about anyone else, but it's definitely something I've noticed trying to search for entries/comments in communities. everything comes back "found no result," even when I test it with random words that should pull up a trillion results.
kareila: (Default)

[personal profile] kareila 2016-12-02 02:09 pm (UTC)(link)
There is a search indexing backlog that we are having to rate-limit right now due to database load... I can't give you an estimate on when it might be caught up, but we are keeping an eye on it. Thanks!

[personal profile] jazzyjj 2016-12-02 05:05 pm (UTC)(link)
All seems to be working great on my end thus far. I posted an entry via email last night, and sure enough when I checked this morning my entry was there. I think I inadvertently created an additional tag, but not to worry. I'll fix that as soon as possible. Thanks again for keeping us users happy!
scarimonious: Wasabi - the sparkliest octopus ever :D (Default)

[personal profile] scarimonious 2016-12-05 02:36 am (UTC)(link)
I keep getting "down for maintenance" on anything that has my username in the URL. I can read this page fine, I can read other people's individual entries fine, I can even post fine but I cannot access my journal/profile/reading page etc.
kareila: (Default)

[personal profile] kareila 2016-12-05 03:16 am (UTC)(link)
Sounds like scarimonious.dreamwidth.org is still pointing to the old server's IP address on your setup. Hopefully this is something that your network provider will eventually update, but if clearing your browser cache didn't resolve it, rebooting your modem or router may help the process happen more quickly.
scarimonious: Wasabi - the sparkliest octopus ever :D (Default)

[personal profile] scarimonious 2016-12-05 03:32 am (UTC)(link)
I've cleared my cache and rebooted and that seems to have done the trick - thanks!
kareila: (Default)

[personal profile] kareila 2016-12-05 03:40 am (UTC)(link)
Yay, I'm glad that worked!