dw_maintenance | Site slowdowns and internal server errors

Hello, my lovely Dreamwidthians! I bring to you information about several site issues this afternoon/evening/other-time-of-day-as-appropriate-for-your-timezone, and the steps we're taking to fix them:

* Some people have noticed intermittent site slowdowns today. This is partially because of increased traffic, and partially, as we feared, because the import queue, and the speed at which we're processing import jobs, is putting a lot of load on the database servers. Short-term fix:

mark is juggling things around to reduce database load and trying to direct some queries to the backup database machine. Long-term fix: we have a ticket in with our hosting provider to upgrade the database machines to make them bigger, better, faster, and more.

* You may get some "internal server error" messages when submitting data to the site, or messages saying that the site sent no data or the connection was reset. This is almost certainly related to the new load balancing solution we put into place last night, and Mark is doing some configuration tweaks and fixes that will hopefully stop it from happening. We're not 100% sure on the exact cause -- there are a few things it could be -- but we're going to keep whacking at it until the errors go away.

As part of these fixes, Mark may need to take the site down for a brief maintenance window -- he's hoping he'll be able to make the fixes without having to put the site into maintenance mode, but he might not be able to. The site may be slow for a while in the next few hours, and there may be brief periods of downtime. We're working to fix the problems as fast as we can!

(Well, okay, Mark is working to fix the problems. I'm cheering him on and passing him towels and Gatorade.)

Flat | Top-Level Comments Only | Expand All

Thank you for being on the ball with this ♥

Thank you for keeping us updated!

Cookies for

mark!

Not quite cookies, but we did buy him pizza for dinner. *G*

(no subject)

larvesta - 2011-12-31 02:08 (UTC) - Expand

moem - 2011-12-31 07:58 (UTC) - Expand

owl - 2011-12-31 23:14 (UTC) - Expand

Thanks for the update! I had trouble choosing a layout theme earlier today, but I managed to get it to work after I left and did something else for a while. Hopefully the problems aren't too big if they're already starting to go away.

Thank you for letting us know! Good luck!

Gatorade is important!

Thanks for keeping in touch.

Y'all are holding up beautifully with the no-invite code + LJ-wah combination. I'm sure you weren't expecting quite this influx of new members. <3

I expected "some extra people" ... this is a lot more than I expected. The hatches are battened, etc!

jer - 2011-12-31 01:51 (UTC) - Expand

lanterne_rouee - 2011-12-31 02:52 (UTC) - Expand

andrewducker - 2011-12-31 11:13 (UTC) - Expand

fight4theusers - 2011-12-31 20:13 (UTC) - Expand

andrewducker - 2012-01-01 20:22 (UTC) - Expand

Did you ever know that you're my heeroooes~?

Go Glitterhoof go!

Between no-invite codes, LJ stuffs and Comm Importing: You all are doing brilliantly! (Seriously, I expected a total site failure by now. Not that I doubted you, but having run even a one, I know how these changes can create major problems. - In other words? I'm highly impressed!)

Thanks again for the communication, it always makes the slowdowns and errors seem not so bad when we know what's going on XD

Exactly!!!!!

Sorry, I get so annoyed when companies have problems and don't take the time to let me know what's going on. I really want people who use Dreamwidth to have the feeling that this is their site, too, and so they should expect to know what's going on with it.

Particularly if they're helping support us financially. That's just a no brainer.

Anyway, a rant for another day...

jhumor - 2011-12-31 01:42 (UTC) - Expand

and as a new user i thank you!!!

misstia - 2011-12-31 03:41 (UTC) - Expand

ironed_orchid - 2011-12-31 10:04 (UTC) - Expand

elialshadowpine - 2011-12-31 13:59 (UTC) - Expand

Thank you for all your hard work.

May I ask what is being done to brace for a DDOS attack, if ever it comes? A quick 'latest things' run-through shows that a lot of Russians have moved here too.

We have a bunch of things set up that can help to handle DDoS traffic if it should ever happen (*knock wood*), and there are various providers and services that you can put in place to help mitigate, but in general, handling a DDoS attack is by nature reactive, not proactive -- unless you're a service that absolutely needs to be 100% available (like, I dunno, infrastructure that runs the power grid or something) it's better to put basic provisions into place and then work with your hosting providers to mitigate the excess malicious traffic if the worst should happen and you get hit. It's just not practical to keep DDoS mitigation in place all the time when it's only needed in absolute emergencies.

(Which means that yes, if we were to be hit with a bad enough DDoS, we'd be down for a while until we could get the traffic handled. But the same is true of most sites: you put things into place to handle a basic level of traffic spike -- which could be a DDoS or it could be a legitimate traffic increase! -- and put the big guns in place if they're needed.)

But we've had a large Russian userbase for a while now (which I think is awesome, I love loading the Latest Things page and seeing languages I don't speak much of -- it shows that our baby is growing up!) and, knock wood, there's been no signs of trouble. The DDoS attacks on LJ have been against a very specific set of journals, not "Russian-language journals" as a collective.

ekmisao - 2011-12-31 02:48 (UTC) - Expand

nilo - 2011-12-31 04:54 (UTC) - Expand

You guys are awesome! :D Thank you!

You can do eet!

(Where's my cheerleader gif...)

Thank you and good luck!

Thanks for the update, and good luck getting the issues solved! It's so nice to have such frequent updates and in-depth explanations of what's going on.

Can I just say I love you for keeping us updated? It's helpful and great--I only noticed a few issues (mostly internal server errors last night) but refreshing after a bit fixed the problem. Even though it was minor, it's nice to know.

Ah! That's why I couldn't get at my Inbox!

Thanks for the head's up!

Good to know - and thanks for keeping all of us in the know!

I would love to see what kind of cumulative numbers this import has generated for DW after this whole transfer period is done.

The stats page is public! You can see the old stats or the new stats (which break things down a little differently), or the raw stats from the old stats page (which doesn't separate out new accounts by type-of-account, so it's got communities, OpenID accounts, feed accounts, etc, included in the "newbyday" numbers).

soc_puppet - 2011-12-31 02:38 (UTC) - Expand

denise - 2011-12-31 02:48 (UTC) - Expand

ironed_orchid - 2011-12-31 10:09 (UTC) - Expand

transferring products over IP, or have you developed a throwing arm to make quarterbacks jealous? ;)

The Gatorade and towels are virtual, but we bought him a real pizza for dinner, too *G*

Awesome! Thanks for keeping everyone updated! I think I had one glitchy page earlier today while I was browsing, but I hit refresh and it was gone. I don't even remember what exactly happened to bring it up, since it was such a small blip on my radar and I knew that

mark was keeping us updated with what was going on.

I had a problem loading pages earlier. I figured it was because of the influx of people this website is getting from LJ. So thanks for confirming it and thanks for the updates. :D

Thank you for keeping up to date with us! :)

I love you guys ♥

Thanks for letting us know what's happening as it's happening and not just after it's all fixed.

Page 1 of 2