slicehost followup

[mark]

Just a followup on yesterday's downtime:

I just spoke with Matt of Slicehost (not sure what his official title is, but he started/runs the place). He had looked into the various incidents we've had over the past few weeks, and it seems that they are all similar: hardware failure of some sort or another.

The good news is, this isn't entirely unreasonable. We made the choice to move to Slicehost's brand new data center in the Dallas/Ft. Worth area. One of the side effects of having new machines is that you have to work out the kinks in them. While it's unfortunate we've hit so many, they are not too concerned about the number, given how many slices we have.

Slicehost is working on rolling out stronger monitoring so they can try to predict failures and better account for them when they do happen. It's sad to have to deal with them, but our system is pretty redundant. There are only two machines (out of 25-30) that take the site down when they fail, so it's pretty bad luck that we've had both of those go down in the past few weeks.

While there is not much action that can be taken at this point, it was a good conversation, and it does serve to emphasize just how top notch Slicehost's customer service is. They are fast, efficient, and take care of problems quickly.

If you have any questions, please let me know!

Comments

[j00j]

Thanks as always for explaining what's going on!

[sara]

Can you schedule "downtime" for most weeknights after I've gotten the kids to bed? About 8:45 to 9:15, PST?

Motivates me to get off my duff and walk the dog, when my crack internets go away.... *GRIN*

[denise]

I'll talk to the gods of hardware failure. ;)

[sara]

Selective internetectomy! You could sell that, D.

[ravan]

Infant mortality of new equipment is always a bitch. The only prevention I know isn't 100% effective: long burn in (including power fluxes) and buying top grade hardware. Good monitoring and redundancy helps, too. Designing system cloning and failover is an art.

[mark]

Yeah, I expect that given the general target market and low prices of Slicehost, they use commodity hardware. Which is usually fine for an LJ based site: it runs very well with commodity webservers, mailserver, jobservers... just not commodity load balancers or global/master databases!

[ravan]

Yeah, well even major online retailers, like the one I used to work for, use commodity server hardware. They pay big bucks, though, for the HA versions of their DB software, hardware firewalls, and specialized stuff. Even they still have problems.

[babysprite]

We host a library webserver on an Enterprise machine running Enterprise SuSE v. 11 and it had infancy problems, too. ^··^~

[flurgh]

Hey, that's pretty nifty. My best friend lives close to that data center. I'll wave at the DW servers the next time I go over there.

[luludi]

I think it's way awesome how on top of everything you guys are and how you share what's going on with everyone. It's nice to be in the loop!

That said, I'm on DW a lot and I have yet to notice any downtime, if that makes you feel any better =)

[ex_heracles30]

I was about to say the same thing.

[jaaaarne]

I'll second that. :) I have yet to notice a downtime here. :)

[pandoraculpa]

Thanks for keeping us informed and in the loop. I didn't notice the glitch, but I so appreciate that you guys work hard not only to keep the site running top-notch, but to let us know how and why things mess up when they do (as they must, sadly).

Thanks for all the dedication, guys. It really is appreciated!

[juste]

Echoing everyone else when I say thanks so much for keeping us in the loop! It's nice to see that you guys are staying on top of things so well and that you always find time to let us know what's going on, from the minor things to the major things. It means a lot!

[rainne]

and it does serve to emphasize just how top notch Slicehost's customer service is.

And this post just serves to emphasize how top notch Dreamwidth's customer service is.

Thanks for the heads-up.

[tetra]

Alright, I'm a sick of LJ and wish to found some of the better communities from there to here.

I'm on a free account now (month long) and should be purchasing a paid once the money is available but in the meantime I was wondering...

Do I have to pay for the communities I found to stay up or will my having a paid account myself transfer to the communities I found?

[afuna]

Your paid status will not transfer over to communities. However (like LJ), you do not need to pay for your communities in order for them to stay up, and you can make communities no matter your account level.

You can only create three communities per week, by the way, to counteract abuse. (The post I linked to explains it more clearly than I could :))

[tetra]

Neat, thanks.

[tetra]

Oh, and what happens if my month expires and I'm unable to get the moneez to purchase a paid account?

I'M SKURRD

[afuna]

If your paid account expires,it lapses to a free account, with fewer icons, lower limits on some things (see this faq for the actual breakdown of differences, but your account will remain active as a free account, so you don't need to worry! (In case you're worried that you'll need to be paid in order to maintain an account on Dreamwidth ( if I misread, I apologize *g*))

[tetra]

Oh, not at all.

I was just worried that all you could have were paid accounts but it seems they only want an "initial donation" of sorts...which I can appreciate.

[bloodcurdling]

I've been using this journal to import all of my LiveJournal entries to. Does it automatically refresh and add new ones or am I supposed to periodically import new entries to this journal? I just noticed that it hasn't imported anything for weeks.