dw_maintenance | Recent Entries

Crossposting to LiveJournal continues to be hit-or-miss: if LiveJournal is having technical difficulties or connection problems at the time you try to crosspost, the crosspost will fail and you'll get a long and messy error in your DW inbox.

(To complicate things, sometimes the crosspost attempt will actually succeed, but LJ will report a failure anyway. You should double-check to see if the post actually did crosspost anyway if you get an error.)

Posts will attempt to crosspost several times before failing permanently. Once you get the notification of the permanent failure, you can try to crosspost again by editing the entry and checking the crosspost checkbox.

Also, a very happy new year (Gregorian) from us all to you!

mark

Hi all,

Due to some recent availability problems with accessing the LiveJournal service, crossposts may be slow. The system does automatically retry, but if it's a slow spot it could be minutes or even an hour or two before we can get the entry posted.

Thank you for your patience!

denise

After some investigation, we've realized that our two new webservers are having a problem with the OpenID modules installed on them. (For those who care about that sort of thing: it's a case of version incompatability; newer versions of the OpenID modules require (without telling us that they require, ugh) a more up-to-date version of the JSON modules than we have installed or have tested against. So, we have to do some regression testing before we can just upgrade.)

So, for at least the next few days, you might notice that the option for commenting as an OpenID account doesn't appear sometimes when you load a comment form, or if you're logged into an OpenID account you might randomly get weird errors or seem to be logged out, etc. If that happens, just refresh the page until the problem clears up. (When you ask for a page, you get sent to the webserver that's next up, so about 1/4th of the time you'll get one of the webservers with the problem.)

We'll have this fixed as soon as possible! Sorry about the trouble.

denise

Alas, the temporary fixes

alierak was able to put in place to baby site search along for a little while have proven to not be a match for your text-generating capacity. Search is down again, and probably will be for at least a little while until we're able to achieve a more long-term fix! Sorry about the inconvenience, folks; we'll have it back as soon as we can.

(EDIT: And, back up it goooooooes! Bouncey bouncey bouncey.

alierak has been exerting herculean efforts to squeeze every last bit out of the current setup, so search may or may not be available at any given moment. It's Schrödinger's search! So, if you want to search for something, give it a try, but don't be surprised if it's slow, out of date, or down entirely at any given moment.)

mark

Hi all!

Edit: This is done, the code has been pushed!

In the next few minutes, we'll be doing a code push on Dreamwidth. You can expect a bit of a rocky experience tonight, since we will be not only deploying new code, but also doubling our caches -- which means that the site will slow down for a while, then will get faster than it was!

If you see any problems, please comment here and let us know. We have all hands on deck to make sure everything goes smoothly.

(PS, special shout out to denise, fu, exor674, and geekosaur who are in IRC helping out or keeping company while this happens. You folks are awesome!)

denise

1) To follow up on the issue with the site search index not updating: We managed to make a few changes that will allow us to baby things along for a little while longer. (And by 'we', in this case I actually mean

alierak!) So, it will take slightly longer for new posts to appear in the search index, but they should be searchable once more.

2) In addition to the code push we have scheduled this weekend (see previous announcement for details and times), we will be adding two new webservers sometime in the next week or so. (And by 'we', I definitely mean

mark this time.) Until our provider can build them and Mark can configure them properly and add them into the webserver pool, the site may be a bit sluggish at peak times, usually early evening.

I like this thing we have going on here where all of our problems this week are a result of people using the site to make and do awesome stuff. CARRY ON, FOLKS.

denise

After some investigation,

mark has realized that our little website has grown up to the point where our site search setup is no longer sufficient for the amount of data y'all have generated. This is an exciting problem to have, but it also means that new posts and newly-imported posts won't be indexed into the search database until we can fix it! (And by 'we', I mean 'Mark'.)

We've got a plan that involves a new search database server (to go along with the two new webservers we ordered today), a new database schema that will make indexing, storing, and accessing search data much faster, and a method of storing and retrieving search data that makes more sense for the way usage has developed now that we have more information on what actual usage looks like. (And by 'we', I once again mean 'Mark'. There is a theme here.)

The plan is going to take a week or two to implement, though, and until then, search results will continue to be out of date. Anything that is already in the search database (as of about 10 days ago) will still be searchable, but newly-made posts and newly-imported posts won't show up until after we can fix it.

We'll let you know when the problem is fixed. Until then, continue on being awesome and making stuff: we'll catch up to you as soon as possible.

denise

We are tentatively planning a code push this weekend, at roughly midnight Saturday EST/9PM Saturday PST/5AM Sunday UTC. (See this in your time zone.)

With luck you won't even notice it (except for the new stuff afterwards, of course!) as it shouldn't involve downtime, but we'll keep you posted!

denise

For people who got bitten by LJ's bug that was causing duplicate content in your account after importing content from LJ, we've finished the tool that will help you clean up your account without having to delete each individual post or comment.

This tool will remove all imported content from your account, not just duplicate imported content. (Basically, there's no way for us to tell which version of a post or comment is the 'original' and which is the duplicate, so we need to remove them both and let you re-import from scratch.) Once you've removed all imported content, you can then re-import your content from scratch.

Warnings: This can't be undone. Once you hit the button to delete all imported content, it can't be stopped. It will remove all imported content from your account, no matter how long ago you imported it. If you've deleted posts or comments on LJ since you first imported, they can't be re-imported into your account on Dreamwidth after you remove all imported content.

Reassurances: It will only touch imported content (with the exception of: If an entry was imported, and then someone commented on that entry on Dreamwidth, running this tool will delete that entry and all associated comments, including any comments that were made on Dreamwidth.) We've tested it pretty heavily (although there's always a chance for weird bugs).

If you want to use the tool to clean up your account, it's here.

EDIT: Also, I forgot to mention: if you're one of the people who were affected by the issue with the system telling you that your comments had imported successfully but the comments weren't appearing in your account, you can now try your import again, and it should work this time!

mark

(I am posting this on behalf of zarhooie, one of our Support Administrators.)

Hello Dreamwidth! Kat here!

As you may have noticed, we are experiencing some issues with importing comments from LiveJournal. They made some changes to their code, which broke the way our importer retrieves comments. Mark knows what is wrong, and is working on it. Unfortunately, it's going to be a day or two before it's fixed.

For right now, hold tight and don't retry your import. Once it's fixed, someone will post here and then you can try again. Thank you for your patience!

Best,
Kat (zarhooie)
Dreamwidth Support

denise

Some people are having problems loading images (icons in particular) and static content such as Javascript right now -- we've been testing out a new content delivery network (CDN) to speed up access to static content, and it seems like a small percentage of people can't load things from the network. Hang tight, we'll get it fixed as soon as possible.

denise

The issue with crossposting not working is due to our crosspost worker being unable to reach LJ intermittently. We'll continue to look into any possible changes we can make on our end to mitigate the problem, but at this point we do believe the issue is on LJ's end and isn't something we can work around completely.

This does mean that crossposting will probably fail, although we are seeing a small percentage of attempts get through. So it's kind of hit-or-miss, mostly miss, at the moment, and we can't begin to guess when the problem might clear up.

The reason people aren't getting notifications of crosspost failure is that the system will try multiple times, with larger intervals in between attempts, before failing completely. The delay in between the fourth attempt and the fifth and last attempt (and final failure) is pretty long, so you'll probably start to get failures in your inbox fairly soon. The reason the crosspost box is unchecked when you edit the entry is because the entry has not yet been crossposted -- the box only appears checked once the entry is crossposted.

Once the attempt to crosspost fails and you receive the final failure message in your inbox, you can try to crosspost it again by editing the entry, checking the box, and saving the entry again. This will kick off a new attempt to crosspost the entry, although you may want to wait until new entries are reliably being crossposted again.

There's no need to let us know that you're having the problem too -- pretty much everybody will be having it at the moment! Crossposts to other sites will probably also be slow to complete, because of the queue being taken up by delays to LJ.

denise

We're seeing a number of people reporting difficulty with crosposting to LiveJournal right now. We're looking into the problem!

mark

Hi all!

We've pushed live the latest code for Dreamwidth. There are some major internal changes to how we serve up files, so you might see some things being a little weird. Please report anything that isn't loading, looks broken, or what-have-you!

Thanks for your patience!

denise

A reminder to all that we will be doing a code push tonight. We'll be starting the prepwork in a few hours (around 6PM PDT/9PM EDT/1AM UTC), and the actual push itself will likely be an hour or two after that.

As I mentioned last time, this is a "higher risk" push. Sometimes we're only pushing frontend changes that have very little chance of causing site-wide problems, and any bugs we didn't catch in testing (because there are always bugs we don't catch in testing!) are only likely to be display bugs or the occasional weird error on a single page. This push has a lot of backend changes, though, and the differences between our production setup (the site itself) and our testing setup mean that there's a greater chance of things going a little wonky. (That's a technical term.)

So, there may be unexpected downtime as part of the push, and there may be weird site-wide errors after we come back up. We will be watching for them, and if they happen, we'll fix them as quickly as we can. Don't worry! We'll be on it.

denise

We will be doing a code push on Saturday 22 Sept. Prep work should begin at around 6PM PDT (9PM EDT (UTC -4), 1AM Sunday 23 Sept UTC), with the actual release itself likely happening anywhere between 7PM and 8PM PDT.

This push is going to be slightly more involved than our usual code deployment process. (It's the first push using our new backend version control system and we need to do a bunch of work for the conversion, much of which has to be done at the time of the push and not before, which is why we can't give you a more accurate estimate.) The actual downtime involved should hopefully be minimal -- as is standard with a code push, at most it should be a few minutes of actual downtime -- but brace yourself for the possibility of brief Utter Weirdness. We'll have all hands on deck to fix any problems that crop up.

mark

Thank you all for your patience, we finished the maintenance and Dreamwidth is back up. This push had a few new features, stay tuned to dw_news for more information shortly!

As always, if you have any problems, please let us know! We will be standing by. :)

denise

We'll be starting our planned maintenance shortly. (As usual, we're running a bit behind; there's always something more to do before we can start!) Remember, the site will be completely down for around an hour or so while we do some database and server maintenance in addition to our usual code push process. Hang tight, and when we come back we will have a whole bunch of totally awesome news.

mark

Hi all!

In approximately 24 hours, we will be doing a code push on Dreamwidth. More than just a code push, though, we will be taking the site offline for one hour. We are doing this to allow us to do some system updates and some general maintenance that we've been putting off for a while.

The impetus for doing it this week is that we have here, in Portland, Oregon, many of our senior contributors. We flew nearly a dozen people out for a conference here, and have been having a good time hanging out, working on Dreamwidth, and doing conference things.

We will be doing the code push at 9PM PST (Pacific), also known as 0400 UTC for more civilized timekeeping devices.

And one final time: the site will be entirely offline for 60 minutes. Thank you for your patience and understanding, and as always, we'll be watching this and further posts for comments and questions!

mark

Hi,

I have kicked off a purge of accounts that have been deleted for a while. This should have no impact on the service, but I wanted to let you know in case you were waiting on a username to free up.

As always, let me know if you see anything terrible or broken! Thank you.

Profile

Dreamwidth Maintenance

June 2013

S	M	T	W	T	F	S
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30

Syndicate

Page Summary

Active Entries

1: Planned load balancer failover shortly
2: Code pushed
3: Payments are back
4: New hardware deployment
5: Code push tonight
6: Downtime this morning
7: Code push happening
8: Load issues / site search
9: Heads up: Replying from comment notification emails will be changing with next code push
10: Code update

Style Credit

Style: Velvet Steel for Practicality by timeasmymeasure

Expand Cut Tags

No cut tags

Page generated Jun. 19th, 2013 03:53 am

Dreamwidth Maintenance

Recent Entries

Crossposting to LiveJournal, redux

Crossposting to LiveJournal

OpenID logins/commenting on the two new webservers

the amazing bouncing site search!

Code push coming shortly

Updates!

Search indexing not currently updating

Code push this weekend

Imported content removal tool

Update on the importer

images and JS loading problems

update on crosspost problems

Crossposting difficulties

Code pushed

Code push tonight

Code push Saturday

Site back up!

Maintenance beginning soon

Upcoming code push + maintenance downtime

Deleted account purge in progress

Profile

June 2013

Syndicate

Most Popular Tags

Page Summary

Active Entries

Style Credit

Expand Cut Tags