Crossposting to LJ throttled; LJ imports slow/failing

[denise]

As LJ has now officially confirmed that they're experiencing a DDoS attack, and a pretty nasty one at that, we've throttled the rate at which we attempt to crosspost entries, and reduced the number of times the process will try before it fails. (It was five tries at an interval of 10 seconds, 30 seconds, 60 seconds, 5 minutes, and 10 minutes; now it's three tries at 1 minute, 5 minutes, and 15 minutes.)

You don't need to do anything differently when posting your entries, but if you receive a failure notice after the third try (which should happen around 20 minutes after you try to crosspost your entry), wait a while and then try to crosspost again, by editing the entry, making sure the crosspost box is checked, and saving the entry.

If the crossposting continues to fail after that, just hang tight and wait a day or so before you try again. LJ's operations and engineering teams are doing everything they can to restore access to the site, including restoring the ability to post from elsewhere using the LJ protocol (which is what the Dreamwidth crossposter uses).

Importing from LJ is also affected by this, obviously, which is driving up the import queue. If you can, please consider waiting a few days to import your journal from LJ -- the process is likely to fail right now anyway. If you get a failure in your inbox when trying to import, just hang tight and try again in a few days.

If the traffic situation at LJ continues to worsen, or if the backup gets too large, we may temporarily disable LJ as an import source, in order to let the queue clear out a bit. If that happens, I'll post here about it again!

We wish LJ and the LJ team the absolute best of luck in dealing with this situation, and we hope that they can recover quickly!


EDIT: And the slowness on our end that some people were noting turned out to be another stuck process and Apache in general sucking up too much memory. alierak is monitoring and fixing.

EDIT #2: (1) The problems on our end involve some bug that's causing the webservers to work too hard that we're trying really hard to chase down the source of. If DW is slow or sluggish for you, that's the cause. We've put some measures in place to deal with it temporarily, and we're working on tracing down the root source of the problem. It's not traffic-related!

(2) If you comment, please try to keep it on the topic of Dreamwidth, not LJ -- we'd like to avoid the comments turning into a LJ-bashing session. Thanks, all.

Comments

[snailbones]

Just wanted to add my thanks to you for keeping us up to date.

[lady_ganesh]

It's worrying.

[gamgeefest]

Can anyone answer this question for me?

I had imported my LJ when I first sent up my DW account, but not since. If I import it again, will it import all posts all over again, or only the posts not already imported? I saw no place to enter a date range on the import form.

Thanks!

[archangelbeth]

It is entirely possible that I am not auto-crossposting for some reason (quite possibly mere control issues; I am mistress of the ticky-box!), and merely automatically check-and-click each time!

[archangelbeth]

From memory, on my tablet (can't check FAQs easily), and just a user...

It will keep privacy settings that you import, definitely -- private or custom stays that way. I imported from default-public to default-public, so I don't know how different defaults would work out. I suspect gracefully but will leave that to a real DW person. O:>

I think you may have to set up custom filters? Or it may indeed import them as a separate step? I don't recall having a problem, and I have many custom filters, so I bet it imported them.

I wonder if DW could do mass privacy default changes based on date. E.g., everything marked "default" in 2009, change to "circle-only". It would probably be a one-way change, though... That sounds like a Suggestion-fodder thing, though. I must try to remember it when off the tablet.

[archangelbeth]

I am not an official DW person, and have not yet remembered I want to re-sync my LJ with DW (I always think of it at times when LJ is lightly frotzed or worse), but I believe it will only bring in new comments and new posts. If you have been using DW to crosspost, it will not duplicate crossposted posts, either. Or so says my memory.

[susanreads]

Thnak you for keeping us up to date.

I hope the DDOS subsides soon, there's a post I keep being linked to on LJ that I haven't been able to load at all. (... and for less selfish reasons ...)

Data points on the slowdown: yesterday evening (British time), DW was awfully slow to load, but everything did load if I left it long enough. Today it's still slow, but not so bad as far as I can tell.

[musyc]

Every time I see a statement like this, I have to stop and wonder what's being talked about, since my color scheme is a delightful purple. Since it looks like you're brand new to DW, you might not be aware that you can change that. Go to http://www.dreamwidth.org/manage/settings/?cat=display and skim down to "site skin". Tropospherical Red is the default, but one of the other options might do you much better. I ♥ Tropo Purple. First social based site I've ever used that let me have purple.

SPEAKING OF. denise? What about getting a link in that section that shows thumbnails of each scheme. Easy enough to do? Worth dropping a note in Suggestions? I can't ever remember what Celerity or Gradation look like without switching to them.

[discodiva76]

V. Strange..for me DW was buzzing along nicely at a fast pace for most of the day and now it's like wading through treacle....what with LJ being down and this my blogging hours seem very limited now.....*sighs*...


Good luck with sorting the problems....am giving up loading icons to my nice new "gifted" account as well....*sighs again*...

[ninetydegrees]

(Wow, nice pics of Tom!)

What about getting a link in that section that shows thumbnails of each scheme.

I think there's a bug about this and someone's working on it... *browses* Yep! click.

[musyc]

(Aren't they? XD )

Oh, excellent. One of the other many things I love about DW. 75% of the time I have a suggestion, someone's already thought of it. Sometimes I'll poke around in Bugzilla just to see WHAT someone's already thought of. XD

[dickgrayson]

For future reference, what kind of privacy settings are there on the importer? Can it be set to keep each entry's settings as it is on LJ, or import everything (regardless of original privacy level) as access-locked or private?
I was wondering this also. ^^;

[dickgrayson]

I did not know about the 'site skin' Thank you so much! I liked purple best too. I may change to Gradation Horizontal at some point too. XD

[dickgrayson]

Thank you for the updates and thank you for not letting this turn into a LJ bashing post.

I do wonder if there is anything I can do for the LJ team. From what I read they are having a very hard time. I would love for them to have some positive words to hear. I really do think they do a great job and miss getting on LJ every day.

[olchik]

:)

[katikat]

Not to sound pushy, guys, but when will your problem be fixed? DW's moving molasses slow right now, reaching LJ's level. It takes a minute or even longer to load a page.

[biscuit1001]

Yes! Thank you, I found it last night while poking around, it never occurred to me to look for a site skin, since (like you) no other social media site I've been to allows that.

I like the purple best also. I think I'm gonna like it it here. :)

[denise]

The problem will be fixed as soon as we can figure out where the memory leak in the code is. It's not easy to find.

I know you guys are working as hard as you can...

[jhumor]

Buuuuut... This is honestly really very bad. I mean, I lost all connection to DW yesterday afternoon for about 30 minutes or so. Are you sure DW is safe from the DDoS attack that LJ has had (With all the cross-posting, it wouldn't surprise me, if they 'found a way'?).

From what I can tell (and I've left LJ), DW is running tons slower than LJ right now (as of this posting).

I know reading code isn't easy, but is there a way you can put a halt on things you know could suck bandwidth (like cross-posting/importing) at least until this bug is fixed?

Re: I know you guys are working as hard as you can...

[denise]

It's not due to load, or at least, not directly due to load -- see my latest post in dw_maintenance just now. There's a bug somewhere in our code that didn't manifest until under load, that's causing the webservers to lock up entirely, and the steps we've been taking to try to work around the problem have been what's causing the slowness. It has nothing to do with bandwidth, trust me!

We're taking a two-prong approach to handle the problem, trying to find the cause of the memory leaks and working to add in more webserver resources. I know it's frustrating, but please be patient.

[jhumor]

Thanks for the update! You must have been working on it when I was working on this.

Sorry if I came off too whiney... it had just been so long since an official update was posted (hard to tell the 'times' when edits are made) and the site is working worse for me than before (other than when it was totally inaccessible to me). Anyway...

I really do appreciate how much you do communicate with us!

For LJ refugees: comment pages

[moragmacpherson]

I'm part of a community that had set up a ICE community over here back in April. It didn't get used much because none of us could figure out how to have a plain style comments page to track our (often extensive) threads (and also the way I prefer to display my fic, in order to maximize screen space). So I started experimenting and after some wrangling, I figured out how to get plain comment pages for both your own journal and your comm here. It's actually rather simple - though the big catch is that the comm can't change the settings for everyone - each member has to do it individually.

1) Go to account settings.
2) On the settings page, select the "Display" tab
3) Scroll down the options until you reach "Entry View Style" -- it's right after "Site Skin". Choose to view Entry Pages in Light Format.
4) Scroll down just a few more options to "Comment Pages" and tick the box that says "View comment pages in your own journal style"
5) Save changes

I think the main reason it took so long for me to figure this out was because of I had no idea what an entry page is - it appears to be the same thing as a comment page so my eyes kept glazing right over it - a little consistency in the terminology might help make this more intuitive.

Also, adding to option for communities to make their default comments/entry page Light Format (or even Site Format) might be helpful.

Sorry for adding to the server snarl in figuring all this out, and I hope this is the right place to post it - I'm still learning my way around here - let me know if there's a better place to put it.

[unixronin]

I am continually saddened by the number of people whom, when presented with something as brilliant, as powerful, as useful and as wonderful as the Internet, can think of no more constructive way to use it than to look for bits of it to smash.

[gamgeefest]

Ok, thanks! Guess I'll give it a go and see what happens. :)

[denise]

The importer keeps the same security level on DW as the entries had on LJ, unless your journal's minimum security level is set to a more restrictive level. If you wanted to import everything as "private", you'd set your journal minimum security to private, run the import, wait until the import was done, and then set the minimum security back to whatever you wanted it to be. (I actually did that with another journal of mine; it was very useful!)