dw_maintenance | (no subject)

(no subject)

We've received multiple reports of crossposting to LiveJournal silently failing, and many people have asked us about the problem. Unfortunately, the issue is not one that we can resolve: it happens when someone has automatic crossposting set up and has changed their LiveJournal password, but hasn't updated the password on their Dreamwidth crosspost settings. The number of failed logins trip LiveJournal's "attempted account hacking" detection system and our IP addresses are locked out from the site. This stops all Dreamwidth users from crossposting, whether or not their passwords are correct: we are entirely blocked from accessing LiveJournal at all.

For a decade or so, when this happened, we were able to contact them and have them remove us from the lockout list manually and whitelist our IP address range so that it wouldn't happen again. At some point in the last several years, they switched to telling us that the blocking is automatic, they aren't able to whitelist any IPs manually, and that we were mistaken when we were able to resolve the issue for a decade by emailing them and having them whitelist us manually; by this point, they no longer respond to us at all about the issue. (The delay in our posting about the issue has been, once again, us attempting to contact them unsuccessfully and getting no reply.)

Many users report to us that when they've contacted LiveJournal about the issue, LiveJournal has told them that the problem is on our end and that there are changes we can make to fix the problem. This is not true, and we aren't sure why they're telling you that. There is nothing we can do differently that will make them stop locking us out when people enter incorrect passwords for their crosspost accounts. At most, we could spend months making a massive, extensive, exhaustive effort rewriting the entire crosspost system, in exchange for an infinitesimal chance it would reduce the number of times this problem happens, not prevent it entirely. We aren't willing to spend six full months rewriting the crossposting system from scratch (and introducing all kinds of inevitable new bugs) to slightly lower the chance this problem will happen as frequently as it does, without assurances from LiveJournal that they are unwilling or unable to give us that such a rewrite would solve the issue entirely.

The only long-term, guaranteed fix for the issue is for LiveJournal to whitelist our IP addresses the way they were able to do for a decade and now claim to never have been able to do at all.

I know people are frustrated about the issue: so are we, and we're disappointed that the collegial relationship we enjoyed with LiveJournal for so long appears to have evaporated at some point in the last several years. However, at this point, we must advise you to treat crossposting to LiveJournal as an "as is" service that may or may not work at any given moment, depending entirely on LiveJournal's whims regarding blocking our ability to access the service due to people having entered their crossposting passwords wrong. If problems happen during the process of crossposting, there is nothing we can do to fix it, solve it, or hasten the site's removal from LiveJournal's access blocking system. All we can do is wait until the block expires, and they will neither tell us how long any given block will be active for nor a date/time when it will expire -- as I said, at this point they no longer respond to me when I email them about the issue, and haven't for quite some time.

If you have a crossposting account set up with LiveJournal as the crosspost destination, please check that the username and password are correct: the fewer people with incorrect passwords in their crossposting setup, the less likely this problem will be triggered. However, when it does happen, there is nothing we can do to solve it, and all you can do is wait it out. I'm very sorry. We're just as frustrated as you are.

ETA: I forgot to specify: yes, this includes the importer as well. The importer is more likely to catch a period of being unblocked because of how it functions asynchronously, but it's not a guarantee. If LiveJournal is currently blocking us, no function on Dreamwidth that requires communicating with LiveJournal will work, period.

UPDATE 1 Feb 2022 21:15 EDT: We have posted a further announcement: further investigation has made us realize there are several potential causes of the block other than the "password error" cause I explained in the post above. Until we're able to a) determine the cause of the block and b) determine whether there's any way we can work around it that will not cost us a significant amount of time and effort for potentially no reward, we have temporarily disabled crossposting to LiveJournal site-wide. We deeply regret needing to do this; however, it's clear that even if the situation is fixable, which it may not be, the resolution will take a significant amount of time. Until we have more information, it doesn't make sense for us to attempt crossposts we are aware have no chance of actually succeeding.

If you try to crosspost to LiveJournal, the process will automatically and immediately fail, and you'll receive a failure notice in your inbox directing you to this post. Please accept our utmost apologies for needing to take this step. We'll continue trying to identify the root cause of the difficulty and see if there is any fix we can make to re-enable crossposting that doesn't involve having to redesign the entire crosspost system from the ground up with no hopes of that even helping.

Flat | Top-Level Comments Only | Expand All

We may already do that, actually? I think. (It's been a long time since I've looked at the crossposter code.) When these whole shenanigans started we did genuinely try to make a bunch of changes to avoid tripping their detectors, but eventually we realized the only way to guarantee it would work would be for LJ to cooperate with us to an extent they were clearly unwilling or unable to do, and honestly, the amount of time we've spent making massive changes based on the vague and handwavey guidance they are willing to give, and then having them change their mind completely about that being the thing we needed to do, means that I am just so completely, immensely over this. When the other party to a discussion starts denying a decade of reality for which you have extensive, exhaustive documentation that you have provided them and their answer is that obviously you are mistaken, the chance of working anything out is over and it's time to just cut your losses and stop even sending messages into the black hole for them to ignore.

I am 99% sure they are very, very mad at us still that we, Firefox, and Troy Hunt of Have I Been Pwned had to reveal the massive, extensive, prolonged and wide-ranging data breech they had after they refused to disclose to their users for over two years of said massive etcetera and that's why they've stopped even pretending to work with us, but, hey.

I apologize for knowing that you don't do that, because I was one of the people who changed their LJ password (in response to the data breach) and then looked at the crossposter errors and thought, you know, I don't really care enough about LJ crossposting to fix it. So I know that it does cause repeated errors.

Which shows how much I care about fixing this for myself, but I do regret that it causes problems for other people.

And, yeah, I totally agree that trying to make code on our side conform to the undocumented whims of a block-system from someone who is at very best completely apathetic to the matter is a fool's errand.

Edited 2022-01-31 01:39 (UTC)

Update that password if you can (or just shut off the crossposting account!) to reduce the number of failed logins, but yeah, "fool's errand" is a very good descriptor at this point. I have just hit total and complete fuckruptcy on dealing with their shit. (This is absolutely not professional of me to say and I probably shouldn't say it! But it is true!)

Yup, I have shut off the crossposting. :)

https://www.youtube.com/watch?v=Vqbk9cDX0l0 seems appropriate here, meanwhile.

I absolutely knew exactly what video you were linking to before I even clicked on it, and hey look, I was right! So much love for that song.

kiya has also pointed me at https://www.youtube.com/watch?v=iB1c68B1iLg, which is a different (and excellent) version of it.

Also https://www.youtube.com/watch?v=sD3WJNa_NS4 (from my wife and studio partner

cyanna) with 100% more Aggretsuko!

"Fuckruptcy" is my new favorite word.

Same.

Having just had my head dug in the crossposter and importer code for a bit: we explicitly check for bad password and Stop Trying in the importer, but not in the way that brooksmoses is suggesting for the crossposter. (Xpost does, however, feed the error message from XMLRPC back up to the user, so on the chance that you get an actual message instead of the silent block, it's reasonably obvious that's what happened.)

Huh, probably worth failing on first attempt with a bad password then, just to minimize these blocks, if somebody cares enough to take the time to change it! it will inevitably fail to have any effect when they decide the next excuse to block us, but it's worth a try

Or, based on

arethinn's comment below, I may have just not traced out all of what's going on. Legacy Perl, here be dragons, etc etc.

You absolutely can't set up xpost with a bad pw, the initial challenge just fails.

Edited 2022-01-31 04:22 (UTC)

don't worry about it too much, honestly; if it's a challenge you're enjoying by all means have at, but otherwise, there's really a limit of how much of their fuckery we can work around and I hit it at the point at which they LIED TO MY FACE and then LIED TO THEIR USERS about their massive server compromise when every single one of the records I checked for my LJ accounts, including the ones that had never been used and the ones used for testing in such a way as to get them into extremely weird states that would not be possible through normal use and had characteristics that could not be determined through any means other than direct db access, was completely correct. If we fix this they'll block us for some other invented reason. They've decided they're done with us and will seize on any excuse.

No, this is just stuff I was noticing on my way past grokking what the IJ fix (noted elsethread) needed. I have zero interest in bending over to try to work with an actively malicious platform.

I am not a coder so what I said could possibly have been misleading. What I meant was that when I see a failure message from the crossposter in my DW inbox, then the xpost box on the new post page becomes unchecked. I don't know anything about how many times it keeps trying before it gets to that point.

That's still more than I thought it was set up to do! So thanks for the report from an end-user viewpoint. :)

I have a whole series of questions about "oh, I'm curious, why don't we do [thing X which in my mind would solve it]?" except I know that the people currently doing the coding have clearly thought of it and have good reasons not to do it / know why it wouldn't work, and even if the reason is just "not enough coding resources," and I shouldn't ask the question if I personally don't have the spoons to attempt a change myself, so I'm going to stifle down my curiosity so I don't make it the problem of the current coders or D to explain to me why my idea is wrong/infeasible. 😁

Hey! I will cheerfully answer questions (to the best of my ability; I'm still a very inexperienced dev in the context of DW).

And as noted in some of my other commenting, we may be doing the Right Thing in a place I hadn't quite wrapped my head around. A codebase that's old enough to drink gets...weird around the edges sometimes. :)

Now I'm thinking of something like the anthropomorphic fic category in Yuletide: "Dreamwidth walks into a bar...."

Thanks for offering! Here's the thing I'm thinking about, which as I ask I can already see some of the problems with: why not have the user's browser send the crosspost request directly? So the IP is the user's?

And as I ask I am already thinking through some of the problems and answering my own questions, ha!

DW is very much an entirely server-side application. Running anything from direct client side API calls to third parties would requires some rewrites on fundamental and old parts of the code. So that's just a question of resources.
It's not like there's any of the modern toolsets in DW which are designed for this sort of direct-from-the-browser communication, and while generally I think the lack of React/Node is one of the things that makes DW so good and stable, it does mean that in this case any browser-origin communication would be writing something brand new.
The current crosspost architecture allows retries with exponential backoff, as I know D said upstream. Client side crosspost means that the user effectively gets one try, and it could get interrupted by any number of things.
Handling passwords securely becomes a mess, doesn't it? I don't remember if the LJ XMLRPC stuff requires any kind of API key, and on top of that the LJ password has to be something which can be passed back to the browser making a call. Very doable, but it's precisely the sort of security issue that even large paid teams often get wrong.

In order:

1/2) yeah, that's an entire paradigm shift, and would need someone who understands the current stuff well enough to make those changes. We have...not too many active developers right now, and even fewer who grok the codebase at a deep level.

3) I do not trust users to do exponential back off. Do you?

4) LJXMLRPC does not handle password security particularly well. Crossposting to a DW account (which is possible, by the by) is set up to use an API key instead of the password specifically because we are so limited by the existing architecture standard.

Yeah, chances are very good we could design something that would work for at least a little while (my brain went running down much the same line of thought as yours when I started thinking about it), but the level of effort it would require, with no guarantee it would work, means it's something I wouldn't be willing to put resources into without some kind of indication it would work. (As I'm sure you've noticed, time and energy is kind of a bit scarce around here this past year or so, and to the extent it's improving now, I'd rather it go to more local improvements, you know?)

It's looking like it's a moot point, anyway: it's very hard to get accurate information even with machine translation and my rudimentary Russian because they keep changing where they post to, but it's starting to look like, in addition to the usual "you have tripped our spam detection", LJ has simply shut off all third-party client access, period. Trying to confirm that, but if that's the case, it would never work again even if they didn't have a grudge.

(no subject)

deborah - 2022-02-01 15:12 (UTC) - Expand

(no subject)

batrachian - 2022-02-01 15:45 (UTC) - Expand

(no subject)

denise - 2022-02-01 15:48 (UTC) - Expand

The xposter does turn itself off if it actually gets a failure message. Maybe it needs to check for the absence of a success rather than the presence of a failure?

If DW doesn't disable crossposting for a user after multiple failed login attempts, I think it's reasonable for LJ's systems to treat further requests as suspicious. "Retry the same thing and hope it works" is only a reasonable client strategy if it gives up after a few failures.

We absolutely do stop trying after several tries that ALLEGEDLY is well within their password failure tolerance, the discussion above is about exactly where that point is (none of us could remember offhand). Crosspost and import failures use an incremental backoff that is far more generous than the limits they gave us back when they were communicating with us, but that was several owners ago and who knows what the tolerance is now; we've asked for confirmation and gotten no response.

The reason not to immediately stop at first failure is there are multiple scenarios that return a result from LJ's end that is indistinguishable from "password incorrect" but are in fact transient and fixable. Their API response codes are less than ideal. Now that I'm remembering, we did retry-with-incremental-backoff because at the time the vast majority of "password failures" were in fact transient errors that resolved themselves on subsequent attempts.

Again, while it's possible for us to go so far as to immediately completely disable any crossposting account that returns any response from LJ other than success and force our user to update all the info on the crosspost settings, that still doesn't change the fact LiveJournal repeatedly blocks us for a single instance of a single password entered incorrectly, and any after-the-fact disabling we do for those crosspost accounts can't change that the blocks have already happened. We have repeatedly made changes to the import and crosspost systems to respect their rate limiting, remain within the limits of their stated client policies, and implement every actual change they've asked us to implement over the years. If password failures were a genuine problem they could not whitelist us for and could not instruct us on how to make changes to avoid hitting, they would have been an ongoing and repeated problem in 2009, or 2014, or 2016, or 2018, or... You get the picture. While we have in the past had times when LiveJournal blocked our IPs, in the past we were always able to contact them and a) be unblocked and b) be whitelisted so we would not be blocked again. Their position now is that this was never possible at any point in the 13 years of DW's existence, despite me having sent them literal screenshots of past conversations of the repeated times they were able to do exactly that.

This is not a technical problem, is what I am saying.

Flat | Top-Level Comments Only | Expand All