Long-standing notification issue fixed

[mark]

While working on the notifications system, I discovered the cause of an issue that has been around for a while that caused a number of users to be unable to receive some notifications. In the worst cases, you would never get an email or inbox notification of new comments, no matter how many times you toggled it on.

This is now fixed.

The story is kind of gory but the short version is that a couple of years ago when we created a new database cluster and moved a bunch of users to it, some subscriptions got 'left behind' on the old cluster. Our subscriptions and notifications system was seeing these old notifications, but later in the process when attempting to generate a notification, was throwing them out because they didn't match the user's 'correct' cluster.

I've pruned the notification rules that were on the incorrect cluster for all users (about 1.1 million notification rules, actually) and this should resolve the issue where notifications didn't work in a number of cases.

Comments

[pronker]

*whoosh, info goes over head* but thanks for your work. :)

[mark]

Hehe sorry, and you're welcome!

[havocthecat]

Yikes! But also, wow, thanks for fixing that.

[ilyena_sylph]

+applauds you some more, with an adorable monster!+

[kore]

Thank you for the information and the transparency!

[althea_valara]

Congrats on fixing a long-standing bug!

Are notifications being delayed right now due to this? Because I noticed i haven't received a notification for this comment in either my DW inbox or email.

[sovay]

Are notifications being delayed right now due to this? Because I noticed i haven't received a notification for this comment in either my DW inbox or email.

I have also just received a comment without any form of notification.

[edit] I have now received a second comment without either e-mail or inbox notification. I am used to seeing both.

Thanks!

[compellingstatement]

Having the same problems. There are notifs that just seem to have dropped into the void. I'm getting them now, but the ones that I wasn't getting before haven't turned up. Even though I'm getting notifs on later tags from the same threads.

[alexseanchai]

\o/

[veritas_poet]

I've very rarely gotten notifications. I gave up expecting them. But no one reads my journal anyway these days, so I guess it doesn't matter.

[casimirian]

If you haven't yet, try finding journals to read and people to share your journal with this community. :)
https://addme.dreamwidth.org

[echan]

Huzzah!

And, separately, I greatly appreciate the details you provide when things are borked and/or fixed.

[mark]

Always happy to. I'm mostly rambling to myself but hope that somebody will think it's interesting, lol.

[asymbiotic]

Right now I'm not receiving notifications on at least this account (I've received two of an estimated 15 in the last two hours, both from the same account oddly enough) and several other people in rp-land are reporting no notifications either.

[mark]

So timelines are hard, and I want to make sure this is still ongoing before I debug it.

Can you test it sometime after I post this and see if it works? If you don't get a notification that you expected, can you give me the link to the comment that should have generated it? (this will help me check the notification logs and see what might have happened.)

[casimirian]

Wooooo! Awesome job!

Root cause analysis?

[dennisgorelik]

> was throwing them out because they didn't match the user's 'correct' cluster

Did you investigate why (and when) your system was programmed that way?

The correct behavior in the situation when code processes inconsistent data -- is to notify developers about unexpected situation that developers have to revisit and recode.

If code silently ignores problems - it leads to years of misdiagnosed misbehavior.

Re: Root cause analysis?

[mark]

Yeah, it was sometime in the early 2000s.

There are a lot of parts of the codebase that are old and full of cobwebs. In the 20 years since LiveJournal started, the art of building reliable web services has improved tremendously. I would love to have the time to rewrite Dreamwidth from scratch, but I'll settle for rewriting systems one by one ... and as I do, adding tons more debugging, metrics, defense-in-depth, modern design practices, etc.

But from time to time, one of the systems that hasn't gotten a modernization will rear its ugly head and we'll have to deal with it. Sucks, but that's the way of it. My present project is to remove the "homebuilt" queueing service (on top of MySQL) which has a tendency to overload the database and cause latency and other issues and replace it with Amazon's SQS. Along the way, this has involved replumbing the notification system, which has led to these adventures.

If you're interested in contributing, we always welcome new developers -- it's a volunteer crew and we're always happy to welcome newcomers.

[krait]

Congrats on tracking down the notification issue! I know it's been bugging (heh) people for a while now.




I am currently having an issue where trying to post a poll returns the message: Error updating journal: Poll error: All poll-question (or lj-pq) tags must be nested inside an enclosing poll (or lj-poll) tag.

I have opened a support ticket, but I'm really hoping there's a faster way to figure out what's going on, because the pole in question is kinda time-relevant (it's for holiday cards). Is there somewhere I can check to see if others are having the problem and find a solution?

[momijizukamori]

Hmm, I was just able to create a poll without any issue - if you add the code that's giving you the error to the support ticket, we might be able to do more debugging.

(in a pinch, I know a lot of people use google docs forms for holiday card info-gathering)

[pauamma]

Could that (or something like it) also be the cause of the double notifications I (and IIRC some others) am getting (in both email and my DW inbox) for some things, most notably newsposts?

[kalloway]

I get double newsposts but that's it, weirdly.

[graildiary]

what if we still have accounts that are not receiving notifs, or accounts getting double notifs? i have filed support tickets about these issues in the past and received no response :/

[mark]

Can you let me know a) which account and b) confirm these are email notifications of comments on that account's entries? Or if not, let me know which notifications you don't seem to get.

Ideally, please go do something to _cause_ a notification, and then let me know. I added tons more tooling to the system so I can look up individual notification events (not the content, but whether or not they fired), but I need to know which journal and when.

[darjeeling]

This problem has been plaguing some of my RP journals for over two years, I could kiss you right now for having fixed it. Thank you!!

[mark]

Haha, well thanks :)

Have you gotten a notification from one of them? Just to prove it's working?

[ct_7567]

Thanks for your hard work!

I've still been getting notifications sporadically and unpredictably - some have come in that are brand new while I know of older comments I got no notifications for and which haven't been showing up in my DW Inbox either.

[tessitura]

Seconding ct_7567 above

[their_nando]

Is this possibly why I'm receiving "This protected entry is viewable by community members only. Membership to this community is closed." errors when I or any other members try to post to a community that I run? It's been happening for a few days now and it's very frustrating.

[momijizukamori]

Highly unlikely, but we've been having some issues with the form auth check (which is basically a small security check that happens on form submission) on random forms failing, for reasons that are not yet clear.

It may be worth trying to toggle the beta entry page (https://www.dreamwidth.org/beta) and seeing if posting from the other entry page form works, as they have different form flows.

[lb_lee]

This wasn't a problem I had, but thanks much for wading in and taking on what sounds like a mucky awful problem!

[rosegardenfae]

Thanks for the fix. I was missing lots of notifications.

[doranwen]

I'm getting a lot of notification comments that are empty. Like, the subject line is blank, and the contents are a couple of bullet points (without any text next to them) and the following:


--
Dreamwidth Team
Dreamwidth Studios


If you'd rather not get these updates, you can change your preferences at https://www.dreamwidth.org/manage/settings/?cat=notifications


At the same time, I'm not getting some notifications that I should. Pretty sure these are all notifications of moderated submissions to the community I'm moderating, which is active right now.