dw_maintenance: notifications delay

Denise (

denise) wrote in

dw_maintenance,
@ 2012-04-19 03:57 am UTC

notifications delay

Thanks to a combination of heavy traffic and a side effect of moving some databases, we had a problem with notifications not being delivered (both via email and the on-site inbox).

mark has identified the issue and fixed the underlying problem. The delayed notifications should be sent out now as the workers catch up with the backlog.

We're sorry about the problem!

EDIT, 7AM EDT: after fixing the problem with notifications in general being severely delayed, we're now seeing that AOL has placed us on their "you send us too much mail" blacklist and is refusing all mail from us. We'll see what we can do to get the block listed, but we can't guarantee anything. If you use an @aol.com or @aim.com email address, you may want to temporarily switch your email address to a different email address under your control in order to be sure you don't miss mail.

(28 comments) - (Post a new comment)
(Flat) (Top-level comments only)

	no subject icarus 2012-04-19 08:08 am UTC (link)
	No biggie! :) (Reply to this)

	no subject lunchee 2012-04-19 08:40 am UTC (link)
	didn't notice anything, but thanks for the update! (Reply to this)

	no subject lifestupid 2012-04-19 08:42 am UTC (link)
	Seriously, I love you guys. You're so fast when problems are noticed. ♥ I didn't even know there was a problem until someone on my plurk timeline saw. Thank you so much for being awesome! (Reply to this)

	no subject burntviolet 2012-04-19 08:49 am UTC (link)
	Thank you for solving the issue! I thought I was the only one having the notifs delayed, but I'm glad it's fixed now. (Reply to this)

	no subject sparowe 2012-04-19 09:06 am UTC (link)
	Thank you! :) (Reply to this)

	no subject giandujakiss 2012-04-19 10:39 am UTC (link)
	Are you sure it's solved? I'm still not getting even new comments, although I guess that could be due to the backlog? Anyway, thanks for fixing it! (Reply to this) (Thread) (Hide 1 comment) (Show 1 comment)

	no subject denise 2012-04-19 10:46 am UTC (link)
	Notifs should now be back to normal -- in your case it looks like (from a quick skim of the logs) your email provider is refusing mail from us at the moment. I'll pass that along to mark so he can look at it once he gets up! (Reply to this) (Thread from start) (Parent)

	no subject scifisentai 2012-04-19 11:13 am UTC (link)
	I'm getting comment notifications but I don't seem to be getting others, would that be a problem with backlog? I have an RP comm tracked so I know when new posts go up and I haven't had notifs for the last couple. (Reply to this) (Thread) (Hide 7 comments) (Show 7 comments)

no subject

denise
2012-04-19 11:20 am UTC (link)

If you're not getting the notifications in your inbox either, then it's the delay from the notification backlog (although that should mostly be cleared by now). If you're getting them in your inbox but not in email, it's likely your email provider is blocking us; we go through that song and dance every few months and it looks like it's time for it to start up again, at least judging by our mail logs.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 6 comments) (Show 6 comments)

	no subject scifisentai 2012-04-19 11:22 am UTC (link)
	I'm not getting them in either, unfortunately. I'm on hotmail for email, if that's any help. (Reply to this) (Thread from start) (Parent) (Thread) (Hide 5 comments) (Show 5 comments)

	no subject denise 2012-04-19 11:26 am UTC (link)
	In that case, wait another six hours or so just to make sure it's not backlog (Mark has gone to sleep and I don't know how to check the current backlogged queue) and then comment back again to let me know stuff is still not arriving! (Reply to this) (Thread from start) (Parent) (Thread) (Hide 4 comments) (Show 4 comments)

	no subject scifisentai 2012-04-19 11:28 am UTC (link)
	Okay, will do. Thank you. :) (Reply to this) (Thread from start) (Parent)

	no subject scifisentai 2012-04-19 09:46 pm UTC (link)
	Just a heads up that it's been over 9 hours and I still haven't had any post notifs, including ones for posts made over 24 hours ago now. (Reply to this) (Thread from start) (Parent) (Thread) (Hide 2 comments) (Show 2 comments)

no subject

denise
2012-04-20 03:14 pm UTC (link)

Hmm. Everything's all cleared out of the queues and things should be running fine, and I don't have any errors in the logs for anything that's not AOL/AIM (which is back to giving us intermittent errors), so I honestly don't know what could be causing the problem. Unfortunately, debugging individual accounts' problems is almost impossible :(

Try going to http://www.dreamwidth.org/manage/settings/?cat=notifications and make sure your desired notification tracking subscriptions (the bottom section) are all checked and all have "by email" checked, then under "My Account" (the top section) do this:

1. Uncheck "Someone replies to any comment or community post I make"
2. Save your changes
3. Uncheck "Someone comments on any entry in my journal"
4. Save your changes
5. Check "Someone comments on any entry in my journal"
6. Save your changes
7. Check "Someone replies to any comment or community post I make"
8. Save your changes

If the settings have gotten funky somehow, that might work. You can also try deleting the tracking subscription for the community and recreating it.

Another possibility is: if a post goes up in the comm and you can't see it when it's created -- for instance, if it's posted Admin-Only and then edited to be members-only or public -- you won't get a notification for it. Notifications are only sent out if you're able to see the post at the time it was created. Likewise, if you've got only a single tag tracked on the comm, and an entry is posted without the tag and the tag is edited in later, you won't get notifications for it. If that's what's happening, there's nothing you can do to get notifications of the posts, and you might want to visit the community directly or subscribe to the community and make a reading filter for it to isolate just the posts from that community if you want to be sure not to miss any.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 1 comment) (Show 1 comment)

	no subject scifisentai 2012-04-20 07:09 pm UTC (link)
	Okay, I've redone the notifications thing, hopefully that should work. Thank you. <3 (Reply to this) (Thread from start) (Parent)

	no subject sylvaine 2012-04-19 11:45 am UTC (link)
	Thank you! (Reply to this)

	no subject consulting 2012-04-19 12:43 pm UTC (link)
	Thanks! ♥ (Reply to this)

	no subject kikibug13 2012-04-19 01:09 pm UTC (link)
	Thank you! (Reply to this)

	no subject thedeadcat 2012-04-19 01:51 pm UTC (link)
	Ah, AOL's at it again. Good luck! (Reply to this)

	no subject eatsprettyhearts 2012-04-19 02:02 pm UTC (link)
	Thank You, again, for being super speedy and willing to jump in for everyone ❥ (Reply to this)

no subject

brendanm720
2012-04-19 02:06 pm UTC (link)

There is a feedback loop that you can get on which should mitigate the issue.

I work for an ISP and we had big problems until we got signed on with them, and again when we migrated our mail servers to a new subnet.

Here's information for it.

http://postmaster.aol.com/Postmaster.FeedbackLoop.php

If you're having issues with Live Mail (hotmail.com, live.com, et cetera) they have something similar.

My advice regarding the AOL feedback loop is to set up a separate email address just for that. You will get a metric ton of email from them regarding the email they're getting from you. :-)

(Reply to this)

	no subject lunareism 2012-04-19 05:40 pm UTC (link)
	Thank you! (Reply to this)

no subject

guardiano_perduto
2012-04-19 05:46 pm UTC (link)

after fixing the problem with notifications in general being severely delayed, we're now seeing that AOL has placed us on their "you send us too much mail" blacklist and is refusing all mail from us. Oh good, so it wasn't just me.

Thank you for keeping us posted! It is working again now, though, and all the notifications I'd missed last night just arrived.

(Reply to this) (Thread) (Hide 2 comments) (Show 2 comments)

	no subject denise 2012-04-19 05:49 pm UTC (link)
	Oh yay, if you're getting email now, then hopefully pestering AOL worked :) (Reply to this) (Thread from start) (Parent) (Thread) (Hide 1 comment) (Show 1 comment)

	no subject guardiano_perduto 2012-04-19 05:54 pm UTC (link)
	Yep! And I just got a notif for this too, so you're all set. ♥ (Reply to this) (Thread from start) (Parent)

	no subject the 2012-04-19 07:09 pm UTC (link)
	aol, curse your sudden but inevitable betrayal. (Reply to this)

	no subject fiones 2012-04-19 10:47 pm UTC (link)
	If I sold you guys my soul, would that be sufficient to pay for more icon packs? (Reply to this) (Thread) (Hide 1 comment) (Show 1 comment)

	no subject kellzilla 2012-04-21 06:45 am UTC (link)
	But your soul is at least slightly used. Offering the soul of an infant or even better, the soul of a fetus, would make it much more enticing! Unfortunately, all I have to offer is the soul of a bitchkitten, as my soul is worn thoroughly out and rotten. (Reply to this) (Thread from start) (Parent)

(28 comments) - (Post a new comment)
(Flat) (Top-level comments only)

Account name:	Log in with OpenID?
Password:	Forget your password?
	Remember me