Notifications slow -- but recovering

[mark]

Hi all,

Due to some behind the scenes maintenance last night, our notifications system got delayed. I've fixed the issue now and it's working on catching up.

For details -- I've been experimenting with Kubernetes as a way to make managing production easier (and hopefully reduce costs!), but it turns out that one of our worker jobs that handles notifications doesn't use much CPU (it mostly spends time waiting on the database).

This caused the pod autoscaler to reduce the size of that particular deployment below what we needed to sustain throughput on our notifications service. The temporary fix is to pin that deployment size to something much larger, the better fix will be to integrate Kubernetes' pod autoscaler with the ability to monitor the queue depth on our task queue.

Sorry for the trouble, and thank you for the person who pinged us on Twitter. When I checked last night, everything was working, but as traffic came back up we fell behind and I wasn't watching anymore. My bad.

Comments

[ninetydegrees]

I hate to be that user but I stopped getting notifications (email and inbox) almost 2 years ago and never got any reply to my request. Is there *anything* I can do on my side to solve this issue?

[mark]

Honestly, the most useful thing is making a comment like this -- to be honest, I forgot this was an issue. That's on me.

Since I'm tied to a coworking space for the day, I'll see about figuring out what's up this afternoon. I can't make any promises though, because that sounds wacky, but I'll try.

[mildred_of_midgard]

Glad you got it catching up!

Is there a particular database bottleneck? (asks the DBA)

[mark]

The bottleneck was just not running enough jobs, and the jobs being sequential. Even though every query is pretty fast, milliseconds add up when you do a bunch of serial queries to fulfill a single request.

We get throughput on this particular job by running lots of copies. Which Kubernetes does by examining CPU usage ... but if your job isn't _capable_ of using lots of CPU, it tricks the autoscaler into getting rid of the jobs. Which ain't great.

[mildred_of_midgard]

Makes sense!

[siderea]

Oh thank goodness - I thought it was on my end and was getting a bit panicky that my email service (which I also use for business email) was bouncing incoming email (again!) Thanks for letting us know, very appreciated!

[kore]

Thank you for keeping us all in the loop!

Watchdog service

[dennisgorelik]

Could you add a "watchdog" service that will check every hour how far behind your messages sending queue is?
Then if your messaging queue is behind more than 2 hours - email notification to yourself (to devops).

Re: Watchdog service

[mark]

Yeah, that's a good call! We have something like that, but I hadn't ported it over to Kubernetes yet. I manually inspected things and saw they were working and hoped they would keep working for 24 hours...

Fool's hope, of course.

Re: Watchdog service

[dennisgorelik]

> I hadn't ported it over to Kubernetes yet

That makes me wonder what other Dreamwidth backend services may be silently falling behind now?

Are you sure that switch to Kubernetes was a right call?
If your bottlenecks are:
1) Database performance.
2) Complexity of monitoring your queues.
Then Kubernetes is, probably, not the right tool to address these issues, right?

[trobadora]

Thank you, so glad to hear it's catching up!

[justice]

I'm glad someone reached out to you on Twitter because I tried the support@dreamwidth.org method, and it turns out it just goes into the support queue. The Twitter said not to reach out to you there - but is it what you'd prefer in scenarios like this one where even the site notifications wouldn't be reaching you?

[denise]

It's fine to ping us on Twitter for a problem or outage that's affecting a bunch of people! It's more that we can't do individual troubleshooting in 280 characters for specific problems only affecting one person.

[justice]

Thanks! I'll keep that in mind for the future.

[sovay]

Thanks for the explanation!

[niqaeli]

Truly, one of the many best things about DW is that I can rely on y'all to consistently and without bullshit cop to whatever caused the last major problem! Even when it's "yeah, so today I have custody of the commit-and-ditch pony." <3

[mark]

That pony and I are quite old friends by this point. :shamecone:

[schematise]

Bad pod autoscaler!

Kubernetes is brilliant when it works, though. We use it for our media streaming servers and it's ability to just self heal and self-manage is amazing once the set up is steady! I hope it continues to be everything you dreamed of.

[devilbear]

Hi! I don't know if this is the right place to put this, since I basically rarely use this site yet. I'm assuming that it's maintenance related because this definitely didn't used to be a function of the site...?

Anyway, I had a lot of my journal layouts in a private community and the coding, put within textboxes, is now broken. It would seem that things like @ keyframes [without spaces; I just don't want to ping a person] has been turned into the raw username code for mentioning a user. Which means that now when I try to copypaste the layout codes featuring that, and presumably anything else using @ in it, I just get a completely broken layout I then have to go through and fix.

Is there a way to disable this and revert the changes without all of my entries needing manual fixing / being ruined?

[ilyena_sylph]

Hey, this is from a few months ago when Mark added an "/@username" feature in the Markdown step.

Last I heard, they're working on a fix for the stuf in textboxes and such, so, speaking as just another DW-izen... hang on a bit still?

[kalloway]

I'd supposed that nobody wanted to go to work this morning, especially my tracking notifs. Thank you for the fix and transparency. ^_^

[brickhousewench]

I'm starting to learn about Kubernetes at my job, so it's kinda cool to know that my blogging platform uses it. (Also cool whenever I read nerdy stuff and understand what people are talking about. Go me!)

[mark]

Yeah! K8s is pretty interesting. A _bit_ overkill for what we're doing here, but also, it's been a good learning experience for me.

[darjeeling]

Just an update as of 12/06, some notifications are still missing. It seems to only be for tracked things... like replies to your comments are coming through immediately as normal, but things like new posts in tracked comms, or replies to tracked posts, are not appearing at all.