dw_maintenance | Notifications slow -- but recovering

Notifications slow -- but recovering

Hi all,

Due to some behind the scenes maintenance last night, our notifications system got delayed. I've fixed the issue now and it's working on catching up.

For details -- I've been experimenting with Kubernetes as a way to make managing production easier (and hopefully reduce costs!), but it turns out that one of our worker jobs that handles notifications doesn't use much CPU (it mostly spends time waiting on the database).

This caused the pod autoscaler to reduce the size of that particular deployment below what we needed to sustain throughput on our notifications service. The temporary fix is to pin that deployment size to something much larger, the better fix will be to integrate Kubernetes' pod autoscaler with the ability to monitor the queue depth on our task queue.

Sorry for the trouble, and thank you for the person who pinged us on Twitter. When I checked last night, everything was working, but as traffic came back up we fell behind and I wasn't watching anymore. My bad.

Threaded | Top-Level Comments Only

I hate to be that user but I stopped getting notifications (email and inbox) almost 2 years ago and never got any reply to my request. Is there *anything* I can do on my side to solve this issue?

Glad you got it catching up!

Is there a particular database bottleneck? (asks the DBA)

The bottleneck was just not running enough jobs, and the jobs being sequential. Even though every query is pretty fast, milliseconds add up when you do a bunch of serial queries to fulfill a single request.

We get throughput on this particular job by running lots of copies. Which Kubernetes does by examining CPU usage ... but if your job isn't _capable_ of using lots of CPU, it tricks the autoscaler into getting rid of the jobs. Which ain't great.

Honestly, the most useful thing is making a comment like this -- to be honest, I forgot this was an issue. That's on me.

Since I'm tied to a coworking space for the day, I'll see about figuring out what's up this afternoon. I can't make any promises though, because that sounds wacky, but I'll try.

Oh thank goodness - I thought it was on my end and was getting a bit panicky that my email service (which I also use for business email) was bouncing incoming email (again!) Thanks for letting us know, very appreciated!

Thank you for keeping us all in the loop!

Could you add a "watchdog" service that will check every hour how far behind your messages sending queue is?
Then if your messaging queue is behind more than 2 hours - email notification to yourself (to devops).

Thank you, so glad to hear it's catching up!

Yeah, that's a good call! We have something like that, but I hadn't ported it over to Kubernetes yet. I manually inspected things and saw they were working and hoped they would keep working for 24 hours...

Fool's hope, of course.

I'm glad someone reached out to you on Twitter because I tried the support@dreamwidth.org method, and it turns out it just goes into the support queue. The Twitter said not to reach out to you there - but is it what you'd prefer in scenarios like this one where even the site notifications wouldn't be reaching you?

Thanks for the explanation!

It's fine to ping us on Twitter for a problem or outage that's affecting a bunch of people! It's more that we can't do individual troubleshooting in 280 characters for specific problems only affecting one person.

Makes sense!

Truly, one of the many best things about DW is that I can rely on y'all to consistently and without bullshit cop to whatever caused the last major problem! Even when it's "yeah, so today I have custody of the commit-and-ditch pony." <3

Thanks! I'll keep that in mind for the future.

That pony and I are quite old friends by this point. :shamecone:

> I hadn't ported it over to Kubernetes yet

That makes me wonder what other Dreamwidth backend services may be silently falling behind now?

Are you sure that switch to Kubernetes was a right call?
If your bottlenecks are:
1) Database performance.
2) Complexity of monitoring your queues.
Then Kubernetes is, probably, not the right tool to address these issues, right?

Bad pod autoscaler!

Kubernetes is brilliant when it works, though. We use it for our media streaming servers and it's ability to just self heal and self-manage is amazing once the set up is steady! I hope it continues to be everything you dreamed of.

Ultimately, the problem is that DW-as-a-codebase and DW-as-a-platform were both born in 1999 (as LiveJournal), and at the time there weren't any of the contemporary backend services that people can choose from now. The prototypes we-as-LJ invented to solve scaling problems were what morphed into those services (in some cases directly, in some cases as parallel evolution), but there have been a lot of newer backend services coming and going in those 20 years. We haven't made use of what's been developed in a lot of cases because the existing "battle-hardened" prototypes worked just fine, they had been optimized for our use cases over that exceptionally long time, a rewrite would run the risk of introducing more issues that weren't worth the performance gain we'd get from the switch, and a lot of the examples would require more rearchitecting than a platform that was written from the beginning to swap those newer services more easily would need. Which is not to say that we don't do the work when we need to, when the benefits we'd get start to outweigh the downsides, but it's not as simple a switch as it would be on a project that had its genesis more recently.

It's a problem we run into pretty often -- I actually have a talk I give at tech conferences called "When Your Code Is Old Enough To Vote" about, among other things, how to do the cost/benefit analysis for deciding whether or not to make major rewrites to benefit from new systems and services that were developed long after you started. If you jump on every new advance in technology, your list of "modernization projects we are in the middle of" outgrows your "new features/enhancements" list really, really quickly. We are in the middle of a lot of modernization projects right now (some you can see; some backend) and we have to be really judicious about which additional ones we take on.

I actually have a talk I give at tech conferences called "When Your Code Is Old Enough To Vote"

Awesome! Is this a talk you make available other ways, or am I going to have to get invited to the right conferences?

I have an older version of the slides at https://www.slideshare.net/mobile/dreamwidth/when-your-code-is-nearly-old-enough-to-vote but I don't think there's a video of me delivering it anywhere -- I'll look, though!

This was fascinating. Thanks for the link!

Hi! I don't know if this is the right place to put this, since I basically rarely use this site yet. I'm assuming that it's maintenance related because this definitely didn't used to be a function of the site...?

Anyway, I had a lot of my journal layouts in a private community and the coding, put within textboxes, is now broken. It would seem that things like @ keyframes [without spaces; I just don't want to ping a person] has been turned into the raw username code for mentioning a user. Which means that now when I try to copypaste the layout codes featuring that, and presumably anything else using @ in it, I just get a completely broken layout I then have to go through and fix.

Is there a way to disable this and revert the changes without all of my entries needing manual fixing / being ruined?

I watched the video of you giving it somewhere!!!!

Hey, this is from a few months ago when Mark added an "/@username" feature in the Markdown step.

Last I heard, they're working on a fix for the stuf in textboxes and such, so, speaking as just another DW-izen... hang on a bit still?

Edited 2019-12-03 01:24 (UTC)

Threaded | Top-Level Comments Only

Page 1 of 3

Notifications slow -- but recovering

no subject

no subject

no subject

no subject

no subject

no subject

Watchdog service

no subject

Re: Watchdog service

no subject

no subject

no subject

no subject

no subject

no subject

no subject

Re: Watchdog service

no subject

Re: Watchdog service

Re: Watchdog service

Re: Watchdog service

Re: Watchdog service

no subject

Re: Watchdog service

no subject