Dreamwidth Maintenance

The case of the missing notifications

2026-04-12T03:58:14Z

Posted by: denise

I keep forgetting to post about this: we've been troubleshooting the "missing notifications" problem for the past few days. (Well, I say "we", really I mean Mark and Robby; I'm just the amanuensis.) It's been one of those annoying loops of "find a logical explanation for what could be causing the problem, fix that thing, observe that the problem gets better for some people but doesn't go away completely, go back to step one and start again", sigh.

Mark is hauling out the heavy debugging ordinance to try to find the root cause. Once he's done building all the extra logging tools he needs, he'll comment to this entry. After he does, if you find a comment that should have gone to your inbox and sent an email notification but didn't, leave him a link to the comment that should have sent the notification, as long as the comment itself was made after Mark says he's collecting them. (I'd wait and post this after he gets the debug code in but I need to go to sleep and he's not sure how long it will take!)

We're sorry about the hassle! Irregular/sporadic issues like this are really hard to troubleshoot because it's impossible to know if they're fixed or if they're just not happening while you're looking. With luck, this will give us enough information to figure out the root cause for real this time.

comments

Performing some traffic maintenance today

2026-03-14T20:12:42Z

Posted by: mark

Happy Saturday!

I'm going to be doing a little maintenance today. It will likely cause a tiny interruption of service (specifically for www.dreamwidth.org) on the order of 2-3 minutes while some settings propagate. If you're on a journal page, that should still work throughout!

If it doesn't work, the rollback plan is pretty quick, I'm just toggling a setting on how traffic gets to the site. I'll update this post if something goes wrong, but don't anticipate any interruption to be longer than 10 minutes even in a rollback situation.

comments

Minor operations; testing new serving path

2026-02-04T06:26:57Z

Posted by: mark

Hi all!

I'm doing some minor operational work tonight. It should be transparent, but there's always a chance that something goes wrong. The main thing I'm touching is testing a replacement for Apache2 (our web server software) in one area of the site.

Thank you!

comments

Привет! LiveJournal imports may be slow

2026-01-01T01:24:48Z

Posted by: denise

Привет and welcome to our new Russian friends from LiveJournal! We are happy to offer you a new home. We will not require identification for you to post or comment. We also do not cooperate with Russian government requests for any information about your account unless they go through a United States court first. (And it hasn't happened in 16 years!)

Importing your journal from ЖЖ may be slow. There are a lot of you, with many posts and comments, and we have to limit how fast we download your information from ЖЖ so they don't block us. Please be patient! We have been watching and fixing errors, and we will go back to doing that after the holiday is over.

I am very sorry that we can't translate the site into Russian or offer support in Russian. We are a much, much smaller company than LiveJournal is, and my high school Russian classes were a very long time ago :) But at least we aren't owned by Sberbank!

С Новым Годом, and welcome home!

EDIT: Большое спасибо всем за помощь друг другу в комментариях! Я ценю каждого, кто предоставляет нашим новым соседям информацию, понятную им без необходимости искать её в Google. :) И спасибо вам за терпение к моему русскому переводу с помощью Google Translate! Прошло уже много-много лет со школьных времен!

Thank you also to everyone who's been giving our new neighbors a warm welcome. I love you all ❤️

comments

Database maintenance

2025-10-25T15:45:31Z

Posted by: mark

Good morning, afternoon, and evening!

We're doing some database and other light server maintenance this weekend (upgrading the version of MySQL we use in particular, but also probably doing some CDN work.)

I expect all of this to be pretty invisible except for some small "couple of minute" blips as we switch between machines, but there's a chance you will notice something untoward. I'll keep an eye on comments as per usual.

Ta for now!

comments

AWS outage

2025-10-20T15:29:18Z

Posted by: alierak

DW is seeing some issues due to today's Amazon outage. For right now it looks like the site is loading, but it may be slow. Some of our processes like notifications and journal search don't appear to be running and can't be started due to rate limiting or capacity issues. DW could go down later if Amazon isn't able to improve things soon, but our services should return to normal when Amazon has cleared up the outage.

Edit: all services are running as of 16:12 CDT, but there is definitely still a backlog of notifications to get through.

Edit 2: and at 18:20 CDT everything's been running normally for about the last hour.

comments

Code deploy happening shortly

2025-09-01T02:40:12Z

Posted by: mark

Per the dw_news post regarding the MS/TN blocks, we are doing a small code push shortly in order to get the code live. As per usual, please let us know if you see anything wonky.

There is some code cleanup we've been doing that is going out with this push but I don't think there is any new/reworked functionality, so it should be pretty invisible if all goes well.

comments

Rebuilding journal search again

2025-06-30T20:25:24Z

Posted by: alierak

We're having to rebuild the search server again (previously, previously). It will take a few days to reindex all the content.

Meanwhile search services should be running, but probably returning no results or incomplete results for most queries.

comments

Captcha issues, switched to reCAPTCHA

2025-04-27T23:20:35Z

Posted by: mark

Looks like there's an issue with our text-based Captcha provider, which is unfortunate. We've switched over to reCAPTCHA for now. I know this is less accessible than our textual ones, so we'll try to switch back if the other provider becomes available again.

comments

another tiny code push, as a treat

2025-02-04T04:42:46Z

Posted by: karzilla

We just pushed out some minor fixes for stuff that wasn't working quite right after last month's update:

various poll display issues
problems submitting poll forms when JS wasn't loaded
comm admins couldn't close a community poll created by a different comm admin
some broken links in the shop are no longer broken
the shopping cart form buttons shouldn't display when viewing a purchase receipt

If you notice anything weird that wasn't weird before yesterday, please open a support request! This will get the attention of our lovely, talented, and chronically underappreciated support volunteers, whereas a reply here will likely only get the attention of me, and I'm going to be spending tomorrow at my kid's quiz bowl competition instead of reading internet comments.

Have a great week, everyone!

comments

Code push imminent... UPDATE: code push done!

2025-01-16T20:40:28Z

Posted by: karzilla

We ~~are planning~~ did do a code push tonight! The exact timing will depend on people's availability but it should be sometime between 9:30PM EST and 11:30PM EST. Most of the changes included in this code push can be found in the most recent code tour.

Here are the new updates you are most likely to notice and appreciate:

fix for /customize applying intended community style changes to the logged-in user
fix for /manage/circle/edit not saving some changes
two new mood themes from Socchan
modernized design for the /poll/create page
modernized design for the /manage/profile page
support for adding Steam usernames on user profiles

We don't anticipate any meaningful user impact, apart from a bit of slowness here and there as things restart, but we'll be sure to let you know here and on Bluesky if things go unexpectedly pear-shaped.

UPDATE: deploy is done, let us know how things are going? 😉

comments

Freeing up deleted usernames for renaming

2024-12-11T21:35:36Z

Posted by: denise

As some folks have noticed: we've actually found some time to run the maintenance script that makes deleted usernames available for re-use! We always swear we're going to do this more regularly and it always winds up getting postponed for something more urgent, so it's been a good while since the last time we've done it, and I apologize.

There's a lot of very long and boring reasons why it gets done so rarely, but the extremely quick explanation is that the system is one of the things that most shows its age (and would be hardest to bring into the 21st century) and needs a lot of manual intervention and babysitting that's really hard to automate: it's hard for us to find the time to do it regularly. (There's also the human factor of it being a combination of "it's annoying and manual and can cause a bunch of load issues" and "it's never hair-on-fire-urgent so we don't need to do it right this second"; no matter how hard we try to stop ourselves from procrastinating on 'ugh' tasks, some of it happens anyway and this is very much one of the ones that is most subject to it.)

If you'd like to rename your account to a username that now shows as "deleted and purged" (or to any other name!), check out How do I change my username? The fee for username changes is to cover the extra system load and support issues that renaming causes: again, it's a really old system and deals with a lot of assumptions that are very baked into the architecture of the site and almost impossible to change, which is why it can be a little confusing to people at times. We've tried our best to make it intuitive, but if you've got questions on how it works or something happens that you didn't expect when you rename your account, drop a support request in the Account Payments category and we'll do our best to give you a hand.

Because this always comes up right about now: if there's a username that you're interested in but the account is not deleted, there's no way to rename to it. Once someone registers a username, it's theirs until they choose to delete it, even if it doesn't look like they're using their account: we don't delete accounts for inactivity. Likewise, a reminder: the rename token system is the only secure way to change your username. Please don't trade or sell accounts or accept a traded or sold account from someone else! A traded account can never be made fully secure and the original owner can always take control of it away from you at any time, plus it carries the risk of being caught up in a Terms of Service enforcement action if the original owner does something severe enough to cause us to suspend all of their accounts on the site.

comments

Offsite status page

2024-09-04T21:09:02Z

Posted by: denise

We've been looking for an alternative to Twitter (excuse me, "X") for our offsite status announcements of the "shit's fucked enough that the site won't load; we're working on it" variety, and have finally settled on Bluesky as the correct choice for that purpose! If you are a Bluesky user, please follow https://bsky.app/profile/dreamwidth.org; if you aren't a Bluesky user, please bookmark that page.

(Side note because I know people are going to ask: yes, we'd like to add Bluesky (and Mastodon) to the username shortcuts, but the way they construct usernames and profiles is complicated enough that we have to give it some extra thought on how we're going to be able to do it!)

comments

still having shop issues

2024-05-18T01:42:08Z

Posted by: karzilla

Sorry all, the shop is currently not applying purchases successfully after processing credit card payments, so we have temporarily disabled the ability to pay with a credit card. It should come back up as soon as someone who can fix things is awake to diagnose the issue. In the meantime, you should still be able to make purchases with points.

Update 11:24 EDT: You should be able to pay with cards again! If you have a payment that hasn't been applied yet, wait 15 minutes...

comments

Resolved!

2024-05-16T19:25:31Z

Posted by: denise

The issues with the shop have been mostly resolved: the only outstanding problems should be one or two missing pieces of text and that the drop-down menus won't drop down (we'll get both of those fixed as soon as we can). If you run into any issues other than those two, please open a support request in the Account Payments category and let us know a detailed description of the problem and what steps you took just before the problem happened, along with your browser, browser version, operating system, and any extensions you have installed. As always, we appreciate your support!

comments

dw_maintenance @ 2024-05-13T20:27:00

2024-05-14T00:27:15Z

Posted by: denise

The shop may be temporarily unavailable on and off for the next day or so! We apologize for the hassle.

comments

Intermittent performance issues

2024-03-23T05:01:03Z

Posted by: denise

Some people have noticed intermittent performance issues and errors on and off today: this is happening because of bursts of traffic (much of it automated or illegitimate) that can temporarily overwhelm our automatic performance scaling system when it can't keep up with adding extra resources quickly enough or when it reaches the limit of additional servers it's allowed to add. (That limit exists so a flood of garbage traffic can't make the autoscaling system spend too much money trying to accomodate the garbage.) The garbage doesn't appear to be a specific targeted DDoS attempt, just part of the general overall increase in garbage traffic internet-wide. We're doing what we can to filter it, and when performance issues happen, the site will generally recover within 5-10 minutes as our defenses kick in. I apologize for the hassle!

comments

Accidental downtime update

2023-11-26T02:57:58Z

Posted by: mark

Apologies for the interruption. I was doing some more work that should have been invisible, but inadvertently turned off all of our web servers (in essence.)

It took a while to get things back online and we're currently working to scale things back up to the point where DW will be loading like normal.

comments

Database maintenance

2023-11-23T05:18:30Z

Posted by: mark

I'm doing some maintenance on the databases to prepare for a (in the future) upgrade to MySQL 8. (We're on 5.7 right now, which is pretty far behind.)

Today's work should be pretty invisible but there may be 1-2 minutes of errors while the failovers happen. I'll update on dreamwidth if something goes wrong, of course.

comments

Continuing dispatches on the war against spam

2023-09-29T03:16:54Z

Posted by: denise

A few days ago we let you know about spam prevention measures that we were taking to help stem some of the flood of garbage. One of those temporary measures included geoblocking all IPs from several of the countries that are our largest source of spam. This did (as we knew it inevitably would) have some collateral damage for real users, and we're very sorry!

We're continuing to experiment: this time we've slightly expanded the range of countries we're geoblocking to include the ones that we held off on geoblocking because it would affect too much legitimate use, but we've limited the geoblocking only to the account creation page. This should mean that if you were having trouble accessing the site because of geoblocks, you should be able to access 99% of the site without a problem, and the only page you won't be able to access is the account creation page. With luck, this should cut back heavily on our spam account creation without disrupting legitimate use of the site. The current list of countries that are geoblocked from account creation are Bangladesh, Cambodia, Egypt, India, Indonesia, Morocco, Pakistan, Singapore, Turkey, and Vietnam. (If you're an existing user from one of those countries and you'd like to make an additional account, email support@dreamwidth.org with the username you'd like to register and we can register it for you. If the number of requests gets to be enough that it's taking up too much of our time, we may have to pause this until we can build automated exceptions, but we'll start there.)

We will continue to monitor the results of these experiments and adjust as necessary: when we do one of these experiments, we always make sure to define in advance what "too much interference with legitimate use" will look like, and we try very hard to stick to it. I apologize to everyone who's been collateral damage in our efforts to filter out more of the goddamn spammers.

comments

A heads up on spam prevention measures

2023-09-24T20:49:38Z

Posted by: denise

Due to recent increases in spam account registration (over and above the already-high baseline levels of spam account registration!) and the amount of our administrative time that dealing with spam is costing us, it's highly likely we will need to start playing around with more aggressive measures to block spammers in the next few weeks. We already use quite a few spam account creation prevention techniques, but there's been a worldwide increase in the amount of abusive/spam traffic over the last six months or so, and we're at the point where we need to start getting much more aggressive about filtering it.

Generally speaking, we try to use the least restrictive measures of spam blocking that we can, because any form of spam blocking can impact legitimate use of the site. If you start getting 403 errors when accessing the site, or you are asked to solve a captcha from our hosting provider (the graphical captcha that shows on a separate page, not the text-based one that shows on the same page) before proceeding to the page you're trying to load, and you are not using a VPN service, please email support@dreamwidth.org with your IP address and let us know. If you don't know your IP address, you can look it up at whatismyip.com.

If you are using a VPN provider and you get these errors, I am incredibly sorry, but we probably won't be able to help. We know that many of our actual-person users use VPNs for privacy and security reasons or to circumvent government restrictions on accessing the site, and we are trying our very best to keep those services able to access Dreamwidth. Unfortunately, VPN services are also a major source of our abusive traffic, especially the free ones, and it's impossible for us to distinguish legitimate traffic from abusive traffic automatically. You are less likely to have problems with paid VPN services, but even those are the source of a lot of spam: our two main VPN sources of abusive traffic are NordVPN and Proton VPN. We're trying very, very hard to not have to block VPN services entirely, but the problem is getting much worse. If you subscribe to either, you may want to contact them and tell them that you've been having problems accessing sites you regularly use because of the amount of abusive traffic that comes from their network.

We will continue to tweak our spam prevention measures as much as we can to avoid interfering with legitimate traffic, and I apologize in advance if we wind up temporarily interfering with your use of the site as we try to stop the garbage we're drowning in.

comments

journal search is down, may be down for a few days

2023-08-07T03:22:39Z

Posted by: karzilla

The machine that powers the journal search feature appears to be having some technical difficulties, and the content database may need to be rebuilt from scratch again. The last time this happened, it took a few days, so be prepared to wait for a bit for that feature to be back up and running normally. Sorry for the inconvenience!

Update: as of August 8 12:00PM CDT, the rebuild is about 25% done.

comments

Code push complete

2023-07-31T02:09:23Z

Posted by: denise

Tonight's code push is complete! Please report issues here. Again, if you try to post and get a message that you need to confirm your email address first, please see How do I confirm my email address? for instructions on how to confirm your email address and fix the problem.

comments

dw_maintenance @ 2023-07-30T21:25:00

2023-07-31T01:25:51Z

Posted by: denise

We are planning a code push tonight -- the exact timing will depend on people's availability but it should be sometime between 9:30PM EDT and 11:30PM EDT (ie, anywhere from 10 minutes from now to about two hours from now). The list of changes that are included in this code push can be found in the most recent code tour.

We don't expect any meaningful user impact (there may be 30 seconds of slowness here and there). Two things of note that may cause some problems after the new code is running, however:

* This release includes a number of pages on the site being converted from the Extremely Old way of assembling a page to the newer way. As always, we do our best to keep them looking as visually similar as we can and to test them on multiple browsers/devices/etc before we let the changes out of testing, but if something looks new and weird (as opposed to just new), please let us know and bear with us while we fix it.

* This release also includes, to address our ongoing spam woes, a change that will prevent people who haven't confirmed their email addresses from posting on the site. (Previously, if your email was unconfirmed, you could post but not comment.) If you run into an error when trying to post that references confirming your email address, see How do I confirm my email address? for instructions on how to confirm your email address. You will be unable to post until you confirm your email address.

comments

another minor code push

2023-07-10T02:08:17Z

Posted by: karzilla

Just deployed a few more small bug fixes and updates. Let us know if anything seems amiss!

(That is, anything other than 504 errors - we usually get those around this time of day anyway due to surges in network traffic.)

comments