denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise

It's only been... okay, I don't want to think how long.

We are preparing to take the beta update page and the beta inbox out of beta with the next code release. We don't have an exact date yet, but this is your last chance to tell us about problems!

To enable the beta versions: go to Beta Features and enable the "new Create Entries" beta test for the new update page and the "New Inbox Page" for the, well, new inbox page.

If this is your first time using the beta update page: nearly everything on it can be adjusted, moved, or removed to some degree. The upper right side of the update form has a " ⚙️ Rearrange Panels" link.

Not every panel is enabled by default! If you don't see something you're used to finding, look in the disabled panels. While in Rearrange mode, you can enable or disable panels, drag them to whatever position you want, and generally customize it to your heart's content.

Once we push the button and switch to the new update and inbox pages for everyone, we cannot keep the old ones available. We are so close to finishing the decade-long "get rid of the undocumented custom templating language from 1996 that no one else uses" modernization project that's been sapping our collective will to live that we can taste it, and we can't convert the old update page and the updated inbox page is the converted version. This is your last chance to alert us of problems. If you let us know after the switch goes live, we'll still fix it, but we can't guarantee how fast. Prevent future problems and test drive them now!

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all,

I had to push some code live today to fix something, and that brought with it a couple of new things like if you have two-factor authentication (2FA/MFA) enabled, you got automatically logged out and it is now required to log in. If you don't, you can set it up if you want (you don't have to.)

This also means that if you have 2FA enabled, you will not be able to use 3P clients using your username and password anymore. You should be able to use API keys in most cases, but there's some nuance to that (and tbh, I just am not precisely sure what clients anybody even uses.)

Please do comment if you're seeing anything weird, as per usual, and we'll work on it. Thank you!

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Folks may have noticed that the site has been slow for logged-out users over the last while. This is partly because we separate traffic by logged-in, "logged out but have visited the site before", and "logged out, never visited the site before" and assign the fewest resources to the last category (because we're pretty confident the overwhelming majority of it is bot and scraper traffic, even if it's often impossible to say for sure). The flood of garbage traffic is a plague and a scourge the entire internet is dealing with, and it's hitting small sites the hardest as operators get better and better at cloaking their requests to look like real, authentic use. We long ago hit the point where adding more resources is a possible solution (because they just eat them up as soon as we do), and splitting traffic lets us keep the site usable for our actual users without wasting too much server power on garbage.

We've now, lucky us, reached the point where the "logged out, have never visited the site before" path is just flooded all the time, and the "logged out but have visited the site before" path is suffering some of the overflow. We've made some changes to the routing to try to improve things for logged out users who have visited the site before and keep it at "it may be a little bit slow, but at least it works" instead of "it keeps timing out", and we've seen some improvements, but if you're accustomed to browsing the site while logged out, I'm really sorry but it may continue to be a little miserable.

You will get the fastest page loads and the best performance by browsing the site logged in. If you are having trouble loading the front page to log in, bookmark the direct login page. We can't route the front page to the "more power" server pool, because it's a common target for garbage traffic, but we've switched /login over to "more power" and we'll try to keep it there as long as we can unless it starts getting slammed, too.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Olá de Lisboa! (I'm in Lisbon right now for work...)

Thank you everybody who has tested on canary - I'm about to push the code up to stable. I know that not everything has been tested and there's a lot in this deploy, so I expect some things to break or be weird. We have ported a lot more of the "older" pages to the newer formats, so there will be some UI changes in a few places, but it shouldn't be anything too major.

Watch for: One thing I know will require tuning, we now have added rate limits (which basically allows us to control how rapidly DW pages are loaded.) This is a protection mostly designed against bots, but any time in my career I've rolled out new rate limits, I've learned someone is hitting them and they have to be tuned. So if you see them, please let me know!

Anyway please do comment and I'll work to fix things today. We also have a rollback ready in case we need to (and I'll be honest, I expect we may need to... we really should do more frequent deploys.)

Known Issues

  • Fixed: Search has gone walkabout.
  • Fixed: Some UI weirdness with Manage Tags.
  • Fixed: Login redirects are not working.
  • Fixed: Login page sometimes returning 'invalid form submission.'
  • Fixed: Rename keywords checkbox on icons page also gone walkabout.
  • Fixed: Uploading large icon files erroring / not going to the image editor.

As always, thank you for your patience and supporting our scrappy little service!

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Quick note that post-by-email and comment-by-email is (sometimes?) failing silently without actually posting right now! I'm pretty sure this is related to last night's shenanigans and will be fixed once Mark can finish the full fix for it, which he's working on, but if you've posted or replied by email in the last 24 hours, fish it out of your sent folder to check if it posted!

EDIT: This should be fixed as of around 7AM EDT! We *believe* everything that was stuck in the plumbing has been sent along to your journal or the comment thread it was meant for; it's definitely not where it was stuck anymore, at least.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise

Robby has managed to put in a temporary fix for the site errors and things failing to refresh or not showing up where they should! The permanent fix is going to need Mark's experience, and unfortunately -- seriously, this literally never fails -- Mark has been on an international flight all day, because of course he has. (Never. Fails. He and I are not allowed to both take vacation at once.)

The site will work just fine with the temporary fix in place, things just might be a little slow here and there. We'll keep you updated.

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We're aware of site traffic issues and are working to fix them for the people who are having problems! (The tactics the damn bot traffic uses are endlessly shifting, and they're really good at looking like real traffic, sigh.)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
It's been a while since we've done a full code push rather than just hotfixes for bugs, so we are well overdue! Depending on availability, we're aiming to do one sometime soon; we'll let you know specifics once we've worked out good timing for everyone who needs to be available.

However! The reason it's been so long is we kept trying to get some of the stuff that's pending to "really finished" instead of just "mostly finished", and then we once again looked around and went "oh no, this is a really big code push with a lot of changes". Those make us nervous, because while we do a lot of testing ourselves, y'all are really creative in how you use the site and we inevitably find a bunch of edge cases when we let you loose on new code with your real-world data!

So, if folks have some spare time in the next few days, it would be a huge help if you could spend half an hour or so using the site the same way you normally do but with the "Site-Wide Canary" beta features flag turned on. Canary mode is a sort of "live testing" mode: it's your real data, but running the most up-to-date code.

Canary mode always does have a few glitches -- there may be missing text strings or errors about missing database properties, which is a limitation of how we run it. We don't need to know about those, but anything else weird that you run into, leave a comment with what you were trying to do and the error message you got.

I'll repeat that the "here be dragons" caution that's on the beta features page: some things may be broken, so don't use it for when you're doing something important. But a few more eyeballs on it before the push will help the push go more smoothly for everyone.

For folks who want to concentrate on what's changing, we haven't finished the second code tour of what's going to be in this push, but the ffirst one has a good chunk of what's going to be going live. (We'll get the second half done ASAP!)
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Happy Wednesday!

I'm taking search offline sometime today to upgrade the server to a new instance type. It should be down for a day or so -- sorry for the inconvenience. If you're curious, the existing search machine is over 10 years old and was starting to accumulate a decade of cruft...!

Also, apparently these older machines cost more than twice what the newer ones cost, on top of being slower. Trying to save a bit of maintenance and cost, and hopefully a Wednesday is okay!

Edited: The other cool thing is that this also means that the search index will be effectively realtime afterwards... no more waiting a few minutes for the indexer to catch new content.

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise

I keep forgetting to post about this: we've been troubleshooting the "missing notifications" problem for the past few days. (Well, I say "we", really I mean Mark and Robby; I'm just the amanuensis.) It's been one of those annoying loops of "find a logical explanation for what could be causing the problem, fix that thing, observe that the problem gets better for some people but doesn't go away completely, go back to step one and start again", sigh.

Mark is hauling out the heavy debugging ordinance to try to find the root cause. Once he's done building all the extra logging tools he needs, he'll comment to this entry. After he does, if you find a comment that should have gone to your inbox and sent an email notification but didn't, leave him a link to the comment that should have sent the notification, as long as the comment itself was made after Mark says he's collecting them. (I'd wait and post this after he gets the debug code in but I need to go to sleep and he's not sure how long it will take!)

We're sorry about the hassle! Irregular/sporadic issues like this are really hard to troubleshoot because it's impossible to know if they're fixed or if they're just not happening while you're looking. With luck, this will give us enough information to figure out the root cause for real this time.

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Happy Saturday!

I'm going to be doing a little maintenance today. It will likely cause a tiny interruption of service (specifically for www.dreamwidth.org) on the order of 2-3 minutes while some settings propagate. If you're on a journal page, that should still work throughout!

If it doesn't work, the rollback plan is pretty quick, I'm just toggling a setting on how traffic gets to the site. I'll update this post if something goes wrong, but don't anticipate any interruption to be longer than 10 minutes even in a rollback situation.

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Hi all!

I'm doing some minor operational work tonight. It should be transparent, but there's always a chance that something goes wrong. The main thing I'm touching is testing a replacement for Apache2 (our web server software) in one area of the site.

Thank you!

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise

Привет and welcome to our new Russian friends from LiveJournal! We are happy to offer you a new home. We will not require identification for you to post or comment. We also do not cooperate with Russian government requests for any information about your account unless they go through a United States court first. (And it hasn't happened in 16 years!)

Importing your journal from ЖЖ may be slow. There are a lot of you, with many posts and comments, and we have to limit how fast we download your information from ЖЖ so they don't block us. Please be patient! We have been watching and fixing errors, and we will go back to doing that after the holiday is over.

I am very sorry that we can't translate the site into Russian or offer support in Russian. We are a much, much smaller company than LiveJournal is, and my high school Russian classes were a very long time ago :) But at least we aren't owned by Sberbank!

С Новым Годом, and welcome home!

EDIT: Большое спасибо всем за помощь друг другу в комментариях! Я ценю каждого, кто предоставляет нашим новым соседям информацию, понятную им без необходимости искать её в Google. :) И спасибо вам за терпение к моему русскому переводу с помощью Google Translate! Прошло уже много-много лет со школьных времен!

Thank you also to everyone who's been giving our new neighbors a warm welcome. I love you all ❤️

mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Good morning, afternoon, and evening!

We're doing some database and other light server maintenance this weekend (upgrading the version of MySQL we use in particular, but also probably doing some CDN work.)

I expect all of this to be pretty invisible except for some small "couple of minute" blips as we switch between machines, but there's a chance you will notice something untoward. I'll keep an eye on comments as per usual.

Ta for now!

AWS outage

Oct. 20th, 2025 10:11 am
alierak: (Default)
[personal profile] alierak
DW is seeing some issues due to today's Amazon outage. For right now it looks like the site is loading, but it may be slow. Some of our processes like notifications and journal search don't appear to be running and can't be started due to rate limiting or capacity issues. DW could go down later if Amazon isn't able to improve things soon, but our services should return to normal when Amazon has cleared up the outage.

Edit: all services are running as of 16:12 CDT, but there is definitely still a backlog of notifications to get through.

Edit 2: and at 18:20 CDT everything's been running normally for about the last hour.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Per the [site community profile] dw_news post regarding the MS/TN blocks, we are doing a small code push shortly in order to get the code live. As per usual, please let us know if you see anything wonky.

There is some code cleanup we've been doing that is going out with this push but I don't think there is any new/reworked functionality, so it should be pretty invisible if all goes well.

alierak: (Default)
[personal profile] alierak
We're having to rebuild the search server again (previously, previously). It will take a few days to reindex all the content.

Meanwhile search services should be running, but probably returning no results or incomplete results for most queries.
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark

Looks like there's an issue with our text-based Captcha provider, which is unfortunate. We've switched over to reCAPTCHA for now. I know this is less accessible than our textual ones, so we'll try to switch back if the other provider becomes available again.

karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla

We just pushed out some minor fixes for stuff that wasn't working quite right after last month's update:

  • various poll display issues
  • problems submitting poll forms when JS wasn't loaded
  • comm admins couldn't close a community poll created by a different comm admin
  • some broken links in the shop are no longer broken
  • the shopping cart form buttons shouldn't display when viewing a purchase receipt

If you notice anything weird that wasn't weird before yesterday, please open a support request! This will get the attention of our lovely, talented, and chronically underappreciated support volunteers, whereas a reply here will likely only get the attention of me, and I'm going to be spending tomorrow at my kid's quiz bowl competition instead of reading internet comments.

Have a great week, everyone!

karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla

We are planning did do a code push tonight! The exact timing will depend on people's availability but it should be sometime between 9:30PM EST and 11:30PM EST. Most of the changes included in this code push can be found in the most recent code tour.

Here are the new updates you are most likely to notice and appreciate:

  • fix for /customize applying intended community style changes to the logged-in user
  • fix for /manage/circle/edit not saving some changes
  • two new mood themes from Socchan
  • modernized design for the /poll/create page
  • modernized design for the /manage/profile page
  • support for adding Steam usernames on user profiles

We don't anticipate any meaningful user impact, apart from a bit of slowness here and there as things restart, but we'll be sure to let you know here and on Bluesky if things go unexpectedly pear-shaped.

UPDATE: deploy is done, let us know how things are going? 😉

Profile

Dreamwidth Maintenance

October 2026

S M T W T F S
    12 3
45 678910
11121314151617
18192021222324
25262728293031

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Nov. 20th, 2026 02:52 pm
Powered by Dreamwidth Studios