denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
The site slowdown tonight (Saturday night) is due to database load -- Saturday nights are one of our busiest times and we're still going to need to do some performance tweaks to our new setup over the next few days as we run into load issues. (Some problems only show up once all y'all start banging on the site!) We've made a few changes now that will improve things for some people, and will do a more permanent fix later on tonight or tomorrow morning once the traffic dies down, since doing it now would just increase database load.

Site performance may continue to be rocky on and off for the next few days until we get everything tuned the way it needs to be tuned -- thank you for your patience!
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
  • The site may occasionally be slow to respond or have trouble connecting as we tune servers and databases, or you may receive Gateway Timeout errors.

  • Comment notification emails may be delayed due to database load.

  • Inbound email (post-by-email and comment-by-email) may be delayed for a bit while your outgoing mail server catches up with the switch.

  • Email sent to your username@dreamwidth.org address may be more likely to be marked as spam, due to our mail server changing its IP address.

  • The site search database may be missing entries and comments and will catch up over the next few days, as database load allows. Entries and comments posted since the site came back up will be added to the index as they're posted.

  • Logging into other sites using your Dreamwidth OpenID is working again! Yay!

  • Loading embedded content over secure connections should be fixed!

  • There is a known problem with trying to rearrange elements on the Beta Update page because a script element isn't loading properly. Update: This should be fixed now!


Let us know if you see anything else odd with the site that isn't listed here!

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We have now cut over to our new hardware! If you can see this, the site should be working for you.

Things you don't need to tell us about:

* The site may look a little weird or naked for a while as your computer/network/ISP/etc catches up with the switch.
* Comment notification emails should be going out properly, but inbound email (post-by-email and comment-by-email) may be delayed for a bit while your outgoing mail server catches up with the switch.
* The site search database is a few days out of date (missing entries from the past two days) and will catch up over the next few days. Entries and comments posted since the site came back up will be added to the index when they're posted.

Known issues:

* Accessing the site will be slow for a while as the caches warm back up, and you may receive Gateway Timeout errors.
* Due to the occasional database connection problem due to high load and site slowness earlier, some comments did not generate emailed notifications and those notifications can't be re-sent. New comments made since we resolved the DB problems are generating email notifications, but slowly (due to the general database slowness).
* Logging in to other sites using your Dreamwidth OpenID is broken (& we'll fix it as soon as we can!)

Let us know if you see anything else weird!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We are now within our maintenance window for today. The site will go into read-only mode when we're ready to begin the maintenance, and will be unavailable for at least most of the morning. Watch our Twitter status account for updates!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
A reminder: the site will be down/in read-only mode tomorrow for server maintenance for at least a little while. (Some people may also have trouble accessing the site even after it comes back up, depending on your network/computer/ISP settings.) We're beginning the maintenance window at 10AM EST/7AM PST/11PM UTC, and to be on the safe side, we're setting a window of 8 hours for possible downtime. It probably won't take that long, but there's also a (very small) chance it may take longer! We'll keep you updated on any changes with our off-site status Twitter account.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
As we prep for tomorrow's server maintenance, the site may look weird or 'naked' (no styling, images not loading, etc) on and off over the next hour or so. If you notice it, there's no need to report it: if it happens, we did it on purpose.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
There will be a maintenance window, with some downtime, on Thursday, December 1, starting at 10AM EST/7AM PST/3PM UTC, for server maintenance and upgrades. To err on the side of caution, we're calling it an 8 hour window (so, until 6PM EST/3PM PST/11PM UTC), although we don't expect that the site will be in maintenance mode for the whole time.

We'll remind you Wednesday night, and again when the site is going into maintenance mode.
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
I'm about to pull the lever on tonight's code push! I'll update this post when it's finished.

Update: All done! Comment here if you notice any issues that need our attention. Sorry about any transient weirdness for a few minutes there.
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
We are planning to do a code push around 21 hours from now, at approximately 6pm Pacific time on Saturday, Oct 1.

Here's a partial list of changes that will go live with this push, apart from the usual minor tweaks and bugfixes:


  • Selective Screening, a new feature that lets you screen comments from a particular user.

  • Two new journal layouts with one theme each: Gold Leaf/Elegant Notebook and Venture/Radiant Aqua.

  • Six new themes for Heads Up, and seven new themes for Corinthian.

  • The minimal spacing between line elements in the list of success links on Foundation pages has been restored.

  • Punctuation marks in message subjects will no longer be overescaped.

  • The admin pages for the translation system are now hidden from anonymous viewers, to avoid scaring unsuspecting search engine users away from the site.

  • The text messaging feature has been removed from user profiles.

  • The email notification worker is now less likely to misbehave.


We'll update again to let you know when the code push is in progress!
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
Sorry for the delayed response on the issue with emailed comment and support notifications not being delivered. We weren't receiving our emailed comment and support notifications saying that there was a problem...

We restarted the relevant software services around 10AM EDT, and the backlog of delayed notifications should have all been sent by now. Thanks for your patience, and extra thanks to everyone who tried to let us know about the problem!
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
Reminder that I'm going to start working on tonight's code push in the next 30-45 minutes or so. I know you just CAN'T WAIT to use the larger icon filesize for your animated gif talents, so I'm going to start a bit earlier than originally planned, closer to 5:30pm PDT. I'll update this post when we're done!

Update: All done! Comment here if you notice any issues that need our attention.
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
We are planning to do a code push around 32[*] hours from now, at approximately 6pm Pacific time on Saturday.

Here's a partial list of changes that will go live with this push, apart from the usual minor tweaks and bugfixes:

  • Icon size limit raised from 40kb to 60kb.

  • Fixed the "hook: enddata returned false" error when uploading multiple icons.

  • Posting DW links on Facebook will now use the "Swirly D" logo for the link image.

  • Added seven new color variants on the popular "Teeny Tinies" mood theme.

  • The user profile page now lists "Other Services" in responsive columns.

  • The user icons page no longer uses "(Default)" in the alt text for every icon.

  • Improved non-ASCII character support in plain text email notifications.


We'll update again to let you know when the code push is in progress!

[*] I had a computer check my math this time, because it was almost wrong again. YAY COMPUTERS.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
People getting mail at Gmail are reporting email delays of about an hour. As far as I can figure out, it's because some of our recent changes to our network have wiped out our existing sending reputation with them and made us start building reputation again, and they're refusing all mail on first delivery and making us re-send it. (It's a common spam reduction technique, because spammers don't bother retrying if the first attempt fails.)

There isn't much we can do about it but wait it out until Gmail decides that we're legit senders again, but we'll poke at it and see if there's anything we can do to make the process go faster. (I doubt there will be, though; Gmail is persnickety.) In the meantime, to get comment notification email faster, you can switch your confirmed email to a different provider, or just refresh your on-DW inbox.

EDIT: And people are now letting me know that mail's delayed to other providers, too, which is probably follow-on effects from having to send everything to Gmail at least twice. There isn't a lot we can do about it; I'm sorry about the hassle, folks.
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
Some DNS changes were made yesterday that manifested themselves in two troublesome ways:

1. Emailed replies to comments were not being delivered correctly for several hours, but I believe they started showing up again about 8 hours ago. If you have an emailed comment reply that still hasn't been delivered this morning, you may want to try sending it again, and let us know if you are still seeing delays.

2. Some users have been seeing DNS resolution errors when visiting pages on our site. As we come up on 24 hours since the changes were made, these should fade away, but if you do run into this problem, I'm told that refreshing the page while holding down the shift key is a thing to try that might help.

Thanks for hanging in there with us! I am hoping for a quieter day today. :)
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
The problem we were having earlier today with failed crossposts was due to a merge conflict in one of the code modules that the crosspost worker uses. I thought the merge conflict was fixed before the changed module was pushed live, but apparently it was not. Alas! Woe! But [personal profile] alierak fixed it a few hours after we were informed of the problem, and crossposts should now be back to normal.

A short while ago, a few of you started reporting a new issue with loading Dreamwidth pages via https. This was not related to the code push, but rather an update to the SSL certificates that we use, which were going to expire soon. One of the intermediate certificates was not in the correct format, and that problem is also now fixed.

If there is anything ELSE not working, do let us know, either by commenting here or opening a support request. Thanks everyone!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We'll be beginning our code push in a few minutes. We're not expecting any downtime. I'll update this entry once we're done for you to report any problems!

EDIT: Code push is complete! Please let us know if you notice any issues.

Code push!

Apr. 8th, 2016 10:59 am
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
We are planning to do a code push around 35 hours from now, at 8pm Pacific time on Saturday.

Here's a partial list of changes that will go live with this push, apart from the usual minor tweaks and bugfixes:

  • Markdown formatting support for crossposted entries.

  • Embedded content support for Google ngrams, Google calendar, vid.me, and codepen.io.

  • Additional space on the user profile page to list your DeviantArt username.

  • Improved support for browsing the site with HTTPS (fewer mixed content warnings).

  • The legacy mobile reading page has been updated to support filter selection.

  • The suggestions generator has been updated to use Github Issues.

  • Dreamwidth-based sites will now be able to disable anonymous support requests.


We'll update again to let you know when the code push is in progress!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
The name of one of the JavaScript modules we use on the beta Create Entries page is being flagged by some adblock software and blocked, due to a recent filter update. If your adblock extension uses the EasyList filter subscription, you may notice that the select tags and select filters modules aren't working.

While we figure out the best way to handle this, you can temporarily work around the problem by whitelisting Dreamwidth in your adblock software. (Dreamwidth does not have ads and you will not see ads if you whitelist us -- it's purely to work around the overbroad matching.)

EDIT: The Adblock filter ruleset has been updated to no longer block the thing it was blocking. You now can update your filters and then remove the exception for Dreamwidth if you want!
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
We're updating the site momentarily! Once the dust settles, please let us know if anything isn't working as expected. I'll edit the entry here if we confirm any issues.

Update, 21:45: All done!

Code push!

Nov. 22nd, 2015 01:03 pm
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
We are planning to do a code push in about six hours, around 5pm Pacific time.

Most of the changes with this push are cleanup and small backend fixes, but we also have a new journal style called "Pattern" with 24 themes for you to choose from, and most excitingly, QuickReply has now been enabled for journal, day, and network views.

We'll update again to let you know when the code push is in progress!
Page generated Dec. 11th, 2016 06:07 am
Powered by Dreamwidth Studios