denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
...have been due to a very badly behaved bot hitting us ridiculously hard. We blocked it a few times, but it kept trying to work around the block. We've now gone for the nuclear blocking option, which we hope will keep the problem from returning, but if the problems start up again, we'll continue to fix them as soon as we can!
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla

We're about to pull the lever on tonight's code push! Many of the changes we are making to the site are under-the-hood improvements, but these are the ones you are more likely to notice:

  • New account setting option for RP accounts, for future feature development.
  • Many more sites upgraded to use HTTPS links instead of relying on our SSL proxy.
  • Moved the Manage Icons page to /manage/icons and modernized the underlying code.
  • Increased the size limit for icon descriptions from 120 to 300 characters.
  • Various requested fixes for the image upload/management pages.
  • Improved processing of emailed entries for changes to entry security.
  • Improved processing of emailed comments for removal of quoted text.
  • Reading page with date filter now has previous / next day links.
  • Banned users hidden by default on the Manage Circle page.
  • Most importer failure messages will now include the name of the journal being imported, for the benefit of users running multiple imports.
  • People who read the RSS/Atom feed of your journal will see correct entry links and embedded content.
  • Whitelist embeds from: coub.com, airtable.com, mixcloud.com
  • New <user> tag sites: medium.com, imzy.com, facebook.com, instagram.com
  • New "other site" fields on user profiles: Imzy, Instagram

Once the code push starts, you may notice that the site is slow to respond, but it should remain available to use unless something goes badly wrong.

I'll update this post when the code push is finished. Stay tuned!

Update: All done! Let us know if anything seems more wrong than usual!

karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
We are planning to do a code push late this weekend, at approximately 8pm PDT / 11pm EDT / 3am UTC on Sunday, Apr 30 (or May 1 for you transatlantic types.).

I don't have a list of changes for you yet, but most will fall into the following categories: things users have complained about to support volunteers, things support volunteers have complained about to developers, things [staff profile] denise has complained about not working the way she expects them to (and as we all know, The Boss is Always Right), and things that were printing warnings over and over in the production server logs, making it hard to spot when less frequent, more urgent errors were being printed. Oh, and also all the unused code I ripped out at the roots, which if you notice that, I did it wrong.

To sum up: we are rolling out a bunch of requested changes, so thank you all for your feedback!

If you're new to Dreamwidth and interested in tracking our development process, our commit logs are published to [site community profile] changelog and [community profile] changelog_digest, and every month or so, one of our volunteers will translate those often-cryptic entries into witty, informative code tours! The most recent one was published on April 1, so we're about due for a new one. Hint, hint.

We'll update here again to let you know when the code push is imminent!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
Our content importer has mostly caught up with its backlog; almost everything that's still listed as being "in the queue" are jobs that were tried, failed once or more with a temporary failure, and are waiting to try again. (The importer tries a few times, at successively longer intervals, when it gets a failure it thinks might be temporary/might correct itself later on.) This means that new imports scheduled now should complete in hours (or even minutes), not the "several days" it's been taking.

If you tried to schedule a second import while the first one was still running, at any time in the past 10 days or so, you may have confused the poor thing. If you think your import should be finished by now and it isn't, and you're seeing "Aborted" on the Importer Status part of the Importer page, feel free to open a support request in the Importer category and we'll look into it for you. (It may take a little bit before you get a response; those of us who have the access to look into importer problems have been really busy for the past two weeks or so, and I at least need a few days to catch my breath a bit before diving back into the fray! But we'll do what we can.)

I hope all y'all are continuing to settle in well to your new home!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
People who are worried because the import of their LiveJournal account has been running for a long time with no real signs of progress: please don't be concerned! The import queue is currently a little long. (In the same way that Mt Everest is a little tall and the Mariana Trench is a little deep.) We're limited in how many import jobs we can run simultaneously and how quickly we can start the next import after one finishes: LiveJournal, like all sites, has restrictions on how frequently we can programmatically request data from their site, so the import queue can get very backed up at times like this when more imports are being started than are finishing. If you look at the import queue and the numbers don't seem to be changing much, or are only going up, it doesn't mean that no imports are finishing: it means a lot of additional people have scheduled an import since the last time you reloaded.

As long as you haven't gotten a failure message in your on-site inbox, your import is still running. (Even if you have gotten a failure message, your import may still be running: if the site thinks that the failure is something that might correct itself, like being unable to connect to the remote site, it will retry for a few times before giving up.) If you have gotten a failure message, the error message in your inbox should tell you what went wrong.

The three common problems right now: 1) you mistyped your username and/or password; 2) you need to agree to LiveJournal's new ToS before they'll permit you to access the data in your account; 3) an entry or entries in your LiveJournal account have a text encoding mismatch and you need to follow the link in the error message to fix it on LJ.

If you haven't gotten a failure message, your import is still waiting in the queue, and will run when it makes its way up to the top of the queue.

People keep asking us how long the queue is (by which they mean, how much time will it take for a job just started to successfully finish: length of time, not number of jobs waiting). I would love to be able to give you a definite answer! It's really, really hard for us to predict how long it will take for a job to get up to the top of the queue, though: how long an import takes to complete depends on a lot of things, including how many posts/comments are in the journal. To give you a ballpark figure that might be off by up to 100% on either side: If I personally started a brand new import right now (in my timezone, the early morning of Monday 10 Apr), I would be pleasantly surprised if it finished before Tuesday morning (24 hours or so), would expect it to finish sometime on Tuesday night or maybe even stretch all the way to Wednesday night (36-60 hours), and wouldn't start to wonder if I should poke [staff profile] mark or [personal profile] alierak to doublecheck that something hadn't gotten stuck in such a way that our monitoring didn't alert us about it until Thursday afternoon or evening (100+ hours).

All of those time estimates, by the way, assume a relatively uncomplicated job that succeeds on the first try. When the site tries again after a failure, it includes a delay that increases after each failure in case the failure was due to transient network issues. So, I know some of you started an import at the end of last week and it's still running: some of you are trying to import very large journals, and some of you ran into errors along the way and are in a retry wait loop. Again: if you haven't gotten the final error message in your inbox (and it will tell you it's the final error), it's still chugging along.

You do not have to leave the importer page open or stay logged into Dreamwidth until your import finishes. (You do have to avoid changing your LJ password until the job is done, or it will fail.) You can close the window/tab and go off and explore Dreamwidth; the movers will be along in a little while with your stuff.

The tl;dr version of my usual longwinded babble: IMPORTER VERY BUSY. MANY PEOPLE MOVING IN. LIKE ON DORM OR APARTMENT MOVE-IN DAY, FREIGHT ELEVATORS VERY SLOW. BUILDING OWNERS RUNNING FREIGHT ELEVATORS AS FAST AS POSSIBLE AND APOLOGIZE FOR THE WAIT.

A housewarming glass of champagne/sparkling cider/fancy handmade soda for all! Welcome to the neighborhood.

EDIT, 10 Apr 2017 7:15PM EDT: the importer is not the only thing that is very busy today! I'm trying to get to all the comments here, but keep getting dragged off to handle other stuff (and will be knocking off for the night soon). If you have a technical support problem, it will probably be faster to open a support request, where there's less likelihood that it will get overlooked in the sea of comments.
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
Thanks to everyone who let us know that Photobucket images were not loading properly on some pages. The problem seemed to be mostly limited to HTTPS requests; Dreamwidth maintains a list of known high-traffic image sites that support HTTPS, so that our secure content proxy service doesn't cache them unnecessarily. Unfortunately Photobucket seems to have recently changed their site configuration such that HTTPS requests aren't being served as expected, and we've now taken it out of our list of "proxy-exempt" sites.

If you continue to have issues, make sure you're not using HTTPS Photobucket links. It's a bit counterintuitive, but if you use HTTP instead, it will be automatically transformed on our end to an HTTPS link that uses p.dreamwidth.org.

Hope that clears everything up for now! Let us know if it doesn't...
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
InsaneJournal has had a hardware failure that means their service is temporarily offline. To avoid sending traffic to them while they're down, we've temporarily disabled them as a crossposting and importing source. We'll re-enable them when they're back up.

Good luck to [insanejournal.com profile] squeaky with the recovery!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
LiveJournal has unblocked and whitelisted our webserver IP addresses, so importing, crossposting, and feed accounts should work properly again! Thanks for the quick fix, LJ.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise

Cloudflare


We've had people ask us about the Cloudflare leak reported a few days ago. We are Cloudflare customers, and it is possible that login cookies or passwords may have been exposed as part of the incident. We believe the risk to you is relatively low -- it was a small percentage of Cloudflare's requests that were involved over a relatively short period of time, and we haven't found any evidence that anything from us was among them. This is not an absolute guarantee that none of your accounts were affected, but we don't think the likelihood is very high.

Because we believe the risk to be low, we aren't automatically expiring everyone's session cookies and requiring you to log back in and change your password -- whenever we do that, it does lock some people who they can't remember their passwords and no longer have access to their confirmed email addresses out of their accounts, and we believe that will affect more people in this case.

Still, it's always a good idea to change your passwords regularly, and now would be a good time to do it, especially if you want peace of mind. We have a FAQ on how to change your password. If your browser logs you in automatically and you don't remember your password, you can reset it. If you've forgotten your password and no longer have access to your most recent confirmed email address, you can have the password reset email sent to any email address you've confirmed on your account by entering both your username and your old email address at the Lost Info page.

Unfortunately, if you've forgotten your password and no longer have access to any email address you've confirmed on your account, you probably won't be able to reset your password. In some cases, if you've previously paid for your account, we can validate your payment details to confirm your identity and reset your password. If you can't reset your password, but think you may have paid for your account in the past, you can open a support request in the Account Payments category and I'll check into it for you.


LiveJournal imports/crossposts/feeds


LiveJournal has temporarily blocked about 2/3rds of our webservers from contacting their site, presumably because they feel that we're requesting data from them too often. This affects the ability to import your journal, the ability to crosspost entries from your Dreamwidth account to your LiveJournal account, and whether syndicated feeds of accounts on LiveJournal will update on Dreamwidth. Those features will fail when they're unable to contact LJ because of the block.

It isn't every one of our webservers, so things will work intermittently -- if you crosspost two entries one right after the other, one might succeed while the other fails. Unfortunately, there isn't much we can do to resolve this other than contacting them and asking them to unblock us (which I'll be doing right after I hit 'post' on this entry).

EDIT 2249 EST 25 Feb: We appear to be down to zero unblocked webservers, so imports, crossposts, and feeds will all fail until LJ unblocks us.

EDIT 26 Feb noon EST: LJ unblocked and whitelisted us this morning, so all is working again!
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
We're about to pull the lever on tonight's code push! I'll update this post when it's finished. For a reminder of what to expect, check the previous post for the list of changes.

Update: All done! Comment here if you notice any issues that need our attention.
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
We are planning to do a code push late this weekend, at approximately 9pm PST / 12am EST / 5am UTC on either Sunday, Feb 12 or Monday, Feb 13, depending on whether you live east or west of midnight. (Time is an illusion anyway, right?)

Here's a partial list of changes that will go live with this push, apart from the usual minor tweaks and bugfixes:

  • HTTPS Everywhere beta! Users can opt-in to have all Dreamwidth content automatically served over HTTPS. We'll post the instructions for this after the feature goes live.

  • New and improved design for the file management pages, which we were hiding from you because we were so embarrassed about them before. Thanks to [personal profile] momijizukamori for making them prettier and more functional!

  • Backend fixes to resolve problems using the aforementioned file management pages. (Did I already mention the embarrassment?)

  • At long last, international character support for journal search! Our systems guru [personal profile] alierak finally cracked this long-standing bug.

  • Support index page converted to Foundation styling, for your mobile viewing pleasure.

  • For users of the Practicality style: color properties now sort properly in the customization wizard.

  • For users of the Drifting style: the QuickReply box will now appear in the appropriate location, instead of wandering off somewhere unexpected.

  • Improved handling of word break (<wbr>) elements in user entries.

  • Allow embeds from: Facebook, CNN, 4shared.com, playmoss.com, onedrive.com, jsfiddle.net, scratch.mit.edu


We'll update again to let you know when the code push is in progress!
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark
Hey all,

Sorry for the 'Welcome to nginx!' issues we've been seeing for a bit now. It was a server configuration issue that hit us when a webserver rebooted.

Basically when the machine rebooted, nginx (software that can act as a web server) started up and then when Apache (the software Dreamwidth code uses for a web server) tried to start, it failed because nginx had already started and registered on the ports (the way web server software gets your requests).

Again, sorry for the site issues. Everything should be back to happy-times now, so let us know if you see any issues!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
The site slowdown tonight (Saturday night) is due to database load -- Saturday nights are one of our busiest times and we're still going to need to do some performance tweaks to our new setup over the next few days as we run into load issues. (Some problems only show up once all y'all start banging on the site!) We've made a few changes now that will improve things for some people, and will do a more permanent fix later on tonight or tomorrow morning once the traffic dies down, since doing it now would just increase database load.

Site performance may continue to be rocky on and off for the next few days until we get everything tuned the way it needs to be tuned -- thank you for your patience!
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
  • The site may occasionally be slow to respond or have trouble connecting as we tune servers and databases, or you may receive Gateway Timeout errors.

  • Comment notification emails may be delayed due to database load.

  • Inbound email (post-by-email and comment-by-email) may be delayed for a bit while your outgoing mail server catches up with the switch.

  • Email sent to your username@dreamwidth.org address may be more likely to be marked as spam, due to our mail server changing its IP address.

  • The site search database may be missing entries and comments and will catch up over the next few days, as database load allows. Entries and comments posted since the site came back up will be added to the index as they're posted.

  • Logging into other sites using your Dreamwidth OpenID is working again! Yay!

  • Loading embedded content over secure connections should be fixed!

  • There is a known problem with trying to rearrange elements on the Beta Update page because a script element isn't loading properly. Update: This should be fixed now!


Let us know if you see anything else odd with the site that isn't listed here!

denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We have now cut over to our new hardware! If you can see this, the site should be working for you.

Things you don't need to tell us about:

* The site may look a little weird or naked for a while as your computer/network/ISP/etc catches up with the switch.
* Comment notification emails should be going out properly, but inbound email (post-by-email and comment-by-email) may be delayed for a bit while your outgoing mail server catches up with the switch.
* The site search database is a few days out of date (missing entries from the past two days) and will catch up over the next few days. Entries and comments posted since the site came back up will be added to the index when they're posted.

Known issues:

* Accessing the site will be slow for a while as the caches warm back up, and you may receive Gateway Timeout errors.
* Due to the occasional database connection problem due to high load and site slowness earlier, some comments did not generate emailed notifications and those notifications can't be re-sent. New comments made since we resolved the DB problems are generating email notifications, but slowly (due to the general database slowness).
* Logging in to other sites using your Dreamwidth OpenID is broken (& we'll fix it as soon as we can!)

Let us know if you see anything else weird!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
We are now within our maintenance window for today. The site will go into read-only mode when we're ready to begin the maintenance, and will be unavailable for at least most of the morning. Watch our Twitter status account for updates!
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
A reminder: the site will be down/in read-only mode tomorrow for server maintenance for at least a little while. (Some people may also have trouble accessing the site even after it comes back up, depending on your network/computer/ISP settings.) We're beginning the maintenance window at 10AM EST/7AM PST/11PM UTC, and to be on the safe side, we're setting a window of 8 hours for possible downtime. It probably won't take that long, but there's also a (very small) chance it may take longer! We'll keep you updated on any changes with our off-site status Twitter account.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
As we prep for tomorrow's server maintenance, the site may look weird or 'naked' (no styling, images not loading, etc) on and off over the next hour or so. If you notice it, there's no need to report it: if it happens, we did it on purpose.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise
There will be a maintenance window, with some downtime, on Thursday, December 1, starting at 10AM EST/7AM PST/3PM UTC, for server maintenance and upgrades. To err on the side of caution, we're calling it an 8 hour window (so, until 6PM EST/3PM PST/11PM UTC), although we don't expect that the site will be in maintenance mode for the whole time.

We'll remind you Wednesday night, and again when the site is going into maintenance mode.
karzilla: a green fist above the word SMASH! (Default)
[staff profile] karzilla
I'm about to pull the lever on tonight's code push! I'll update this post when it's finished.

Update: All done! Comment here if you notice any issues that need our attention. Sorry about any transient weirdness for a few minutes there.

Profile

Dreamwidth Maintenance

May 2017

S M T W T F S
 123456
78910111213
14151617181920
21222324252627
28293031   

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated May. 26th, 2017 01:43 am
Powered by Dreamwidth Studios