Import issues

[denise]

There are three commonly-reported issues we're seeing with importing your journal right now:

* Some icons are not transfering over -- the message will say that they're successful, but only the keywords/comments will come over, not the userpic itself.

* Entries will appear to have no comments, until you click through to the entry itself, at which point the comments will be visible.

* Tags will appear not to import, until you wait for a little while.

We're aware of all three problems, and are working on diagnosing and fixing them. There's no need to open a support request about these problems.

The other issues described in this news post still apply.

The geeky bits, for those who care:

The first problem is happening because the userpics are in the database, but not in MogileFS, our image-storage filesystem. We know what's happening, but haven't figured out what's causing it yet. Once we do, we'll fix it from happening again, and we'll likely be able to fix it automatically for every affected account (although we can't say that for sure until we locate the root cause).

We think that the second and third problems are happening because of memcached, the function we use to store commonly-accessed information in memory to drastically increase site performance. It may turn out to be something else entirely, of course, but our leading theory right now is that the information is making its way to our database, the database is just not telling memcached that the information's been updated. We're looking into the best ways to solve it. In the meantime, viewing the comment page and then returning to your journal will cause the comment counts to display properly, and some people have reported that viewing the Manage Tags page and then returning to your tags list will cause your tags to show up.


EDIT, 2300 EDT 18 April: We believe we've figured out what was causing these problems. Future imports (anything scheduled after now) should no longer display these symptoms, and we're working out the best way to fix accounts that were affected. We'll make a new dw_maintenance post with the postmortem once we've deployed the fix.

Comments

[princessofgeeks]

but until the comments show as having been imported on the Import Helper there's no point in checking the entires, yes? and i did go and check, on the offchance, and an entry i know had comments still has none.

I've tried twice to import comments, waiting three or so days in between, and they're still queued. (I filed a support ticket for this.) Maybe it's me... b/c people who've created accounts since I did have their comments.

You guys are amazing! It's all so happening, and people are chomping at the bit to be part of it! yay! please remember to sleep, eat, breathe, smile, and take a day off once in a while!

[justabi]

I don't have any of my comments, either, so it's not just you!

[denise]

Mark's looking into this for you, by the way -- thank you so much for being so patient!

[princessofgeeks]

no problem at all. am checking emails from overnight and found both these; will investigate my journal now.

you guys are truly amazed and it must be so gratifying to see all the interest. my f list are all chomping at the bit to be let in here to run and play. lol. i hope your servers are ready for a huge flood of cash of April 30; hundreds of people will be hitting them at once. :D. Massive congratulations.

[princessofgeeks]

update: The importer shows they imported, and they indeed are there.

i feel so moved in now.

you can cross me off your to do list. i'll just be potting around here investigating happily; the next thing I'm all agog about is the crossposting function.

i'm happy with one of the templates for my layout so far, and am content to use an off the rack style later when more choices are available.

Again let me express my amazement and delight at what you guys are doing. It's so incredible to watch this happening. I never expected to be part of something like this and yet here I am. Do take good care of yourself, take breaks and eat. We're all in it for the long haul now. Love you.

[whitechii]

yeah my problem is that I am having delayed importing as of this AM. got a note saying that something from 2005 is being imported

[ratherbe4gotten]

My comments did the import but not show up thing, but they do show up now. I'm not sure when they appeared. I did mess around and make a new layout, based on core1, and it might be that they started showing up after that.

lol I don't know if that helps you/anyone at all, but that was my experience.

[xochiquetzl]

I saw that your code push fixed my reported issue. Thanks, guys! (And I love the geeky bits!)

[ineptshieldmaid]

I just noticed that my comments didn't import- I thought they might be hiding from me, as this post suggests, but going into specific entries doesn'tt bring them into existence.

Should I file a support request or is this a known problem? Should I just run the importer again?

[denise]

Yeah, can you try running the importer again? The fix we made to fix this might also fix your problem.

[ineptshieldmaid]

Sure thing :)

[ineptshieldmaid]

Fixed, thank you!

[ineptshieldmaid]

ah. I now have duplicate copies of all my entries. the more recently-imported copy has comments on it. So, er, the comment importer is fixed. I assume there isn't a handy way for me to delete all the first import in one go? (It's ok if there isn't, I've got less than a year's worth of posts, I can delete them gradually by hand.)

[denise]

Oops. No, there isn't an easy way to delete -- sorry about that!

[alchemise]

That happened to me with importing tags. I had the Manage Tags page open in a separate tab while I was trying to import. Sure enough, a few minutes after it said they were imported, all of my tags suddenly appeared.

[kaylashay81]

Is the icon issue what my import with not all my icons coming over fell into?

Or was there a limit on the number of icons that could be imported?

[denise]

If the problem was that your icons' keywords and descriptions were being imported, but not the icon itself -- so that it appears as a broken-image -- yes, that's this problem. We're looking into the best way to fix it for affected accounts, and we'll post again once we've determined how we're going to fix it.

[kaylashay81]

Ah... I had only one that did that. Then there was one that imported the image and not the keywords/descriptions. Then there were many that didn't even import.

That's why I was wondering about the icon limit. I have 143 on LJ and it was my icons that were more recently added to LJ that didn't import at all.

ETA: Number wise, I have 143 icons on LJ and only 65 transferred and the last one that did transfer didn't get the keywords/descriptions with it.

[denise]

Hm. Can you try just importing your icons again?

[msilverstar]

I just started an import this evening and it came over with everything, very very quickly!

[mec]

I was having the icon issue and have just redone the import - everything turned up in seconds, fully intact! Thank you so much! ^_^

[yvi]

Same here. It worked like a charm this time :)

Really sorry about this...

[whatistigerbalm]

... but where do we report things that don't work? I had some errors with updating recently. Sorry about the newbish question.

Re: Really sorry about this...

[denise]

You can leave a comment on the latest dw_news post!

[sarisia]

Bug?

Having just received notice of my entries and comments being imported (yay!), I noticed that my tags list seems to include three extras that were not part of my original tags: "after all," "could be twisted and molded into somethi," and "ever since ritsuka found out how soubi f."

All three of these tags link to the same post. The text for these tags comes from an entirely different post.

So far, the comments seem to be all there. :) Thumbs up, guys!

[tablesaw]

Okay, this is weird.

The other day, I was playing around with the importer to see how it handled inactive icons, so I imported from a small LJ account. Everything seemed fine.

Today, with the crossposter going into alpha, I decide to import my actual journal. When I do, I notice that a block of seventeen entries from the very beginning of my journal did not carry over. ("Nap Bites Man" to "This is getting personal." on my LJ archive.) I make a note to post a bug report about it.

Before I do, I decide to get rid of the information from the dummy imports. I clear out the extra icons, then I start deleting the entries. I delete 7 of them before I notice that the next one has comments. This is odd, because they didn't have comments in the original journal, and I hadn't been notified of new comments on DW.

I look at the entry, and I see that the comments that have been added to the post from the dummy LJ are the comments from one of the missing entries of my actual LJ. (Compare the LJ entry with the DW entry.) Several other comments got transferred over to the other dummy-lj entries; and there were, in fact, 17 entries transferred from the dummy LJ remaining on my DW. (There was an 18th entry that I deleted before the import, which may be why the first entry of my real LJ wasn't affected.)

So, that looks like a bug, one which could be a problem if people start consolidating journals.

I'm leaving everything as it is for now. Support in getting those seventeen entries would be appreciated, but I figure I can copy over seventeen posts pretty easily. I can re-date and copy over the posts that had the misplaced comments (though one of the entries I deleted apparently had a misplaced comment), so very little of those are lost. But it's still a wacky bug.

[denise]

Yeah, this is a known bug with importing multiple journals -- we'll have it fixed by open beta.

[tablesaw]

Darnit. I thought I checked that page. not carefully enough, I guess.

[sofiaviolet]

I tried ticking only "Import friend groups as trust groups." and got filters with the appropriate names, but every one of them has been populated by my entire access list.

Is this normal/intended behavior?

Having checked this post, it does seem to be normal, but I'm glad to see that more work is planned. :)

[denise]

They should be blank, not populated by your entire access list. Were you by chance looking at the "not in access filter" list, perhaps? There are two lists, the "not in access filter" and "in access filter" -- the "not in access filter" is on the left, and the "in access filter" is on the right.

[sofiaviolet]

Yes, I was. *facepalm* I haven't been getting enough sleep. Too busy exploring DW. :)

[torakowalski]

I imported my journal 10 days ago and have been waiting patiently for my comments to arrive, but by now I've a feeling they're not going to. I'm happy to try importing them again but will that give me duplicates of every entry? (740 entries!)

[denise]

Nope, running your import again won't double your entries. It'll just grab any new entries you made since your last import.

Since you last tried importing 10 days ago, and since we have made changes to the importer since then, go ahead and try running your import again. If the comments don't come over in about 24h, open a support request.

[torakowalski]

Fantastic, I have all my comments now! Thank you.

[shezan]

Hi! First, thank you for all the good work! Also, I seem not to be able to import my LJ comments at all (I clicked into some entries to see them individually, still no comments.)

No hurry but it would be nice to be able to migrate entirely...

[denise]

Go ahead and try running your import again. If the comments don't come over in about 24h, open a support request.

[snakey]

Hi, my attempts to transfer my comments keep aborting.... Will keep trying. The icons say they've been successful, but don't transfer except the default (neither do the comments/keywords on the icons). Have opened support request as did it before reading this post, so sorry. *facepalm*

Thanks for all the great work you're doing here.

[vass]

My userpics didn't transfer over, and the fix hasn't resolve this, and neither has importing everything again. I now believe that this happened because I hadn't upgraded my account to paid before starting the import process. Is there any way I can delete all 3,745 entries and start the import process again? Short of deleting them one by one, which I can't do, because like I said, 3,745 entries.

[denise]

This is bug #1 here -- we're going to be fixing it so that if you rerun the importer, the icons will update.

[bgmt]

My comments are shown as being queued since some four days ago now. The depth of the queue varies, from 3 to 1 and then back to 2 or 3. An attempt to launch anopgther import which gtheoretically should abort that one, results in "aborted because we are processing your previouis request". No way to abort by force is shown. The comments HAVE NOT been imported. What whould I do?

No message in my mail, either.

Thanks

[denise]

Give it another 3 days (if the importer encounters problems such as not being able to contact the remote site, it tries again with increasing intervals, and 7 days from initial attempt is when the last of them should fire) and if it still hasn't succeeded or given you a failure message, open a support request and hang tight. (It'll likely be a while before someone can get back to you; import troubleshooting is time-consuming and can only be done by people with direct server access.)

[bgmt]

Thank you for the reply. I do not think the reason of the problem could be the impossibility to contact Livejournal (it has been working without any problem all that time); and I do not understand why, when the text in front of my eyes clearly says "you can open another request, but beware that this will abort your previous request", the reality is quite different. I would suggest either change the message, or change the behavior.

Incidentally, the comments were PARTIALLY imported during that import session. Nothing suggests in what I read that the import may be interrupted and a new queue may originate in the middle of an import.

What I am trying to say, is that there is clearly a technical problem on your side.

Thanks again

bgmt

[denise]

I used not being able to contact the remote site as an example -- it's not the only thing that can block an import! A technical issue is entirely possible, but since other imports are completing successfully, it's not a guarantee and if an issue does exist, it's only an issue with importing from your LJ account. (Most commonly, because something about the data in your LJ account is irregular, malformed, or otherwise unexpected -- after ten years, the importer is pretty hardened against "stuff it didn't expect to find", but LJ's been accepting data for so long and has gone through so many changes in terms of what's accepted, what's cleaned, and what's rejected that even after a decade we're still being presented with edge-case data.) In cases like that, the problem will need one-on-one troubleshooting and debugging that can take hours and needs to be done by one of two people, both of whom are very busy, and it can take a long time for one of them to have the time to troubleshoot an i ndividual import problem. Please wait out the 7 day cycle, check your on-site inbox (https://www.dreamwidth.org/inbox) for any error messages -- they won't necessarily be sent to your email -- and then open a support request, including the text of any inbox messages you've gotten.

The "your previous import job will be aborted" applies only if the past job has been scheduled but not started. If some, but not all, of your comments have been imported, scheduling a new import doesn't abort the old one, because otherwise someone will lose their place in the import queue. I took a look, and the comments task for the first import you scheduled a few days ago did fail and is waiting out the delay for the next retry. There should be a message in your inbox to give you more information.