research

Journal migration tool

I've written a command-line tool for migrating journal entries from any LJ-style server to any other LJ-style server. This tool needs testers. It should run on any system with a recent Python installed. That means OS X out of the box, most Linux distros, and any Windows system where the user has installed python.

Get it
Github: ljmigrate
Tarball: ljmigrate.tar.gz
Zipfile: ljmigrate.zip
Latest version: 1.5 090110a Sat Jan 10 23:51:35 PST 2009
Documentation: README, README for Windows

Features
- Archives entries, comments, and user pics locally.
- Can optionally generate simple html versions of entries with comments and a userpic.
- Can optionally migrate from one LJ-style server to another. E.g., from LiveJournal to InsaneJournal. Post metadata (user pic keywords, mood icon, tags, music, location) is preserved when the destination supports them. Privacy info is preserved: if a post is private or flocked in the source, it will be private or flocked at the destination. Custom filters aren't moved, though.
- Can optionally migrate only posts with particular tags, if you want to move only a portion of a journal to a new service.
- Archives communities. Can also migrate them. Community comments are also archived. See README.
- Cannot migrate comments or memories. These limitations are LJ limitations.

Tutorials
karma_apple's illustrated tutorial for Windows users
alter_writes's illustrated tutorial for Mac users
lumnata's detailed instructions

Instructions for Terminal-comfy Mac users:
Download the tarball: ljmigrate.tar.gz
Safari will warn you that it might contain an application. (Oh noes!) Firefox/Camino won't.
Unpack it. (Double-clicking works.)
Run the Terminal.
Change directories to wherever it was you unpacked the file. E.g., cd ~/Desktop/ljmigrate
Read the README.
Edit the config file ljmigrate.cfg to mention your accounts.
Run the tool in the Terminal: ./ljmigrate.py
Watch. Wait. Report results to me in a comment here.
To use a new version of the tool, just move or copy your account folder into the folder for the newer version.
./ljmigrate.py --help provides usage info.

Instructions for Terminal newbies are in the README.

Thanks to: ldybastet & kannnichtfranz for early testing & bug-reporting.

Known bugs:
1) Migration will fail for posts that use LJ features you don't have permission to use on the destination. For instance, polls: if your LJ post has a poll but you haven't paid for polls on IJ, it'll fail.

Call for Windows help
I'd like to write up detailed instructions for non-technical Windows users who'd like to use this (say, to migrate communities). Could one of you give me a hand with this task? Thanks!
  • Current Mood: cynical
  • Current Music: Polar Bear : Ride : Nowhere
Tags: , ,
For the moment, it is. Fandom is panicking again. Four hundred freakin' million links about it here. Short version: two somebodies posted art of under-18-looking Harry Potter characters having sex. They got tossed by LJ without warning. Fandom freaks.

I have no intention of moving at the moment. I'm just handing fandom tools. I'll give plenty of warning if Buffy fandom moves and I think it's the right thing to do.
Hello,
Working on a Mac OS X terminal, and this keeps coming up, even though I follow your instructions to the letter.

Sebastian-Redux:~/documents/ljmigrate lyds$ ./ljmigrate.py
Traceback (most recent call last):
File "./ljmigrate.py", line 526, in ?
main()
File "./ljmigrate.py", line 316, in main
fetchConfig()
File "./ljmigrate.py", line 279, in fetchConfig
error("Problem reading config file: %s" % str(e))
NameError: global name 'error' is not defined


Is there something in particular I'm doing a bit wrong?

Thanks, by the way, for doing this.
Ah. Okay, there are two errors here: you haven't got a config file! And I've got a bug *reporting* the error to you! Did you copy ljmigrate.cfg.sample to ljmigrate.cfg and edit it?

I'll upload a fixed error-reporting step in a moment as well... Okay. New version uploaded.
perhaps I'm being a bit thick, but could you explain this again, please?
thanks!
Those are the expert version of the instructions, because I am looking for slightly more expert users to help me test. If you're feeling brave, I can write up the Terminal Newbie version of the instructions right now.
I'm too much of a newbie to be able to test it at this stage, so I apologize if this seems like pestering, but I'm curious, will it grab comments as well, or only posts?
It grabs comments, yes, and archives them along with each post. It cannot migrate the comments, because there's no way to add comments via the API. Only posts can be migrated. (Well, friends as well, but since account names aren't guaranteed to be the same from one system to another, that's a dicey proposition.)
Awesome tool!! I was planning to use MacJournal to transfer, but this might be better. MacJournal doesn't preserve privacy settings.

I have only tried it with an old, barely used RP account so far, which only had seven entries. I think it got through all the posts and then died on comments.

Fetching journal entry L-7 (update)
    re-posting journal self.__dict__...
Fetching journal comments for: investor_ko
Traceback (most recent call last):
  File "ljmigrate.py", line 767, in ?
    main()
  File "ljmigrate.py", line 698, in main
    allEntries[id].emitPost(htmlpath);
  File "ljmigrate.py", line 414, in emitPost
    content = content.replace("\n", "
\n"); AttributeError: Binary instance has no attribute 'replace'


The new journal appears to have all the new posts in place, though. This is the content of my journal directory:

> tree investor_ko/
investor_ko/
|-- entry00002
|   |-- comments.xml
|   `-- entry.xml
|-- entry00003
|   |-- comments.xml
|   `-- entry.xml
|-- entry00004
|   `-- entry.xml
|-- entry00005
|   `-- entry.xml
|-- entry00006
|   |-- comments.xml
|   `-- entry.xml
|-- entry00007
|   |-- comments.xml
|   `-- entry.xml
|-- entry00008
|   `-- entry.xml
|-- html
|   |-- 00002.html
|   `-- 00003.html
|-- metadata
|   |-- comment.meta
|   |-- entry_correspondences.hash
|   |-- last_sync
|   |-- user.map
|   `-- userpics.xml
`-- userpics
    |-- default.png
    |-- inquisitive.png
    |-- roses.png
    |-- serious.png
    |-- smile.png
    |-- smug.jpeg
    `-- welcome.png


Let me know if there's any other info I can give you.

Thanks for all your efforts!
I, too, got an Error when getting the comments:

Fetching journal comments for: ldybastet
Traceback (most recent call last):
File "./ljmigrate.py", line 767, in ?
main()
File "./ljmigrate.py", line 698, in main
allEntries[id].emitPost(htmlpath);
File "./ljmigrate.py", line 423, in emitPost
result = result + c.emit()
File "./ljmigrate.py", line 460, in emit
result.append('%s: %s
' % (self.user, self.subject))
AttributeError: 'Comment' object has no attribute 'user'
Ooh, nice one. I didn't anticipate that. I've uploaded a fix. The script died at the very last stage, while generating the html, so your journal migration/archive *should* have gone swimmingly. Did it produce the results you expected?
Yes, I suspect the first error is LJ's fault, and nothing I can do anything about; I merely skip that entry. The second bug is one I noticed & fixed this morning; argh that it bit you in the field, but it was the VERY last step and informative output only. So no practical bad effect, at least!

Thank you for testing!
Just scanning through this post and comments... looks like you are in you element here and having a blast. :)
I'm using this and having no trouble, although I am getting the same error as weirdquark:

Fetching journal entry L-134 (create)
re-posting journal entry...
Error getting item: L-134
[Error: Irreparable invalid markup ('<socket.gaierror>') in entry. Owner must fix manually. Raw contents below.]

I'm using this and having no trouble, although I am getting the same error as weirdquark:

Fetching journal entry L-134 (create)
re-posting journal entry...
Error getting item: L-134
<socket.gaierror instance at 0x4ed378>


But if this is an LJ fault, then no worries. Just chiming in. So far my entries are reposting to journalfen without any trouble. (Except the ones that can't be fetched in the first place.)
Hi. Found you on elke_tanzer's list. I'm getting this error when I run ./ljmirgate.py:

Fetching journal entries for: nm973
Fetching userpics for: nm973
Traceback (most recent call last):
File "./ljmigrate.py", line 803, in ?
main(retryMigrate)
File "./ljmigrate.py", line 533, in main
userpics = dict(zip(r['pickws'], r['pickwurls']))
TypeError: unhashable instance

I'm only trying to backup my LJ, not migrate. I did change that to False in my config file. Thanks for your help!!!
Honestly. I might just have to propose to you.
Tried it, and it mostly works but I get errors on the posts with current location such as:
Error getting item: L-168
[Error: Irreparable invalid markup ('<fault [...] current_location'>') in entry. Owner must fix manually. Raw contents below.]

Tried it, and it mostly works but I get errors on the posts with current location such as:
Error getting item: L-168
<Fault 205: 'Client error: Unknown metadata: current_location'>
This could just be that GreatestJournal (where I'm migrating to) just doesn't support it, I suppose.

And when it was saving to HTML, I got this:
skipping post 119 because of error: 'ascii' codec can't decode byte 0xc3 in position 90: ordinal not in range(128)

Other than that, works wonderfully! Thanks!
Ooh, nice bug report, thanks. I'll handle that in my next release of the script. (Both the GJ location thing & the unicode character problem.)
Trial and Error: The Remix
Let's try that again. Here is the error message I got:

Fetching journal entries for: daiseechain
Created subdirectory: daiseechain
Traceback (most recent call last):
File "./ljmigrate.py", line 833, in ?
main(retryMigrate)
File "./ljmigrate.py", line 497, in main
gSourceAccount.makeSession()
File "./ljmigrate.py", line 97, in makeSession
response = self.handleFlatResponse(r)
File "./ljmigrate.py", line 107, in handleFlatResponse
while True:
NameError: global name 'True' is not defined


Hopefully there will be actual line breaks this time...
Re: Trial and Error: The Remix
I know exactly what this problem is: you're running an older version of Python. Which means you're on a very dusty version of OS X as well. You can run "python -V" in the shell and find out exactly which version. (It's before 2.3, I know that. I've tested my tool with 2.3, 2.4, and 2.5.)

I'll see what I can do about making the tool compatible with older Pythons. *think*
I'm...stupid. But This is what it's saying as I try to migrate to journalfen:

migrating journal entry...
Error getting item: L-4
[Error: Irreparable invalid markup ('<protocolerror [...] www.journalfen.net//interface/xmlrpc:>') in entry. Owner must fix manually. Raw contents below.]

I'm...stupid. But This is what it's saying as I try to migrate to journalfen:

migrating journal entry...
Error getting item: L-4
<ProtocolError for www.journalfen.net//interface/xmlrpc: 404 Not Found>
Fetching journal entry L-5 (create)
migrating journal entry...
Error getting item: L-5
<ProtocolError for www.journalfen.net//interface/xmlrpc: 404 Not Found>
Fetching journal entry L-6 (create)
migrating journal entry.

Duh?
Ah. It's the double //. I can fix that in the source, but in the meantime, could you remove the trailing slash from your config file setup for your journalfen account?
Just tried this for the first time and for the most part, it's worked pretty well! Icons are working, posts are there, locked posts are locked. No comments, though. Here's everything that showed up once the fetching/migrating finished:

1977 entries migrated or updated on destination.
Fetching journal comments for: alto2
Now generating a simple html version of your posts + comments.
skipping post 970 because of error: 'ascii' codec can't decode byte 0xc3 in position 1452: ordinal not in range(128)
skipping post 979 because of error: 'ascii' codec can't decode byte 0xc3 in position 5362: ordinal not in range(128)
Local archive complete!
1977 entries, 10173 comments, 1601 comments by user, 122 userpics

There are many more skipped posts than I've included--the comment was too long to include them all, but the error was the same for all of them. Let me know if you'd like me to try anything else--and thanks for this! I'm using it as a backup at this point, and that's a nice thing to have.
I think I have fixed that error, but there's no need for you to run the tool again if you don't want to-- the problem was with generating local html, not with migration. Please tell me if I'm wrong :)