Ethan magic

Things I have learned in the last couple of days

- LJ's APIs are demented and poorly documented.
- People who used memories instead of tags are screwed. Tags are available as post metadata and can therefore be migrated. Memories are not accessible via the client API; see point 1.
- There is no earthly reason why community comments should not be accessible, but they aren't. Probably the programmer who hacked up the feature went out and got drunk before he could finish it. See point 1.
- Dave Winer should be shot for failing to allow for Unicode strings in XML-RPC.
- Python's xmlrpclib is similarly awful about converting strings to unicode; fully 2/3rd of my bugs have been with unconverted data that is a) treated as binary that should be utf-8, or b) being treated as regular python strings when they should be unicode strings.
- In consequence, Python 3000 cannot come too soon for me.
- But next time I'll get it it right from the beginning.
- Half the code is error-handling. (Though strictly speaking I'd learned this years ago. It's always true.)
- Despite the above, the project was absurdly easy. Anybody could have written a migration tool at any time. Give me a week and a case of Diet Coke and I'll make it dance the foxtrot for you.

Also: LJ fandom is full of sweet people who are very patient with my bugs.

Ah, fandom! You do puzzle me so.
You may as well be speaking Parseltongue, but you awe me with your smart. And you write great stories too. ::slouches off to stare at my current mental blue screen of death:: ::totter back in case the above is read as a suicide note. S'not::
Awwww, it's just because the task is in my area of professional expertise. Everybody looks smart when talking about what they do for a living.

*goes off to beat head against latest story*
I'm fond of how you anguish. I love sharing misery though mine doesn't have the faintest whiff of a story to beat my head against. I think I might have to start drabbling. Or watching the shows again to refresh my connection.
Here from friends of friends of friends of friends...

Thank you so much for creating the backup/migration tool!

I've just pointed to it here in macosx since people there were asking for exactly what you've created. Well, I suspect some folks will dream of a graphical UI for it, but one can't have everything!
They can dream :)
Thanks for pimping it for me! I got so much response just from my quiet little post here that I haven't had time to even consider publicizing it more. Heh.
LJ API and Tags
I'm in the process of writing a Java API for the LiveJournal Flat protocol and I'm having trouble posting events with tags. I've looked in the docs but I don't see mention of them anywhere. You mentioned in your post that Tags are available as post metadata but I'm not seeing that in the Chapter 12 docs. Any info you might have on posting tag information with events would be greatly appreciated.

Re: LJ API and Tags
'props' array, 'taglist' property. Comma-delimited string. Undocumented, yes. What I did was examine a complete list of properties observed in my own post dataset.

Are you enjoying the completely un-versioned protocol yet?
Re: LJ API and Tags
Thanks for the reply. I was able to programattically post an event with a taglist attached.

From your original post it sounds like you're building some type of Python client for LJ. If it's not too forward, may I ask if that's the case and if it's not the case what type of work are you doing with the LJ api. As I mentioned in my last post, I'm working on a Java api and I'm interested in other work that is going on in this area.

Thanks again for the reply!
I'm going to pimp your tool (don't that sound dirty) in my journal. I'd love to help with code but I can barely wrap my brain around a bash script at this point in time. You're doing a fantastic job from my POV.
Heh, yeah, that makes writing software sound way way more naughty than it is. And thanks! I'm struggling to keep up with comments, though. Argh.
Dave Winer should be shot for failing to allow for Unicode strings in XML-RPC

Definitely a sentiment heard floating around my former workplace (we were all Unicode freaks there -- witness the many 'weird' projects we did, via-a-vis Sanskrit, Romanian, and ancient Greek projects).

Your tool is super awesome, we are very patient with bugs. Bugs are the staff of life, sadly.
- LJ's APIs are demented and poorly documented.

Oh, god. Yes. On behalf of ljlogin, I totally feel your pain on this score and any and all others that descend from it. (For instance, I recently discovered that ScrapBook re-uses the old ljsession auth cookie, but needs it to have a specific domain session cookie value assigned to is only available via login.bml, not the client API at all.)

Also, I am intrigued and delighted by the post announcing ljmigrate, and will have to give it a closer, potentially even more gleeful look once I'm less busy getting my own code development methods back into gear.
It isn't versioned. It isn't versioned! The way I migrate from LJ to an older version of the software (like the one JournalFen runs) is to try the full post data set, then respond to error messages to remove unsupported content until it works. ARGH!

I've started tinkering with the source of the OS X client iJournal, in the hopes of maybe imitating some of Semagic's feature set. (And teaching myself Cocoa, which might be handy.) That had some useful things in it.
Oh, man, that's horrible. I was afraid I'd discover something like that if, after completing the revamp of LJlogin, I ever decided to turn my eye towards client/migration methods. (On which note re: the revamp, I did browse the README for ljmigrate, and I offer a mighty word of agreement on the virtues of ConfigParser over XML for configuration. We use ConfigParser extensively at work, and I weep for the lack of it in anticipating the much more complex configuration needs I'm going to have for LJlogin 2.0, which is, of course, in JavaScript, not Python.)

I was using Xjournal for a while, but a bunch of its features didn't seem to actually work, like they'd been stubbed in but never fully coded out, and the last version is wicked old and it's nigh unto unmaintained. I never got around to giving iJournal a try, but I'd totally be down for a client that included a union of features from both of those, and Semagic, etc.

If you don't mind my asking without having gotten 'round to fully reading all the comments to the announcement post or trying it out myself :-) , how does ljmigrate handle migrating comments? Does it stick them in the post, post them anonymously with some info on who posted it on the original journal, something else? That was something that the much-vaunted lj-sec appeared to be lacking, and that I'd been despairing at having to deal with when the time came to set up backup/alternate journals for my own stuff.
It doesn't migrate comments at the moment; it merely backs them up. Back up is in two forms: in XML (which I will get around to re-using sometime when I manage to write the next set of features) and in the bare-bones HTML archive.

I sure could do something like decorating the original post with the comments, but then I'd have to handle character limit issues (64K character count limit per post). Most users are okay with just having the backup. The real bummer is not being able to snag community comments. If you figure this out, I'd love to know how you do it. I tried experimenting with adding parameters to the /export_comments.bml request. Oh, darn it, I know how to figure this out-- this thing is open source, so I should just be able to download it all and read it to see what it's really doing. Bleah.
Ah. Alas. I apparently misread. It does, however, appear to be a much neater and better-configured archival system than ljdump, which is itself a plus. And yeah, I eventually ended up just doing an svn checkout of the LJ sources to trace just how the hell the whole ScrapBook (or "FotoBilder", as the product used for ScrapBook is called) auth crap got handled.