Mega-fanfic archive dreams

Mega-fic archive wished for.

Scalability is the item on the wishlist that is the hard one. Or rather, the one that requires the brains from whatever poor sap they coax into writing it for them. Those guys need a programmer who doesn't suck in the first moments that the project exists. I don't know that they understand what it means to scale the way they're talking: bandwidth costs, CPU. Disk is at least cheap... The architectural choices need to be made by a programmer who's done this before. The tempting early mistake is putting it all into an sql database; this mistake is fatal for scalability. All the work is in that initial data design.

Goes without saying that you'd open-source it.

The initial feature list includes more items about social issues than about software specification, yet the few that are about the software include some mighty assumptions. Modulating down the expectations to something satisfiable given the constraints of the project would be a trick.

The political fu involved in volunteering on something like that would scare me off. Huge demands + huge wankage + huge pressure + no money. The first thing I'd do is build a very very very high wall around myself so I wouldn't have to put up with the chattering classes. Then fling frequent iterative releases over the wall, XP/Agile style. Often programmers desperately need more interaction with their customers, so they can better understand what they want. In this case, the programmers would need insulation from the customers. Note the trend in the comments: "I'm not a 'coder', but I would like it to..." Have rainbow ponies, do my laundry, save me from having to categorize my deathless prose in any way yet magically make it easily findable by category-driven readers... but the commenters probably don't realize that's what they're asking for, because they don't have the domain knowledge. Some translator needs to tell them so, nicely.

The key insight is that the audience for this archive is the reader. Not the writer. Some of the odder writer-centric demands need to be politely ignored in favor of serving the reader as well as possible.

Finding stories is the major task. By fandom, pairing, author, genre, content, tags, "readers who liked this also liked....", however it works. That's the task an archive has to do well. Oh yeah, and serve those millions of stories to the hundreds of thousands of daily visitors without choking. That too.
  • Current Music: Count It Out : Erik Satie & Dave Brubeck (mixed by Alex C) : Erik Satie
Tags: ,
WORD. it's a lovely idea in the pie in the sky phases, but realizing those dreams?

one of the things i'm most grateful for was being employed as a receptionist type person in small companies during the end of the boom/into the bust, because i learned first how to listen to our programmers/engineers/it people without having the working hanging over our heads (in the sense that i wasn't a project manager or a sales person - so i learned details about projects without needing to get stuff done)... and then coming to realize that 98% of the time the sales/top level project managers/execs/marketing depts don't really have a clue even how to *listen* to the people who work on creating the systems that help everybody else do their job. And sure, the reverse is true too. It was fun (in that 'wow-this-is-sometimes-not-even-the-same-language kind of way) being the mediator.

so, question - you bored and looking for projects?
Yeah, the translation definitely needs to go both ways! Programmers to marketing/sales people (or to an even better representative of their end user). And the non-technical staff back to the technical staff. Every profession has its jargon, which evolves so that people can communicate with each other more quickly. Technical professions are the worst; there's a huge ton of specialized stuff that the programmers know that is not easily communicated outward.

Good project managers can do this and are worth their weight in gold.

I am moderately bored, and somewhat idly seeking a new hobby programming project. (Not one the size of that archive! For one thing, I don't have the scalability experience they'll need. I know enough to know I don't know how to do that project.) An interesting project would tempt me, if it weren't too huge. Can't promise much free time to devote to it until after my first SOG day on June 21 ;)
Well, I'm not a programmer, but I took one look at the first couple sets of ideas for what the proposed archive should do...and I said to myself that if I was a programmer, I'd be ducking and covering about then.

'Hi, we want the universe on a silver platter with watercress and caviar around it for nothing with no problems or hiccoughs ever...and a pony!' Yikes!
Oh yes. No volunteering on this one for me. It would be beyond thankless and right in the territory of soul-sucking.

What I would do if I were cannier (or maybe a tad less emphatically employed than I am at the moment) is take that list and go away and write something that's what they secretly want, though they don't know it. It is an interesting problem to solve, if you take away all the drain of interacting with the people who don't realize that what they want is incompatible with everything else they want. (Or who don't realize that it's not a political problem. Asking more emphatically doesn't change reality. I've seen this in people I've worked with. "If only I yell, it'll make these people tell me I can have it in three weeks instead of this horrible three months they're saying now.")
Pardon my density, but why do we need this? I mean, we really need all fandoms, everywhere, to be united on one site? Did I miss something? I must have, I'm always missing something like that.

Don't most fandoms already have a main archive site? Isn't that the point -that we don't have to wade through loads of crap to get to our favorite authors/challenges/pairings/etc?

I get that some of those features would be nice for any archive, but I don't see why it's beneficial to have all fandoms on one site. That just seems...mind boggling.
Imagine only sensibly designed.
Imagine LJ extended in a direction that assumes the content posted is stories.
Imagine that fandom is making a pre-emptive strike against non-fans moving in and filling unmet needs. (There is this thing I haven't bothered to look at that has the meta-people in a tizzy, Fanlib or something.)

I've just recently been through the New Reader / Visitor From Mars pattern, and I think finding stuff, particularly recent stuff, has been made much more difficult by LJ's takeover of much fandom interaction. Google helps, but then you have weird writers turning off web spidering for their journals so nobody can find them who isn't already fully plugged in. Improved access to content is a worthy goal.
here via metafandom
Often programmers desperately need more interaction with their customers, so they can better understand what they want. In this case, the programmers would need insulation from the customers.


It's a fantastic idea, but as a technical writer I've sat in enough kick-off meetings for Projects from HELL to see that they're going to need one hell of a project manager to pull the whole thing through.

I couldn't spot from the comments if they have any hard-core coders already on-board, but I'm worried that all those ideas - while good for brainstorming - may just frighten away the people who're supposed to make this thing do what it's supposed to do. Because I'm sure that when and if the archive actually happens, it won't have all those features that people want and there will be wank about why feature X was implemented and not feature Y (in the way of the eternal "why can't we have more userpics instead of the pnoneposts because I never use them" complaints at news posts).
Re: here via metafandom
Hello from metafandom, oh amazing Giles-icon-making one!

I completely agree that the people with the skills needed to write this successfully would be running far, far away after reading that thread. Lots and lots of competing interests in the feature suggestions, for one thing. And there will be wank. Sigh.
Re: Thoughts on the problems of a large archive, or why that discussion bothered me... Part 2
I didn't need to read your case histories to guess that fans won't stump up at the rate they would need to to keep these sites going. :(

I think, for a number of reasons, this project won't move an inch without a wealthy bored retired-post-IPO programmer deciding to make it his or her hobby project. And throwing a bunch of money into it. Or you'd need to do a paid subscription model (sub or have ads on your pages), and they'd scream. Sadly they've been taught they can have things for free.

*Waves umbrella* Back in my day we mail-ordered fanzines for real cash!

Re: Thoughts on the problems of a large archive, or why that discussion bothered me... Part 3
The programmer staffing thing works like this: you hire the architect/lead engineer first. Then that person makes design decisions and top-level implementation choices, with a number of factors in mind. The eventual feature set, the anticipated load, the resource constraints, and so on. You don't want non-engineers making those decisions. Then the architect + the project manager recruit the remainder of the staff based on the skills required. You politely turn away the eager and helpful but tragically inexperienced volunteers.

Recruit the project manager first of all, and hope to gawd you've found a saint willing to work for the meager glory and the resume item.
Re: Thoughts on the problems of a large archive, or why that discussion bothered me... Part 4
And good luck finding programmers and a person to be the person where the buck stops. They are destined to get shat on and beaten up on by fandom.

The key insight is that the audience for this archive is the reader. Not the writer.

Sorry? Where is this written?

My interpretation was more along the lines of: the users of this archive will be both creators and consumers. My assumption, based on the impetus for the post (i.e., the FanLib thing), was that the desire was for a place that would be responsive to those who create the content. The writers, in other words, and vidders and artists, and those who wish to enjoy their work.

Have I misread?

(via metafandom)
That would be my insight about what would be required for a successful design. Writers' needs would also matter, in my approach, but they wouldn't be the primary driver. Readers outnumber writers by at least a hundred to one, and that estimate is probably conservative.

The point can be argued, and no doubt will be. At length.
astolat is the technical person for Yuletide, so she does have some experiance with this kind of thing.
Your analysis is sadly true, and scalability is one of those issues that's not pretty but is necessary to architect in from the beginning. Someday, we'll find a rich person to bankroll this stuff.