Caveat Lector

I beat the snow home yesterday by an hour or so. Travel was travel: lengthy, occasionally annoying, but generally uneventful.

The NISO/PALINET workshop was sold out, packed to the gills. I was fair shocked, after hearing one librarian say at ASIST that her boss had said “No, I don’t want one of those institutional repositories—they all fail.”

NISO is going to put up both slides and talk video, and I wholeheartedly recommend taking a look when they do, because saving my presence, every single presenter was firing on all cylinders. (I got in at 11 the night before, was up at 5, and survived the day fueled by sugar, caffeine, and indignation.) I’ll keep an eye out and link the stuff here when it becomes available. I also want to say that NISO has the best licensing agreement I have ever seen; it was a pleasure to sign it!

When I got home, I took the opportunity to glance through Peter Suber’s annual predictions list. I am going to disagree slightly with one of his predictions and add another that I doubt he would make.

I do not think that there will be significantly more open-access institutional repositories in the United States at the end of 2008 than there are today. This is only a slight disagreement with Peter Suber, because he didn’t specify IRs, just open-access repositories, and there likely will be a few more of those, especially outside the States. I also think that if, as Suber suggests, self-archiving hits the tipping point once we get an NIH mandate and a few mandates like it, institutional repositories will not be winners. Nothing will counteract scholars’ natural gravitation toward their disciplines.

I also predict that there will be at least one high-profile IR failure in the United States before the end of 2008. The exact form of this failure I’m not sure about. It could be an outright closure, which will touch off a furious debate about repository succession planning that we really should have had years ago. It could be a more graceful handoff, or a consolidation into a consortial repository. It could be a major defunding; the repository’s materials will remain accessible, but staff time and money thrown at the repository will be reduced significantly or eliminated. (I add that this is most likely at schools that have zero to one dedicated repository-rats, rather than a team-based IR program involving digitization, mediated deposit, meaningful copyright assistance, software hacking, and all that good stuff. But most IRs have zero to one dedicated repository-rats, so I haven’t excluded much.)

But I’ll stake my reputation that it’ll happen. To be fair, I eliminate any repository I’ve ever been involved with, numbering at least three at last count, from consideration—insider knowledge shouldn’t matter. And as is the way of these things, one high-profile failure really means ten more that nobody noticed. I’d predict that too, except I don’t have a terribly good way of finding out whether I’m right.

Look, it’s simple. Institutional repositories are money pits, and the returns are negligible. The cost-per-item-archived is absurd. Libraries may be idealistic, but they’re not stupid, and they do move on from failed experiments, especially when those experiments have a heavy technology component.

After my talk on Monday, which I admit was caustic, one person came up to me and asked why the heck I was still in this business if I was so bearish on it! Well… that’s a good question, and I won’t deny that I have spent a lot of this year wondering whether what I’m doing is viable in the long or even the short (one to three years) term.

Here’s the thing. I agree wholeheartedly with Andrew Dillon that this business of gathering and storing and curating digital materials is not going away. And that’s the sort of work that I get a personal kick out of. And right now, barring a few specialized data-librarian positions, repository work is as close as I can get.

I do think that will change. MPOW has a cyberinfrastructure initiative; I’m involved in it on behalf of the library. But that initiative isn’t being led by the library; it’s being led by central IT. That may well be where my future is, because one thing is crystal-clear: IR platforms as they currently exist and as I see them developing cannot serve the expressed and anticipated needs. Cannot. CANNOT.

Don’t throw RepoMMan and Monash University at me, either. Yes, they are a lot closer to what I envision as necessary. But they are out of reach for most academic libraries in the United States; yea, even many Research Is. Most of these libraries with their half-a-cylinder IT capacity can’t even do anything in DSpace that doesn’t come out of the box, okay? And when I think about throwing DSpace at serious cyberinfrastructure problems, I’m sorry, it’s completely risible. I laugh. And I scoff. And I quietly fume, because DSpace is what I’m stuck with whether I like it or not—and I don’t.

Moreover, libraries (with a few notable exceptions, e.g. Ohio State) mostly don’t have a viable service model for this work yet, and some of what I’ve heard lately, including on Monday at the workshop, makes me despair that one is coming. After Peter Murray’s utterly brilliant talk on intervention in digital workflows as the “fourth wave” of library work (er, that was a spoiler, wasn’t it? oh well), an audience member asked with a certain amount of Gormanesque hauteur whether intervening in faculty digital workflows was really consonant with the library mission, since after all, we deal in authoritative, quality-controlled information, don’t we.

Peter’s answer was great, and I won’t spoil it (watch the video when it’s out!); I will merely add that if this is our work, and I believe large swathes of it are, we have a small and shrinking window of opportunity to get in on the plans being made and the resulting services that will be offered. If we turn up our noses because the information our faculty churn out every day isn’t good enough for us, IT will take up the banner and disrupt us right off the playing field.

And in fact, I’m decidedly worried that is exactly what will happen. That’s our lunch, there. It’s getting eaten. We’re not only not at the table, we’re hardly even in the restaurant. And IRs? Barely even fit on the same block. Consider the librarian at ASIST I mentioned. Consider the effects when my prediction comes true and a big IR folds. How likely do you think it is that libraries will take up a vastly larger project than an IR, with much more nebulous goals and means, once they decide (as I believe they will) that IRs burned them?

I would love to be patient, as Suber suggests I do. Unfortunately, if we’re going to keep even enough preservation infrastructure (by which I chiefly mean librarians engaged with these issues and employed specifically to deal with them) to start addressing digital collection and preservation in libraries, I just don’t think I can afford to be patient. My New Year’s resolution for 2008 is already made: yell about all this, yell loudly, and yell a lot until important decisionmakers actually start listening.

I may blow out a cylinder or two. But it’s necessary.

Just downloaded the Library of Congress report (PDF) on bibliographic control that everybody’s reading.

First paragraph of the guiding principles:

The phrase bibliographic control is often interpreted to have the same meaning as the word cataloging. The library catalog, however, is just one access route to materials that a library manages for its users. The benefits of bibliographic control can be expanded to a wide range of information resources both through cooperation and through design. The Working Group urges adoption of a broad definition of bibliographic control that embraces all library materials, a diverse community of users, and a multiplicity of venues where information is sought.

I’m sold. I haven’t even read that entire page yet, and I’m willing to call this report a roaring success just on the strength of that paragraph. I was recently at a gathering of librarians (details left vague for obvious reasons) in which someone expressed disbelief at Roy Tennant’s “MARC must die!” attitudes because “obviously we’ll always need structured bibliographic description!”

I wanted to sink into the floor and die, I was so embarrassed for my profession. That the equation that librarian drew is just plain flat-out wrong and it didn’t even occur to her to question it… it’s just shameful.

Go LoC.

Well, I’ll be a monkey’s aunt. Reading through the Library of Congress report, I was inspired to see if I had an authority record yet.

Yup, I sure do, thanks to the fantasy-authors book. The information in it is already outdated. So it goes.

I’m oddly conflicted about this. I have a nihilistic thing about leaving permanent footprints on the world. Even so, not my choice and all that… and in a library-geeky sort of way, I have to admit I find it kinda cool.

Also on the topic of authority control, and from the same report:

1.3.2.3 LC: Make the LC Name Authority file available as a Web resource, for
downloading or linking to through various Web service interfaces.

As a repository-rat, I wish to remark OH YES PLEASE MAY WE HAVE THIS RIGHT AWAY and let’s make it not session-based while we’re at it!

I’ll feed back headings I create for people who haven’t written books! Really I will!

My slides for the NISO/PALINET talk are now up, here and in PDF at NISO’s own site.

Enjoy. Be warned. Or something like that.

The sad thing is, I’ve never installed a Firefox plugin for web developers before. Yes, I know, I know. I’m kicking myself, believe you me.

I installed Firebug the other day, because my Firefox profile managed to hose itself (grrrr) and I had to reinstall a bunch of plugins anyhow, so why not pick up a few new ones?

This morning, I had one of those horrible moments when you click on a perfectly innocent link in your new design and the resulting page is oh-noes b0rked.

Firebug found the problem within thirty seconds of my figuring out how to use it (which itself took less than five minutes). Isolating it on my own would have taken me hours of cussing and fiddling.

The Frozen North, she has been welcoming me back with an extra-special display of frozenity. If barely making it out of town after an ice storm wasn’t enough, it snowed again the day I got home, and then again again while I was walking home from class. In between was the coldest morning I remember in quite a while.

Though I dressed for it, so well that I’m still not sure the weatherfolks weren’t exaggerating, because the walk to work didn’t feel bad at all. The home-from-class snow was fairly gentle, and it wasn’t hardly cold out at all, and I enjoyed the little diamond pinprick lights of fresh snow, and the snow-fog over the bay.

The ice fishers are out already. Those people? Are crazy. For my part, though, I still have a sneaking fondness for winter, though I know that fondness will have snuck entirely away by late February.

This morning I wandered into the kitchen and saw a right forest of icicles from our kitchen windows. (This does not surprise me, in a rental house. I love this house, but it needs better care than it’s getting.) I went for the camera to capture the pretty, and Mouser promptly went for my leg to climb high enough to capture the lens cap dangling from its string.

This hurt. Quite a lot.

But I still like winter.

I am extraordinarily fond of Terry Pratchett’s Monstrous Regiment. It is, in fact, one of my favorite books.

I’d love to see it made into a movie. It’d be a cracking good ’un, except…

… I just cannot imagine who on earth could play Sergeant Jackrum.

Only in Wisconsin is the thumping bass coming from a car at the stoplight the underlying oom-pah beat to a polka.

This morning’s reading was the admirably honest and straightforward “Taking EPrints to the Next Level” report. In a field drowning in useless happytalk, it’s good to see an effort as important as EPrints stepping back and taking a good hard look at its missteps as well as its (considerable) successes.

(I am also personally chuffed because I came to one or two of their same conclusions completely independently in “Roach Motel.” I always welcome evidence that I am not in fact stupid or useless, when events conspire to paint quite a different picture. Also, I need to bug the Library Trends editors for permission to post a “Roach Motel” preprint, now that I’ve sent the draft in. Somebody kick me about that.)

There’s a lot of talk in the report about marketing, and failures thereof. Seems to me that what happened—and this insight could be extended to the entire institutional-repository premise—wasn’t so much a failure of marketing as a failure of market research, a serious and ongoing failure.

Simply put, just as libraries charged gaily into running repositories without understanding how the entrenched structures and reward systems of academia would hinder them, repository software developers charge gaily into development without understanding how libraries work, or how repositories work inside libraries. (Yes, yes. I know libraries are not the only venue for repository software. You can’t tell me the market research got done for those other venues either. If it had, I wouldn’t have to keep telling random people who call me on the phone that no, sorry, DSpace doesn’t have any paid vendor-support or consulting options.)

So here, free gratis and worth what you paid, are a few hints about libraries and repositories that ought to inform development and support decisions.

First, the usual open-source “scratch own itch” development model doesn’t work as well in libraries. The reason for this is that with a few exceptions, librarians are not programmers and do not think like them. Most librarians aren’t aware they have itches; we’ve been so beaten down by our software vendors that we won’t scratch our own noses without an RFP! And when we do know we itch, we typically outsource the scratching, not least because we’re not taught to program in library school.

I have personally experienced having my own programming skills wildly overestimated by a DSpace committer. And I’m more technical than most in my field! By a long shot! So repository software developers cannot count on a ready-made bunch of talented, trained itch-scratching developers. Folks, I’m as close as it gets, and I’m not half good enough for your needs. How you fix that I don’t know, but you won’t fix it if you don’t first acknowledge it.

Second, the community-based development models that are so fashionable just at present in the repository community are equally if not more precarious. This just isn’t how libraries are accustomed to acquiring their software and having their needs met! The EPrints report goes into the results of this disconnect in considerable sheepish detail, so I don’t need to; I will merely remark that I’m not bullish on Fedora Commons or the DSpace Foundation.

What are libraries accustomed to? RFPs. Vendors. Hosted services. Black boxes. Fee-for-service, not fee-for-input. Passive, reactive technology management involving minimal technical staff. Saying “no” to anything, from chat-reference services to blogs and wikis to open-source repository software and its associated pay-to-play community structures, that doesn’t fit into those categories. We libraries don’t get involved with standards development except in our own community (and that only rarely); we don’t even have that model to look at!

When communal software development happens in libraries, as it occasionally does, it’s worth noting that the genesis appears to be handshake agreements among individual libraries (or even librarians) who know and trust each other. That’s a very different model from “Hi. I’m the Community Federation and I’m here to help you (if you give me lots of money)!”

So talk about community-based development models is just so much “blah blah blah how much is this going to cost me and why should I pay it at all?” to a library administrator (who, I reiterate, typically knows as much about software development as my cat knows about Christmas). These federations and commonses and foundations and stuff had better get to work on some kind of fee-for-service funding model, because they’re fish in a barrel if they don’t.

Third, this is not a good time to be asking libraries for resources for repositories. Institutional repositories are in enough trouble as it is. MacKenzie Smith asked me rather peevishly on the DSpace tech list why I couldn’t just go get a developer assigned to the repository for a year. Trust me: if I could, I would. For reasons it would be unbelievably inappropriate of me to discuss onblog or on-list, that’s about as likely as the foot-plus of snow on the ground in Madison melting to bare earth by tomorrow.

I’m not alone in this. “Repositories operate on limited budgets, if they have specific budgets at all,” says the EPrints report (page 10). This squares with my experience—and if open-access advocates want to know why this is, they need only examine screeds from some of our more vocal advocates proclaiming that repositories are cheap and easy and fill themselves like magic. That (false) ideology is now coming back to bite would-be repository communities hard.

Fourth, a good many library technologists hide themselves—from their administrations, from their fellows, from the world—because the risk is too great of being shut down abruptly if one is discovered doing this sort of work. I wish I were making this up, but I’m not; I’ve started and supported a couple-three skunkworks projects myself, because there just wasn’t an open way to get the work done. This reticence means first, that the oft-touted egoboo benefits of open-source development do not typically motivate work in the library context; second, that developers in libraries have a tremendous incentive not to share work with other libraries or developers.

Fifth, most libraries don’t have any library technologists. Any. At all. This means that if you put functionality in a server config file, or require a server restart to change it, most libraries won’t be able to tinker with it. At all. “Out of the box” means very different things to developers and librarians; I’ve yet to meet a developer who understands that it means “I don’t have to mess with it, ever, because even if I wanted to I couldn’t” to librarians.

What all this means for open-source software such as DSpace, EPrints, and Fedora is that many itches do not get scratched, and those that do are only scratched locally, these local enhancements never fed back into the software. It certainly doesn’t help the situation that the average open-source developer regards user questions as an annoyance, rather than free requirements-gathering for the next dev iteration. I was roundly mocked for saying something along these lines on the DSpace technical list, but eppur si muove.

Take DSpace. (Please.) To my certain knowledge there are at least four DSpace hacks to create embargoing out there. One (the Minho hack, and for future reference, Yankee devs: that’s pronounced MEE-nyoh or MEE-nyoo, depending on whether you prefer to sound Portuguese or Brazilian) is available publicly in some kind of documented fashion. All were developed wholly independently of each other. None is on track to be put back into DSpace. And those are just the ones I know about; I’d be shocked if there weren’t several I haven’t heard of. There is clearly an itch here. DSpace refuses to scratch it—so we get a lot of useless flailing about, and multiply-redundant efforts that could unquestionably be better-employed. We don’t have so much development talent that we can waste it solving the same damn problems over and over again!

So. What do we do about this knot of snakes?

The strategic-minded development community will do whatever it can to protect skunkworks library developers. I find that a carrot-stick approach consisting of public accolades and the delicate threat of public embarrassment for library administrators is the best shield: sending kind letters of thanks and support on official Foundation letterhead to every single code contributor with a separate copy to his or her boss, coupled with public acknowledgement of all contributors on a website, is a cheap, easy aid. (Any library administrator who reams out the dev over this, or forbids him/her to continue developing, is risking opprobrium in the wider community, and believe me, they’ll get that right away and won’t do it. A project near and dear to my heart at MPOW has been protected from summary execution in more or less this fashion.)

The next avenue of attack is through library vendors, paid developers, and the campus IT units who do most local hacking (since librarians, as I may have mentioned before, are mostly not coders). The strategic-minded development community courts these people like royalty, and does everything possible to ensure that their code gets thrown over the wall instead of hoarded locally. All three open-source repository packages have severe problems with this, for somewhat differing reasons—although none of the three, it is worth noting, has good procedures in place for vetting and adopting third-party code. In DSpace’s case, the biggest problem has historically been MIT’s Not Invented Here-ism, which notably caused certain of them to brush off (in a fairly nasty manner, I might add) the developers of NSpace (PDF), which was a visionary proposal that the current DSpace developer group is only now starting to catch up to nearly three years later.

And where are BioMed Central’s Open Repository developers? I bet you they’ve got some damn interesting code.

Sure, a lot of the code that gets thrown over the wall will be the kind of crap that I produce. Here I invoke Open Source 101: even crap code is better than no code, because crap code can be improved, and it’s a significant hint where the scratch-needing itches are.

(I mean, DSpace knows what it needs, if it cares to look. All anybody has to do is go through the tech list archives. ETDs. File-less submissions. Embargoes. Real dark archiving, not the half-assed kind where your metadata is flapping in the breeze via OAI-PMH. Language-switching interfaces. Statistics that don’t suck. Deposit interfaces and workflows that don’t suck. None of this is news!)

Campus IT is a bit of a harder nut, because they don’t necessarily feel any solidarity with the software package in the way that the librarians do. But developers speak developers’ language. Contact can be made, and should be.

So here endeth Libraries for Developers 101. There’s more to the story, but just what I’ve said here could in my opinion make a substantial difference to the viability of open-source software in libraries if taken seriously.