One Big Library. - openurl

The other end of the mic: OpenURL, Crossing Over

dchud — Sun, 18 Feb 2007 01:43:56 -0600

In case you didn't catch the updated text at the top of the last post here, Jon Udell kindly followed up on a comment he left here recently and interviewed me for his Friday podcast about OpenURL and other topics persistent and linkish.

...in which I:

try to explain OpenURL linking
fail to recognize or usefully distinguish the differences between archive.org and arXiv.org (I just heard "archive.org" and went with that even though Jon meant "arxiv.org", at least early on, I think)
reference the 50+-year research and publishing history of Eugene Garfield
dredge up the old DSpace identifier wars
discuss some issues with the Handle System and the centralized/distributed governance spectrum
describe COinS and how to use COinS browser extensions at your library
highlight the COinS Generator, the OCLC OpenURL Resolver Registry, and LibraryFind projects
crib Jon's notion of a publishing surface area and reference FRBR as a framework for considering the surface area of a distinct intellectual or artistic creation
say "Mess is Lore" and reference the Book Arts Press valentines the *week* of valentine's day without saying they're valentines
mention the Computer History Museum
try to convince Jon that he should find an archives to work with to start collecting his papers (online and otherwise) and correspondence without actually saying that so succinctly or usefully
give Jon the scoop about my upcoming new job

It was a real pleasure to get to talk with someone whose writing I've read for many years and to appear on a podcast I've been listening to since its inception. Jon's long-time crossover interest in libraries (c.f. Library Lookup) is one of the strongest bridges we library hackers have to the broader software and systems communities. It's also a privilege to follow great folks like Tony Hammond, both John Blyberg and Ed Vielmetti, John Wilkin, and Lou Rosenfeld (both of whom I was lucky enough to interact with while at grad school at umich, check the podcast feed for these older, pre-Jon's new blog links) as a library-interested interviewee.

Unfortunately I've been too busy to put together more Library Geeks interviews, but I plan to start that back up again once I settle into the new gig, I promise. It's easier to be the one asking the questions!

Rethinking OpenURL

dchud — Fri, 16 Feb 2007 01:38:28 -0600

Recently I was interviewed for a podcast episode that might (hoping it indeed is published! [Updated 2/16, 5:30pm: yep, and here it is, and there goes the news about my new job...]) soon be heard by a lot of people. The topic was OpenURL, and why it's both so exciting and so frustrating to everyone who dives down into it.

I've been particularly frustrated with OpenURL lately. Just when efforts like COinS are starting to pay off (c.f. new Wikipedia cite templates, worldcat.org, new Wordpress plugin from Zotero, Zotero itself, and LibX), it feels like the door to wider use and acceptance of COinS specifically and OpenURL in general remains welded shut.

Don't take this the wrong way - I remain a huge OpenURL fan, including the whole vision that led to the ANSI/NISO Z39.88-2004 standard, even if that standard turned out to be way too complicated. I've been a fan of this way of doing things since we added multiple linking paths to the now-defunct jake project in early 1999, so we could link users from bibliographic references on the web into the books and journal articles the way your library has best set you up to do so, in pretty much an OpenURL-0.1-compatible way, even before we ever heard about OpenURL. (It was, after all, obvious to have a bunch of HTTP GET fields with names like [title, spage, volume, issue, issn, etc.].)

And I'm not saying we need to do away with OpenURL. I think instead that we need to recontextualize what it is we think OpenURL is for, and to do so in a way that focuses on the common use case of service linking on the web today. And I think we need to avoid dogmatic adherence to the Z39.88 standard, since we all mostly think it's overwrought anyway.

Let me try to explain it this way. Have you noticed those little clusters of links next to stories at NYTimes.com, or at the end of blog posts, or in reference databases, or on journal sites? These little blocks of links are popping up all over the place. What they are, in general, is "a list of the functions you can do from here, given the thing you're looking at now, depending on who you are."

New York Times service links.

Blyberg.net weblog service links.

Pubmed.gov service links.

JAMA service links.

Citeseer service links.

Google Scholar service links.

This, my friends, is dynamic service linking, in a nutshell.

(And, yes, I do mean "in a nutshell" as in "ooh, ooh, help, I'm trapped in a nutshell, get me out of here.")

I believe that this analysis maps *exactly* to the vision for OpenURL specified in the documents that led up to the ANSI/NISO Z39.88-2004 standard that nobody can actually read. They nailed it - they knew this was coming, and they were right, and here it is, all over the place, on the most popular sites on the web.

But there are two major problems with this analysis.

First, all of these diverse sites publishing useful service links for their own materials have *no* idea that I'm a Yale University person and that *my* library's OpenURL resolver is over here, yes, please, thank you very much. This means that there's no public, cross-site workflow for getting out from most of these sites' own concepts of what "context-sensitive links for me for this content object" what mean back into what Yale's library thinks should be useful links for me. Nor is there a clear workflow for me to get back out from a Yale resource to these online, not-at-Yale items. This is a fatal flaw in the current state of how we both conceive of and implement OpenURL solutions in libraries today (which is to say, not a fatal flaw in the model, just how we use it now).

Until we address this flaw in our current thinking, and bring this wider world of web resources and their own notions of service links and our traditional, more narrowly defined research library workflows into some kind of common system, we'll never see the full vision of OpenURL realized.

Secondly, and this is perhaps a bit less practical, but no less important, but I think the notion of "context-sensitive" linking does itself have a basic flaw. If you've studied finite automata or linguistics and know what language parsing is all about, "context-sensitive" might ring an alarm for you. "Context-sensitive" grammars tend to be significantly computationally harder to work with than "context-free" grammars. (If you're not familiar with this distinction, see the Wikipedia entry for Chomsky hierarchy.) Do you know how the "natural language query" problem remains mostly unsolved? Well, think about the difference this way - most human languages are context-sensitive, whereas most computer programming languages are context-free.

What we've done with the OpenURL model and how we use it in libraries today is to design a standard around our unusual, more complex cases (dealing with weird bibliographic references). The workflow we have applied to OpenURL is context-sensitive in that the publisher needs to recognize my network address, and needs to then redirect to my institutional resolver, and that our institutional resolver needs to be pre-configured with our holdings information, and the outgoing links from our resolver needs to be aware of my network address again as I go out there to get the thing I wanted in the first place. Without all of this preceeding context-setting, the workflow breaks, so our context-sensitive worldview simply requires all of it.

In the meantime, sites applying the more common, simpler cases (dealing with web links) have essentially adopted the entire OpenURL model but without so much as saying so. That blog engine doesn't care what my network address is, and doesn't know whether I have accounts at the endpoints of the digg and del.icio.us links at the bottom of the blog post. It'll send me there just in case I do, and at the moment of need, just in time, if I need to identify myself to digg, or to whatever the target link might be, I will, if I have an account. But not before. Digg doesn't need to know that I'm a TimesSelect subscriber. The New York Times doesn't care that I use unalog, not del.icio.us. This whole scenario does not depend upon - and thus performs none of - the context-setting of our library research and licensed resource workflows.

The computational aspect of it doesn't bail out if you don't have a del.icio.us account - del.icio.us just doesn't let you in, but I can still get there if I want. In the heavier-weight scenario, I can't even figure out where I might want to go through my OpenURL resolver if my institutional context isn't sniffed out by the original remote resource, because I never even get to the OpenURL resolver in the first place.

I think I'm starting to see a better way to address all of this. This post is already pretty long, though, so I'll just state the objective for now.

The goal of rethinking OpenURL, I think, should be to reimplement the same original OpenURL model of dynamic service linking as - in the typical case - a context-free workflow. The instant we do that conceptually, we see how unified all of this really could become. It gives us librarians something to offer the nytimes.coms and diggs of the world: a singular framework for mixing and mashing service links for diverse sites independent of who we all are and what we know about each other.

Then, to address the more complicated context-sensitive research workflows we hold so dear, we simply define a context-free way to hook into that workflow, so if some arbitrary transaction needs it, it can hand us over into it, but *not* until we have to have it.

I'm not sure who said it first, but there's a librarianly spin on the old Perl paradigm I think I heard at code4libcon or Access in the past year: instead of "making simple things simple, and complex things possible," we librarians and those of us librarians who write standards tend, in writing our standards, to "make complex things possible, and make simple things complex."

That approach just won't cut it anymore.

Coming soon: a concrete way forward.

OpenURL over OpenSearch using Parameter extension

dchud — Thu, 7 Dec 2006 14:27:39 -0600

I will get back to the ZeroConfMetaOpenSearch series soon. But before I could go much further, I had to try this first.

Here's what the OpenURL KEV format for journal and journal article reference queries might look like if it were to follow OpenSearch 1.1 Draft 3. More to the point, this is what the block specifying the journal KEV params might look like inside the OpenSearch description response if it were to use the Parameter extension.

<Url xmlns:parameters="http://a9.com/-/spec/opensearch/extensions/parameters/1.0/"
     type="text/html"
     template="http://example.com/search">
  <parameters:Parameter name="q" value="{searchTerms}"/>
  <parameters:Parameter name="count" value="{itemsPerPage?}" minimum="0"/>
  <parameters:Parameter name="start" value="{startIndex?}" minimum="0"/>
  <parameters:Parameter name="aulast" value="{aulast?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="aufirst" value="{aufirst?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="auinit" value="{auinit?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="auinit1" value="{auinit1?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="auinitm" value="{auinitm?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="ausuffix" value="{ausuffix?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="au" value="{au?}" minimum="0" maximum="*"/>
  <parameters:Parameter name="aucorp" value="{aucorp?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="atitle" value="{atitle?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="title" value="{title?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="jtitle" value="{jtitle?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="stitle" value="{stitle?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="date" value="{date?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="chron" value="{chron?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="ssn" value="{ssn?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="quarter" value="{quarter?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="volume" value="{volume?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="part" value="{part?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="issue" value="{issue?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="spage" value="{spage?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="epage" value="{epage?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="pages" value="{pages?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="artnum" value="{artnum?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="issn" value="{issn?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="eissn" value="{eissn?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="isbn" value="{isbn?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="coden" value="{coden?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="sici" value="{sici?}" minimum="0" maximum="1"/>
  <parameters:Parameter name="genre" value="{genre?}" minimum="0" maximum="1"/>
</Url>

I *think* that's right. It isn't clear to me whether the q/searchTerms param is required in OpenSearch or not. If it is, then we could use it to paste through an url-encoded text representation of the reference, I suppose. If not, well, it could just be optional, or dropped entirely.

We'd still need to do work on the response record format - perhaps something might be borrowed from the IESR Metadata specs and inserted into RSS2.0 or Atom responses.

What's amazing to me about OpenURL is that it's been implemented so successfully (in the sense that we had nothing before, and now we have something). That said, I repeatedly hear arguments that by defining and using more OpenURL application profiles you gain benefits through the shared OpenURL infrastructure. I don't buy that because I've never seen such benefits. If you swap out OpenURL and swap in OpenSearch and use the same application profiles, like in this example, then I could see the benefits of shared OpenSearch infrastructure.

Talk: COinS, unAPI, and a Plan for Zero Configuration Service Discovery

dchud — Thu, 2 Nov 2006 22:15:37 -0600

Today I gave that talk at the NISO D2D meeting. I think it went pretty well despite having gone way beyond what I'd originally thought it would cover.

The first part basically introduces and gives background for why COinS and unAPI are useful steps forward. The second part argues that:

We should merge our metasearch and openurl resolver interfaces
We should layer an opensearch interface on top of that
We should register the opensearch interface as a DNS-based wide-area zeroconf discoverable service

Why? If we did all of that, then everybody visiting our domain could find our base search interface, and remote services visited by people from our domain could find our resolver interface.

The slides are attached as pdf. Sadly I botched my own audio recording, but tomorrow I'll plead for a copy from the soundboard guys, and will add that too if I can get it.

[Updated 2007-01-08] The audio is a little blotchy, but here it is.

Library Geeks 001 - Fun with OpenURL (updated)

dchud — Mon, 7 Aug 2006 22:24:58 -0500

Hear ye, hear ye, we have met the Library Geeks and we are us: Listen to Library Geeks 001 - Fun with OpenURL.

Ross Singer joins me for the first episode, wherein we discuss OpenURL, the state of the OpenURL resolver marketplace, and the innovative work Ross is leading at the Georgia Tech library to implement a next-generation resolver.

[Updated.] Get it through the feed at geeks.onebiglibrary.net/feed.xml.

Notes:

Learn more about the umlaut at the umlaut trac
The JCDL paper on statistics is available through the umlaut itself

A Clean, Well-Linked 'Base (or, Solving the "Appropriate Resolver" Problem with the OCLC Resolver Registry)

dchud — Thu, 3 Aug 2006 08:47:39 -0500

Along with several colleagues I've come to the opinion that the current OpenURL implementation workflow and user experience is wholly insufficient. There's too much maintenance overhead on both sides of the library/publisher equation. There's too much work for small publishers to be able to participate. There's almost zero possibility that a tiny publisher (like a single weblogger somewhere) will be able to put useful OpenURL links on their own little sites.

What we need next if we really want to spread OpenURL-based services more widely is a no-configuration, no-overhead, inexpensive solution that works for the widest possible range of libraries, publisher/vendors, and users. (The usability of the prevailing OpenURL resolution click-flow is a wholly separate matter with its own insufficiencies, but we can't solve everything at once.)

COinS, as cool as it is, remains inadequate for meeting this need statement because it requires every user to twiddle bits on their desktop, although it pushes us a step closer by *allowing* users to benefit by twiddling bits, which wasn't possible before.

What to do? Use registries.

The OCLC OpenURL Resolver Registry comprises records for roughly 1000 OpenURL resolvers at various institutions, mostly but not solely in North America. It also provides a simple web service that takes an IP address as a parameter and returns zero-to-many resolver records for every resolver that serves users coming from that IP address.

What does that mean? If you're like me, and you work for a small service like the Canary Database, you used to be essentially unable to provide user-appropriate OpenURL linking without having to configure many many ranges of IP addresses after many many conversations with librarians. "Used to be," that is.

Are you on a campus (or using a campus proxy) for an institution with an OpenURL resolver? If yes, visit the Canary here and tell me what you see.

What you *should* see (only just turned this on, mind you, so please report broken stuff!) is working links to your own institution's OpenURL resolver. Easy, right?

Here's how to implement it on your site:

read about the OCLC OpenURL Registry Gateway service and find the details of the query service in the Word doc on that page.
implement code in your database that queries the gateway with the IP address of your webapp's incoming users (REMOTE_ADDR in CGI-land)
parse the response and if there's a resolver in there, formulate and render links and link buttons to that user for all the references on your site
watch users' eyes light up when they see links
get excited

Well, it's a bit more complicated than that. You're not quite done:

since you don't want to hit OCLC's service multiple times for the same user, build a little caching system into your application It could be as simple as a single table with a UNIQUE constraint on users' IP addresses that maybe stores the raw xml Registry responses and parsed values for at least base_url, icon_url, and link_text
instead of hitting OCLC every time, first check your db, and only query OCLC if you don't already have a record.

Better, right? Actually, the best thing to do would be to also parse out the per-institution IP address block information and do local queries against the *ranges*, not specific IP addresses. PostgreSQL has a built-in type that supports this really well.

Any competent web geek should be able to implement this in a few hours. Call me if have questions.

So, to review, here's what happens:

You implement a function in your webapp that queries OCLC and caches the resolver information locally and then renders appropriate resolver links to all your users
Your webapp users follow the links as if they were always there because it looks just like what they're used to seeing in Fancy Expensive Resources and they'll Just Know (tm) what to do

Pretty cool, eh? It's not without some important problems, though.

It's still not good enough. It doesn't solve the "but I'm off-campus" problem. The good news is, though, that if your campus, like ours, uses a web proxy for remote access, your remote users will probably be using the proxy anyway if they're already doing research, so this should work for them in a lot of cases.
There's no good fallback if there's no resolver for you. Ideally something like OCLC's Find in a Library function could work here too, but that's fraught with difficulties. Trust me, if you've ever been to New Haven, you'll know... if you live on the wrong side of the street across from Ivy U., you won't necessarily get Ivy U. access even if you beg and plead. But, that's a much bigger problem of which this is just yet another instance.
This simple function isn't as smart as the work Google Scholar does to attempt to check against each institution's holdings before showing links. But then again, Google Scholar isn't a last mile service, so they don't want to have to deal with the untidy problem of finding something that might not be online. But we librarians do!
It isn't clear whether the OCLC Registry coordinates queries across the pond to the UK Registry, or any other registry, or whether OCLC's registry comprises their remote data, but it would be good to be able to fire off just one query and be reasonably assured that users *anywhere* will find their resolvers.
There's a fundamental flaw in the whole approach of using a web service to query IP- and DNS-related information.

The flaw with the IP/DNS query bit is that a massive, distributed, caching system for queries about membership of IP addresses in IP blocks already exists, and it's all connected to user- and application-queriable layers through DNS, too. And a protocol which uses these existing layers for just these kinds of purposes already exists along with a related DNS Service Discovery piece. Check the name of the first protocol: Zero Configuration Networking.

That's exactly what we're doing here - providing a zero configuration experience to users. But we're not doing it with ZeroConf, though we probably should be. Or at least we need to make a concerted effort to try it before we dismiss it.

Still, this seems to work. And from multiple, repeated usability tests on the Canary, the first thing everybody always says *still* has been "I want full-text links." Now they can have 'em.

A quick aside: the folks behind the Registry and Gateway at OCLC are supportive of this approach and want to see more people using it, so don't be shy (but cache your queries so as not to be rude, either :). If your institution's resolver isn't in the Registry, or your resolver's record there is out of date, use the form they provide to enter or update information about your service.

Have at it!