It wasn't long before websites running these directories realized they could write a program that browsed the websites that were listed on their directory and archive the discovered text into a database, and then expose this database to the public via a convenient form in their handy dandy Internet Browser (alternatively, you could "submit" your website to search engine to tell them to index it...you still can, too).

However, most of the search engines weren't very...good. If you searched Excite and Lycos didn't find what you were looking for, you'd try another search engine (WebCrawler?). Still didn't find what you were looking for? Keep looking! Maybe Altavista has it!

The tech folks realized this, too, and came up with a somewhat reactionary response: make a search engine that searched other search engines. Metacrawler? Hell yeah. Your one query would show you results from two, three, or more other search engines. Even now, Metacrawler searches Google, Yahoo!, Bing, and Ask (which, for the record, appears to be the same as WebCrawler now).

Then along came Google. Google had three main allures: it was simple (no bloated "search portal"), it had great (and fast!) search algorithms, and indexed damn near everything (1998-2000). Lots of people were contributing content to the web, creating their own sites on Tripod.com, GeoCities, and Angelfire. The amount of content search engines were indexing continued to increase. Unique page counts increase from thousands to millions to billions (Google is purportedly around 20 billion pages now, see http://www.worldwidewebsize.com/).

Innovation slows a bit (well, a lot) when the dot com bubble pops, when companies realize that simply having a website doesn't guarantee multi-million dollar successes (shocking!). While people are still playing with JavaScript and making pretty animated snowflakes tumble down the tops of their GeoCities pages, Google does something unprecedented: it releases Gmail on April 1, 2004, creating one of the first major web applications (what we now call "Rich Internet Applications") the world had ever seen, and launched it into the spotlight by offering an unprecedented amount of space in the inbox: 1 gigabyte! Other web-based email services existed, like MSN Hotmail (with 2 megabytes for email storage), and Yahoo! Mail (with 25 megabytes of email storage on the premium plan, $30/yr), but these suffered from various issues from security to slowness to being downright ugly.

Gmail, besides revitalizing the webmail industry, showed us all two things: 1) Javascript can be used to be, well, useful, and 2) XmlHTTPRequest is incredibly useful (and iframes, bleh). This begins the trend of taking web applications seriously as a replacement for certain desktop applications.

Fast forward a few years -- there are now many many Web 2.0 apps and/or RIA apps. So many, in fact, that there is a veritable glut of apps. We have a new discoverability problem, but not because we can't find the apps we need, but because we can find too many apps that could fulfill our needs. How do we deal with hundreds of todo list apps, dozens of task-management apps, scores of invoicing apps, etc etc?

Well, there are a few things. SEO (and search), word of mouth, getting featured on news sites, and awards. But those are all so...qualitative. None of those things means a product is necessarily even what you're looking for, except for getting covered in a story perhaps, but even that is only the opinion of one person. What if you want reviews from lots of people, and not just journalists? Or reviews of all sites, not just sites popular enough to get noticed by the media? And that's why we'll be seeing an increase in web app stores -- they're low barrier to entry, help people learn about online tools from other people, and are more structured. People will be able to find what they're looking for without having to figure out the best combination of search keywords. This is definitely a good thing, provided that superior web products get the attention they deserve and incumbency means nothing. Looking forward to what the web has to offer, again!

1. Webmaster creates Javascript that requires database with schema A
2. User goes to website and a local database with schema A is created
3. Webmaster updates Javascript to use schema B
4. User goes to another page -- but what now? User's database is using schema A but the javascript expects schema B

How does the webmaster update from A to B? Or create the initial schema, for that matter? changeVersion.

Here's how you migrate the original schema, A in our case:

That works...but already there's a problem. If a user revisits a page, that changeVersion line will attempt to execute again. Easy enough to fix:

(Note: the second argument to openDatabase is the "expected version" of the database. If you specify it, the openDatabase method will fail unless the database matches the version you provided, without chance to recover. I haven't found a use for this yet, so you'll always see the expected version as empty string from me. Also note: this particular problem could be solved by setting the initial schema in the openDatabase callback, but you'd still hit it later when you add your first schema alteration)

Now users can view and refresh a page without the database connection erroring out. Gooood. But wait, I need to modify the schema again. What now? Let's try this:

This...actually doesn't work at all (except within the scope of web workers, perhaps), because within the "regular" web sql database (window.openDatabase), everything is asynchronous. Which is awesome for performance, but a pain if you're trying to do what we're trying to do right now. Also, changeVersion only changes the version after the callback has run, successfully. In other words, for a user new to the site with no schema, the "if db version == 1" line will get executed before the database has been updated to version 1! If you refresh the page a few times this might work, but...that's obviously not acceptable. But we're getting closer:

Alright, this actually works, but it's cumbersome and still has a fair bit of code duplication (less in the second version...).

And now for the punch line -- what if we take that "v3" migration, stash all the migrations in an array, and...say, put it in a class, maybe called Migrator? And that's what I've done: And usage example:

Granted, it is small and fairly simple (and uh, disclaimer, not tested super thoroughly), but if nothing else it should be a good starter for your own migration scripts.

Thanks for reading!

UPDATE on June 5, 2010:
I've created a github project with a more recent version (one that works with Safari/Webkit), available here. It does this by using a schema table to track state instead of relying on changeVersion. While sort of a hack, the changeVersion design is rather poor anyway (pretty much the only useful openDatabase version string is "", since everything else could throw an exception). Additionally, I added some logging, an example, and what I discovered to be a much-needed feature: callbacks, which let you execute further SQL with confidence that the migrations have already occurred.