occasionally useful ruby, ubuntu, etc

CouchDB and logs…

I've taken it upon myself to do some log analysis at my work. This involves taking a large volume of logs (multiple gigabytes) and somehow organizing them so that the data stored within them are readily available. This is no small task. At present, there are thousands of individually gzipped files, and the best way to find what you're looking for is to essentially 'zcat *.gz | grep -in "session=123"' to get what you want and is, frankly, absurd. In some types of log files, you have 15 lines that all pertain to the same log record, with each line in the format KEY=monkeyfacevalue, and in others you have everything on the same line, with key/value pairs comma delimited. Doesn't really matter, but there are key-value pairs in both case, and no two log records necessarily have the same set of keys. This sounded like an opportunity to try out a new type of database...CouchDB.

But, in the end, I think CouchDB is not the best way to solve this problem...

CouchDB is what's called a document-oriented database. This is as opposed to relational databases, like MySQL, PostreSQL, Oracle, even sqlite. For convenience I'm going to abbreviate each as DOD and RD, respectively.

How they differ:

1. DODs are schema-less. This means there are no columns/fields to speak of with size constraints. A "document" is a record in the database, which is essentially a hashmap (or associative array if you prefer) with a couple other little features (i.e. revisioning).
2. There are no "queries" in DODs (CouchDB, at least). If you were curious about how something without rigid structure could be fast, this is how. Instead of queries, you have "views", which are only slightly related to views in the RD sense. There are two types of views -- permanent views and temporary views. A view is the result of calling a user-defined function on every record in the database -- if the function evaluates to true, the record is included in the view. Permanent views are stored in the form of design documents within the database itself. Every time you add or modify a record, every view function is called on that record to (re)evaluate whether that record should be included in a particular view. Temporary views are the basically the same, but not stored permanently in a design document -- they disappear after they're used.
3. Some other things, see Wikipedia or their official page.

So here's the problem. You can't just say "Show all blog posts by user X", because that would require you pass a variable. Not a big deal; you create a view that returns true if the blog post belongs to that user. But still, you have to create a view for every single user if you want to be able to select all the blog posts by any particular user -- I'd imagine that could take up a lot of space, and CPU cycles. Alternatively, you could have a view that just returned all blog posts, and then filter through those in your application. Neither would be that good though, I feel, but I might be missing something.

Similarly, in a log file, you can't say "Give me all records with session ID Y" -- you'd need a view for that, and a temporary view would have to be generated on the fly, and the permanent view...well, you'd want a permanent view for every session ID, and there could be tens of thousands of those. Plus every time you add a new record, you'd have to run the view functions for all those on it, and that could take a while.

On the bright side, CouchDB doesn't have any locks in it (helps that it was written in Erlang). This means that no matter who or how many are reading or writing, nothing will ever get blocked. Additionally, I guess you can seamlessly distribute the contents of the database across multiple servers and get redundancy pretty easily, but I haven't looked into that tooo much. Also, CouchDB is technically Alpha, so all of the functionality hasn't actually been implemented yet, like interfacing with Apache Lucene (a full-text indexer search thing).

So CouchDB looks like it has a lot of potential, but I'm not exactly sure for what, unless you never have that many documents in the database or not that many different parameters you want to be searching on.