I noticed that there was a new link on DZone today advertising a blogging engine for Ruby named "Feather". Well, I thought I'd take this opportunity to re-introduce my Java web framework named Feather. :) (Isn't name collision fun?)

(My) Feather is yet-another Java web framework. It isn't trying to take over the world, just the part of the world that uses Java and writes web apps. Oh, and the ones that want the web framework to do as little as possible. As such, Feather really only deals with the Controller aspects of an MVC web app (there are some utilities to help with the model and view, but they aren't required). I started it because I got frustrated with Struts and the 6 files I needed to edit to add a new action. It is used internally for a few applications, but is available under the Apache license.

The basic idea is that you don't need to extend an abstract class or implement an interface to create controllers for web apps. You write a plain java object that has normal methods. The methods are then wired up to incoming requests through regular expression @Path annotations. The methods can have any number of parameters that are auto-injected by Feather based upon request parameters, session attributes, etc... This is also configured using annotations. The only requirement for a Feather action is that the method must return a Feather Result object. This can send the user text, html, forward them to another URL, dispatch to a JSP page, etc...

The documentation is quite light (non-existant), but I'm more than willing to help anyone understand what is going on. Out of the box, it will work with Spring or Guice to build your controller objects, so you can inject anything that you want into your objects. You can create a fresh object for each request, or use a singleton for all requests. The easiest way to get started is to extend the FeatherFilter class, and add that to your web.xml file.

More information is available at http://feather.fourspaces.com

---

Sorry about the formatting errors on some comments... they've been fixed (I think).

I think that I best addressed my lack of willingness to use Erlang here: http://reddit.com/info/6fvf1/comments/c03q623. Suffice it to say that my decision to rewrite the application was a little more complicated than my unwillingness to use Erlang. 

So here I want to address a little better the differences between FeatherDB and CouchDB, and more importantly, where I think we go from here.

 

---

 

Data model 

However, the biggest reason for the rewrite was that I didn't think that CouhcDB the data model quite right. I think that the ability to allow common attributes in documents is an important one. At least, it is for the application that I was planning on using it for.  The use-case that I'm specifically thinking of is the ability to tag pages in a CMS.  If you want to add a tag to a page, using CouchDB, you simply create a new revision of the document, and add the tag.  Simple.  But inefficient.  You're effectively duplicating the contents of a document in the database to add common (meta) data to a document.  The concept of a tag logically can apply to all revisions of a document, so there shouldn't be the need to create a duplicate.

(Note: This inefficiency only applies to the storing/retrieving of documents, running/creating views is a different issue, and I'm not quite sure how CouchDB would handle it) .

 

Java

Why is FeatherDB written in Java?  Another way of saying this is why didn't I just work on the Erlang version?  One reason was that I was primarily using a Windows machine for development at the time, and the development process for CouchDB (even to just get it running) was quite tortuous.  I had to switch over to a Linux box just to write the Java library to connect to it.  Why? Because I couldn't get the most up-to-date version running on Windows.  This isn't necessarily a big deal for me, since I use a Windows, Mac and Linux machine on a daily basis.  However, for some this is a deal-breaker.   So, I wrote it in the language that I was working with at the time... it just happened to be Java.

Would there be the same issue had I written this in Python or Ruby? 

Now, there is one big (and I mean big) advantage that Java brings to the table, and that is the ability to run multiple languages (including Javascript) in the JVM.  This adds one feature that CouchDB cannot add without increasing overhead: the ability for a Javascript view to directly query the database.  This feature allows for the ability to have 'joins' in your views.  Meaning that the Javascript view function that you run can pull other documents from the database.  This makes your view code much more flexible.

It is perfectly reasonable to argue that you may not want this feature, but I'm not sure how you'd do it with the current view system in CouchDB.  Maybe it is possible, and I just don't know it.

 

Scalability

One of the biggest criticisms was the lack of distributed scalabilty of FeatherDB.  My flippant reply was:

Re: Distributed - Via the plugin backend storage mechanism, it actually wouldn't be that hard to make the system distributed. There would need to be a new backend written that delegates requests to other servers. The trick is to figure out the logic to correctly 'guess' which server has the data you're interested in. However, this is entirely possible.

Now, this is admittedly a little simplistic. Writing a new backend to farm a single request to a number of other servers and then 'reduce' them back to return to the user isn't difficult. The pseudo-code is very similar to the in-memory-cache implementation that is currently in SVN. For a brute force, dumb system, it would work. What is difficult is figuring out what other servers to farm the request to. This is the secret sauce of any distributed system, and isn't trivial regardless of language.  As was said by one of the commentors, there is no such thing as a simple distributed system. Perhaps this is were Erlang's sweet spot is... I don't know.

Unfortunately, from what I've seen, the simple distributed scalability that is advertised with CouchDB isn't quite a reality. But this is to be expected... CouchDB is very much a work in progress. People act as though it is a finished product, but this is far from the case. 

 

Motivations 

I think the bigger question is what are my motivations for writing this, and what are my motivations for releasing it to the public.

I love the things I read in Damien Katz's blog and his descriptions of CouchDB and non-RDBMS style of data management.  I first heard of CouchDB from Assaf Arkin's blog, and I have continued to follow it ever since. 

Now, I don't want to compete with CouchDB, and I'm not looking to make any money from this.  I've released this because this is something that I worked on a few months ago, and felt bad seeing it sit in my private SVN repo gathering dust.  That and, these are some concepts that I've been looking at for a while.  I guess that my ideal outcome would be for some of these concepts to make their way into CouchDB so that I don't have to think about it.  But, I'm more of a 'working-code wins' kind of guy.  So I prototyped somethings up, and thought that maybe someone else could find it useful.

Future

I'm not too sure where FeatherDB will go.  As I said, I'd be happy to use CouchDB (or maybe I'll have to look at StrokeDB or GrassyKnoll), but for now, it doesn't fit my needs.  Until it does, I'll keep updating FeatherDB... and I'll be following CouchDB closely.

---

I've been holding onto this code for a while now (months)...  I haven't finished polishing is, and chances are I won't.  But there are a few important concepts in here that I think are important to get out into the open.

FeatherDB was my attempt at a Java clone of CouchDB.  When I first found out about CouchDB, I was very intrigued.  A non-relational database fits one of my projects perfectly... when you find yourself fighting with Hibernate just to get a schema mapped, there is an issue on one side or the other.  In this case it was me. [I should note that I actually like relational databases for most things, it just so happens that my particular application doesn't map well to a static schema]

So, I did what any self-respecting geek would do - I downloaded the code and tried to get it running.  And that led to the first problem: I was on a Windows machine and the (at the time) binary download for Windows washorribly out of date.  Not to fear though, I had a linux box handy, so I just moved over to that machine.  I downloaded CouchDB, and got it running.  Then I had the problem that the java library wasn't compatible with the newer JSON syntax (the old version was all XML, all the time).
  No problem there either... I spend a day or two getting my feet wet with the REST API, and wrote a new library couchdb4j.

Now, I was off and running working with CouchDB, and non-relational nirvana. 

Or at least I should have been.  At the time, CouchDB was (and is) very much a work in progress... there were a bunch of things that were planned but were missing, such as: support for stored/named views and any sort of authentication.  No problem!  I'll just add this myself!

Problem: CouchDB is written in Erlang.

I don't do Erlang.

I don't have anything against Erlang... we've never met.  I just don't want to learn another language.  Especially one that is so specialized.  I know that I should learn a language a year, but Erlang was just too much of a hurdle for me to get in and start mucking around in code.  Plus, would you really accept a patch from a guy that is just learning a language?  Neither would I.

No problem!  I'll write my own Java version of CouchDB... and do it better.  The result was/is FeatherDB. 

FeatherDB is a Java based document database.  It has an all HTTP/REST interface.  It supports querying by a Java class (added as a jar to the server), or via JavaScript (uses Java6's JavaScript support).  In theory you could also search by any language that is on the JVM and implements the correct interface.

FeatherDB allows for flexibility in the backend storage, allowing you to use any backend you'd like.  I wrote three: a filesystem based backend, an all-in-memory backend, and a caching backend that uses an in-memory hash and punts to a supporting backend when needed.  (I'd like to add a BerkeleyDB or Derby backend to make things a bit more resilient).

FeatherDB uses an embedded Jetty HTTP server to handle all interaction. 

There were a few things about the way CouchDB handles documents that I didn't like, so FeatherDB does things a little bit differently. 

Like CouchDB:

1. Documents are accessed through the HTTP interface by a REST API.
   
2. Documents are versioned.  Always.  You can access any version of any document.
3. Document information is stored and retrieved in JSON format.
4. The Document space is "flat".  Documents can link to other documents, but there is no concept of "joining" documents.
5. You query the database through 'views'.  Views are also documents.

Unlike CouchDB (last I checked):

1. Access is controlled through an authentication layer.
   • Users are authenticated
     
     • though HTTP-Auth
       
     • or by loading /_auth?username=foo&password=bar
     • or via a previously generated 'token' (an authenticated user can generate a token, and pass it to an unauthenticated user for temporary access).
   • Users can be restricted to read-only, write-only, or read-write access to a database (document-level ACLs proved to be too slow).
2. Documents can have "common" data that is shared among all revisions.
   • Any attribute whose name starts with "_" is determined to be "common".
   • When you update that attribute, it is updated for all revisions.
   • This makes it trivial to support a feature like tagging, where you want to update all revisions of a document, including future versions.
3. Documents don't need to be JSON documents.  A "document" in FeatherDB lingo is any file of any HTTP Content-type. This has a few nice side-effects.  The FeatherDB server can now serve any type of file to any client.  For example, you can upload a binary image to the server, load it in your browser, and it renders correctly! 
   
   • No need to base64 encode a binary payload. 
     
   • Content-types are determined from the HTTP Content-type header. 
     
   • Everything is stored in binary format, unless the incoming HTTP Content-type has an associated handler. 
     
     • So far, there are handlers for:
       • text/html
       • text/plain
       • application/javascript (JSON)
         
       • and image/{png,gif,jpeg}
   • Only JSON documents can be properly queried
   • A document's id can contain the '/' character.  This means that you can 'fake' an HTTP server's paths by being creative with your document ids (Similar in this respect to Amazon's S3).
4. Because the content of a document is no longer JSON, there is the need for a "Meta-data" JSON file to accompany the main file.  This meta-file is what stores the document's id, revision, etc...  For JSON only documents, the JSON data is stored directly in the meta-data entry.  For other documents, this is a separate file.  Information in this meta-file can be dynamically added by the content-type handler.  So, for example, the image handler can add width/height information to the record, and binary files can have their MD5/SHA1 sums calculated automatically on insert.
   • If you're keeping score, this means that using the default file-system backend, there are 2 files per revision for non-JSON documents, plus one more to store the common data.
5. The JavaScript views can query the database for associated documents.  Okay, so this actually is like "joining" documents.  Because the JavaScript engine is hosted in the same JVM, you can query the Java backend from the JavaScript view.  This lets you create joined views of your data, if needed

Those are the major differences, and things that I think should be in CouchDB.  As far as the status of FeatherDB, it has sat in SVN untouched for a few months, so I wanted to try and make it public, in case anyone else was interested.  AFAIK, it works.  However, there aren't any included libraries for accessing the server, so the API is a little undocumented.  It does follow the CouchDB API as closely as possible though.

You can access the project via SVN at: http://svn.fourspaces.com/public/featherdb/trunk

If you are interested in more information about it, leave a comment, or send me an email at: mbreese at fourspaces dot com.  Currently, the project requires Java 6 for the Scripting API support.

 

Getting started

1. Download the code via SVN from the above url
2. Run "ant run".

This starts the server with the default configuration:

• Port: 8889
• Admin username: sa
• Admin passowrd:password
• Backend: File system (directory "testdb")
• Allow anonymous access

To access the server, connect to http://localhost:8889.

Important URIs 

• GET /_all_dbs -> list all databases
• GET /_auth -> authenticate
  • You can authenticate via HTTP-Auth, or by passing the request parameters "username" and "password" (?username=sa&password=pass)
  • If anonymous access is enabled, all requests are treated as if authenticated as an administrator
• GET /_invalidate -> invalidate the current credentials
• GET /_sessions -> show active authenticated sessions (not in anonymous mode)
  
• GET /_shutdown -> shutsdown the server (must be admin)

Databases

• GET /{dbnme} -> db stats
• PUT /{dbname} -> add a database (must be authenticated as admin)
  
• DELETE /{dbname} -> remove a database (must be authenticated as admin)

Documents

• GET /{dbname}/{documentid} -> Get the document's current revision
  • If this isn't found, but /db/docid/index is found, this will be returned instead
  • Optional parameters:
    • showMeta=true -> returns the meta-information for the document (see above)
      
    • showRevisions=true -> includes a list of available revisions in the meta-information

• GET /{dbname}/{documentid}/{revision} -> Get the document's content (see above)
  
• POST or PUT /{dbname}/{documentid} -> write the request's body as a new revision of the given document
  • If the documentid doesn't exist, it is created
• POST /{dbname} -> write a new document, but use a generated id
  
• DELETE /{dbname}/{documentid} -> delete the document (must be able to write to db)

Views

• POST /{dbname}/_temp_view -> perform an adhoc query 
  
  • The contents of the POST should be in the format of a javascript function
  • Ex:
    

function(doc) { if (doc.value=='foo') {map(doc.id,doc.value); }}

• POST or PUT /{dbname}/viewname/functionname -> add/update a new view.
  
  • Note: the view name must start with an underscore ('_').
  • The default functionname is "default"
• GET /{dbname}/_all_docs -> returns a list of all document ids
  • This actually calls an included view named "_all_docs"

Views can be either written in either Java or JavaScript.  View documents are JSON documents that have the following attributes:

'view_type': 'application/javascript' or 'java:fully.qualified.class.Name'

Java views must implement the interface: com.fourspaces.featherdb.views.View
JavaScript views are implemented as JSON documents in the format:

 { 
'view_type': 'application/javascript',

'view1': function(doc) {
	// your code here
	if (doc.val='foo') {
		return doc; 
	} ,
'view2': function(doc) {
	// your code here
	if (doc.val='foo') {
		map('key',doc.val); 
	} 
} 

So, as you see, JavaScript views are functions that take a JavaScript object as input, and either returns a JSON object, or builds a map with a key and value. (See CouchDB docs for more information).

There is another associated JavaScript method that you can call, and that is: function get(id,rev,db).  From JavaScript, you can retrieve other documents by id, revisions (optional), and database (optional).

Views are maintained by a "ViewRunner".  The ViewRunner is responsible for maintaining the index of results for documents in the database.  The only included ViewRunner doesn't store an index, but instead iterates over all of the documents in the database on each request.   

Configuration

FeatherDB is configured by writing a "featherdb.properties" file. In the src/java/com/fourspaces/featherdb directory, there is a 'default.properties' file that shows the default properties and their values.  Your featherdb.properties file will overwrite the default values.

 

Additional Notes

If you are using an access mechanism that doesn't support cookies, you'll need to pass a "token" with each authenticated access.  This can be passed as a request parameter (?token=asdfasdf), or via an HTTP Header "FeatherDB-Token".  The token is given to you when you access "/_auth".  This isn't needed in anonymous access mode.

There is a severe lack of unit tests.  I'm sorry :).  Specifically, there is a lack of tests for running views, but it _should_ work.

I make no promises about this code.  If it works for you, great.  If not, let me know, and I'll see what I can do about it.  As always, patches are gratefully accepted. 

 

 Future directions

I'm not sure what to do with this project.  I'd prefer to not maintain a separate code base that may or may not track the CouchDB API.  I'd personally like to see the changes in the document structures implemented in CouchDB, but I'm not sure if it is possible given the split in a document data and meta data.  Then again, I'm completely unfamiliar with the CouchDB code base, so perhaps this will be possible.

I wanted to make this all public to see if anyone else may be interested in using this type of database and may be interested in helping to flesh it all out.

Hope you enjoy it! 

---

My last post was all about the problems inherent with using Java as a builder for Java programs.  My conclusion was that it isn't a good idea... mainly because you have to solve a bootstrapping problem - you need to compile your builder before it can work.

Well, I thought about it a bit more, and figured, eh, what the hell?  After all, working code wins.  So, I sat around and attempted to make my own Java based Java builder.  This ultimately turns out to be a two-step process:

1. Compile/package your builder
2. Call your builder with it's own classloader

So, how does it work?

You put your builder code in it's own source tree.  I decided the "builder" folder would be a good choice.  So, you put your builder's java source code in "builder/src", dependencies in "builder/lib", and your compiled code will end up in "builder/classes".  My "Builder" class will compile everything in builder/src, and call whatever methods in one of your classes.

What?

Maybe it would help if I included an example... Here is how you build the builder:

java -cp builder.jar com.fourspaces.builder.Builder com.fourspaces.builder.BuilderBuilder
(or more simply)

java -jar builder.jar com.fourspaces.builder.BuilderBuilder

What this does is pretty simple: it uses the com.fourspaces.builder.Builder class to compile all the source code in "builder/src" and then calls the @Default task in com.fourspaces.builder.BuilderBuilder. In this case, the default task compiles the Builder project itself and creates the builder.jar file. You can call any method (that has no parameters) in the class that you call. So, for example, this will clean-up any artifacts from a previous build (clean method) and create the builder.jar (jar method):

java -jar builder.jar com.fourspaces.builder.BuilderBuilder clean jar

Not only can you call any method, you can also configure @Depends task-dependencies.  So, in the above case, the "jar" task depends on "compile", which depends on "init".  So, the methods that are called are (in order) clean, init, compile, jar.

The final little feature is that you can optionally create a single custom-builder jar file. If you do this, your command-line can be as simple as:

java -jar custom-builder.jar clean jar

 

---

 

Okay, so you can successfully create a bootstrapping builder in Java.  Woohoo :)  The real question is: is it worth it?

After much deliberation, I can say this: maybe.  It doesn't suck as much as I thought it would, and you can really do some amazing things that you can't do in Ant.  You can create whatever type of building infrastructure that you want without having to adapt your process to your build tool.  The downside is that you have to do your own coding, so there isn't much already done for you.

As an example of what you can do, here is the BuilderBuilder class.  I wrote three helper classes as well "JavaCompile", "JarBuilder", and "FileSupport".  These helper tasks make compiling and jar building quite nice... from this code you can see how to define task dependencies, how to configure the default task, how to compile, how to jar up files, and how to do some basic file operations. 

public class BuilderBuilder {

	private String srcDir = "src/java";
	private String buildDir = "build/classes";
	
	@Depends("init")
	public void compile() throws IOException {
		new JavaCompile()
			.setTarget(buildDir)
			.addSource(srcDir)
			.compile();
	}
	
	@Default
	@Depends("compile")
	public void jar() throws FileNotFoundException, IOException {
		new JarBuilder()
			.setFileName("builder.jar")
			.addSource(buildDir)
			.addInclude("**/*.class")
			.addSource(".")
			.addInclude("LICENSE")
			.setMainClass("com.fourspaces.builder.Builder")
			.create();
		
		FileSupport.copyFile("builder.jar","builder/lib/builder.jar");
	}
	
	public void init() {
		mkdirs(buildDir);
		mkdirs(srcDir);
	}

	public void clean() {
		deleteDir(buildDir);
	}
	public static void main(String[] args) throws SecurityException, NoSuchMethodException, IllegalArgumentException, IllegalAccessException, InvocationTargetException, ClassNotFoundException, FileNotFoundException, IOException {
		Builder.invokeBuilderTasks(new BuilderBuilder(),args);
	}
}

So, all in all, not bad.  I'm curious to see if there is any interest in this builder.  If there is, I'm interested in polishing things up a bit and making this project a bit bigger (and give it a name).  I'm planning migrating my own projects over to this type of builder, so the built-in tasks will be pretty much centered around what I need.  Specifically, the tasks that are on the list are: SVN update/commit, Javadoc generation, creating a WAR file, starting a Jetty container (using said WAR file), and JUnit testing.  One other things that I'd like to add is integration with Maven/Ivy jar repositories. If you're interested in checking it out, you can check it out from SVN at: http://svn.fourspaces.com/public/javabuilder/trunk/

Note: it should be mentioned that this requires using Java 6, since it relies on the Java Compiler API. When running, you should make sure that you are using the "java" executable from the JDK and not the JRE (a common problem)

---

I am very intrigued with the idea of new Java building infrastructure. I'm a pretty committed Ant devotee, owing to bad experiences with some early Maven builds. So far, there hasn't really been anything that I want to do that I can't get Ant to do. Every so often I needed to write a custom ant task, but that's pretty rare. However the main reason that I don't like Maven is that I'm a control freak. I like things in the directories that I want them. I don't like my project's build to the build tool. That just seems so counter-intuitive to me. But this post isn't about Maven...

Now, my needs are pretty simple: 1) setup directories, 2) compile classes, 3) run unit tests, 4) assemble jars/wars, 5) sometimes start/restart Tomcat. I don't deal with much inheritance simply because I don't need it. The few times I've tried, it ended up a giant sticky mess (that ultimately didn't work). My basic build directory setup is also pretty standard, so I can (for the most part) reuse a build.xml file from one project to another.

Even though Ant has served me well in the past, I have always had some uneasiness with the concept of using an XML "script". I've been slowly removing all XML configuration from my projects, opting instead for Annotations or DSL-style Java configuration (much like Guice). So, you could say that I've been in the market for an Ant replacement.

One top contender is Buildr. If you don't know, Buildr is a Ruby based build system for Java projects. (Such is the life of a polyglot). However, I've been holding off on trying Buildr until Ruby 1.9 comes out... I honestly don't have a good reason for not trying it...

Today I ended up reading about Gosling. Now this is an interesting idea that doesn't seem to be going anywhere (the last release was in January 2007). I starting to think about this a little more... why isn't there a Java based Java builder? I think this is a chicken and the egg problem. You need a builder to build the builder. And then it's even a little more complicated. You need a builder to build the builder so it can build your project.

Let's take a step back. What would the workflow look like for the developer? Well, if you're doing command-line builds (you can build all of your deliverables from a single command, right?) then you expect to be able to type a command and maybe an optional task. Such as: make all
Even if you're doing some nifty bootstrapping, you're still looking at something like: java -cp . Bootstrap Builder task
Now, if it were me I'd add a bash script to at least cover the java bootstrapping, so we're looking at something like build.sh task
That's a little better. But, you still have to compile your project's builder.

So now instead of having one set of source code, you have two. Your project, and your project's builder. And if your builder requires libraries, you have two sets of dependencies (at least... your unit test may have their own as well). But, once your builder is compiled, you can finally compile your project. java -cp builder/classes MyBuilder task
but that looks like crap (and takes too many keystrokes), so you'll have to write another shell/batch script. But at least we're getting somewhere.

---

This all seems very much like the saying "now you have two problems". I guess the point of this all is that while a Java based Java builder is an interesting idea, I'm not sure how much sense it makes. Look at the granddaddy of them all: make. Make exists largely to build C programs. Why not write a C builder in C? Well, then you'd have to first compile your builder...

(note to self: just try Buildr)

Copyright ©2006-2008 Fourspaces Consulting, LLC and the author. All rights reserved.
Comments are owned by the poster.