Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.

I recently watched an InfoQ interview with Greg Young where he discusses an interesting architecture that uses event stream processing. I thought their architecture sounded very interesting, but I did not go into ‘research mode’ to dig into it. Later I came by Jonas Bonér’s blog on “Real world scala” and saw an example of event sourcing using actors in Scala. I thought it would be a good exercise to implement a similar example in Clojure, and perhaps it would also be interesting to compare the solutions and see the effect of the languages on the implementation.

As Clojure focuses on concurrency it was natural to start thinking about what ‘concurrent’ event sourcing would mean and how to implement it.

Sequential Clojure implementation. For context, read Fowler’s article on event sourcing, and for comparison Jonas Bonér’s Scala implementation.

Modeling. Scala uses classes and actors for modeling ships and their state. A ship is an actor that receives events (messages) and reacts to those by updating its internal state, i.e., its location (e.g., departure event, arrival event).

Clojure has agents which are similar to actors. Agents are simpler and there are more functions that act on agents than on Actors; for example, in Clojure it is possible to directly read the state of an agent (using (deref x) or @x) — this is not possible with actors since they are distributed (at least programmed as if they were distributed).

I’ve used two additional Clojure features in implementing event sourcing: watchers and meta-data. Watchers are a form of callback attached to agents. The callback is called synchronously with the agent actions and “derefs of the agent in the callback will see the value set during that action.” I am using watchers to record the events that are sent to each ship. This decouples the code dealing with event storage from the state-changing functions sent to agents. The state of an agent is the location of the ship it represents. So we can get the location of a ship at any time simply by deref’ing the agent representing the ship (with actors this is more complex as one has to send a message to the actor and receive a message with the answer). I decided to use meta data attached to the location to represent the event that caused a move to that location. For example

user> @(first agents)
{:country "At sea", :city "At sea"}
user> ^@(first agents)
{:agent 0, :time #<Date Sun Feb 22 19:49:59 CET 2009>, :type :depart_for,
:loc {:country "Sweden", :city "Malmö"}}
user>

In general, I think it is possible to implement sequential event sourcing reasonably elegantly in Clojure using agents, watchers and metadata. The agents store the state that can change, the watchers record the events that cause the changes by looking at state meta data. The invariant should be that if an agent moves from state s1 to state s2 then (meta s2) should store the event that caused a transition from the s1 to state s2. Then if one knows the initial state, and all the events it is possible to reconstruct the entire sequence of states.

Our watchers are quite simple: (notice the cool syntax #(meta @%). A shame I couldn’t write #(^@%) it looks like I’m swearing! that would be cool)

;;Record event history
(def events (atom (map #(meta @%) agents))) 

(defn event_logger
  [idx a changed]
  (if changed
    (swap! events conj (meta @a))))

(defn add-watchers []
  "associate a watcher with each agent
   the watcher logs the events for that agent"
  (dotimes [i NUM_SHIPS]
    (add-watch (agents i) i event_logger)))

Concurrent event sourcing A natural question to ask for Clojure is “what if there are multiple concurrent event sources?” Is it still possible to do event sourcing? I don’t think so; at least not is a manner as powerful as with sequential events. Since Fowler defines event sourcing as ensuring “(…) that all changes to application state are stored as a sequence of events.” We already have a problem: when events can occur concurrently one must serialize them in order to store them as a sequence. One could try and timestamp all events and store them as a set, but this is sensitive to timing an scheduler issues. Even if stored as a set of timestamped events, how would one replay these events in a way that ensures that the global program states are the same in the replayed program? This would be dependent on thread scheduler timings.

It is of course possible to do the less advanced use cases of event sourcing, e.g. event querying and analysis (I guess that is concurrent event stream processing).

Anyway, if anyone is still reading the Source file link.

A what? A partial evaluator (or program specializer) is a program which takes two inputs: another program and an environment mapping variables to values; it outputs a specialized (i.e., more efficient) version of the input program with respect to the environment. One can think of a partial evaluator as a mix between an interpreter and a compiler: it interprets the static parts of the program and emits code for the dynamic parts.

For a simple example of code specialization, consider the following function for creating a string corresponding to an HTML tag:

var mk_tag = function(tag_name,clazz,contents) {
   return '<'+tag_name+' class="'+clazz+'">'+contents+'</'+tag_name+'>';
};

If we are only interested in making ‘div’ tags with a class of ‘green’ then a more efficient version would be:

var mk_div_green = function(contents) {
   return '<div class="green">'+contents+'</div>';
};

since it would require fewer string concatenations per call. We can think of ‘mk_div_green’ as a version of ‘mk_tag’ which is specialized for writing ‘div’ tags with a class of ‘green’. A partial evaluator for JavaScript could automatically derive the ‘mk_div_green’ function from the ‘mk_tag’ function.

This is exactly what we can do with the evaluator in this posting.

//This code actually works  
Function.prototype.specialize = net.higherorder.jeene.Jeene.make();

var mk_tag = function(tag,clz,cont) {
   return "<"+tag+" class='"+clz+"'>"+cont+"</"+tag+">";
};

var mk_div_green = mk_tag.specialize({tag:'div', clz: 'green'});

mk_div_green("Pratt rocks!");
//result: <div class='green'>Pratt rocks!</div>

mk_div_green.toSource ? mk_div_green.toSource() : mk_div_green.toString();
//result:
//(function (cont) {return ("<div class='green'>" + cont) + "</div>";})

This last line, shows that the output function is much more efficient than what is created by general JavaScript curriers which have been seen before: these functions merely wait evaluating the function until all parameters are supplied; instead, a partial evaluator will create specialized function taking advantage of the information given.

The first goal is for the partial evaluator to process any function written in an extension of “simplified JavaScript” (a subset of full JavaScript corresponding to what Crockford calls the “good parts“). This partial evaluator will have a number of useful features: it

• can easily be extended to full JavaScript;
• is written in simplified JavaScript (think Futamura projections);
• is a fast extension of Douglas Crockford’s Pratt parser;
• because functions implement the toString method, it can be run on dynamically generated functions (e.g., a specialized function can be further specialized);
• can be embedded in any full JavaScript program as long as it is only used to specialize functions which are syntactically in simplified JavaScript;

I have started a new open source project, Jeene, which aims to create an efficient partial evaluator for full JavaScript that works in any ECMAScript 3 compliant implementation (e.g., all major browsers, Rhino, TraceMonkey, V8 etc). Right now the project is at a very early stage; a proof of concept, for example it cannot specialize itself. Let me know if you are interested in contributing: I will use a 1 patch threshold like Rubinius: if you submit one patch which is accepted, you get commit rights.

Stay tuned here for more information about how Jeene is designed and implemented.

[update: March 18, 2008. I asked Crockford what he thinks about this pattern, and he actually discourages the use of 'instanceof' -- instead, he prefers to "... rely instead on good design and polymorphism."]

The inheritance model of JavaScript is based on a combination of the ‘new’ keyword and the prototype property of (constructor) functions. JavaScript Guru Douglas Crockford (aka ‘Yoda’) argues that this model (which he calls pseudo-classical) is awkward. Instead, he proposes an elegant, powerful and simple model (parasitic inheritance), using so-called power constructor functions. Note, familiarity with the above concepts is necessary for complete understanding of this post [and something that every web developer should know anyway ].

The advantages of power constructor functions include support for private, shared and public variables as well as simplicty (avoiding new and prototype). There is a mismatch, however, between constructors and the JavaScript keyword, instanceof. Consider the following example:

//recall that the object function creates a new object which has
//the input object, o, as its prototype
var object = (function() {
     function F() {}
     return function(o) {
         F.prototype = o;
         return new F();
     };
})();//included for completeness.
var OriginalPerson = {
   sayHello: function(){
      return "Hello, my name is "+this.getName();
   },
   getName: function(){return 'Adam';}
};

function Person(name) {
  var p = object(OriginalPerson);
  p.getName = function() {return name;};
  return p;
}

function Guru(name,topic) {
  var g = object(Person(name));//Technically we don't need object(.) here
  g.getTopic = function() {return topic;};
  return g;
}

var karl = Person('Karl');
var crockford = Guru('Douglas','JavaScript');

karl instanceof Person;//<- false 
crockford instanceof Guru;//<- false

Hmm… Clearly, any environment that makes crockford instanceof Guru evaluate to false must be getting something wrong!

In general, one has to do something to make super-constructors work with instanceof. The expression exp1 instanceof exp2 where exp1,exp2 are JS expressions (exp2 must evaluate to a function and exp1 should evaluate to an object) works with following semantics:

First exp1 is evaluated, say, to o then
exp2 is evaluated, say, to f. If o is an object and f is a function, the entire expression evaluates to true only if following o‘s [[prototype]] chain, we can reach f.prototype.

This means that to make this work, we must ensure that the object created has Person.prototype or Guru.prototype in its prototype chain.

What I would really like to end up with at the end of this blog entry, is to be able to write code similar to:

var OriginalPerson = {
   sayHello: function(){
      return "Hello, my name is "+this.getName();
   },
   getName: function(){return 'Adam';}
};

var Person = OriginalPerson.parasite(function(Host,name) {
  var p = object(Host());
  p.getName = function() {return name;};
  return p;
});

var Guru = Person.parasite(function(Host, name,topic) {
  var g = object(Host(name));
  g.getTopic = function() {return topic;};
  return g;
});

Guru('Douglas Crockford','JavaScript') instanceof Guru;//<-- true 
Guru('Douglas Crockford','JavaScript') instanceof Person;//<-- true

The extra parameter Host is supposed to represent the “Host” of the parasite (i.e., Person in the case of Guru), the idea being that the parasite function will somehow ‘wrap’ the Person function to set it up so that ‘instanceof’ works, and then finally feed this wrapped function to the parasite (via the Host variable). I won’t be able to write the code exactly as above, but we will get close… Anyway, hopefully this will make more sense very soon!

We will get there in two steps. First we code it up manually (so to speak) and secondly we will do the meta-programming with parasite.

Incidentally, we can exploit Crockford’s ‘shared secrets’ technique to get the prototype chain working. Consider the following code.

function Person(name, proto) {
  var p = object(proto || Person.prototype);
  p.getName = function() {return name;};
  return p;
}
Person.prototype = {
   sayHello: function(){
      return "Hello, my name is "+this.getName();
   },
   getName: function(){return 'Adam';}
};
Person('Karl') instanceof Person;//<-- true

The key here is the statement: object(proto || Person.prototype). This ensures that the object created has Person.prototype in its [[prototype]] chain. The proto ||-part is intended to be used as a ‘shared secret’ between a parasite ‘sub type’/'sub power constructor’; the invariant is that proto || Person.prototype will always denote an object which is Person.prototype or has it in its prototype chain. It can be used as follows:

function Guru(name,topic,proto) {
  var g = object(Person(name, proto || Guru.prototype));
  g.getTopic = function() {return topic;};
  return g;
}
Guru.prototype = object(Person.prototype);
Guru('Douglas Crockford','JavaScript') instanceof Guru;//<-- true
Guru('Douglas Crockford','JavaScript') instanceof Person;//<-- true

Actually, I feel some pleasure in the assignments:

Person.prototype = {
   sayHello: function(){
      return "Hello, my name is "+this.getName();
   },
   getName: function(){return 'Adam';}
};

and Guru.prototype = object(Person.prototype);: Intutively, these objects are the ‘prototypes’ of the objects created by the power constructors, so it feels natural to make this assignment.

So far so good – we have instanceof working with power-constructors, but we can do better. The problem now is to do the meta-programming that will handle the ‘secret-sharing’ behind the scenes.

In this example, we will enhance Object.prototype and Function.prototype. For simplicity I’ve introduced the constraint that all power-constructor functions should take only one argument [however, I'm convinced this can be generalized to more than one argument].

Note, slightly off topic here…
In either case, I’ve developed at taste for single-argument functions. Consider:

/** power-constructer for Guru objects 
  *@param conf {Object} a configuration object with properties:
  * name and topic
  * @return a Guru object with name: conf.name and topic: conf.topic.
  */
function Guru(conf){
  var g = object(Person(conf)),
      t = conf.topic || 'none';
  g.getTopic = function(){return t;};
  return g;
}
Guru({name:'Yoda', topic:'JavaScript'});

The disadvantage is that there is slightly more code to write. The advantages are (1) readability: Guru({name:’Yoda’, topic:’JavaScript’}) can be read without having to consult the constructor function about which argument is the name and which is the topic; (2) optional/default arguments: you can leave out either of the arguments: Guru({name:’Yoda’}) or Guru({topic:’JavaScript’}) are both valid (in the multiple arg constructor with name as the first parameter, you’d have to write Guru(‘Yoda’) (which is fine), and Guru(undefined,’JavaScript’) (which is not).

Back on track
The following is my implementation. I extend Object.prototype and Function.prototype so that we can write:

var OriginalPerson = {
   sayHello: function(){
      return "Hello, my name is "+this.getName();
   },
   getName: function(){return 'Adam';}
};

var Person = OriginalPerson.parasite(function(Host, conf) {
  var p = object(Host()),
      name = conf.name || 'Anonymous';
  p.getName = function() {return name;};
  return p;
});

var Guru = Person.parasite(function(Host, conf) {
  var g = object(Host(conf)),
      topic = conf.topic || 'none';
  g.getTopic = function() {return topic;}
  return g;
});
var h = Guru({
 name: 'Douglas Crockford',
 topic: 'JavaScript'
});
h instanceof Guru;//<-- true
h instanceof Person;//<-- true

I’ve implemented it as follows (with comments):

Object.prototype.parasite = function(parasite){
    /* This is the function returned as the result
       of this call; it represents a wrapper for the
       function in parameter parasite. wrapper will simply
       call the parasite function, but supplying a Host function
       as the first argument. If wrapper is called with proto === undefined
       then the Host function will create an object with its prototype === this,
       otherwise an object with prototype === proto is created (this lets
       sub-parasites supply the proto parameter).
    */
    function wrapper(conf,proto) {
        var p = proto;//Exercise why is this necessary?
        Array.prototype.splice.call(arguments,0,0,function(){
           return object(p || wrapper.prototype);
        });
        return parasite.apply(this, arguments);
    }
    /* it is important that wrapper.prototype is set to this object, both so that
       o instanceof wrapper works, and so that objects created with
       object(p || wrapper.prototype) above will inherit properties of this */
    wrapper.prototype = this;
    return wrapper;
};
Function.prototype.parasite = function(parasite) {
    var host_cons = this;//the constructor function for the host of parasite

    /* Again, this function is the result of the computation. 
       When called it splices a Host function on the 0'th pos in the arguments array.
       The Host function will call the host_cons and (important!) supplies an 
       additional last argument (proto). If proto === undefined we are in the case
       where client code is calling wrapper, so we call the host_cons function
       supplying wrapper.prototype; if instead proto is provided we call host_cons
       with this object (this is the case where wrapper is called by a sub-parasite).
    */
    function wrapper(conf,proto) {
         var wrapper_this = this,
             p = proto;//exercise: why?
         Array.prototype.splice.call(arguments,0,0, function() {
              Array.prototype.splice.call(arguments,arguments.length,0, 
                                              p || wrapper.prototype);
              return host_cons.apply(wrapper_this,arguments); 
         });
         return parasite.apply(this, arguments);
    }
    /* our prototype is an object which inherits properties from this.prototype,
       e.g., Guru.prototype inherits from Person.prototype.
    */
    wrapper.prototype = object(this.prototype);
    return wrapper;
};