To start with something that wasn’t too complex, I decided to try and create a parallel version of mapping a function for vectors, i.e., an eager function that would take a vector and a function as input and produce a mapped vector as output (instead of a seq). This would replace a pattern I’ve often used:

(into [] (map f vs))

with (vectormap f vs), which avoids the overhead of constructing a seq and the reconstructing a vector by conj’ing from the seq.

My initial goal was to produce a parallel version e.g., (pvectormap f vs). As Rich has pointed out: Clojure’s persistent data structures are excellent candidates for parallel processing using divide and conquer since they are trees which are already “sitting there divided!” Further immutability means no synchronization is needed. For example, remember that PersistentVector is a balanced 32-way tree consisting of size 32 arrays of Objects (Nodes in the tree or the actual values stored in the vector).

I decided to warm up by implementing vectormap, i.e., the serial version, first.

Remember this?

Apparently Rich Hickey presented this piece of code at JVM Lang summit:

static public Object ret1(Object ret, Object nil) {
    return ret;
}

public static int count(Object o){
    if(o instanceof Counted)
        return ((Counted) o).count();
    return countFrom(Util.ret1(o, o = null));
}

Has Rich gone mad? A two argument static method which simply returns the first argument?? When I first read Ola’s blog post I simply couldn’t figure out why he would use that code… However, when I was writing the vectormap code I was thinking: suppose the input vector is really large — in fact, so large that we don’t have memory enough to hold both the input and the output vector. Then vectormap would produce an OutOfMemoryError. But suppose the calling code didn’t actually need the input vector what if we would release references to the elements of the input vector as we construct the corresponding mapped elements in the output vector (but without destroying the input vector)? We would need to only keep references to Node vectors we hadn’t already processed, and then null out our local variables to those we had.

This was when I realized that this is exactly what Rich’s function can help with: when calling the countFrom function, he provides as argument the seq referenced by the local variable o via the ret1 function. The side effect of using ret1 is that since Java is strict, both argument expressions to ret1 are evaluated (left-to-right), and consequently o is null’ed out. No more holding on the the head I could use this to only hold on to the vector arrays I hadn’t processed.

For example:

private static Node mapNode(IFn f, Node node, int level) {
	if (node == null) {return null;}
	if (level == 0) {
		return new Node(null,mapArray(f, Util.ret1(node.array, node=null)));
	}
	Object[] newArr = new Object[node.array.length];
	System.arraycopy(node.array, 0, newArr, 0, node.array.length);
	node=null;
	level -= 5;
	for (int i=0;i<newArr.length;i++) {
		newArr[i] = mapNode(f,Util.ret1((Node) newArr[i], newArr[i]=null),level);
	}
	return new Node(null,newArr);
}

This code is available in my clj-ds project.

Parallelize with Fork/Join

The pvectormap function uses Fork/Join to parallelize the mapping. I’m not sure about the granularity of the tasks, but I decided that processing a size 32 array was too small a task, and went with processing 32 size 32 arrays instead.

Starting at the root array of nodes, the code simply forks 32 tasks which recursively process each child of the root. This forking continues recursively until we hit the second lowest level in the tree — this is processed sequentially using the mapNode function from above. This is implemented as a RecursiveTask in the Fork/Join framework.

static final class PMapTask extends RecursiveTask<Node> {

	private IFn f;
	private int shift;
	private Node node;

	public PMapTask(IFn f, int shift, Node node) {
		this.f = f;
		this.shift = shift;
		this.node = node;
	}

	public Node compute() {
		if (node == null) {
			return null;
		}
	   if (this.shift <= 5) {
		   return mapNode(f,node,shift);
	   }

	   PMapTask[] tasks = new PMapTask[node.array.length];
	   shift -= 5;
	   for (int i=0;i<tasks.length;i++) {
		   tasks[i] = new PMapTask(f,shift,(Node) node.array[i]);
	   }
	   invokeAll(tasks);
	   Node[] nodes = new Node[node.array.length];
	   try {
		   for (int i=0;i<tasks.length;i++) {
				nodes[i] = tasks[i].get();
		   }
		   return new Node(null,nodes);
	   } catch (InterruptedException e) {
			Thread.currentThread().interrupt();
			throw new RuntimeException(e);
		} catch (ExecutionException e) {
			throw new RuntimeException(e);
		}
   }
}

On my dual core system with a non trivial function f that actually does some work, using pvectormap is about twice as fast as vectormap: see PersistentVectorTest of clj-ds.

Next stop: optimize and add vectormap and pvectormap to Clojure core

A lot of interesting things have happened with data structures in JVM-land since then: Erjang uses Clojure’s data structures, it looks like Scala is porting PersistentVector, upcoming Seph is using them too. My clj-ds project should help in this regard: I’ve extracted the data structures of Clojure from its compiler for use with JVM-based languages (providing some extra stuff like reverse and “positioned” iterators). There are already people interested in using this in Java land.

Some people have asked for the “part ii” post, and my son just fell asleep, so … Most of what is described in the PersistentHashMap post is still true, however there have been optimizations and simplifications which I explain here. Note: before reading this post, you should read the previous post on PersistentHashMap.

First, some changes. Previously there were five implementations of the INode interface; this has changed and there are now only three implementations: ArrayNode, BitmapIndexedNode and HashCollisionNode. This means that: EmptyNode, LeafNode, FullNode are out, with ArrayNode replacing FullNode. An array node is an array where the entries are null or instances of INode, i.e., it stores other nodes but not any key-value pairs. An empty persistent hash map is simply a persistent hash map where the root node is null — this removes the need for EmptyNode. Finally, leaf nodes used to store the actual entries stored in the map. Leaf nodes are out, and instead BitmapIndexedNodes directly embed the map entries in their arrays.

The idea is the following: if, in the old implementation, a BitmapIndexedNode would store a leaf node at an index, then instead, it now embeds the key and value directly in its array. This array used to be of type INode[], storing only nodes, but is now a mixed object array for which the value can be one of: a map key, a map entry, null or an INode object.

The above drawing corresponds to the old structure. In the new structure all the white circles (the leaf nodes) are embedded directly in their parents.

There is an invariant:

for all the indices that are in the bitmap, the even indices store keys or null; the odd indices store values or INode objects. Key-value pairs are layed out in sequence, so that if index 2*i is a key then index 2*i + 1 is a value. If an index 2*i is null then 2*i+1 must be a non-null INode object.

Note: if you don’t remember how bitpos, bitmap and index work see the part i post.

The strategy used in BitmapIndexedNode is that it can store up to 16 entries in the bit map: if the size grows above 16, it is converted to a FullNode.

Assoc. The assoc method creates a persistent hash map which is like the current one, except that it additionally stores another map entry. As before assoc works using path copying: all that is changed in the new map is the path from the root node to the newly added map entry.

Beware: the following drawing looks confusing: take the time and read the explanation. It is showing two persistent hash maps, where one is obtained by assoc’ing to the first. Again this is a modified version of one of Rich’s slides.

The first map is rooted on the left (where the left-most box points to). The second is rooted where the right-most box points to. The nodes are grouped in three, each indicated by a colored circle that surrounds it. The new node is in the right-most, lower corner.

Purple is the path in the old tree to where the new map entry would be;
green is the new nodes that are created in the new hash map.
The dashed lines indicate that the nodes in the new tree share children with nodes in the old tree. The red circles show all the nodes that are shared in the tree: note that this is most of the nodes.

So how much work needs to be done to create the new tree? Suppose we are storing key k with value v. Using recursion, descend down the original tree as if looking for the key k. The key k isn’t found. In the worst case this takes time: O(log₃₂ n), in practice it we could stop at any level in the tree so 2-3 steps would be common (remember from the old post that the work done at each level is constant and fast). At the bottom we create a new BitmapIndexedNode and store the new map entry in it. If the bottom node in the old tree was a BitmapIndexedNode with less than 16 elements the new BitmapIndexedNode is a copy of the old one, except that the new map entry is added. This step takes constant time since the array to copy at the bitmap indexed node always has less than 32 elements (because we store at most 16 map entries as: key, value, key, value, …). If the bottom node was an ArrayNode we simply copy the array node, and make the new BitmapIndexedNode a size-one child of this array node (still constant time).

On the drawing above, we have now recursively descended the purple path in the left/old tree, and we have created the new node which is a bitmap indexed node. What remains is to establish the path to this new node, in the new tree: this path is almost identical to the “purple” path in the old tree: the only difference is that we have created a new bitmap indexed node. The parent of the new node must of course reference it, so that is copied and modified to get a reference to the new node. This means that we must also copy the grandparent of the new node, modifying it to reference the parent. And so on… This copying takes place on our way “up” through the recursion, i.e., after the recursive calls complete at each node level.

Let’s decompose the code. We only look at assoc for BitmapIndexedNode as it is the most interesting.

public INode assoc(int shift, int hash, Object key, Object val, Box addedLeaf){
		int bit = bitpos(hash, shift);
		int idx = index(bit);
		if((bitmap & bit) != 0) {
			//..
		} else {
			//
		}
}

This should be familiar from the previous post: we check if the index of the hash (at the current level) is in the bitmap. The else branch is the case where the index is not in the bitmap, corresponding to reaching the bottom of the path: here we must create a new node and path-copy as described above. In the “if”-branch we simply either call recursively if the index references an INode: if it references a key-value pair this is a “replace” and we do path copying here too. Again we look only at the else-branch as it is the most interesting.

public INode assoc(int shift, int hash, Object key, Object val, Box addedLeaf){
		int bit = bitpos(hash, shift);
		int idx = index(bit);
		if((bitmap & bit) != 0) {
			//..
		} else {
			int n = Integer.bitCount(bitmap);
			if(n >= 16) {
				//convert to ArrayNode
			} else {
				Object[] newArray = new Object[2*(n+1)];
				System.arraycopy(array, 0, newArray, 0, 2*idx);
				newArray[2*idx] = key;
				addedLeaf.val = addedLeaf; 
				newArray[2*idx+1] = val;
				System.arraycopy(array, 2*idx, newArray, 2*(idx+1), 2*(n-idx));
				return new BitmapIndexedNode(null, bitmap | bit, newArray);
			}
		}
}

Remember (from the previous post) that Integer.bitCount(bitmap) counts the number of children of this node. If we are storing less than 16 elements, we have room for one more: To create the new bitmap indexed node simply copy the object array, and modify it to store the new key-value pair (using the invariant mentioned above). (Ignore the “box” part, it is used to communicate to higher-levels in the recursion what happened.) Finally, if we have 16 elements stored already, convert to an ArrayNode:

public INode assoc(int shift, int hash, Object key, Object val, Box addedLeaf){
		int bit = bitpos(hash, shift);
		int idx = index(bit);
		if((bitmap & bit) != 0) {
			//..
		} else {
			int n = Integer.bitCount(bitmap);
			if(n >= 16) {
				INode[] nodes = new INode[32];
				int jdx = mask(hash, shift);
				nodes[jdx] = EMPTY.assoc(shift + 5, hash, key, val, addedLeaf);  
				int j = 0;
				for(int i = 0; i < 32; i++)
					if(((bitmap >>> i) & 1) != 0) {
						if (array[j] == null)
							nodes[i] = (INode) array[j+1];
						else
							nodes[i] = EMPTY.assoc(shift + 5,  Util.hash(array[j]), array[j], array[j+1], addedLeaf);
						j += 2;
					}
				return new ArrayNode(null, n + 1, nodes);
			} else {
				//we covered this...
			}
		}
}

The first block up to the ‘for’ loop prepares the array of INodes that will be the children array of the new ArrayNode. We create a new BitmapIndexedNode easily by calling assoc on an empty persistent hash map (which is an easy way of creating an one-entry BitmapIndexedNode). Note that the indexing strategy for ArrayNodes is different from BitmapIndexedNode: we don’t need the bitmap We simply use the 5-bit block corresponding to the level (again see the older post).

Finally, the for-loop copies all the other nodes that are stored in this bitmap indexed node into the new array node. This entails mapping between the bitmap index scheme and the “5-bit hash-block” scheme. For each possible index in the bitmap node, only do work if the index is present in the bit map: if(((bitmap >>> i) & 1) != 0). The variable j runs through the even indexes, and according to the invariant: null means that j+1 is an INode, and non-null means it is a value in a map entry.

I am wondering if there is an optimization possible here? We are looping through all possible indices of the new ArrayNode, but we know that we only have to do something on the indices that correspond to an non-zero index in the bit map; and there can be only 15 of those… Would it be possible to iterate only those using some strategy? If I figure something out, I’ll let you know

There are many advantages to programming with persistent data structures (which implies immutability) but that isn’t the topic of this post… Currently the Clojure data structures are implemented in Java, so in principle they should be usable also outside of Clojure, say from Java. However, in practice it is inconvenient (see below).

I’ve created the project clj-ds to make Clojure’s data structures available in a more practical form to other JVM languages than Clojure. The README file from the clj-ds GitHub project explains the motivation:

Advantages of clj-ds when constrained to working with Java (as opposed to just including clojure.jar)

* Currently the Clojure data structures are implemented in Java. In the future,
all of Clojure will be implemented in Clojure itself (known as “Clojure-in-Clojure”).
This has many advantages for Clojure, but when it happens the data structures will
probably be even more intertwined with the rest of the language,
and may be even more inconvenient to use in a Java context.

The clj-ds project will maintain Java versions of the code, and where possible attempt
to “port” improvements made in the Clojure versions back into clj-ds. Thus keeping maintained
versions of the Java data structures.

* In the current Clojure version, calling certain methods on PersistentHashMap requires
loading the entire Clojure runtime, including the bootstrap process. This takes about one second.
This means that the first time one of these methods is called, a Java user will experience a
slight delay (and a memory-usage increase). Further, many of the Clojure runtime
Java classes are not needed when only support for persistent data structures
is wanted (e.g., the compiler).

* The clj-ds library is not dependent on the Clojure runtime nor does it run any
Clojure bootstrap process, e.g., the classes that deal with compilation have been removed.
This results in a smaller library, and the mentioned delay does not occur.

* Clojure is a dynamically typed language. Java is statically typed, and supports
‘generics’ from version 5. A Java user would expect generics support from a Java
data structure library, and the Clojure version doesn’t have this.
clj-ds will support generics.

* Finally, a slight improvement. Certain of the Clojure data structure methods use Clojure’s ‘seq’ abstraction. In the implementation of the Java ‘iterator’ pattern. It is possible, to make
slightly more efficient iterators using a tailor made iterator. clj-ds does this.

Code: http://github.com/krukow/clj-ds

Michael Nygards stability pattern “Circuit Breaker” is useful for failing fast when calling integration points that are unstable (which is every integration point I’ve ever dealt with!). This is done by detecting when integration points fail, and subsequently cutting off access for a time-period. Use the circuit breaker to

… preserve request handling threads in the calling system. Very often, when you make a call to an external integration point that’s broken, it will tie up a thread in a blocking synchronous call for an indefinite period of time. [Michael Nygard, QCon interview]

I’ve written a fast, non-blocking functional implementation of the Circuit Breaker in the 1.2 branch of Clojure which will be released shortly. It uses the new Clojure constructs protocols and datatypes for modeling states, for interop and to obtain platform-speed polymorphic calls.

The implementation exposes a simple Java interface which makes it usable from Java, Scala, JRuby et al.

Why?

Design Patterns are a disease, and Clojure is the cure. (the smiley is mine!)
[http://www.nofluffjuststuff.com/blog/stuart_halloway/2009/10/the_case_for_clojure]

I believe this implementation has a couple of advantages compared to these Java and Scala implementations that use the GoF “State Machine” pattern.

First, I find the functional version simpler (see examples below). Second, this version guarantees that only a single call is made to the integration point when the circuit breaker decides to re-test if it is working again (the other versions seem to allow an unbounded number of calls if more threads concurrently try to access the integration point calling “invoke”). Finally, this version encapsulates the (immutable) state in a single Clojure atom (corresponding to a Java AtomicReference), whereas the GoF implementations use at least three atomics for various counters. Why does that matter? Well, it gives you the ability to obtain a consistent (immutable) snapshot of the state of circuit breaker at any given time which can be used to e.g. logging and analysis – this isn’t possible when you have several atomics in play.

These benefits come naturally from following Clojure’s programming model and concurrency constructs. Let me illustrate the Clojure features that I’ve found useful for this problem.

Protocols and records.
I’m using pure polymorphic functions on-success, on-error, on-before-call as transition functions mapping a state to the next state for an event (successfull call, error call and before a call is initiated). A pure function proceed is a predicate on states that decide whether or not the state allows calls to go though to the integration point.

Together these functions form a Clojure protocol (which is similar to a Java interface, but has additional benefits).

(Show with JavaScript: for non-JS User agents, see http://gist.github.com/390485)

Apart from the definition of the protocol, we define a default implementation of the protocol functions that our states can use. The default transition functions are simply the identity function and proceed defaults to false.

We now define datatypes corresponding to each type of state: closed (calls go through, count failures), open (calls fail-fast, stores a time-stamp when IP failed), initial-half-open (a single call goes through), pending-half-open (waiting for a probing call to return). The datatypes are parameterized by a “transition policy” defining how many failures are “needed” to transition to the open state, and how long to wait in the open state.

(For non-JS User agents, see http://gist.github.com/390492 )

This simply defines the states as datatypes. Note that we use defrecord not deftype. This makes our datatypes work like persistent clojure maps which is extremely useful – for example our states can be destructured (see, e.g. defrecord at http://clojure.org/datatypes).

We can make our new types participate in our CircuitBreakerTransitions protocol

(http://gist.github.com/390494)

A couple of notes:

- We use merge to take the default implementations and “override” with the implementations given (this would correspond to an abstract super-class in Java but is more flexible).
- We use destructing in the function definitions for easy access to the “ClosedState” data, e.g., in the body of (fn [{f :fail-count p :policy, :as s}] ... the states fail-count is available as f and similarly for the policy. The :as s clause makes the state itself available as s.
- Finally, we can construct new instances of the states since they are simple dynamically compiled classes, e.g., (ClosedState. p 0) creates the initial state with a policy p.

Pure functions. A clear advantage of the functional approach is the ease of testing. Consider this simple test of some states and transition functions.

(http://gist.github.com/390505)

This is the core of the circuit breaker itself:

(http://gist.github.com/390529)

A circuit breaker is simply an atomic reference to a state. The function wrap-with takes a function, f to wrap – this represents a function that will call an integration point, and a circuit breaker, named state. It then returns a “wrapped” function which is guarded by the circuit breaker. It uses the “transition-by!” function which makes a state-transition from the current state.

An example usage:

(http://gist.github.com/390535)

Notice that the snapshot of the state is available simply with a deref, e.g., as @cb.

A Java-interface

To expose the functionality to Java I have used Clojures gen-class to create a class that exposes two methods given by this protocol which lets you wrap a function and look at the state:


(defprotocol CircuitBreaker
(#^clojure.lang.IFn wrap [this #^clojure.lang.IFn fn])
(#^net.higher_order.integration.circuit_breaker.states.CircuitBreakerTransitions current-state [this]))


Then using gen-class I generate a class that implements the interface corresponding to that protocol. This gives the possibility of using the circuit breaker from Java:

import clojure.lang.IFn;
import clojure.lang.RT;
import net.higher_order.integration.circuit_breaker.AtomicCircuitBreaker;
import net.higher_order.integration.circuit_breaker.CircuitBreaker;

public class C {
	public static void main(String[] args) {
		CircuitBreaker atomicCircuitBreaker = new AtomicCircuitBreaker();
		IFn wrap = (IFn) atomicCircuitBreaker.wrap(new clojure.lang.AFn() {
			public Object invoke(Object arg0) throws Exception {
				if (arg0 == null) throw new IllegalArgumentException("null arg");
				System.out.println("Invoked with: "+arg0);
				return arg0;
			}
		});
		succeed(atomicCircuitBreaker, wrap);
		fail(atomicCircuitBreaker, wrap);
		fail(atomicCircuitBreaker, wrap);
		fail(atomicCircuitBreaker, wrap);
		fail(atomicCircuitBreaker, wrap);
		fail(atomicCircuitBreaker, wrap);
		fail(atomicCircuitBreaker, wrap);
		sleep(1000);
		status(atomicCircuitBreaker);
		fail(atomicCircuitBreaker, wrap);
		sleep(5000);
		succeed(atomicCircuitBreaker, wrap);
	}

	
	private static void sleep(long howlong) {
		try {
			Thread.sleep(howlong);
		} catch (InterruptedException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	private static void succeed(CircuitBreaker atomicCircuitBreaker, IFn wrap) {
		try {
			System.out.println(wrap.invoke("KARL"));
			System.out.println(wrap.invoke(42));
		} catch (Exception e) {
			System.out.println(e.getMessage());
		} finally {
			status(atomicCircuitBreaker);
		}
	}

	private static void status(CircuitBreaker atomicCircuitBreaker) {
		System.out.println(RT.printString(atomicCircuitBreaker.current_state()));
	}

	private static void fail(CircuitBreaker atomicCircuitBreaker, IFn wrap) {
		try {
			System.out.println(wrap.invoke(null));
			System.out.println(wrap.invoke(42));
		} catch (Exception e) {
			System.out.println(e.getMessage());
		} finally {
			status(atomicCircuitBreaker);
		}
	}
}

This is getting long. I’ll save the comparison for the next post

Github: http://github.com/krukow/clojure-circuit-breaker

As an appetizer, the first chapter is available for free:

Clojure—A Lisp for the Java Virtual Machine

I’ve read the first chapter and the book looks very promising! To quote the last paragraph of chapter one:

We’ve talked a little about how this book will go beyond what Clojure is to why it’s designed the way it is and how that design can be exploited through idioms that will help you think in Clojure. So lets stop talking about what this book will do and get on with the doing.
Fasten your seat belts.

I’ve fastened my seat belt and ordered my copy

In a previous post, I gave a high-level description of how Clojure’s PersistentVector is implemented. While the code has changed, the description was high-level enough that the explanations still hold (although some code snipplets don’t correspond to what’s in Git master.)

In this post, I’ll try to explain (also at a high level) how clojure.lang.PersistentHashMap works internally. Reading the mentioned post on PersistentVector is helpful as some of the concepts are the same (e.g., bit-partitioning).

Persistent
PersistentHashMap is a persistent version of the classical hash table data structure. Persistent means that the data structure is immutable, yet has efficient non-destructive operations that correspond to the operations on the classical hash table. E.g., put(K,V) in hash table corresponds to a side-effect free function assoc(P, K, V) which computes from P a new PersistentHashMap P’ which is like P except that it maps key K to value V. The word “efficient” means “on par” with their mutating counterparts. For Clojure data structures, Rich tries to make them within 1-4 of the Java data structure operations; and read-only operations can even be faster than Java’s. Later I will cover ‘transients’ which are a new optimization that make “batch” operations faster.

Array-mapped hash trie
In his paper Ideal Hash Tries Phil Bagwell describes a data structure “Hash Array Mapped Trie” which is an efficient implementation of a Hash Tree, based on a combination of hashing and the trie data structure. Hash Array Mapped Tries, are not persistent or immutable. What Rich did was create a persistent version of Bagwell’s data structure; clojure.lang.PersistentHashMap.

PersistentHashMap basic idea
PersistentHashMap (PHM) maintains a very-wide tree, each node having up to 32 children. Each node is a concrete implementation of a static inner interface, INode, and there are five implementations of this interface: EmptyNode, LeafNode, FullNode, HashCollisionNode, BitmapIndexedNode. I’ll only cover EmptyNode, LeafNode and BitmapIndexedNode; the latter being where most of the interesting stuff happens.

The INode interface look like this:

static interface INode{
    INode assoc(int shift, int hash, Object key, Object val, Box addedLeaf);
    LeafNode find(int hash, Object key);
    //I've left out a few methods
}

The assoc method “adds” a new key-value pair to the map. The find method searches for the Leaf-node holding a key.

An EmptyNode simply represents the empty hash map. LeafNodes are also pretty simple; they hold the actual entries stored in map.

The root node of the tree is initially an EmptyNode. When assoc is called on EmptyNode, it returns a new LeafNode, which the key-value pair. So EmptyNode “becomes” a LeafNode with assoc. In turn, a LeafNode typically “becomes” a BitmapIndexedNode with assoc. We will go into details with BitmapIndexedNode, but first we need to understand…

Bit-partitioning of hash-codes
When PHM assocs a key object K with value object V, it first computes the hashCode of K, just as a hash-table would. The hash code of K yields an int, which has a 32-bit representation in Java (as I explained in the post on PersistentVector). Here are some example bit representations of numbers:

The trick that PHM uses is to partition this bit representation in to blocks of 5-bits, represented with colors in the above example. Each block corresponds to a “level” in the tree structure; for example, the right-most green block corresponds to root-level, and the orange block corresponds to the children of the root. Exactly what “corresponds” means is described below. Levels are multiples of 5. I.e., the root level is level 0, the children of the root are level 5, the grand-children of the root are level 10, etc. Note that a block of five bits corresponds to a number in the range 0-31.

The reason that levels are multiples of five is the following: You have a bit-representation of a hash-code and you are interested in a particular block corresponding to a level n. You obtain this number in two steps: first move the block of bits to the right, until it is the right-most block. Then null-out all other bits except this right-most block. The is done with two bit-operations: you simply right-shift the bits with the level n and then do a bit-wise ‘and’ (&) with the pattern 00..11111. For example, suppose you want the block corresponding to level 5 (the orange block) of the number 1258 (binary: [00001][00111][01010]). You right shift with the level, 5, which is [00000][00001][00111]; then do the null’ing, yielding [00111], which was exactly the orange block of 1258.

The following function does this.

static int mask(int hash, int shift){
	return (hash >>> shift) & 0x01f;
}

Illustrating the tree structure
I’ll use the following picture (adapted from one of Rich’s slides).

The colored nodes are BitmapIndexedNodes and have between 2 and 31 children (should they get a 32nd child, they become FullNodes). A naive implementation of BitmapIndexedNode might be the following: use an int variable, level, to denote the level that this node lives in, and allocate a full 32 element array of INode references for the children. To add a new child: lookup the index via the bit-block corresponding to the level, i.e. given a hashCode hash for the child, and given the level, call mask(hash, level) to get the index in range [0, 31]. But this strategy wastes a lot of memory: each node has a full 32 element array where most entries are simply null, i.e., if there are 4 children there are 28 null references which are just wasting space.

The hard part is to only use as much space as is needed for each BitmapIndexedNode, i.e., if a BitmapIndexedNode has N children it maintains an array of size N. But then we can’t use mask(hash,shift) as the index into the array since it returns a number in the range [0,31] and we need a number only in range [0, N).

bitpos
So we need a function to map numbers in range [0, 31] to indexes in range [0, N). The function has to be fast constant time, since we are using it to find the child of a node from a hash code, which we will do at each level in the tree. The function is a composition of two functions: bitpos and index. Function bitpos maps numbers [0, 31] to powers of two, i.e., numbers that have a binary representation of the form:

static int bitpos(int hash, int shift){
    return 1 << mask(hash, shift);
}

bitmap
Each BitmapIndexedNode also maintains an int variable bitmap which we also look at in binary form. The bitmap tells us how many children this node has, and also what their indexes are in the child array. All this is encoded into one int variable! How? The bit-map has a binary representation, e.g.,

Suppose we are at level 5, and looking up an element with hash-code 1258. Suppose also the bitmap-indexed node has four children, with bitmap
bitmap =_{binary rep} 00000000000000010000000010000101

Now check if there is a child for hashCode 1258 at this level:

mask(1258,5) = 7 (i.e., binary 00111 as we saw before).

bitpos(7) =_{binary rep} 10000000

00000000000000010000000010000101 &
00000000000000000000000010000000 = 1

Which means that the child exists. Now what is its index? This is where the index function comes into play…

index
This is the final piece of bit-trickery, I promise The index of a child is the number of 1‘s to the right of the child’s bitpos in the bit map. In our example above, the index corresponding to hash code 1258 would be 2, since there are two 1‘s to the right of 10000000 in the bitmap. Now the trick is that on many processors there is an efficient instruction called CTPOP which counts the number of ones in the bit representation of an integer (CTPOP is “count (bit) population”). Note that if we subtract 1 from the bitpos, 10ⁿ, we get 01ⁿ, then binary ‘and’ with the bit map gives us the same bit map, but where only the 1‘s to the right of bitpos are present. If we do a CTPOP on this, we get the index. Hence,

final int index(int bit){
    return Integer.bitCount(bitmap & (bit - 1));
}

To be continued…
From this you should be able to understand how find works. In combines all this:

final static class BitmapIndexedNode implements INode{
    final int bitmap;
    final INode[] nodes;
    final int shift;
    //some stuff left out
    static int bitpos(int hash, int shift){
	return 1 << mask(hash, shift);
    }
    
    final int index(int bit){
	return Integer.bitCount(bitmap & (bit - 1));
    }
    //...some methods left out
    
   public LeafNode find(int hash, Object key){
       int bit = bitpos(hash, shift);
       if((bitmap & bit) != 0)
	   {
	       return nodes[index(bit)].find(hash, key);
	   }
       else
	   return null;
   }
}

In part 2 we look at how assoc works…

• Monday, Sept. 7th in Aarhus, location: Trifork. Registration: https://secure.trifork.com/aarhus-2009/freeevent/register.m?eventOID=2129
• Wednesday, Sept. 9th in Copenhagen, location: Trifork Cph. Registration: https://secure.trifork.com/aarhus-2009/freeevent/register.m?eventOID=2130.
• Free Clojure workshop at JAOO – featuring Rich Hickey. October 6, 2009, 17.30 – 19.30. Registration: https://secure.trifork.com/aarhus-2009/freeevent/register.m?eventOID=2093

Abstract for Aarhus/Cph talks.
Clojure is..

… a new functional, dynamic programming language for Java Virtual Machines. The primary novelty of Clojure is its strong focus on and support for in-process concurrency: a unique concurrency model, combining a notion of persistent (i.e., immutable, fast) data structures, with a lock-free concurrency model. This simplifies concurrent programming greatly and has good scalability properties.
Influenced by LISP and Haskell, Clojure supports pure, lazy functional programming and has a powerful macro system which makes extending the language to support DSLs easy and powerful.

This talk..

… is split in three parts. In the first part, Clojure is introduced for those who don’t know the language. There is so much to cover that this will be a fast tour with pointers to more information, but we will emphasize the unique aspects of the language.

In the second part we go into more depth regarding the implementation of persistent (and transient data structures) – “the secret sauce of Clojure”

In the third part we get to see Clojure in action running on some very cool technology – a unique opportunity! Azul Systems (www.azulsystems.com) has promised to make available one of their large Vega 3 compute appliances (864 core, 368 GB memory, let’s go concurrent). We will explore how the Clojure concurrency model fares in practice, scaling a demo of a parallel Traveling Sales Problem algorithm. We will also push the implementation to its limits in a high-contention demo. Great fun!

]]> http://blog.higher-order.net/2009/08/17/clojure-talks-in-copenhagen-and-aarhus-now-with-azul-systems-clojure-demo/feed/ 0 http://blog.higher-order.net/2009/07/13/jaoo-discount/ http://blog.higher-order.net/2009/07/13/jaoo-discount/#comments Mon, 13 Jul 2009 05:25:56 +0000 admin http://blog.higher-order.net/?p=331 Continue reading →]]> I suggested to the JAOO program committee that the JAOO Aarhus 2009 conference should have a concurrency track. Their reply was “that’s a good idea – you are hosting it!” – this is how Trifork works

The good thing is that the track host gets to pick (or at least propose) speakers for the track (the bad thing is that it entails some work!). Given my recent interest in Clojure and since he is such a great speaker, I immediately suggested we invite Rich Hickey. I’ll write a bit more on the program in an upcoming post, but right now I just want to mention that via the Danish Clojure Users’ Group I am now a JAOO affiliate, and anyone signing up via dcug gets a 15% discount on JAOO tickets.

To sign up, follow this procedure:

1. To join dcug, simply register as a dcug user at http://www.clojure.dk.
2. Read about the discount here: http://clojure.higher-order.net/?p=32

As an appetizer, check out this post at dcug.

#version 1 - 'functional'
cur_period = Time.now.month - 1
RecurringActivation.find(:all, :select => "product_id, period").select { |r|
  cur_period % r.period == 0
}.map &:product_id

#version 2 - 'imperative'
cur_period = Time.now.month - 1
result = []
RecurringActivation.find(:all, :select => "product_id, period").each { |r|
  result << r.product_id  if cur_period % r.period == 0
}

In the first version I compose functions (methods) ‘select’ and ‘map’: the blocks have no side-effects, and this is why I call it ‘functional.’ This is shorter and more clear. Of course, this is implemented by the library using iteration and imperative assignment, but at least my code feels functional.

In the second version I use ‘each’ with a block that does both filtering and ‘mapping’ in one step: I add the product_id explicitly to the products array for each object which satisfies my criterion.

Notice that the first version first produces a complete filtered list which is then passed on to the map method which produces a complete mapped and filtered list: the list is iterated twice, and an intermediate array containing only the filtered objects exists in memory and is later collected as garbage. In the second version the list is only iterated once, and such an intermediate array of filtered objects is never allocated. Hence one might choose the latter for performance reasons.

Now consider the same in Clojure or any lazy functional language. We might have (no active record here):

krukow:~/Projects/private/okooko-prod/tmp$ cl
Clojure
user=> (def recurring_activations '({:product_id 1 :period 2}
                                    {:product_id 2 :period 3}
				    {:product_id 3 :period 2}))
#'user/recurring_activations
user=> (def res
         (map #(do (println "map") (:product_id %))
             (filter #(do (println "filter") (= 0 (mod (:period %) 2)))
                 recurring_activations)))
#'user/res
user=>

Nothing has been mapped or filtered yet since Clojure is fully lazy. So res is now a lazy seq which will compute exactly what is needed on demand. Now think about what will happen if I just peek at the first element of res… In a non-lazy language, already the entire recurring_activations list would have been filtered (printing “filter” three times), then that result would have been mapped (printing “map” twice) and finally the first element would be looked up. So the output would be
“filter”
“filter”
“filter”
“map”
“map”
(return first element of list).

What happens in Clojure? It prints “filter” “map” and gives the first element:

user=> (first res)
filter
map
1
user=>

In effect we don’t need an intermediate list containing the filtered elements which is then garbage collected. Notice also that if we just look at the entire list (recomputing res first) we get:

user=> res
(filter
map
filter
filter
map
1 3)
user=>

So the side effects are evaluated in a different order than had we realized the entire filtered list first. This can be counter intuitive when one is used to eager languages (which all mainstream languages are); however, once understood laziness can be extremely powerful and elegant.

In effect, in lazy languages we can compose functions that work on entire sequences, e.g., map and filter, to obtain a lazy sequence which evaluates the all of the composed functions on each element in sequence.

If that sounded abstract and poorly phrased, I can try to say it more concretely: In Ruby (or Java) I would need to write a new function: filter_and_map taking two “blocks”/”procs” and a list, then using e.g. “each” on the list and apply first the filter “proc” then the map “proc” — the existing functions “map” and “select” can’t help me. In Clojure (or Haskell) I can simply compose the existing library functions filter and map to obtain the same thing:

(def filter_map
	(comp (partial map :product_id)
	          (partial filter #(= 0 (mod (:period %) 2)))))

or just use it inline

user=> (def res
	(map :product_id
	 (filter #(= 0 (mod (:period %) 2)) recurring_activations)))
#'user/res
user=> res
(1 3)
user=>

Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.

I recently watched an InfoQ interview with Greg Young where he discusses an interesting architecture that uses event stream processing. I thought their architecture sounded very interesting, but I did not go into ‘research mode’ to dig into it. Later I came by Jonas Bonér’s blog on “Real world scala” and saw an example of event sourcing using actors in Scala. I thought it would be a good exercise to implement a similar example in Clojure, and perhaps it would also be interesting to compare the solutions and see the effect of the languages on the implementation.

As Clojure focuses on concurrency it was natural to start thinking about what ‘concurrent’ event sourcing would mean and how to implement it.

Sequential Clojure implementation. For context, read Fowler’s article on event sourcing, and for comparison Jonas Bonér’s Scala implementation.

Modeling. Scala uses classes and actors for modeling ships and their state. A ship is an actor that receives events (messages) and reacts to those by updating its internal state, i.e., its location (e.g., departure event, arrival event).

Clojure has agents which are similar to actors. Agents are simpler and there are more functions that act on agents than on Actors; for example, in Clojure it is possible to directly read the state of an agent (using (deref x) or @x) — this is not possible with actors since they are distributed (at least programmed as if they were distributed).

I’ve used two additional Clojure features in implementing event sourcing: watchers and meta-data. Watchers are a form of callback attached to agents. The callback is called synchronously with the agent actions and “derefs of the agent in the callback will see the value set during that action.” I am using watchers to record the events that are sent to each ship. This decouples the code dealing with event storage from the state-changing functions sent to agents. The state of an agent is the location of the ship it represents. So we can get the location of a ship at any time simply by deref’ing the agent representing the ship (with actors this is more complex as one has to send a message to the actor and receive a message with the answer). I decided to use meta data attached to the location to represent the event that caused a move to that location. For example

user> @(first agents)
{:country "At sea", :city "At sea"}
user> ^@(first agents)
{:agent 0, :time #<Date Sun Feb 22 19:49:59 CET 2009>, :type :depart_for,
:loc {:country "Sweden", :city "Malmö"}}
user>

In general, I think it is possible to implement sequential event sourcing reasonably elegantly in Clojure using agents, watchers and metadata. The agents store the state that can change, the watchers record the events that cause the changes by looking at state meta data. The invariant should be that if an agent moves from state s1 to state s2 then (meta s2) should store the event that caused a transition from the s1 to state s2. Then if one knows the initial state, and all the events it is possible to reconstruct the entire sequence of states.

Our watchers are quite simple: (notice the cool syntax #(meta @%). A shame I couldn’t write #(^@%) it looks like I’m swearing! that would be cool)

;;Record event history
(def events (atom (map #(meta @%) agents))) 

(defn event_logger
  [idx a changed]
  (if changed
    (swap! events conj (meta @a))))

(defn add-watchers []
  "associate a watcher with each agent
   the watcher logs the events for that agent"
  (dotimes [i NUM_SHIPS]
    (add-watch (agents i) i event_logger)))

Concurrent event sourcing A natural question to ask for Clojure is “what if there are multiple concurrent event sources?” Is it still possible to do event sourcing? I don’t think so; at least not is a manner as powerful as with sequential events. Since Fowler defines event sourcing as ensuring “(…) that all changes to application state are stored as a sequence of events.” We already have a problem: when events can occur concurrently one must serialize them in order to store them as a sequence. One could try and timestamp all events and store them as a set, but this is sensitive to timing an scheduler issues. Even if stored as a set of timestamped events, how would one replay these events in a way that ensures that the global program states are the same in the replayed program? This would be dependent on thread scheduler timings.

It is of course possible to do the less advanced use cases of event sourcing, e.g. event querying and analysis (I guess that is concurrent event stream processing).

Anyway, if anyone is still reading the Source file link.