This post describes some mistakes I’ve made in the Clojure circuit breaker project, and how I managed to fix them, mostly due to guidance from Christophe.

The following is an (embarrassingly long) list of issues that my previous version of circuit breaker had. I think others can learn from my mistakes.

Platform speed protocol functions. In order for a protocol function reaching a datatype to run at platform call-speed, the datatype must inline the protocol implementation. In the old version, I extended the state transitions protocol to the circuit breaker states using (extend …): this is usually 3-4x times slower than platform-speed method calls (but can be much slower when cache misses). I now inline the protocol extension to the states.

This inlining gave a new problem. I had two data types, say A and B, where A creates an instance of B and B creates an instance of A in the implementation of the inlined protocol functions. You can’t have this type of “mutually recursive data types” since they are compiled on the fly to Java classes. The fix was to…

Create factory functions for your data types. There are at least two reasons for this. First a factory function can do processing of its arguments (e.g. pre condition), say for validation. The data type constructors can’t have any logic. Second, as noted by Alex Miller on his blog: to use data types defined in another namespace you must import them. But you can use require/use to bring the factory functions into your namespace.

I thought I knew: protocols aren’t interfaces (and Java interop).. I used a protocol: CircuitBreakerTransitions to model the states. The protocol contains functions needed for state transition on various events. I extended the protocol to my states, e.g.,

(defrecord ClosedState [policy fail-count]
  CircuitBreakerTransitions
  ...)

It inlines the protocol so the generated class implements the protocol’s backing interface: It’s fast and works great.

The problem appeared when I wanted to expose a Java API for the circuit breaker: one of the API methods getCurrentState returns the current state – now, the question is what is its signature? My first attempt was public CircuitBreakerTransitions getCurrentState(). Now what is wrong?

The problem is that my protocol can be extended to reach other types. Those types may or may not inline the protocol functions, and hence it is possible for the protocol to reach a type which doesn’t implement the CircuitBreakerTransitions interface… So should I just return “Object” – that isn’t really useful.

I haven’t implemented a solution yet, but this is what I am thinking. Keep the type: CircuitBreakerTransitions getCurrentState(), but in my Java-interop (gen-class), the function that implements getCurrentState should grab the state s from the circuit breaker. Now I know that (satisfies? CircuitBreakerTransitions s) is true, but (instance? (:on-interface CircuitBreakerTransitions) s) may be false. If it is true, I simply return it. Otherwise: reify CircuitBreakerTransitions by delegating to the protocol functions on s. This may sound a bit wierd at first, but I think of it like creating a new object that satisfies the interface by dynamically dispatching to s.

Type-hits are for removing reflection, not enforcing type safety My first iteration tried to type-hint the records, e.g.,

(defrecord ClosedState [^TransitionPolicy policy ^int fail-count])

This is just plain terrible First, those hints are ignored (but may be used in later versions of Clojure). Second, int isn’t suported in the current version of Clojure. Third, TransitionPolicy is a protocol – and, yes, protocol’s aren’t interfaces (see above): problem being again if I supply an object that satisfies the protocol but doesn’t implement the backing interface. Ugh.
The reason I put the hints wasn’t to remove reflection, it was for the generated constructor to have types (which is a valid thing to want, but unsupported now). But it is the wrong approach for the reasons above.

The right thing is to remove hints and expose a factory function which checks that the first argument satisfies the TransitionPolicy protocol and that the second argument is non-negative.

Premature optimization. My circuit breaker implementation tried to optimize a corner-case for the swap! function: suppose you (swap! state f) and f applied to the current state is identically the same state. Then there is no reason to do a compare-and-set!. My transition-by! function implemented this optimization. However, this was a premature optimization that changes the semantics of time (wow, Clojure really is powerful ). The fundamental question is this: [s] -f-> [s] (where s is a state and f is a side-effect free function that advances time), is this transition really advancing time? (it isn’t observable, is it?…)

Anyway, I’ve never benchmarked the code so it was a premature optimization leading to less understandable code. Also, as Jacob and I discussed in the plane home: it actually skews the behavior of the circuit breaker to prefer switching states rather than staying in the same state – something that you may or may-not want.

Finally an off-topic: It is now possible to build the Clojure circuit breaker including AOT compilation of Java interop classes. I ran into a problem with lein where the order of namespaces in :namespaces isn’t preserved. This is a problem when you are gen-interfacing in one file and gen-class :implements that interface in another.

I’ve created a very simple patch to lein to preserve the order.

http://blog.higher-order.net/wp-content/uploads/2010/07/0001-don-t-convert-namespaces-aot-to-a-set-as-it-does-n.patch.txt

Phil is applying this patch so it should be out soon. Until then, if you don’t want to patch lein, there is a step-by-step guide to repl AOT compilation in the README.

Those interested can grab the latest code.

http://github.com/krukow/clojure-circuit-breaker

There are many advantages to programming with persistent data structures (which implies immutability) but that isn’t the topic of this post… Currently the Clojure data structures are implemented in Java, so in principle they should be usable also outside of Clojure, say from Java. However, in practice it is inconvenient (see below).

I’ve created the project clj-ds to make Clojure’s data structures available in a more practical form to other JVM languages than Clojure. The README file from the clj-ds GitHub project explains the motivation:

Advantages of clj-ds when constrained to working with Java (as opposed to just including clojure.jar)

* Currently the Clojure data structures are implemented in Java. In the future,
all of Clojure will be implemented in Clojure itself (known as “Clojure-in-Clojure”).
This has many advantages for Clojure, but when it happens the data structures will
probably be even more intertwined with the rest of the language,
and may be even more inconvenient to use in a Java context.

The clj-ds project will maintain Java versions of the code, and where possible attempt
to “port” improvements made in the Clojure versions back into clj-ds. Thus keeping maintained
versions of the Java data structures.

* In the current Clojure version, calling certain methods on PersistentHashMap requires
loading the entire Clojure runtime, including the bootstrap process. This takes about one second.
This means that the first time one of these methods is called, a Java user will experience a
slight delay (and a memory-usage increase). Further, many of the Clojure runtime
Java classes are not needed when only support for persistent data structures
is wanted (e.g., the compiler).

* The clj-ds library is not dependent on the Clojure runtime nor does it run any
Clojure bootstrap process, e.g., the classes that deal with compilation have been removed.
This results in a smaller library, and the mentioned delay does not occur.

* Clojure is a dynamically typed language. Java is statically typed, and supports
‘generics’ from version 5. A Java user would expect generics support from a Java
data structure library, and the Clojure version doesn’t have this.
clj-ds will support generics.

* Finally, a slight improvement. Certain of the Clojure data structure methods use Clojure’s ‘seq’ abstraction. In the implementation of the Java ‘iterator’ pattern. It is possible, to make
slightly more efficient iterators using a tailor made iterator. clj-ds does this.

Code: http://github.com/krukow/clj-ds

What? From the Stomp website:

Stomp provides an interoperable wire format so that any of the available Stomp Clients can communicate with any Stomp Message Broker to provide easy and widespread messaging interop among languages, platforms and brokers.

As mentioned, this includes JMS. Stomple is a JavaScript library which enables using browsers that support WebSockets as messaging clients, e.g., subscribing to an JMS topic in a JavaScript program running inside Chrome.

Websocket communication has less overhead than HTTP when HTTP semantics aren’t needed. More importantly, WebSockets allow the server to push updates to the browser (as opposed to the usual HTTP request-response paradigm). For a simple example, see the screen cast of the “transactional chat example” where chat messages are pushed from the server to participating clients.

Why release Stomple when there is already stomp-websocket? First of all, Jeff did an excellent job with stomp-websocket. However, I wanted additional features and more production maturity (e.g., support for timeouts, various low- and high-level callbacks and automatic receipts).

A few notable features:

• Customizability and Callbacks. A key value of Stomple is that it shouldn’t cut off access to any low-level functionality. Hence everything is customizable and accessible to the user code. All core functions support success and failure callbacks. Further, there are low-level callbacks: socketOpen, socketMessage, socketClose, socketError corresponding to the low-level websocket events. These can be used to customize behaviour for special use-cases. At a higher-level, there are callbacks onConnect, connectFailed, onError, onReceipt correspoding to the Stomp events.
• Automatic receipts. If client.autoReceipt is true (which it is by default), then all Stomp messages sent to the server include an automatically generated ‘receipt’ header. In the Stomp protocol, this causes the server to send a receipt for each message it receives, hence reassuring that the message has been received. If not enabled, the client can’t be sure the message is actually received.
• Timeouts. By default all messages sent to the server are associated with a (configurable) 8 second timeout. So for example, suppose you subscribe to a destination, but do not receive a server receipt within 8 seconds: then your failure callback is called with reason ‘timeout’ and information about the message that failed. This way you are guaranteed that either your success or failure callback will always be called (but not both ). These time-outs are pervasive and configurable per API call.
• Transaction management. In Stomp, transactions are started using the ‘BEGIN’ frame, corresponding to a begin function in Stomple. When you call begin, you don’t have to specify a transaction-id: if you don’t, one will be generated automatically for you. Further, any API calls made between a begin and a commit/abort automatically use this transaction identifier, hence grouping the calls in the same transaction (unless, of course, you explicitly provide a different or no transaction id). Further, this nests arbitrarily, and works in a stack-like manner (e.g., begin, begin, send, associates send with inner-most begin).
• Feature complete. Stomple aims to be feature complete. For example, it supports the recommended Stomp ‘content-length’ header, where the current stomp-websocket version does not.
• Correctness. Stomple is tested manually and has a growing automatic test-suite written in YUI3-test. Stomple passes JSLint.

I decided not to build on Jeff’s library because when APIs become more complex, I prefer using configuration-objects since it makes code more readable and easy to maintain (most Stomple functions take a single object literal parameter which names the function’s “many arguments”).
For example to create a Stomple client and to configure it to use destination ‘jms.topic.chat’ as default use:

var client = Stomple.create_client({
    url : "ws://localhost:61614/stomp",
    destination : "jms.topic.chat",
    login : "guest",
    passcode : "guest"
});

A different example of using single-argument configuration objects would be subscribe:

client.subscribe({
    success: function(frame) {//called if subscribe succeeds within timeout-period
        //subscribe succeeded... do something
    },
    failure: function(spec) {//called if subscribe fails or times out
        console.log(spec.reason);
    },
    handler: function(msg) {//called when a message arrived ('this' is scope - see below) 
        this.received = msg.body;//"this" is "someObject"
    },
    scope: someObject
});

Note by the way that if client.autoConnect is true (which it is by default), then you don’t have to explicitly connect. Stomple will automatically connect upon your first action. The subscribe function supports an optional destination parameter in-case a default wasn’t supplied when the client object was created (or in case you want to subscribe to a non-default destination).

How?
For each Stomp client frame (command): SEND, SUBSCRIBE, UNSUBSCRIBE, BEGIN, COMMIT, ABORT, ACK, DISCONNECT there are corresponding lower-case functions on the client object, e.g., send, subscribe… Defaults are specified when creating the client object. For example.

var client = Stomple.create_client({
    url : "ws://localhost:61614/stomp",
    destination : "jms.topic.chat",
    login : "guest",
    passcode : "guest",
    autoConnect: true,//which is actually the default
    timeout: 3000,
    autoReceipt: true//which is actually the default
    //...
});

See Stomple.ClientPrototype for all overrides.

Once a client object is created, the user can issue commands. For example, in the chat example a typical sequence would be.

client.subscribe({
    handler : function(msg) {//update screen on chat msg receiption
	    var c = document.getElementById('content');
	    c.innerHTML = c.innerHTML + '<br>' + msg.body;
	},
	scope : anObjectIfNeeded,
	success : function() {// did subscription succeed?
        //send a test message
        client.send({
            body: "User krukow joined chatroom #stomple",
            success: function() {
              console.log("sent welcome message...");          
            },
            failure: function() {
              console.log("Failed...");
            }
            
        });
	},
	failure : function() {// did subscription fail?
		console.log("sub fail");
	}
});

Things are pretty straightforward from here (if you get Stomp ). For more advanced examples see the automated test-suite. Here is an example of a transaction.

this.client.begin({
    failure: function() {
        Y.Assert.fail("Begin transaction failed.");
    },
    success: function(){ 
		for (i=0;i<N;i+=1) {
			this.client.send({
				body: ""+msgBodies[i],
				failure: sendFailed
                //we don't wait for receipts here... Socket provies order..
			});
		}
        this.client.commit({
            failure: function() {
                Y.Assert.fail("Commit transaction failed.");
            },
            success: function(){//...
            }
        });
    }
});

You can watch a screencast of the transactional chat example.
Screencast of transactional chat.

What is Stomp? Stomp provides an interoperable wire format so that any of the available Stomp Clients can communicate with any Stomp Message Broker to provide easy and widespread messaging interop among languages, platforms and brokers.

You can use any of the Stomp Clients to work with any JMS compliant message broker, for example HornetQ. There are Stomp clients for many languages, and Stomple adds JavaScript via WebSockets. This means that you can send and receive JMS messages directly in the browser (when it supports WebSockets).

Stomple is inspired by Jeff Mesnil’s cool stomp-websocket, but I wanted more customization, feature-detection, a nicer API, and more robustness (e.g. timeouts) (and I wanted to write it myself ). Just like Jeff’s client, Stomple is not directly a Stomp client since it runs over WebSockets which requires a handshake between the browser’s client and the server. This is supported by HornetQ and ActiveMQ (details in Jeff’s docs: http://www.jmesnil.net/stomp-websocket/doc/).

This library is not a pure Stomp client. It is aimed to run on the Web Sockets protocol which is not TCP. Basically, the Web Sockets protocol requires a handshake between the browser client and the server to ensure the browser’s “same-origin” security model remains in effect.

This means that this library can not connect to regular Stomp brokers since they would not understand the handshake initiated by the Web Socket which is not part of the Stomp protocol and would likely reject the connection.

I’ll post more documentation soon, but stomple should be quite simple to use.

if (Stomple) {//feature detection: is stomple available?
    Stomple.debug = true;//enable debug logging of messages
    var client = Stomple.create_client({
        url: "ws://localhost:61614/stomp",
        destination: "jms.topic.chat",
        login: "guest",
        passcode: "guest"
    });
    client.subscribe({
        handler: function(f) {
            console.log('received frame');
            console.log(f);
            console.log("Message: "+f.body);
            console.log("'this' is bound to 'scope'");
            console.log("Session:"+this.session);
            client.send({
                success: function() {},//handle successful send
                failure: function() {},//handle failed send
                timeout: 5000//wait max 5 seconds before failing
            });
        },
        scope: client,
        success: function() {//did subscription succeed?
            console.log("sub ok..");
        },
        failure: function() {//did subscription fail?
            console.log("sub fail");
        }
    });
} 

Check it out on Github: http://github.com/krukow/stomple.

More info to come

I read a post on Ajaxian about Fusebox, a JavaScript library which is described as:

[...] The problem is that frameworks / libraries / third-party scripts may overwrite native methods or each other’s custom methods resulting in unpredictable outcomes. Fusebox, a limited version of the sandboxing component found in FuseJS, avoids these issues by creating sandboxed natives which can be extended without affecting the document natives.

I think Fusebox is a good project. The code in this blog post shows a different approach which in some situations may be more useful and in others, it may not.

With Fusebox script developers avoid polluting the globals by writing their scripts using FuseBox and its wrappers. E.g, you write code with a “Fusebox prefix” like this:

  var fb = Fusebox();
  fb.Array.prototype.hai = function() {
    return fb.String("Oh hai, we have " + this.length + " items.");
  };
  fb.Array(1,2,3).hai(); // "Oh hai, we have 3 items."
  typeof window.Array.prototype.hai; // undefined

// like the native Array constructor the sandboxed constructor will return [ , , ]
  var a = fb.Array(3);
  // equiv to square-bracket notation [3]
  var b = fb.Array.create(3);
  // converting a native array to a sandboxed array
  var c = fb.Array.fromArray([1, 2, 3]);

That immediately reminded be of some work I’ve have been doing with colleague Jimmy Junker at Trifork (jju at trifork com) motivated by the following problem:

In portal environments where multiple portlets want to use different JavaScript libraries or different versions of the same JavaScript libraries, you are very likely to run in to problems since almost all libraries (except later YUI versions) are designed to live in the global namespace.

After a brainstorming session we developed a simple prototype (no pun intended), uninspiringly named “VersionManager”, with which we succeeded in loading two different versions of the PrototypeJS library. With VersionManager you can write code like this:

with (VersionManager.version("1.6.1")) {
//"1.6.1" refers to prototype which must have been loaded
  console.log("abcdaba".gsub("a","42"));// console.logs '42bcd42b42'
  console.log(typeof Object.extend);// console.logs 'function'
  $A([1,2,3])._each(function(x){
    console.log(x);
  });// console.logs '1','2','3'
}
VersionManager.clear();//otherwise previous version is lingering...
try {
  console.log(typeof "".gsub);
//console.logs 'function' <= this is generic delegating proxy function
  console.log("abcdaba".gsub("a","42"));//throws error
} catch (e) {
  console.log(e);
  //TypeError: "attempt call to delegate with no version defined (see documentation)"
}
console.log(typeof Object.extend);// console.logs 'undefined'
console.log(typeof $A);// console.logs 'undefined'

So version manager lets you load several libraries on the page, and sandboxes changes made to globals. By telling VersionManager which version you want to use for a particular block you code, you can use each of those libraries, even if they define the same global variables or properties on the prototypes of objects, e.g., Array.prototype.

VersionManager also creates sandboxes, but as opposed to Fusebox, what is sandboxed is "changes made to" the built-ins, and global variables defined. Also, the goal is not to avoid extending the prototypes, but almost the opposite: to allow several scripts/libraries that would otherwise conflict to live in the same page without seeing each other, even if they define the same global names or want to extend built-in prototypes in different ways.

With VersionManager you simply write regular user code, but wrap it in a with statement. This means that existing code can easily be rewritten.

The loaded "sandboxed" libraries must adhere to a few syntactic and semantic constraints.
For an example, see a sandboxed version of prototypejs here

Transformed prototypejs

The transformation applied to prototypejs is very simple and can be done automatically (i.e., it is easy to write a program that performs this script-transformation for you). It is a bit technical, but it results in is adding a declaration with-statement "header" at the beginning and a "footer" and the end. In fact, we are writing a program that transforms any JavaScript program/library into an equivalent one that adheres to these constraints.

An alpha version is on github. It is a proof-of-concept that shows that the fundamental approach seems to work.

jdalton pointed out a couple of things :

• You may run into problems with minifiers given that your code is wrapped in 'with'. However, you can minify the code first, and then wrap it.
• It doesn't track changes to event or elements.

So far, we've tested it only in latest Firefox and Safari, but I see no reason why it couldn't work in other recent browsers. We have sucessfully loaded prototypejs in a sandbox, and managed to run all prototypejs unit tests successfully inside the sandbox.

Github project: http://github.com/krukow/versionmanager

/Karl

As an appetizer, the first chapter is available for free:

Clojure—A Lisp for the Java Virtual Machine

I’ve read the first chapter and the book looks very promising! To quote the last paragraph of chapter one:

We’ve talked a little about how this book will go beyond what Clojure is to why it’s designed the way it is and how that design can be exploited through idioms that will help you think in Clojure. So lets stop talking about what this book will do and get on with the doing.
Fasten your seat belts.

I’ve fastened my seat belt and ordered my copy

As should be obvious, Clojure is a labor of love on my part. Started as a self-funded sabbatical project, Clojure has come to occupy me far more than full-time. However, Clojure does not have institutional or
corporate sponsorship, and was not, and is not, the by-product of
another profitable endeavor. I have borne the costs of developing
Clojure myself, but 2009 is the last year I, or my family, can bear
that.

I would be such a shame if development were to slow down or stop completely because of such a small amount of money… So, please donate to the project.

See details,

http://groups.google.com/group/clojure/t/cc77df25e98ce46b