read


read
or learn more

Compiling Clojure to JavaScript, pt. 2 – Why No Eval?

Jul 29, 2011

Why no eval?

this is the second entry in an n-part series explaining the techniques and design principles of ClojureScript. translations: [日本語]

I was not expecting to talk about eval so soon in this series, but apparently its exclusion from ClojureScript was more controversial than I imagined. However, by excluding eval the Clojure/core team was not acting miserly, but instead had valid reasons for its exclusion. This post will go over why eval was excluded and why it may never see the light of day in the core distribution.

What is eval?

The first question you may have is “who cares?” but preceding that may be “what is eval?” Simply put, eval is a function that takes a data structure representing a program and executes it in the context of the dynamic environment. For example, in Clojure you can perform the follow:

(eval '(+ 1 2 3))
;=> 6

This takes the list: the symbol + and the numbers 1, 2, and 3 and executes it as if it was typed into the REPL. Let’s look at a more interesting example:

(def blip [\b \e \e \p])
(def buzz {:beep 42})

(eval (list* (keyword (reduce str blip)) 
             [(symbol "buzz")]))

;=> 42

This fragment builds from pieces the list (:beep buzz) which is evaluated as a keyword lookup into a map. This is an interesting example not because the result is useful, but in that it demonstrates an interesting property of Clojure (and Lisp in general) — code is made of the same data structures that the language itself manipulates.

eval is Indeed Very Cool

One of the first games that programmers new to Lisp play is the eval game. That is, they build a bunch of data structures using the Lisp functions and pass them into eval. It’s great fun. Having eval in ClojureScript would also be a very useful tool. Indeed, I suspect it would be extremely useful to base an in-page REPL on eval. This REPL could then be made to operate within the context of the currently loaded ClojureScript-enabled page allowing direct manipulation of the page elements and even the runtime code itself. This is potentially a very powerful debugging and incremental development tool for ClojureScript in the same way that Firebug serves for JavaScript. However, the greatest part about this “fictional” REPL is that it doesn’t require that ClojureScript provide eval at all.1

But honestly, if ClojureScript programs are composed of data forms directly manipulable by the language itself then why not just provide eval anyway? The answer can be found, as many answers in life, in the tradeoffs.

FojureScript

Imagine a language named FojureScript that is exactly like ClojureScript in every way except that it contains eval (and has awesome error messages). FojureScript programs are compiled into JavaScript with a compiler written in another language named Fojure. FojureScript’s compiler also allows for differing levels of optimization from raw JavaScript emission, to advanced dead-code elimination. What happens when a call to eval is performed in FojureScript? In the first scenario the naive eval falls down:

(eval '(+ 1 2 3))
;!! Unknown function +.
;!! Maybe you meant fojs.bore._PLUS_?

So clearly the problem is that during the compilation process the function named by + is munged into a JavaScript function fojs.bore._PLUS_. OK, no big deal. FojureScript can be made to resolve the properly munged names at evaluation time with (relatively) minimal fuss. However, what about the call to eval in the case of a maximally minified compilation?

(eval '(+ 1 2 3))
;!! WTF dude!?  I have no idea what + is!
:!! That function doesn't even exist.

What’s the problem?

If you looked in the FojureScript bore.fojs library you would see that + is clearly defined. However, in the production code a call to + never actually occured and therefore its implementation was removed completely from the runtime environment!

Blurring the Lines

Although Lisp programs are made up of data structures available in the language, there is a marked difference between the data composing the code, and the data flowing through the code. The data composing the language is subject to static analysis and a lot of information about it can be garnered at the time of compilation. The data running through the code is known only at the time of execution and therefore can only be known in very superficial ways. There is a stark line between what is known at compilation time and what is known at runtime. Therefore, compilation can and will eventually remove a function that is needed at runtime via eval.2 And therein lies the rub. The presence of eval blurs the line between runtime and compilation time requiring that everything available in the latter be present in the former. (or is it the other way around?)

Trade-off

A goal of the ClojureScript team is to provide maximally minified source code that maintains program semantics. The trade-off therefore is that it is much more important as a ClojureScript design principle that the runtime environment be available at dev time than the dev environment be available at runtime.3

The larger goal of Clojure (the family) is to generate high-performance code on every single platform and to support eval in ClojureScript is antithetical to that goal. It would be a fool’s errand to try and support eval in the face of aggressive code elimination — you can have one, but not the other. ClojureScript is designed to solve the types of big problems that are simply too difficult to fathom, much less achieve, using raw JavaScript or any number of JavaScript frameworks. It’s conceivable that these huge applications will require a bevy of libraries. You cannot effectively leverage these libraries without an optimizing compiler utilizing dead-code elimination. You cannot have the optimizing compiler if you wish to support eval. Q.E.D.

While you may not agree with all decisions made in the development of ClojureScript (especially regarding eval), I hope you will agree that it was not designed and developed by fools.

I hope…

Anyone?

Bueller?

:F

thanks to Chris Redinger and Brenton Ashworth for reading a draft of this post


  1. I knew this would happen, just not this quickly. Great timing I’d say. 

  2. And if you extrapolate from this point you’ll see why there is no runtime compilation. That is, if a macro builds a form containing functions that were optimized away at runtime, then bad things will happen. 

  3. This principle is manifest in Chris Granger’s brepl

13 Comments, Comment or Ping

  1. I don’t really understand why anyone would want eval, and for that matter, any other bad part of JavaScript. As a new language ClojureScript has the privileged (and the common sense) to focus on the good parts of JavaScript. I haven’t checked it, but I hope that there’s no ‘with’ in ClojureScript.

  2. Base

    Thanks for this great explanation. This decision certainly makes sense in the context that you laid out.

    (B.)

  3. Dane

    Couldn’t you have both with different compilation modes? Then the developer could decide whether eval is worth the performance tradeoffs.

  4. Nick Main

    I imagine that there is no runtime support for user macro-expansion in the reader either.

    Is that correct ?

  5. Laurent Petit

    I’m having a hard time to buy what I consider to be the “main argument” of your post, e.g. “look, when dead code is aggressively removed, eval cannot be used, so it’s not possible to write eval”. (it’s how I understood it the first time I read it).

    For me the argument is “eval in prod would prevent us from being able to provide a ‘maximally minified source code’ option”.

    And removing the need for eval has probably led to (or facilitated) the design decision of writing the ClojureScript compiler in Clojure instead of ClojureScript. Which decision totally sealed the fate of an eval in ClojureScript (and which I think is /now/ the real technical reason for not being able to change mind and e.g. provide eval for some development mode only).

    Note: I’m fine with that, but the argumentation starts when wrong or partial explanations are given to people :-)

    Cheers,

    – Laurent

  6. Ok, so the design decision makes sense for the deployment case, but just out of curiosity why couldn’t there be an eval for an everything included, non-optimized version? If the compiler is written in clojure and it emits javascript it seems like it should be able to load into a running environment. In his talk Rich said that the compiler has to generate closure-compiler friendly javascript. Can this be executed directly without using closure, or is it necessary to run the compiler? Would ClojureScript as it is now be sufficient to run its own compiler, or are there missing features that would preclude this anyway?

    Thanks for an informative post.

  7. Paul M Bauer

    It seems a very pragmatic decision was made to effectively fork key parts of Clojure to make ClojureScript, as opposed to doing Clojure-in-Clojure first, then making JS a target.

    Will there be a test suite comprehensive enough to guarantee a measure of semantic fidelity between the two versions? Interop, eval and macros are just the immediately visible differences (see binding semantics).

  8. Brian Goslinga

    I can see a world were all of the reflective features (including eval) are provided, but a pay-as-you-go system is in place. For the vast majority of programs, all of or almost all of the support for reflection is optimized out as it is not used by the program.

    Granted, the increase in complexity of the compiler may make it not worth it (although the Closure compiler seems like it may be able to do most of the hard work) as only a minority of programs would need the extra features.

  9. @Laurent

    You’re right that writing the compiler in Clojure is a reason why eval is not provided, but that is a purely technical reason. That is, if eval was of the utmost utility and importance, then it would get in one way or another.

    If you would, please let me know how my phrase and your rewording are different. I’m confused.

  10. Laurent Petit

    @fogus maybe it’s me which was confused. I thought that having the compiler written in Clojure was also driven by other reasons, such as not having to solve harder problems (*), which may have lead to releasing the project later, and that maybe this was not affordable.

    So I had the feeling that not having eval was an acceptable trade-off, while I see it exposed as the main reason(**). But it’s just musings, after all!

    (*) adding a bootstrap phase to ClojureScript. (**) because I keep thinking that there could have existed other ways to provide eval by having 2 different kinds of distributions : a compiler+runtime distrib, and a runtime-only distrib. Of course, I can imagine lib providers starting to rely too much on compiler+runtime distribs, thus locking their users … ==> but even by not providing the choice to embed the compiler at runtime, wouldn’t have written the compiler in ClojureScript enabled writing macros in ClojureScript ?

  11. Pete F

    I love the smell of premature optimization in the morning!

  12. @Pete

    It’s not as if this is untrodden ground.

  13. I am wishing for a Node.js based REPL. I am almost certain this requires eval — how about with no optimizations, say, just for development? ^_^

Reply to “Compiling Clojure to JavaScript, pt. 2 – Why No Eval?”