Yegge, Clojure, Arc, and Lolita: or Days of Future Past

Feb 6, 2009

This post was discussed thoroughly on Hacker News. Even Mr. Graham was kind enough to comment — though not in the affirmative. There was moderate discussion on Reddit as well.

A long time ago, in a galaxy far far away, Steve Yegge wrote an essay on Common Lisp named Lisp is not an Acceptable Lisp. As far as Yegge essays go, it was one of the more Molotov-esque. There are a few items in this ancient tract that I feel are spot on with reality and another few that seem to miss the mark these many centuries later.

Precognition

Arc’s going to be a new religion, of course, because programmers just haaaaaave to make it that way.

Looking back on the reaction to the announcement of Arc, Yegge was reading the writing on the wall with this particular comment. Paul Graham has always tended to be viewed as a living/breathing/walking religion because of his spectacular essays on high school life, education failures, and general purpose geekery. However, his essays are likely to be remembered for the way that they generated a frenzy-tinged lust for LISP. There are a 1000 reasons why this could be, but the most likely lies in the fact that when his most groundbreaking essays were written, the world of programming was mired in the world of Java, C, Visual BASIC, and C++. His essays on LISP elucidated an almost Utopian world of programming power, long forgotten or little imagined to those mired in Blub.

… My prediction: someone will get tired of waiting, and they’ll Torvalds Arc into obsolescence before it’s ever released.

The release of Arc was a grand day in the history of the Internets. Birds sang that day my friends. The air smelled of the finest of perfume. I believe even Christopher Hitchens thanked the gods that day (speaking of cult-like figures in the world of geekdom). As a result of this unparalleled software release, mzscheme was downloaded over a trillion times. However, the honeymoon was short and the grumbling was indeed furious thereafter. “Is this it?” Now bear in mind, over the years leading up to the release, the fledgling Arc community had nurtured such expectations of Mr. Graham’s language that anything short of The Singularity occurring as a result of its execution was tantamount to Global Thermonuclear War followed by the cancellation of Heroes and the concurrent deaths of Adam and Jamie. It was running on top of mzscheme? It didn’t support UNICODE? No module system? Few libraries? Unhygienic macros? Lists for dinner — again? These criticisms festered into essentially an Internet meme ¹; everyone was jumping in on the lashing. Eventually, the buzz of Arc wore off and all that remained was the negativity. Eventually, Mr. Graham himself seemed to spurn ² the language, and it has since been relegated to the dustbin of the github fork. ³

And then along came Clojure. For all intents and purposes, Clojure’s creator Rich Hickey is Arc’s Torvalds quipped on by Mr. Yegge. Adding powerful namespacing coupled with the fact that it’s built on top of the JVM dispels the questions and gripes aimed at Arc from the start. Any momentum that Arc may have gained from its release and subsequently maintained once the love affair ended was squashed under the heel of the Clojure juggernaut. Sure, Arc still has its proponents, desperately clinging to their language of choice like a desperate Humbert Humbert to a very pregnant Lolita ⁴, but even they have the preliminary itch of disillusionment.

Truth be told, Lisp should probably have a skinnable syntax. That implies a canonical abstract syntax tree, which of course hasn’t been defined (and in many implementations isn’t even available to you, the way it is in the Io language, say).

Now I’m not a LISP expert, but it seems to me that the S-Expression is popularly regarded as the canonical representation of the LISP AST (and many other languages for that matter). That is, the LISP syntax is as close to an AST as one can get. Perhaps the presence of macros, #, `, ‘, and , muddy the waters a bit, but not much.

Misses

Your only other option is to design a new language, and you won’t get any help from Lisp people, because they will hate you.

It seems to me that the greater LISP community is embracing Clojure and its creator Rich Hickey most amiably. Not coming from a LISP background, it’s difficult to say what the condition of vitriol and hatred was when the original article was written. Additionally, I do not have a herd of LISP hackers roaming my office halls in order to observe any low-frequency mummers disparaging Rich Hickey. In any case, Clojure has garnered a lot of excitement from some highly credible LISP hackers. If Mr. Kenny Tilton would come around, then the coup would be complete.

Or maybe you could go the Haskell route and not have OOP at all. That seems to alienate most programmers, though, despite the attractions of not having to create nouns for everything.

It’s tough to be negative on Mr. Yegge on this quote considering that 2006 was still during the time when OOP principles were drilled into every C.S. majors’ skull and could be heard in the laughter of children. However, the functional programming para-dig-em is finally starting to gain a handhold in an otherwise homogeneous programming language landscape ⁵.

Still True?

Macros are notoriously hard to debug, and honestly it needn’t be that way. If your editor knows all about macros, then you should be able to click to see the expansion, and click again to see its sub-expansions, all the way down to the primitive functions.

It may simply be my own ignorance, but I still do not know of an editor that does this for any current LISP. (pointers welcome)

Conclusion

In order to wrap up this post, I would like to quote a comment posted in the comments section of the original article:

Lisp indeed leaves you with your own ideas and your own limitations. It doesn’t pose any artificial restrictions on your programs that you have to work against, and it doesn’t provide any “color by numbers” examples that make you feel like you have achieved something. Lisp requires you to be creative.

The quote above by Pascal Costanza says it all: about Clojure, about Common Lisp, about Qi, Scheme, and even about Arc.

Having spent a majority of my programming career writing Blub, I have only recently ⁶ discovered the power of languages in the LISP vein. Like many programmers, my interest was stoked by the excellent and iconic (as far as Internet noise goes) LISP essays written by Mr. Graham. I, like many others, dreamed of the 100 Year Language and was pulling for Mr. Graham to deliver it. However, I think that a language such as Clojure (or more likely one of its inevitable derivatives) will instead be the language of 2109.

-m

Am I the only person amused by the fact that it’s spelled Me Me? ↩
Let me elaborate. As far as I know, Mr. Graham does not Twitter, does not have a Facebook account, does not host Arc on github, and does not bounce his ideas off of his adoring followers. There are some quarters who would view this relative isolationism as a dismissal of his creation. As far as I’m concerned I have enjoy his output thus far under these exact circumstances, and would not wish otherwise. ↩
Don’t get me wrong, I love Github ↩
I should make it clear that I find many things that I like about Arc. Foremost, I find it incredibly easy to write information-dense code with it. There is much to be said for that — which I will not do here. ↩
I often wonder how genuine the buzz around functional programming is. I mean, the majority of programmers I’ve worked with do not seem the types to really care two shakes about something like functional, be it pure or no, even if they could understand it to start. I sometimes think that much of this buzz lies in fashion, but hope makes me dismiss those damnable thoughts. ↩
Not true. I used LISP throughout college and grad school as an AI major and actually served as a T.A. for a number of LISP-centric courses. However, hacking LISP in the spirit of a school assignment’s deadline and grading (mostly) hackish filth is far different than doing so for one’s own entertainment. ↩

19 Comments, Comment or Ping

Peter Seibel

Emacs with SLIME and Common Lisp gives you the ability to expand macro calls in your editor.

Feb 6th, 2009
fogus

Thanks for the tip. I am surprised that Yegge, being an EMACS advocate wouldn’t have mentioned that. I will go and play around with it.
-m

Feb 6th, 2009
Allen Rohner

There is SLIME support for clojure now. See http://github.com/jochu/swank-clojure/tree/master There are plenty of posts on the clojure group helping users get started with slime.

Feb 6th, 2009
Eric

Re. footnote 4: as I understand it, the current buzz around functional programming is being driven mostly by the need for parallelization due to multicore processors.

Feb 6th, 2009
Ken

“Eventually, Mr. Graham himself seemed to spurn the language”

Wait, what?

“it seems to me that the S-Expression is generally regarded as the canonical representation of the LISP AST”

That’s like saying “trees are the canonical representation of the C++ AST”. It says how they’re stored, but not what they contain or how to interpret them.

Feb 6th, 2009
Marek Kubica

I also like the macro stepper in PLT Scheme. It helped me a lot to grok how the macro I wrote actually worked.

Feb 6th, 2009
ml

herd of LISP hackers raoming my office halls in order to observe any low-frequency mummers

Those mime types only pass through once per century, they’re difficult to observe.

Besides, Lisp hackers are harder to herd than cats. Throw three of them together, you’ll have at least four distinct dialects. Common Lisp contains multitudes.

Methinks it’s ingrained in the s-exps. The Gadarene swine story comes to mind.

Feb 6th, 2009
trittweiler

In fact, you can expand macros again within a Macroexpansion buffer in Slime, recursively ad nauseam. And not only that, you can even undo such an in-place expansion.

See http://common-lisp.net/~trittweiler/slime-talk-2008.pdf.

Expanding local macros is also possible, though I haven’t come around polishing and committing that work yet.

That macros can be hard to debug is unrelated to this, however.

Feb 6th, 2009
Scott

The macroexpand functionality in Slime/Emacs (C-c RET) works with Clojure.

Feb 6th, 2009
fogus

Eric,

I agree that the “emergence” of the multi-core processor has indeed fueled the current interest in functional programming. My hopeful side yearns for it to continue. -m

Feb 6th, 2009
fogus

Ken,

Spurn was probably a little too strong a term to use.

As for s-expr as the AST form, I see your point and claim idiocy in my mistaking syntax with semantics. Take for example, the loop form in Common LISP, simply knowing that it conforms to s-expr syntax tells me nothing about how to interpret it.

-m

Feb 6th, 2009
Required Name

It is “Lisp”, not “LISP”. I hate to be that guy, but enough people have been talking about it for long enough to know better.

Feb 7th, 2009
Daniel Weinreb

I’m not sure what the “Common Lisp contains multitudes” phrase was intended to mean. (Yes, I do recognize the reference to Whitman.) Portability between Common Lisp implementations is excellent when it comes to things that are actually covered by the standard, and there are lots of useful libraries to provide portable ways to do things beyond the scope of the languages (e.g. usocket, CFFI, many more).

We have ported our 500KLOC airline reservation system core from SBCL to Clozure CL (formerly OpenMCL). It took a bit of work because of some fancy stuff we do, but it wasn’t very hard. We now maintain it in both implementations; every night our buildbots compile and test the code in both implementations, just for extra testing. No problem.

(OK, not all of the 11 implementations are quite that good, but about 8 of them really are.)

Yes, there is plenty of interest in Clojure in the Common Lisp community. At the International Lisp Conference (ilc09.org) I intend to get a lot of discussion going about the future of Lisp and how we can all try to move together (and less like a herd of cats!) to pool our efforts. The ultimate success would be to bring the Scheme world and the regular-Lisp world together on one language, although I suspect that’s too hard to accomplish and the Scheme people need to own their own language, so that they can experiment with it, but, we’ll see.

Yes, mostly-functional languages with software transactional memory is the big new trend. Many new languages are being created based on this, and Rich was far-seeing to do it in Clojure. Yes, it’s primarily because we’ll all be writing more concurrent software to take advantage of all the cores.

The Lisp s-expression structure is not exactly analogous to an AST in the sense that the compiler community means. An AST is a deeper, richer structure on which a lot of information is hung by the compiler, usually. The s-expressions are more of a “surface syntax”, which is the right place to put extensibility (macros) since ordinary language users won’t understand an AST, and it’s not even clear that it’s a good idea to make the AST’s data structures part of the externally-defined language standard.

Macros of Lisp-like power for languages with infix-style syntax have been explored. See Jonathan Bachrach’s Java Syntactic Extender, and the Dylan language. Dave Moon will be presenting a new paper on this at the Lisp conference.

Feb 7th, 2009
Luc Préfontaine

We have now a bus product deployed at a customer site using Clojure since Jan. 6th.

http://www.infoq.com/news/2009/01/clojure_production

I did major dev projects using Java since 1999 and became gradually disgusted by the tons of Java code lines needed to achieve anything significant in the last few years.

The absolute need to use an IDE like Eclipse is a clear indication of the low expressiveness of Java. The best argument to support the above is that without such an IDE, any refactoring task is anything else than mundane.

Relying on tools like Rational Rose to spit out Java code is also an indication that something is wrong…

I used to work with Lisp both on academic projects and on prototypes for production systems. However using Lisp to deliver a production system proved to be a significant task mainly because anytime we needed some library we end up having to code it 4 times out of 5. The variety of Lisp dialects did not help in this area.

Clojure is for us the best compromise, it gives us all the major advantages of a Lisp while allowing us to tap on the huge Java libraries available out there. Since there are no “java dialects” we can always rely on some java extension to compensate for the lack of a native Clojure implementation, assuming such an implementation is really needed. If it exists in Java, well it may be better to use it than recoding it.

That drove us to choose Clojure as our main language for our product line. Practical, usable right now, build for concurrency, concise syntax, predictable performance, escape to Java allowed, robustness of the implementation,…

Clojure V1.0 is around the corner, the pre V1.0 code worked so well that we went in production without waiting for 1.0 to be available.

The Lispers religion wars do not interest us, we want something usable now to deliver commercial applications faster and take advantage of distributed processing. Commercial software has been lagging significant improvements in the last 20 years or so.

Rich Hickey’s approach of bringing two worlds together looks to us as a significant improvement.

Luc

Feb 7th, 2009
fogus
It is “Lisp”, not “LISP”. I hate to be that guy, but enough people have been talking about it for long enough to know better.

There are two reasons why I like to use LISP instead of Lisp:
1. I use LISP for the general LISP-ness, and Lisp for a specific Lisp implementation (see how I used it for Common Lisp
2. Using LISP helps to quickly identify the type of people who are usually not interested in having serious discussions about LISP, Lisp, LIsp, lisP, or LIsP.
-m
Feb 9th, 2009
rzezeski

“It is “Lisp”, not “LISP”. I hate to be that guy, but enough people have been talking about it for long enough to know better.”

I guess Steele and Sussman should republish the Lambda papers.

I for one am loving Clojure. It’s made programming fun all over again.

Feb 10th, 2009
Austin King

DrScheme (PLT Scheme) will also Graphically expand your macros.

Feb 11th, 2009
Xah Lee

you wrote: «Now I’m not a LISP expert, but it seems to me that the S-Expression is generally regarded as the canonical representation of the LISP AST (and many other languages for that matter). That is, the LISP syntax is as close to an AST as one can get. Perhaps the presence of macros, #, `, ‘, and , muddy the waters a bit, but not much.»

that is technically not true. Although you mentioned lisp syntax irregularities like “# , ,@ ‘”, but in general this sentiment that lisp has regular syntax, or that its syntax is closest to abstract syntax tree, is perpetuating a myth.

Mathematica language’s syntax, XML, and XML derived general purpose languages (e.g. O:XML, X#), are in fact more regular or closest to a pure tree representation.

For detail, see:

• Fundamental Problems of Lisp http://xahlee.org/UnixResource_dir/writ/lisp_problems.html (the first section; on syntax irregularity)

• The Concepts and Confusions of Prefix, Infix, Postfix and Fully Nested Notations http://xahlee.org/UnixResource_dir/writ/notations.html

also note, some pure form of nesting is not the only way to represent a tree. Indendation, such as used in Python or ascii-picture tree used to represent directories and sub dirs and files, is another form, arguably has more clarity than nesting with delimiters. As far as i known, there were a couple such proposal/spec/library with working implementation of this syntax that works for Scheme lisp.

Feb 12th, 2009
Jochen Schmidt

Yegge’s essay was really fun to read but full of factual errors that showed that he cannot be so experienced in lisp like he likes to tell people. Most of his arguments seemed to be more like hearsay and not like first class experience. I think he is a Paul Graham victim ;-). I used to like Yegge’s essays but after “Lisp is not an acceptable Lisp” and his relativizing follow-ups he lost some levels of credibility in my books.

What Clojure really proved is that the Common Lisp community is not so religious about other dialects of Lisp like many people thought. Many lispers tolled respect to the design and like to play with it; even use it in projects. Most older “new lisps” seemed to solve mini-problems like unusual naming or were only incompatible because making them compatible would be actual work – it was never clear for more experienced lispers what they actually tried to solve. Clojure shows that it is well designed, solves some real problems and offers new ideas – to me it is no wonder why Lispers like Clojure.

(regarding expanding macros) In the LispWorks IDE (Common Lisp) you can macroexpand (expand once) and walk (expand completely) directly in the editor or listener (by key chord or context menu). LispWorks also has a very slick Stepper-Tool. You can place break-points just in your source code (at expressions, not only lines!) and the stepper is automatically opened if the program hits it. The stepper asks if it should automatically expand a macro if it sees one. The whole IDE is running within the same Common Lisp image and is using LispWorks’ own cross platform UI-Toolkit CAPI for the Tools. The editor is programmable like emacs; you can integrate this editor as a text-pane in your applications. The only caveat: It’s a commercial offering – as some people cannot get themselves using non-free tools, this could be a problem. There is a free Personal editon though – its restricted in memory and you cannot deploy your application with it – but the IDE works like the full edition.

Feb 12th, 2009

Send More Paramedics