Translations: δΈζ
A few years ago, a little-known language called newLISP completely changed my understanding of what “good” programming languages look like.
Why newLISP?
Before saying another word, I’d like to address the question that some of my LISP-familiar readers may be asking right now: Why newLISP? Why not Clojure, Scheme, or Common LISP?
The answer is that after evaluating these dialects of LISP, I’ve come to the conclusion that newLISP has several important advantages over other LISPs.
Today, unfortunately, whenever someone mentions newLISP on an online forum frequented by adherents of another LISP, an almost clan-like flame war will erupt. An adherent of one particular dialect, usually someone who has never used newLISP, will shout obscenities such as, “newLISP goes back on just about every advancement made by LISPs”, or “dynamic scope is evil!”
The historical context out of which these sentiments are born is mostly unknown to those observing the debate. As signal gives way to noise, the discussion collapses, the melee disperses, and the bystanders go back to sipping their Java and eating their Pythons.
It is fortunate, I think, that my introduction to the language did not sprout out of one of these battles. It was, for the most part, largely unbiased.
Something called LISP
While attending the University of Florida, I had signed up for a course on artificial intelligence. Our professor predictably introduced the class to a programming language called LISP.
Up to that point in time my knowledge of LISP consisted of the usual hearsay and mantra of those unfamiliar with the language:
“People who like it are crazy zealots who think they’re superior to everyone!”
“It’s mainly used for artificial intelligence stuff.”
“No one uses it for practical purposes.”
“It’s mainly used as a research language.”
“It stands for ‘Lots-of-Irritating-Silly-Parenthesis’. Haha!”
You get the idea. I did not have a clue as to what it was, but I was excited to finally be forced into the position of having to find out. My grade depended on it, after all.
What I discovered was that LISP did in fact live up to its reputation of zealot-inducing awesomeness. Simply being exposed to some of the basic concepts and philosophies of LISP had an immediate and positive impact on my abilities as a programmer.
Syntax: Programmer Enemy #1
LISP’s relative lack of syntax was perhaps the greatest insight, for I immediately felt as though a great weight had disappeared. I realized that it was syntax that was at the root of most programming errors. It was syntax that created a subconscious burden that I had simply been unaware of; causing errors, bugs, and making it difficult to simply turn my thoughts into code. Despite having many years of experience with C-based languages, and being intimately familiar with their syntax, I realized that it was nevertheless a totally unnecessary burden that was slowing me, and everyone else, down.
This is one great advantage of Lisp-like languages: They have very few ways of forming compound expressions, and almost no syntactic structure. . . . After a short time we forget about syntactic details of the language (because there are none) and get on with the real issues.
—Abelson and Sussman
It’s not just the size of the syntax that matters, it’s what you can do with it. What requires layers of special syntax in languages like PHP, Python and Ruby, LISP can do with its basic concepts of lists, functions and symbols. It can do everything those languages can do, in a more elegant fashion, and still have enough tricks left up its sleeve to accomplish feats that are simply not possible in other languages.
Common LISP: A Series of Unfortunate Mistakes
Despite all of these exciting discoveries I still had an uneasy feeling.
Common LISP (CL) was a great departure from what I had known previously, but it reeked of antiquity, and worse, its syntax simply wasn’t very well thought out. The “Zen-like” perfection that it seemed to be yearning for was missing. It was a feeling that Mac users are all too familiar with: There were too many buttons, and most of them were unnecessary.
Compared to C++, that’s fantastic. Then again, compared to C++, most languages appear favorable. Having had a taste of the liberty offered by the drop in syntax from C/C++/Java to CL, I did not see why all of this syntax was necessary, and indeed, most of it wasn’t.
Every little piece of syntax that’s introduced into a language adds to the programmer’s mental burden, be he conscious of it or not. I was not prepared to spend the effort learning Common LISP if a better alternative could be found.
What do you mean by “syntax”?
You may object at my inclusion of the functions defvar, =, eq, eql, etc. as they are functions. I include them because they constitute low-level functionality that cannot be expressed in the syntax of LISP itself. In other words, the functions = and eq are low-level primitives that must be defined in the language that LISP itself is implemented in, and their meaning and usage cannot be deduced from any other existing LISP syntax.
Consider PHP’s != and !==, they are both operators and are used in the same way, yet that doesn’t tell you anything about what the difference between them is. There’s no way to deduce their meaning from the existing semantics of the language and thus they each represent new syntax that must be learned.
How I Discovered newLISP
The professor revealed to us our “major class project”: we were to implement a text-based version of the game Chainshot:
An implementation of Chainshot
Chainshot starts with a grid completely filled with colored cells. Your goal is to clear the board and you do this by clicking on each cell. If the cell has any adjacent cells of the same color, they all disappear. The effect spreads to include all of the cells adjacent to those, and so on. Cells then drop down to fill in the gap left by the vanished group. If an entire column disappears then all cells to the right of it move left to fill it in. There’s a wonderful and free implementation of this for OS X called Otis.
For the midterm we were to make a version of this playable by a human, and for the final we had to write an AI to play the game.
“One more thing…” he said, “If you do it in LISP, you’ll get a 10% bonus.”
The problem though was most of the lectures focused on various algorithms and theory for doing AI. Those who wanted the 10% bonus would have to teach themselves the language.
Most students chose to forsake the bonus in favor of using a language that they were already familiar with, and like most students I had very little time, so I was partial to that sentiment. However, I decided to do a Google search to see if I could find a Common LISP alternative that was more appetizing.
To my delight I found a language that seemed to check all the right boxes, and surprisingly it wasn’t Scheme (although that is a wonderful language as well). Like Scheme, this language had greatly simplified Common LISP’s syntax, but at the same time it came with a standard library full of useful functions for performing modern scripting tasks, and all you needed to run it was a single tiny executable!
Discovering newLISP
Here is what ultimately turned me into such a fan of newLISP.
We had several months to complete the first part of the project, and the night before it was due, I did not have a single part of it complete. I had spent the night working on other things, and for perhaps an hour I spent some time looking over the newLISP website, reading about it.
The next day, approximately four hours before I was to hand in my finished, playable version of Chainshot, I sat down at a desk, put on my headphones, and proceeded to simultaneously learn newLISP while creating this game with it.
I finished in about 3 hours, of which only about ten minutes was spent debugging. I was dumbfounded. I had discovered something new, a language that allowed me to rapidly write what, after years of C-based languages, seemed like virtually “bug-free code” that just worked. It was a language that I had come close to mastering in a matter of minutes, having never used it before! In the time I had learned newLISP and written a text-based game in it I would probably only be finishing the tutorials for Ruby or Python.
newLISP’s Strengths
At the beginning of this post I made the claim that newLISP has several advantages over the other LISPs. They can all be summarized as follows: If you want a LISP-based scripting language, choose newLISP.
Before getting into the details let me warn:
newLISP is not a general-purpose programming language.
In the same sense that you wouldn’t use JavaScript to write an iPhone app (some beg to differ), you wouldn’t use use newLISP to write an operating system, a music player like iTunes, or a web browser like Firefox. For such endeavors I recommend without hesitation Clojure, Scheme, C, Objective-C, etc. In other words, languages geared for solving complex, low-level problems, as quickly as possible.
The very first sentence on newLISP’s website states (emphasis mine):
newLISP is a Lisp-like, general-purpose scripting language.
Long ago, when computers were slow, the LISP community was mercilessly mocked by their C and Assembly-wielding counterparts for the crime of being too slow. Ever since that time the subject of performance has been a sore spot for the LISP community from which, I dare say, it has yet to fully recover. Its focus turned towards compilation and proving to the world that it too, could be fast. As a result, few seem to have noticed the need for a general-purpose LISP focused instead on interpretation and scripting.
Luckily, newLISP seems to fit that role rather well. It is a general-purpose interpreted scripting language. It’s my understanding that because of how dynamic it is, it cannot even be compiled to bytecode (this does not mean it is not fast, though).
Without this understanding, some of its design decisions will not make sense. Why choose fexprs over macros? Why dynamic scope instead of lexical scope? Why One-Reference-Only memory management instead of garbage collection?
Design and Syntax
newLISP comes in a single, tiny 200+KB binary executable. Out of all the LISP derivatives I’ve tried, it is the easiest to setup, deploy, and develop for. Somehow that tiny package contains the entire language and includes functions for reading and writing files, parsing text, regular expressions, running code in parallel, over a network, and much more. For the final project I was the only person in the class to submit a fully parallelizable AI (scalable to any number of cores) to solve each grid. The only reason I did it was because I could do it without breaking a sweat. newLISP made it mind-numbingly easy, and this was before it had all the actor and Cilk stuff.
newLISP’s syntax is minimalist and well thought out. For the sake of comparison with the Common LISP syntax card, I’ve kept most of the attributes (such as font size) the same:
Functions do not need any of the &rest, &optional flags. Simply pass in variables or don’t, the parameters that don’t get anything are set to nil, and extra stuff can be accessed through the function (args) or the symbol $args.
Functions, like most other things, evaluate to themselves. You don’t need a special #’ syntax to access them. Functions are also real lists. This means you can get their source after they’ve been defined, and even modify them while they are executing.
fexprs and eval
Instead of macros newLISP chose to use fexprs, or functions that simply don’t evaluate their arguments (although to the chagrin of some, newLISP calls them macros). This decision makes sense because in an interpreted LISP, almost everything happens at runtime and there are situations where fexprs can be much faster to execute than macros. It also means that newLISP’s “macros” don’t need the special backquote syntax, making them easier to write and read.
You may have heard the mantra against using eval in other languages. In newLISP, this just doesn’t apply. newLISP’s eval is faster than other LISPs.
This has many consequences, one of which is that sometimes newLISP’s fexprs can be faster than compiled macros in the other LISPs, but also it means that using eval is no longer frowned upon, which opens up all sorts of coding possibilities.
Equality and Memory Management
Notice that there’s a single equality operator, the equals sign. newLISP can get away with this luxury because of its memory-management model, called One-Reference-Only (ORO).
In short, most things are passed by-value and so you end up not needing all of those ridiculous comparison functions. If two things have the same value they are equal—end of story (except in the case of Objective newLISP).
This is not as crazy as it sounds. Internally, newLISP passes data by reference between built-in functions and does other optimizations. You can pass data by reference through the use of contexts and symbols, or by using Objective newLISP. newLISP’s ORO also means repeatable code execution times; you’ll never experience “GC Hell” because there is no garbage collector.
Dynamic Scope
Much fuss is often made over newLISP’s use of dynamic scope. It is true, dynamic scope can be dangerous!
In a similar way, pointers and alcohol can be dangerous too! That doesn’t mean you shouldn’t ever program in C or enjoy yourself at a party. Check for NULL, don’t drink and drive, and beware of “free variables.”
Remember, newLISP is an interpreted language. Lutz Mueller, the author of newLISP, made a simple cost/benefit analysis and chose dynamic scope because it’s faster than lexical scope, and because it’s very easy to avoid the potential pitfalls of dynamic scope. Instead of this:
(define (my-unsafe-func)
(println my-var)
)
Do this:
(define (my-safe-func my-var)
(println my-var)
)
It’s a small price to pay for the performance improvement, and oftentimes it’s actually quite handy to have dynamic scope (especially in combination with the no-longer taboo eval). If you need lexical scope though, newLISP has you covered.
Parallel Processing
newLISP takes an interesting approach for running code in parallel. Whereas Clojure uses advanced methods for multi-threading and ensuring safety, newLISP simply uses its small size and lets the Operating System do all the work!
There are no threads. Writing safe, parallel code is simple through actors and spawn/sync because newLISP simply forks itself. Its modest stature makes this a fairly cheap operation, allowing the OS to handle scheduling and memory-protection. Try forking a JVM… π
Excellent Documentation and Community
newLISP’s documentation is one of the best examples of excellent documentation that I’ve ever seen, simply surpassing the documentation for any other LISP that I’m aware of. You don’t need to shell out money for a book to learn it, and that’s because it doesn’t need a book! The short manual included with the reference documentation is all you’ll need to learn the language. Its documentation is one of the primary reasons that I was able to successfully procrastinate for my midterm.
If I had to pick a word to describe newLISP’s community it would probably be “cozy” (and for Common LISP it would probably be “abrasive”). Everyone’s question is heard and answered in a friendly and rapid manner, and there is no formality. Lutz Mueller will often answer your question or incorporate your suggestions directly into the language. It’s a small community, yes, but it’s also agile and capable of rapid change without politics.
Other Goodies
There are many other goodies that I won’t dwell on:
Conclusion + Related Links
newLISP is a true diamond in the rough, sorely under-hyped, but a thing of beauty nevertheless.
If you found this post interesting, you may find some of the links below worth visiting:
