Lisp As An Alternative To Java
Point of View
Lisp as an Alternative to Java
In a recent study Prechelt (1999) compared MB of RAM and ours had only 64 MB; how-
the relative performance of Java and C++ in
ever, none of the programs used all the avail-
execution time and memory usage. Unlike
able RAM, so the results should not have
many benchmark studies, Prechelt compared
changed. Common Lisp benchmarks were run
multiple implementations of the same task by
using Allegro CL 4.3. Scheme benchmarks
multiple programmers in order to control for
were run using MzScheme (Flatt 2000). All
the effects of differences in programmer skill.
the programs were compiled to native code.
Prechelt concluded that “as of JDK 1.2, Java
programs are typically much slower than pro-
Results
grams written in C or C++. They also con-
Figure 1 shows the results of our experiment
sume much more memory.”
We repeated Prechelt’s study using
Lisp as the implementation language.
Our results show that Lisp’s perfor-
mance is comparable to or better than
C++ in execution speed; it also has sig-
nificantly lower variability, which
translates into reduced project risk.
Furthermore, development time is sig-
nificantly lower and less variable than
either C++ or Java. Memory consump-
tion is comparable to Java. Lisp thus
presents a viable alternative to Java for
dynamic applications where perfor-
mance is important.
Experiment
Our data set consists of 16 programs
written by 14 programmers. (Two pro-
grammers submitted more than one
program, as was the case in the original
study.) Twelve of the programs were
written in Common Lisp (Steele
1990), and the other four were in
Scheme (ACM 1991). All of the sub-
jects were volunteers recruited from an
Internet newsgroup.
To the extent possible we duplicat-
ed the circumstances of the original
study. We used the same problem
statement (slightly edited but essential-
ly unchanged), the same program
input files, and the same kind of
machine for the benchmark tests: a
Figure 1: Experimental results. The vertical lines from left to right indicate, respectively, the 10th
SPARC Ultra 1. The only difference percentile, median, and 90th percentile. The hollow box encloses the 25th to 50th percentile. The
was that the original machine had 192
thick grey line is the width of two standard deviations centered on the mean.
i n t e l l i g e n c e • Winter 2000
21
and the data from the original Prechelt study.
the low variability in the results. The standard
The results are presented as cumulative prob-
deviation of the Lisp runtimes was 11 seconds
ability distribution functions. The Y-value at a
versus 77 for C and C++. Furthermore, much
particular point on the curve represents the
of the variation in the Lisp data was due to a
fraction of programs with performance on a
single outlier at 212 seconds (which was pro-
Erann Gat
particular metric that was equal to or better
duced by the programmer with the least Lisp
Jet Propulsion Laboratory,
than the X-value at that point. The horizontal
experience: less than a year). If this outlier is
California Institute of
extent of the curve indicates the range of val-
ignored, the mean is 29.8 seconds, essentially
Technology
ues. A smooth curve indicates evenly distrib-
identical to the median, and the standard
Pasadena, CA 91109
uted values. A curve with discontinuous
deviation is only 2.6 seconds.
gat@jpl.nasa.gov
jumps indicates clustering of the data at the
Memory consumption for Lisp was signifi-
jumps.
cantly higher than for C
Two striking results
and C++ and roughly
are immediately obvious
comparable to Java.
from the figures. First,
However, this result is
development time for the
somewhat misleading for
Lisp seems to offer
Lisp programs was signif-
two reasons. First, Lisp
icantly lower than the
and Java both perform
reduced development
development time for the
internal memory man-
C, C+, and Java pro-
agement using garbage
time and reduced
grams. It was also signifi-
collection, so often Lisp
cantly
less
variable.
and Java runtimes will
variability
Development time for
allocate memory from
Lisp ranged from a low
the operating system that
in performance.
of 2 hours to a high of
is not actually being used
8.5, compared to a range
by the application pro-
of 3 to 25 hours for C
gram. Second, the memo-
and C++ and 4 to 63 hours
ry consumption of Lisp
for Java. Programmer experience cannot
programs includes memory used by the Lisp
account for the difference. The experience
development environment, compiler, and
level was lower for Lisp programmers than for
runtime libraries. This allocation can be sub-
both the other groups (an average of 6.2 years
stantially reduced by removing from the Lisp
for Lisp versus 9.6 for C and C++ and 7.7 for
image features that are not used by the appli-
Java). The Lisp programs were also signifi-
cation, an optimization we did not perform.
cantly shorter than the C, C++, and Java pro-
grams. The Lisp programs ranged from 51 to
Analysis and Speculation
182 lines of code. The mean was 119, the
Our study contains one significant method-
median was 134, and the standard deviation
ological flaw: all the subjects were self-selected
was 10. The C, C++, and Java programs
(a necessary expediency given that we did not
ranged from 107 to 614 lines, with a median
have ready access to a supply of graduate stu-
of 244 and a mean of 277.
dents who knew Lisp). About the only firm
Second, although execution times of the
conclusion we can draw is that it would be
fastest C and C++ programs were faster than
worthwhile to conduct a follow-up study with
the fastest Lisp programs, the runtime perfor-
better controls. If our results can be replicated
mance of the Lisp programs in the aggregate
they would indicate that Lisp offers major
was substantially better than C and C++ (and
advantages for software development: reduced
vastly better than Java). The median runtime
development time and reduced variability in
for Lisp was 30 seconds versus 54 for C and
performance resulting in reduced project risk.
C++. The mean runtime was 41 seconds ver-
Our results beg two questions: (1) Why
sus 165 for C and C++. Even more striking is
does Lisp seem to do as well as it does? and (2)
22
W i n t e r 2 0 0 0 • i n t e l l i g e n c e
Point of View
If these results are real why isn’t Lisp used
posing a matrix represented as a list of lists:
more than it is? The following answers should
(defun transpose (m) (apply ‘mapcar
be considered no more than informed specu-
‘list m))
lation.
The final performance result is the low
When discussing Lisp’s performance we
variability of runtimes and development
need to separate four aspects that have poten-
times—which has several possible explana-
tially different explanations. First, Lisp’s run-
tions. It might be because the subjects were
time performance appears comparable to C
self-selected. It might be because the bench-
and C++. This result contradicts the conven-
mark task involved search and managing a
tional wisdom that Lisp is slow. The simple
complex linked data structure, two jobs for
explanation is probably the correct one: the
which Lisp happens to be specifically designed
conventional wisdom is just wrong. There
and particularly well suited. Or it might be
was a time when Lisp was slow because com-
because Lisp programmers tend to be good
pilers were unavailable or immature. Those
programmers. This in turn might be because
days are long gone. Modern Lisp compilers
good programmers tend to gravitate toward
are mature, stable, and of exceptionally high
Lisp, or it might be because programming in
quality.
Lisp tends to make one a good programmer.
The second performance result is the low
This last possibility is not as outlandish as
development time. Lisp has a much faster
it might at first appear. Lisp has many features
debug cycle than C, C++, or Java. The compi-
that make it an easy language to learn and use.
lation model for most languages is based on
It has simple and uniform syntax and seman-
the idea of a compilation unit, usually a file.
tics. There is ubiquitous editor support to
Changing any part of a compilation unit
help handle what little syntax there is. The
requires, at least, recompiling the entire unit
basic mechanics of the language can be mas-
and relinking the entire program. It typically
tered in a day. This leaves the programmer free
also requires stopping the program and start-
to concentrate on designing and implement-
ing it up again, resulting in a loss of any state
ing algorithms instead of worrying about the
computed by the previous version. The result
vagaries of abstract virtual destructors and
is a debug cycle measured in minutes, often
where to put the semicolons. If Lisp program-
tens of minutes.
mers are better programmers it may be
Lisp compilers, by contrast, are inherently
because the language gives them more time to
designed to be incremental and interactive.
become better programmers.
Individual functions can be individually com-
Which brings us to the question why, if
piled and linked into running programs. It is
Lisp is so great, is it not more widely used?
not necessary to stop a program and restart it
This has been a great puzzle in the Lisp com-
to make a change, so state from previous runs
munity for years. One contributing factor is
can be preserved and used in the next run
that when artificial intelligence fell out of
rather than being recomputed. It is not
favor in the 1980s for failing to deliver on its
unusual to go through several change–com-
lofty promises, Lisp was tarred with the same
pile–execute cycles in one minute when pro-
brush. Another factor is the dogged persis-
gramming in Lisp.
tence of the myth that Lisp is big and slow.
The third result is the smaller size of the
Hopefully this work will begin to correct that
Lisp code. This can be explained by two fac-
problem.
tors. First, Lisp programs do not require type
declarations, which tend to consume many
Conclusions
lines of code in other languages. Second, Lisp
Lisp is often considered an esoteric AI lan-
has powerful abstraction facilities like first-
guage. Our results suggest that it might be
class functions that allow complex algorithms
worthwhile to revisit this view. Lisp provides
to be written in a very few lines of code. A
nearly all of the advantages that make Java
classic example is the following code for trans-
attractive, including automatic memory man-
i n t e l l i g e n c e • Winter 2000
23
agement, dynamic object-oriented program-
contract with the National Aeronautics and
ming, and portability. Our results suggest that
Space Administration.
Lisp is superior to Java and comparable to
PERMISSION TO MAKE DIGITAL OR
C++ in runtime, and it is superior to both in
For information on Java resources, see intel-
HARD COPIES OF ALL OR PART OF THIS
programming effort and variability of results.
ligence (Summer 2000) or visit:
WORK FOR PERSONAL OR CLASSROOM
This last item is particularly significant
tigger.cs.uwm.edu/~syali/ali-links/
USE IS GRANTED WITHOUT FEE PRO-
VIDED THAT COPIES ARE NOT MADE
because it translates directly into reduced risk
OR DISTRIBUTED FOR PROFIT OR COM-
for software development.
References
MERCIAL ADVANTAGE AND THAT
ACM. The revised report on the algorithmic language
COPIES BEAR THIS NOTICE AND THE
Acknowledgments
FULL CITATION ON THE FIRST PAGE.
Scheme (W. Clinger and J. Rees, eds.). 1991. ACM
TO COPY OTHERWISE, TO REPUBLISH,
Thanks to Lutz Prechelt for making available
Lisp Pointers 4, 3, pp. 1–55.
TO POST ON SERVERS OR TO REDIS-
the raw data from the original study and to
Flatt, M. MzScheme Language Manual. 2000. Available
TRIBUTE TO LISTS, REQUIRES PRIOR
Dan Dvorak for calling the Prechelt study to
at http://www.cs.rice.edu/CS/PLT/packages/mzscheme/
SPECIFIC PERMISSION AND/OR A FEE.
© ACM 1523-8822 00/1200 $5.00
my attention. Lutz Prechelt, Dan Dvorak, and
Prechelt, L. 1999. Java vs. C++: Efficiency issues to
Kirk Reinholtz provided comments on an
interpersonal issues. Communications of the ACM
early draft of this paper. This work was per-
(October).
formed at the Jet Propulsion Laboratory,
Steele, G.L. 1990. Common Lisp: The Language. 2nd ed.
California Institute of Technology, under a
Digital Press.
24
W i n t e r 2 0 0 0 • i n t e l l i g e n c e
