The Zero Bit Stream

Beautiful Quicksort in Common Lisp

steveth45@gmail.com (Stephen A. Goss) — Sat, 01 Sep 2012 00:00:00 -0700

Pre-disclaimer. Angry Haskell nerds: please go away, I didn’t write this for you. This isn’t a X is better than Y piece, just a friendly comparison and maybe a tool to help people understand some functional programming concepts better.

Second pre-disclaimer. This is Quicksort, just not one you’d use in production. You can see it does twice as many comparisons as necessary, or much worse if it encounters a pre-sorted list. The list manipulation (via ++ or APPEND) is doing more work and using more memory than necessary, even for a purely functional implementation.

Beautiful Quicksort in Common Lisp

Quicksort in Haskell vs. Common Lisp

Haskell apologists often cite how Haskell code is terse and expressive. I’ve used Haskell a bit, and while I do not find its syntax very friendly, it is quite powerful. (As a side note, I really like Standard ML, and I wish I could transplant its eagerness and impurity into Haskell.) Common Lisp is a master of multi-paradigm programming and it wears the functional hat pretty well, too. So, let’s see how it stands up against Haskell.

The canonical “beautiful” Haskell function is the quicksort. Here it is:

quicksort :: Ord a => [a] -> [a]
quicksort []     = []
quicksort (p:xs) = (quicksort lesser) ++ [p] ++ (quicksort greater)
    where
        lesser  = filter (< p) xs
        greater = filter (>= p) xs

I gotta admit, that’s a pretty good one. Let’s see what it looks like in Common Lisp.

(defun quicksort (list)
  (when list
    (destructuring-bind (p . xs) list
      (let ((lesser (remove-if-not (lambda (x) (< x p)) xs))
            (greater (remove-if-not (lambda (x) (>= x p)) xs)))
        (append (quicksort lesser) (list p) (quicksort greater))))))

I probably wouldn’t write it this way, but this is the closest approximation of what the Haskell function is doing. Let’s break it down into pieces and show which parts correspond in the two examples.

The Haskell code is using pattern matching to deal with the empty list case, and destructuring to split the list into head and tail.

quicksort []     = []
quicksort (p:xs) = expr

The Common Lisp code is using WHEN on the input (which returns NIL for an empty list and otherwise evaluates the expression) and is using DESTRUCTURING-BIND to split the list into head and tail.

(when list
  (destructuring-bind (p . xs) list
    expr))

Haskell uses “where” to bind local variable in an expression:

... = expr
    where
        binding_a = expr_a
        binding_b = expr_b

Common Lisp uses LET to bind local variables in an expression:

(let ((binding-a expr-a)
      (binding-b expr-b)
  expr)

Haskell uses partial application of the comparison functions with “filter” to filter the lists.

lesser  = filter (< p) xs
greater = filter (>= p) xs

Common Lisp uses lambdas to construct the filtering closures and REMOVE-IF-NOT which is essentially equivalent to Haskell’s “filter”.

(lesser (remove-if-not (lambda (x) (< x p)) xs))
(greater (remove-if-not (lambda (x) (>= x p)) xs))

Even Terser

OK, but there’s an even shorter, one-line Haskell quicksort.

qsort (p:xs) = qsort [x | x<-xs, x<p] ++ [p] ++ qsort [x | x<-xs, x>=p]

Let’s ignore for now the fact that using 1 and 2 letter variables is a very silly way to write “shorter” code. What’s great about this solution, though, is the use of list comprehensions. Common Lisp doesn’t have list comprehensions, but it doesn’t matter because it has mother-flipping macros, and we can add list comprehensions.

Now, I could pull in a library that gives list comprehensions, but wouldn’t it be funner to write the macro right here?

I’m not going to build a general replacement for all Haskell list comprehensions, just the ones here. I see 4 parts, the value, a binding, a list, and a test. Here’s my first shot at a list comprehension macro:

(defmacro @ (value bind list test)
  (let ((newlist (gensym)))
    `(let ((,newlist nil))
       (dolist (,bind ,list)
         (when ,test
           (push ,value ,newlist)))
       (nreverse ,newlist))))

I selected the @ symbol because it’s short and sweet, and mainly just short. The Haskell folks seem to think 1-letter names are the bees knees, so there you go. A quick test:

;; Something like this in Haskell: [x*2 | x<-[1, 2, 3, 4, 5, 6, 7], x>3]
(@ (* 2 x) x (list 1 2 3 4 5 6 7) (> x 3))
;;-> (8 10 12 14)

In case you don’t know how DEFMACRO works, it takes the input forms, unevaluated and returns a list which is evaluated it its place. You can debug macros to see the intermediate form by just doing the transformation with MACROEXPAND:

(macroexpand '(@ (* 2 x) x (list 1 2 3 4 5 6 7) (> x 3)))
(LET ((#:G791 NIL))
  (DOLIST (X (LIST 1 2 3 4 5 6 7)) (WHEN (> X 3) (PUSH (* 2 X) #:G791)))
  (NREVERSE #:G791))

Alright, armed with our new list comprehension macro, let’s write our super-short quicksort:

(defun qsort (l)
  (when l (destructuring-bind (p . xs) l
            (append (qsort (@ x x xs (< x p))) (list p)
                    (qsort (@ x x xs (>= x p)))))))

That’s pretty good. I still have to use WHEN to deal with the null case which Haskell seems to auto-magically handle, and I still have to use DESTRUCTURING-BIND explicitly. I think it holds up to Haskell’s qsort.

More Generic

Oh, I know, the Haskell qsort function applies to lists of any Ord type, which is to say, any type that can be ordered (or at least implements the proper interface) (disclaimer: I don’t pretend to know Haskell-speak when it comes to Classes, Types, Typeclasses, etc. I got it wrong, I don’t care.). Common Lisp ships with CLOS which provides the ability to create generic methods, but it doesn’t really ship with many implemented interfaces. That could be considered a shortcoming in the language, but more properly, it is a shortcoming in libraries. You can easily define generic methods and implement them on all the types you might want to sort, even types from other packages and libraries.

(defgeneric lt (some other))

(defmethod lt ((some number) (other number)) (< some other))

(defmethod lt ((some string) (other string)) (string< some other))

(defun qsort (l)
  (when l (destructuring-bind (p . xs) l
            (append (qsort (@ x x xs (lt x p))) (list p)
                    (qsort (@ x x xs (not (lt x p))))))))

(qsort (list "this" "is" "only" "a" "test"))
;;-> ("a" "is" "only" "test" "this")

(qsort (list 12/11 -1.09 3000))
;;-> (-1.09 12/11 3000)

So we can be significantly generic with Common Lisp, too. No, it doesn’t have Haskell’s compile time type-checking goodness (or badness, however you want to slice it), it is a fundamentally dynamically typed language. But I think it matches Haskell in expressivity, and what it lacks in terse syntax, it makes up for with flexibility through macros, generic methods, and dynamism.

In conclusion: I love Haskell, please don’t shoot me. Also, Common Lisp is pretty neat, too. Both are definitely worth learning and excellent tools for functional programming.

Concurrency in Common Lisp with ChanL

steveth45@gmail.com (Stephen A. Goss) — Sun, 26 Aug 2012 00:00:00 -0700

Concurrency in Common Lisp with ChanL

Rob Pike did a great talk at Google on concurrency in Newsqueak. The gist of it is that you spawn concurrent tasks in separate lightweight processes and synchronize by communicating through channels. That stuff worked its way into the programming language Go. After seeing that video I wanted to try it out myself in Common Lisp, of course. It turns out there is a Quicklisp-installable library called ChanL that provides just that functionality. I don’t know how lightweight the threads are, but the interface is similar.

A simple, but effective example of using channels is to perform a number of IO-bound tasks in parallel, such as retrieving several web pages. It’s surprisingly simple to do in parallel. To start, I’ll define a little timing macro:

(defmacro time-it (&body body)
  (let ((start-time (gensym)))
    `(let ((,start-time (get-internal-real-time)))
       ,@body
       (format t "Runtime: ~a milliseconds.~%" (- (get-internal-real-time) ,start-time)))))

Then, let’s load the ChanL library and Drakma (for simple HTTP requests) using quicklisp:

(ql:quickload "drakma")
(ql:quickload "chanl")

Our example set of URLs:

(defparameter *urls*
  (list
    "http://blog.thezerobit.com/"
    "http://quicklisp.org/"
    "http://www.cliki.net/index"
    "http://sbcl.org/"
    ))

Now, here’s a function that performs a single HTTP request and times it.

(defun do-request (url)
  (let ((start-time (get-internal-real-time)))
    (format t "Starting request: ~a~%" url)
    (drakma:http-request url)
    (let ((elapsed (- (get-internal-real-time) start-time)))
      (format t "Completed request in ~a ms: ~a~%" elapsed url))))

This is what you might do without concurrency if we wanted to perform these HTTP requests:

(time-it (dolist (url *urls*)
           (do-request url)))
;; Starting request: http://blog.thezerobit.com/
;; Completed request in 616 ms: http://blog.thezerobit.com/
;; Starting request: http://quicklisp.org/
;; Completed request in 819 ms: http://quicklisp.org/
;; Starting request: http://www.cliki.net/index
;; Completed request in 429 ms: http://www.cliki.net/index
;; Starting request: http://sbcl.org/
;; Completed request in 291 ms: http://sbcl.org/
;; Runtime: 2155 milliseconds.

It takes over 2 seconds to make all these requests in serial fashion. There may be a way to queue up these requests in parallel with the Drakma library, but instead we’ll just use the ChanL library. Let’s define a function, like DO-REQUEST that instead of printing its progress will send messages over a channel.

(defun do-request-chan (url chan)
  (let ((start-time (get-internal-real-time)))
    (chanl:send chan (format nil "Starting request: ~a~%" url))
    (drakma:http-request url)
    (let ((elapsed (- (get-internal-real-time) start-time)))
      (chanl:send chan
                  (format nil "Completed request in ~a ms: ~a~%" elapsed url)))))

We can use the PEXEC call to spawn separate processes for each HTTP request and then wait for the 8 messages to come in on the channel we create to synchronize.

(time-it
  (let ((chan (make-instance 'chanl:channel)))
    (dolist (url *urls*)
      (chanl:pexec () (do-request-chan url chan)))
    (dotimes (x 8)
      (format t (chanl:recv chan)))))
;; Starting request: http://blog.thezerobit.com/
;; Starting request: http://quicklisp.org/
;; Starting request: http://sbcl.org/
;; Starting request: http://www.cliki.net/index
;; Completed request in 291 ms: http://blog.thezerobit.com/
;; Completed request in 302 ms: http://www.cliki.net/index
;; Completed request in 306 ms: http://sbcl.org/
;; Completed request in 703 ms: http://quicklisp.org/
;; Runtime: 703 milliseconds.

Here, the whole operation takes only as long as the longest single request and the overhead of spawning separate threads is negligible compared to the time saved by making these requests concurrently. It is important to note that, by default, the channels are not buffered so calls to SEND will block until there’s a RECV called on the same channel in a different thread. ChanL also provides buffered channels and some other goodies which I haven’t touched on here.

Lazy Sequences in Common Lisp

steveth45@gmail.com (Stephen A. Goss) — Sat, 28 Jul 2012 00:00:00 -0700

Lazy Sequences in Common Lisp

0. Delayed Evaluation with Closures

Last week I expressed some anti-lazy sentiment, or at least it was perceived as such. Really, I am just not a fan of laziness by default. Common Lisp is decidedly not lazy by default, but it is easy to defer or avoid computation with closures. With macros, we can add it to the language in an easy-to-use form.

Common Lisp is not a lazy language, but it is easy enough to add lazy sequences. We can model them after the normal list which is constructed piecemeal with the cons operator. The following recursive function builds a list of numbers counting down to 0 using cons.

(defun countdown (n)
  (if (>= n 0)
    (cons n (countdown (1- n)))))

(countdown 10)
;;--> (10 9 8 7 6 5 4 3 2 1 0)

(reduce #'+ (countdown 100))
;;--> 5050

This is a common pattern, but there are some shortcomings. First of all, it’s not a tail recursive function, so this call will always build the stack since tail call optimization or tail recursion elimination does not apply, and will do so linearly with the argument n. Also, it constructs the entire list. You may want this list of numbers, but you don’t need all of them right away sitting in memory. In the example above, the call to reduce gets a full list of 101 elements, from 100 to 0 before processing. Depending on the length of the sequence, a different algorithm might work better.

One way (of many) to construct lazy sequences in Common Lisp is to store the CDR (second part) of the CONS cell as a thunk, or a closure that takes no argument and computes the next CONS in the list.

(defun lazy-countdown (n)
  (if (>= n 0)
    (cons n (lambda () (lazy-countdown (1- n))))))

(lazy-countdown 10)
;;--> (10 . #<CCL:COMPILED-LEXICAL-CLOSURE (:INTERNAL LAZY-COUNTDOWN) #x30200F59400F>)

We can grab the first element of the lazy list with CAR per normal, but to get the next CONS cell, we need a version of CDR which FUNCALLs the thunk. That will call LAZY-COUNTDOWN which will return another CONS cell.

(defun lazy-cdr (lazylist) (funcall (cdr lazylist)))

(lazy-cdr (lazy-countdown 10))
;;--> (9 . #<CCL:COMPILED-LEXICAL-CLOSURE (:INTERNAL LAZY-COUNTDOWN) #x30200F55B7BF>)

So far, so good. Now we need a version of REDUCE which uses LAZY-CDR instead of CDR, and can reduce a lazy list for us.

(defun lazy-reduce (f lazylist)
  (labels ((inner-reduce (lazylist acc)
             (if lazylist
               (inner-reduce (lazy-cdr lazylist) (funcall f acc (car lazylist)))
               acc)))
    (inner-reduce (lazy-cdr lazylist) (car lazylist))))

(lazy-reduce #'+ (lazy-countdown 1000))
;;--> 500500

1. Use CLOS and Macros for a Nice Interface

In order to use this pattern easily, we will create a nice interface. We’ll package up the lazy list into its own class, which I will call LCONS and use a macro of the same name to construct lazy lists by automatically wrapping the second argument in a thunk.

(defclass lcons ()
  ((val :initarg :val)))

(defmacro lcons (head tail)
  `(make-instance 'lcons
                  :val (cons ,head (lambda () ,tail))))

For our interface, I am going to use methods HEAD, TAIL and EMPTY? to approximate CAR, CDR, and NULL and make them compatible with normal lists.

(defgeneric empty? (lcons))
(defgeneric head (lcons))
(defgeneric tail (lcons))

(defmethod empty? ((lcons lcons))
  nil)

(defmethod empty? ((lcons list))
  (null lcons))

(defmethod head ((lcons lcons))
  (car (slot-value lcons 'val)))

(defmethod head ((lcons list))
  (car lcons))

(defmethod tail ((lcons lcons))
  (funcall (cdr (slot-value lcons 'val))))

(defmethod tail ((lcons list))
  (cdr lcons))

Now, we can redefine LAZY-COUNTDOWN, just the same as the original COUNTDOWN but use LCONS instead of CONS.

(defun countdown (n)
  (if (>= n 0)
    (cons n (countdown (1- n)))))

(car (cdr (countdown 10)))
;;--> 9

(defun lazy-countdown (n)
  (if (>= n 0)
    (lcons n (countdown (1- n)))))

(head (tail (lazy-countdown 10)))
;;--> 9

Now, we can define functions to do things that are normally only for regular lists that work with lazy lists, also. A good candidate is MAPCAR. Here’s a version that works with lazy lists:

(defun lmapcar (f &rest lists)
  (if (notany #'empty? lists)
    (lcons (apply f (mapcar #'head lists))
           (apply #'lmapcar f (mapcar #'tail lists)))))

(head (lmapcar #'+ (lazy-countdown 10) (countdown 10)))
;;--> 20

Notice that LMAPCAR is lazy, too. It returns an LCONS (lazy list). In this way we can build up a collection of functions just like the normal functions on lists in Common Lisp, but ones that take and return lazy lists.

Just like lazy sequences in other languages, these sequences can be infinite. Here’s a function INTEGERS which builds an infinite lazy list of integers.

(defun integers (n) (lcons n (integers (1+ n))))

(head (tail (tail (integers 0))))
;;--> 2

I have encoded this into a project which is available on github: slow-jam . It is the above code with a few extra functions that take and/or return lazy sequences. Here are a few examples of things you can do with the library:

;; infinite sequences that print nicely on the REPL
(range)
;;--> (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ETC...)

(take 10 (range))
;;--> (0 1 2 3 4 5 6 7 8 9)

;; FILTER is lazy
(take 5 (filter #'evenp (range)))
(0 2 4 6 8)

One advantage of this approach is that the tail of the lazy list is never cached which means that, unless you use the TO-LIST function provided in the library to turn a lazy list into a normal list, all the elements of your lazylists (except the first) can be garbage collected. These lists never take much permanent memory. The downside is that results are not cached, and you may end up recalculating the same list elements more than once. For that reason, these lists should be built in using side-effect-free functions. If you do want to cache the contents of a lazy list, just call TO-LIST which returns real list and keep a reference to that.

This is just one implementation of lazy sequences in Common Lisp. It is partially inspired by Clojure, Haskell, and SICP. If you like the slow-jam library and want it to be better, report an issue or make a pull request at github.

Immutable Persistent Data Structures in Common Lisp

steveth45@gmail.com (Stephen A. Goss) — Sat, 21 Jul 2012 00:00:00 -0700

Immutable Persistent Data Structures in Common Lisp

0. The Rationale

Clojure, Scala, and Haskell (and other languages) have recently brought the idea of immutable (and persistent) data structures into some amount of popularity. This is not a new idea, by any means, as Lisp has always supported this with its list structure, which is essentially a linked list. In Common Lisp and Scheme, lists are actually mutable but in practice, this is rarely a problem because it is generally considered bad form to go around mutating lists except under certain circumstances. Most Lisp books, tutorials and references encourage using lists in a functional style (that is, assuming and treating them as immutable structures).

In Clojure, the big three data structures are the hashmap, vector, and set, which each come with their own literal syntax. Clojure also has normal lispy lists, too with language enforced immutability. These immutable data structures can be approximated with normal lists in Common Lisp with the caveat that they don’t retain the more efficient performance characteristics of Clojure’s data structures. There are a few libraries for Common Lisp which provide these structures with similar time and space complexity as Clojure’s implementations. The one that I recommend is FSet which, according to the Wayback Machine, has been around since at latest 2007.

What’s the point of learning to use these data structures in Common Lisp? Isn’t Clojure better? Well, in a lot of ways, Clojure is a better language and environment, but in a lot of ways, Common Lisp is better, too. I enjoy using both languages. I’m not quite lucky enough to have a day-job that allows me write much code in a Lisp, so I use it for fun. Therefore, my criteria for languages that are fun tend to push me toward Common Lisp. I generally use SBCL or ClozureCL as implementations which are native (not bound the the JVM) and have a much faster startup time and interact easier with native libraries. I don’t care much for the JVM, and I prefer native libraries to JVM libraries, which are themselves, often just JNI wrappers on native code. Tracebacks in SBCL and CCL are much easier to read than what you get with Clojure. I also prefer non-lazy to lazy.

An aside on laziness: I like the idea of lazy collections, or iterators, or generators, or streams (from SICP). They can be useful for certain constructions, but they don’t really enable anything amazing that you can’t do otherwise, just with a slightly different algorithm. It’s slower. Every implementation of Standard ML (not lazy) I’ve tried has varied from just being somewhat to several times faster than Haskell where all computation is lazy. I think setting up all that delayed computation is expensive, and the only way I found to make Haskell perform within the same ball park as Standard ML, was to add hints here and there to get computations to go ahead and happen instead of building a stack of thunks. I don’t have to encourage Standard ML to do a computation, it just does it. Same with Common Lisp. Life is much simpler and faster when computations just happen and nothing is lazy by default.

So, Clojure has its advantages. One of those is a slight reduction in the number of parens with certain standard Lisp constructions like LET and COND. Also, the collection literals are nice. Clojure is a Lisp-1, so the shared namespace is generally better than Lisp-2’s like Common Lisp. Even so, there are nice things about split function namespace, that is that you don’t have to worry about shadowing built in functions. This is a problem in languages with a single namespece. I don’t know how many programs in Python I’ve seen that shadow the built in “id” function. It turns out that’s a really popular name for a database column, and hence a variable name. In Common Lisp, you can name your variable “list” even through there is a standard function called “list”, none of this “lst” crap.

TLDR; It’s all a matter of taste, but for me Common Lisp is just a little more fun to hack in than Clojure.

1. The List

I’ll try to keep this short. A Common Lisp list is a linked list constructed of CONS cells. Each CONS has two parts the CAR and the CDR. The CAR generally contains a list element and the CDR contains the next CONS cell, or NIL, signifying the end of the list. An empty list is the same as NIL.

;; contruct a list with a single element, the keyword symbol :x
(cons :x nil)
;;--> (:x)

;; alternately:
(list :x)
;;--> (:x)

;; or with quoting (bypass evaluation):
(quote (:x))
;;--> (:x)

;; shorthand for quoting is a single quote mark
'(:x)
;;--> (:x)

;; CAR and CDR extract the cells of the CONS
(car '(:x))
;;--> :x
(cdr '(:x))
;;--> NIL

;; A list of three elements:
(cons :x (cons :y (cons :z nil)))
;;--> (:x :y :z)
;; Also: (list :x :y :z) or '(:x :y :z)

So the way to use lists as an immutable, persistent data structure is that you can construct a new list with an element added to the front of the list without affecting the original, and the “tail” of the list is shared. Also, you can get the list minus the front element, just by taking the CDR of the list.

(let ((our-list (list 1 2 3)))
  (list
    (cons 0 our-list)
    (cons 1 our-list)
    (cdr our-list)
    our-list))
;;--> ((0 1 2 3) (1 1 2 3) (2 3) (1 2 3))

In that example I took a list and first constructed a new list by adding a 0 to the front, then another new list by adding a 1 to the front, then I got a list without the first element, and then the original list, unscathed. No list was mutated and no space was wasted, as all these lists share the same last 2 or 3 CONS cells.

2. Vector / SEQ

Of course, you might want to do something other than have an ordered sequence with the ability to add or remove elements to the front. In Clojure, you’ve got the vector, which is an ordered sequence that allows you to add or remove elements anywhere in the list, and lookup arbitrary elements in O(log n) time and with shared structure. You can do the same thing in Common Lisp with normal lists but some of the operations will take O(n) time complexity and some operations will return a brand new list with no shared structure. Let’s start with that and then show how to get Clojure-style complexity characteristics with the SEQ collection provided by the FSet library. For small enough sequences you might just want to use normal lists, and avoid the FSet dependency.

In none of the following examples is any collection mutated, instead the value of the expression must be captured and used instead of the original, so when I say “drop an element”, I really mean “return a new collection without an element”, etc.

You can remove an element from the end of a list with BUTLAST, a funny sounding function which basically creates a whole new list minus the last element. You can also specify a number of elements to remove from the end of the list.

(butlast (list 1 2 3))
;;--> (1 2)

(butlast (list 1 2 3 4 5) 2)
;;--> (1 2 3)

You can get the NTH element of a list:

(nth 0 (list 1 2 3 4 5))
;;--> 1

You can take a slice of a list with SUBSEQ:

(subseq (list :a :b :c :d) 1 3)
;;--> (:B :C)

Or drop the first n elements with NTHCDR:

(nthcdr 4 (list :a :b :c :d :e :f))
;;--> (:E :F)

If you want to drop or add or change an element from the middle of the list you might have to write or use one these simple functions to do that:

(defun drop-nth (n list)
  (append (subseq list 0 n) (nthcdr (1+ n) list)))

(drop-nth 3 (list :a :b :c :d :e :f))
;;--> (:A :B :C :E :F)

(defun add-nth (n elem list)
  (append (subseq list 0 n) (cons elem (nthcdr n list))))

(add-nth 3 :bar (list :a :b :c :d :e :f))
;;--> (:A :B :C :BAR :D :E :F)

(defun set-nth (n elem list)
  (append (subseq list 0 n) (cons elem (nthcdr (1+ n) list))))

(set-nth 3 :foo (list :a :b :c :d :e))
;;--> (:A :B :C :FOO :E)

Now if you want the tree-structured, persistent version of that, use FSet’s SEQ. If you are using Quicklisp (which I hope you are), you can load up FSet in the REPL like this:

(ql:quickload :fset)
(in-package :fset-user)

There is a function, EMPTY-SEQ that creates an empty SEQ, and there’s a macro SEQ which can be used to construct a SEQ with multiple elements:

(empty-seq)
;;--> #[ ]

(seq :a :b :c)
;;--> #[ :A :B :C ] ;; this is how FSet pretty-prints sequences in the REPL

You can get the first or last or nth element with FIRST, LAST and @:

(first (seq :a :b :c))
;;--> :A

(last (seq :a :b :c))
;;--> :C

(@ (seq :a :b :c) 1)
;;--> :B

You can add or remove elements at the beginning or end of sequence with WITH-FIRST, WITH-LAST, LESS-FIRST, and LESS-LAST:

(with-first (seq :a :b :c) :foo)
;;--> #[ :FOO :A :B :C ]

(with-last (seq :a :b :c) :foo)
;;--> #[ :A :B :C :FOO ]

(less-first (seq :a :b :c))
;;--> #[ :B :C ]

(less-last (seq :a :b :c))
;;--> #[ :A :B ]

You can replace an element, drop an element, or insert an element at a particular index with WITH, INSERT and LESS:

(with (seq :a :b :c) 1 :foo)
;;--> #[ :A :FOO :C ]

(insert (seq :a :b :c) 1 :foo)
;;--> #[ :A :FOO :B :C ]

(less (seq :a :b :c) 1)
;;--> #[ :A :C ]

2. Set

In Common Lisp lists are also used to emulate sets. Not all operations will have the optimum time complexity, but it’s generally adequate for most purposes.

Common Lisp has various equality functions: EQ, EQL, EQUAL, and EQUALP. I will not summarize them here, but for the sake of consistency with Clojure, we’ll probably want to consider two elements to be equal with EQUALP. Many Common Lisp functions take an optional :TEST parameter to specify which equality test function to use. All of the set theory related functions do.

To test if a value is present in a set (really, just a list that we’re treating as a set), there is the MEMBER function. It will return NIL if the value is not present (which is the only “false” value in Common Lisp) or the subset of the list starting with that value (which will be treated as “true” in conditional statements, etc) if it is present.

(member :a (list :a :b :c) :test #'equalp)
;;--> (:A :B :C)
(member :d (list :a :b :c) :test #'equalp)
;;--> NIL
(member '("asdf") (list '("asdf") :b :c) :test #'equalp)
;;--> (("asdf") :B :C)

You can get the union or intersection of two sets with UNION and INTERSECTION:

(union (list :a :b :c) (list :d :e :a) :test #'equalp)
;;--> (:C :B :D :E :A)
(intersection (list :a :b :c) (list :d :e :a :b) :test #'equalp)
;;--> (:B :A)

You can get all the elements from one list that do not exist in another list with SET-DIFFERENCE.

(set-difference (list :a :b :c) (list :d :e :a :b) :test #'equalp)
;;--> (:C)

You can even check if a set is a subset of another set with SUBSETP.

(subsetp (list :a :b :c) (list :d :e :a :b) :test #'equalp)
;;--> NIL

(subsetp (list :a :b :c) (list :d :e :c :a :b) :test #'equalp)
;;--> T

The set of elements that exist in only one of two sets is constructed with SET-EXCLUSIVE-OR.

(set-exclusive-or (list :a :b :c) (list :d :e :a :b) :test #'equalp)
;;--> (:E :D :C)

Adding and removing elements from a list is accomplished with CONS and REMOVE:

(cons :foo (list :a :b))
;;--> (:foo :a :b)

(remove "elem" (list "foo" "bar" "elem") :test #'equalp)
;;--> ("foo" "bar")

It’s clear that Common Lisp was designed to use lists as sets when necessary, but under certain conditions you might want to use a more efficient implementation (say if you wanted to do a lot of membership tests in a larger set, and would benefit from O(log n) membership test instead of O(n)) you can use FSet’s SET. You can construct an empty set with EMPTY-SET or pre-populate with the SET macro (which shadows the archaic and generally unused SET function built into Common Lisp).

(empty-set)
;;--> #{ }

(set :a :b :c)
;;--> #{ :A :B :C }

The FSet library comes with an equality function that is used automatically: EQUAL?. It is slightly better than EQUALP in that it will find two sets equal that are equivalent sets (same for other FSet collections).:

(equalp (list :a :b :c) (list :b :c :a))
NIL

(equal? (set :a :b :c) (set :b :c :a))
T

Membership is tested with CONTAINS?:

(contains? (set :a :b :c) :a)
;;--> T

(contains? (set :a :b :c) :d)
;;--> NIL

UNION, INTERSECTION, and SET-DIFFERENCE work as expected:

(union (set :a :b) (set :b :c))
;;--> #{ :A :B :C }

(intersection (set :a :b) (set :b :c))
;;--> #{ :B }

(set-difference (set :a :b) (set :b :c))
;;--> #{ :A }

Two tests exist: SUBSET? and DISJOINT?.

(subset? (set :a :b) (set :b :c :a))
;;--> T

(subset? (set :d :b) (set :b :c :a))
;;--> NIL

(disjoint? (set :a :b) (set :c :d))
;;--> T

(disjoint? (set :a :b :c) (set :c :d))
;;--> NIL

Adding and removing elements from a SET is accomplished with WITH and LESS:

(with (set :b :c) :a)
;;--> #{ :A :B :C }

(less (set :a :b :c) :a)
;;--> #{ :B :C }

3. Map

There are two ways lists are used in Common Lisp to create mapping collections.

The plist is just a list with every two elements representing a key and a value. It has one useful function, GETF which uses EQ for key equality and no way to use any other comparison, making it only useful for using symbols for keys:

(getf (list :name "Steve" :age 33) :name)
;;--> "Steve"

(getf (list :name "Steve" :age 33) :weight)
;;--> NIL

(getf (list :name "Steve" :age 33) :weight "some default")
;;--> "some default"

Mappings can be changed or added by appending a key/value pair to the front of the list since GETF searches from the front:

(getf (append (list :name "Stephen") (list :name "Steve" :age 33)) :name)
;;--> "Stephen"

You could easily build a repertoire of functions to do all the common operations you might want with plists, but there is another form called the alist which comes with a better set of operations built in. An alist is a list of cons cells with the key in the CAR and value in the CDR. When you have a CONS cell with a non-list in the CDR, it is called a dotted list and looks like this: (car . cdr). Alists can be constructed in a variety of ways:

(list (cons :key "val") (cons :other-key "other-val"))
;;--> ((:KEY . "val") (:OTHER-KEY . "other-val"))

'((:key . "val") (:other-key . "other-val"))
;;--> ((:KEY . "val") (:OTHER-KEY . "other-val"))

(pairlis (list :key1 :key2 :key3) (list "val1" "val2" "val3"))
;;--> ((:KEY3 . "val3") (:KEY2 . "val2") (:KEY1 . "val1"))

The lookup function for alists is ASSOC which takes the :TEST parameter for key equality though it returns the entire matching CONS cell, not just the value. You can use CDR to extract the value.

(assoc :key '((:key . "val") (:other-key . "other-val")) :test #'equalp)
;;--> (:KEY . "val")

(cdr (assoc :other-key '((:key . "val") (:other-key . "other-val")) :test #'equalp))
;;--> "other-val"

(assoc :missing-key '((:key . "val") (:other-key . "other-val")) :test #'equalp)
;;--> NIL

Like plists, key/val pairs can be added to alists by adding them to the front of the list. There is a helper function for this called ACONS:

(acons "foo" "bar" '(("baz" . "val")))
;;--> (("foo" . "bar") ("baz" . "val"))

There’s no builtin function to remove keys from an alist, but we can write one easily enough:

(defun alist-remove (alist key)
  (remove-if #'(lambda (cell) (equalp key (car cell))) alist))

(alist-remove '((:a . 100) (:b . 200)) :a)
;;--> ((:B . 200))

Alists are probably an 80% solution, which is plenty for most situations, but FSet’s maps are quite nice and give you that O(log n) lookup and update time complexity with the structural sharing that we’ve grown to love. I’ve found them as useful as Clojure’s hashmaps. Like the other collections, there’s a function EMPTY-MAP that’s self explanatory and a macro MAP for constructing larger maps:

(empty-map)
;;--> #{| |}

(map (:a 100) (:b 200))
;;--> #{| (:A 100) (:B 200) |}

Like SETs, elements can be added or removed or accessed from FSet MAP collections with WITH, LESS and @.

(with (map (:a 100)) :b 200)
;;--> #{| (:A 100) (:B 200) |}

(less (map (:a 100) (:c 200)) :c)
;;--> #{| (:A 100) |}

(@ (map (:a 100) (:c 200)) :c)
;;--> 200

4. Conclusion w/bonus Feature

I only scratched the surface with the FSet library. There are more functions than the ones I listed and some of the ones I describe actually return multiple values, for example to signify if a key is present in case you are storing NIL as values, etc. I recommend the FSet Tutorial for more details.

Lists in Common Lisp are really quite versatile and can approximate most other structures for limited sizes. The nice thing about the FSet library is that it always puts the collection as the first argument, so it is amenable to Clojure’s threading macro: -> which I will translate here for your benefit into Common Lisp:

(defmacro -> (x &optional (form nil form-supplied-p) &rest more)
  (if form-supplied-p
    (if more
      `(-> (-> ,x ,form) ,@more)
      (if (listp form)
        `(,(car form) ,x ,@(cdr form))
        (list form x)))
    x))

(-> (empty-map)
  (with :a 100)
  (with :b 200)
  (less :a))
;;--> #{| (:B 200) |}