REPL-Based Development
A REPL is more than just a REPL window. In this article, I explain the concepts behind REPL-based development, and how REPL-based development means something more interesting than writing code in a REPL window and moving code from a REPL window to source files.
I spent many years writing Common Lisp using LispWorks, which has a very nice REPL-based IDE. And long ago, before the AI winter, I learnt Lisp using Interlisp on Xerox Lisp machines. These environments are built, from the ground up, around the idea of REPL-based development.
For the last several years I've been using Clojure. IDEs for Clojure still fall far short of what I was used to with Lisp, but things are improving.
In this article, I won't be talking about those Lisp IDEs (maybe that's a topic for a future article); rather I will explore what REPL-based development is and how it fits in with other approaches to development.
I apologise for a degree of repetition in the following, but I think the repetition is preferable to the disjointedness that might result if I tried to make things more modular.
A Definition
In Lisps, a form is a thing that can be evaluated. The following are all forms:
42(+ (/ 7 8) (factorial 10))(def the-answer 42)(defn greet [x] (str "Hello " x))
Forms are composed of subforms, so in the above (factorial 10) and 10 are both forms.
Forms include data definitions, function definitions and test definitions.
Two Meanings of the Word “REPL”
The word “REPL” has two meanings:
REPL Window A window that we type forms into. The REPL reads a form, evaluates it, and prints the result. Then it loops and repeats. (“REPL” stands for Read-Eval-Print Loop.)
REPL Process The process that sits behind a REPL window. As well as interacting with this process from a REPL window, we can interact with it from other places. Within our IDE, these other places include editor windows, debuggers, and inspectors. Outside our IDE we can run our code in ways that match the way our deployed application works, for example from a UI or by hitting an HTTP endpoint.
Many people are only thinking of the first of these when they use the word “REPL”.
Using a REPL Window
A REPL window is most useful when exploring relatively simple functionality in the language and in libraries. For example, if we can't remember whether (- x y) means “x minus y” or “y minus x”, we can type (- 7 5) into a REPL window and see the result.
That's a trivial example, but for more advanced functions it can be really useful to play around like this when learning a new-to-us part of the language or a library, or when trying to remember some detail.
This isn't limited to one-liners; it can involve building up a bunch of data and function definitions.
Using the REPL from an Editor Window
When we are working on real code or doing more complicated experimentation, we can fire up an editor window and enter code in a file within our project (perhaps a new file).
When doing this we can make use of the ability to evaluate individual forms before we've saved them to file. We have other options too — see the “Flexibility Over When to Evaluate Things” section.
Feedback from the REPL
After evaluating a form, whether in a REPL window or an editor window, we will want some feedback. The feedback might be obtained in any of the following ways:
In simple cases, when exploring the language or a library, the feedback will be the result of the evaluation.
If we have made a less simple change, we might need to run the new code by evaluating another form somewhere — perhaps a test form. Or we might refresh a UI or hit an HTTP endpoint to see the effects of the change.
If the form is a test definition, then, depending on what test framework we are using, the result may tell us whether the test passes or fails.
Note that this can all be done without saving anything to file. This form-at-a-time evaluation provides us with a useful micro-level feedback loop, but often it will be insufficient for our needs; see the “Flexibility Over When to Evaluate Things” section below for more on this.
Flexibility Over When to Evaluate Things
With REPL-based development we have a high degree of control over when and how our code is evaluated.
Some possibilities are:
We can evaluate a form in a REPL window.
We can evaluate a form in some other tool, such as an editor window. Depending on our IDE, we might be able to evaluate forms in debuggers, inspectors and other tools.
After saving changes, we can load a file from disk. This causes evaluation of all the forms in the file.
We can have something that autoloads changed files and their dependers.
An autoloader might also run and re-run automatic tests. For example, in a project that uses Midje, we can use
midje.repl/autotest.We can run our code in ways that match the way our deployed application works, for example from a UI or by hitting an HTTP endpoint.
Earlier we spoke about feedback from the REPL. That was in the context of evaluating individual forms, but now we can talk about feedback after evaluating larger chunks of code — feedback from our test suite, from UIs and from hitting endpoints.
Note that autloaders complect the idea of saving a file and the idea of evaluation. That's not a bad thing — and it is part of my workflow — but I also like to be able to make use of evaluation without saving to file. See also the “Don't Auto-Save Changed Files" section.
Don't Auto-Save Changed Files
Many people have their editor set up to save files automatically when they make changes.
I advise against this because it takes control away from us, removing the option of using a micro-level feedback loop where we make a small change and evaluate an individual form without saving to file.
Regularly Reload from Scratch
Every now and then, but frequently and probably before every commit, it's important to start with a fresh REPL and re-load everything from our source and test files. If we don't do this we will get caught out by our files being out of step with what we have in the REPL. This can happen when we change names of things, when we delete things and when we move things around.
We can get a fresh REPL by restarting from scratch, but that might be very slow. For example, with Clojure, it might require us to kill our JVM and start a new one. Add on the loading of Clojure, any libraries, and then our application, and the restart time can easily become long enough to break our flow.
To find out how to avoid time-consuming restarts when using Clojure, take a look at Stuart Sierra's Clojure Development Workflow and his later work on the Component library.
REPL-Based Development
REPL-based development means making use of all of the above, so:
We make use of a REPL window for simple exploration.
We build up a program bit by bit in editor windows, and we repeatedly send new and modified forms to the REPL process.
We save our changes to file, and we load and run larger chunks of code in various ways.
We regularly reload from scratch.
These provide progressively coarser-grained feedback loops. (For a short description of how this fits in with even coarser-grained feedback loops, see REPL-Based Development and Feedback Loops.)
REPL-Based Development and Other Approaches
REPL-based development is orthogonal to some other classifications of development approaches. For example, it can be used in conjunction with bottom-up development or top-down development, and with or without TDD. The distinguishing thing is how we interact with our code, not what order things happen in.
REPL-Based vs REPL-Driven
A while ago I heard people using the phrase “REPL-driven development”. When I first heard it, I assumed they meant what I have been describing here as REPL-based development, but it seems that the “driven” meant something different.
Of course, there's no authority that defines what these phrases mean, but, in the discussions I'm referring to, people were using “REPL-driven development” to mean doing real development (rather than just exploration) in a REPL window, and then copying code from the REPL window to editor windows.
I think there's a more interesting way to characterise this process: it's using a REPL window for mini-spikes. So we have REPL-based development driven by mini-spikes in a REPL window, and we have disentangled the “REPL” part from the “driven” part. However, although this may be interesting and accurate, it's not very catchy.
I tend not to do real development in REPL windows, preferring to create and modify real code in editor windows as I have described.
Evaluating Test Definitions
Two popular test frameworks for Clojure, clojure.test and Midje, differ in an interesting way when test definition forms are evaluated.
In Midje, evaluating a fact form causes the test to run, and we see immediately whether it has passed or failed.
In clojure.test, evaluating a deftest form defines the test, but running the test happens separately when we call run-tests.
If we use the workflow I have described, where we evaluate forms as part of a micro-level feedback loop, this makes a difference. If our lowest-level feedback loop involves saving files and running tests using the contents of files, it doesn't make a difference.
For me this is a reason to prefer Midje over clojure.test, but of course there are many other things to take into account if choosing between the two.
Conclusion
This article has explained why a REPL is more than just a REPL window. I hope it has also given food for thought on REPL-based development workflows.
References
I first wrote up some of these thoughts in some messages to the London Clojurians mailing list:
Some of the other discussion at those links is well worth reading.
I've been meaning to write this blog post for more than four years, but haven't been able to face the horrors of my old blogging process and blog host. Finally I've set up a thing that makes it easy and pleasurable to write and publish blog posts. This feels good.