Editor remote REPL into production

that's awesome and terrifying. — Dan Sutton

What are we talking about

Lispy languages have a REPL. That is like a console or a terminal into a running application, which sounds familiar at first but it is quite a different beast. When you just query for a variable, or set one, there is not much difference to a terminal or shell. But if you evaluate a function definition the function gets compiled into the running code. This is how Lispers, Schemers, Racketeers and Clojurians develop their programs. In the beginning you do this from the repl itself. But that is about the same stage as having learned to use the shell and progress to writing scripts.

Editor connected nREPL

When you start serious work you use an editor, in my case emacs, to write the code. The editor, in my case the awesome Cider package, does things like code highlighting, completion etc. And in some languages like C, Shell, Java, Python it has a button to run your code, possibly compiling and linking it in the process. In Clojure/Clojurescript that happens so rarely, that I don't even know what the command for that would be. When I load a project it happens automatically on startup.

Once.

No second time.

But I edit my code and have commands to evaluate the current expression, or the current buffer. To know the current value of a variable in the running application I just put the cursor there and eval. To update a function, I eval…

Functional programming and hot reloadable code

So called object oriented languages do not lend themselves to this workflow, because like hiding eastereggs in the garden for the children to find, object oriented code often hides its state in different objects in the code. If you tried to reload such an object you would loose the bit of state hidden in it. Functional programming has many functions that get applied to data and your state is in the data, not in the code. With your stateless functions and objects you can reload them in a running application without messing up your state. Of course there is limits to that. Mastery is to know where that limit lies.

The discipline of hot reloadable function composition

On the server backend I can connect to a database, eg. datomic, datascript or datahike, that is immutable and behaves like a big functional datastructure. This behaves like being inside and part of the application, which in the case of datascript it is, while datahike keeps it in memory but also persists it to disk in the background. Datomic is the closed source database that brought us this api. These databases lend themselves to serve as graph databases and are perfectly suited to the above mentioned style.

The webserver part is build from a composable functional stack like ring. So it, too, fits the bill.

Then we need something to make the ring-stack as a whole remountable. I use mount.

I use fulcro and pathom with this whole setup copied from the fulcro template.

Development

Until now I have not described anything new to a clojurian. This is one of the ways to develop an application. What I will only mention as a sidenote in this post, is that in development I run two REPLs, one into the development backend on my laptop and one into the browser frontend. I have the same functionality when connecting to the browser and to the jvm. I compile or reload per buffer or per function and can share code between the backend and frontend. But this is for another post. And we have not touched on the subject for this post jet.

An nREPL into Production

When you compile functions into code you can't make any mistakes the compiler catches. If you get an error, it just will not compile, will not be inserted and nothing is lost. But you can, of course, mess up.

So where and how does it make sense to use this power?

The Database Cases

Schema updates

The world changes, so do databases and their schemas. To do schema updates I have a namespace that contains the database schema. Then I wrote a function update-schema that does not blindly install this but reads the schema from the db, diffs my schema and the installed one and then installs the missing parts. This function can run on startup. But guess what… It is perfectly save to run on a live application. So if I changed the frontend code and added an entity type to the schema I can:

  1. Call all users and tell them to keep their fingers of for 5 minutes while I restart or
  2. Call the function in a running application.

The danger of course being that the code the application runs from and the installed schema get out of sync. But this is not how my work flowed. I write a new feature in development and test it. Then I push the code into production. So it is now, before the restart, that the application is out of sync with its code. It is just that instead of restarting the application I navigate to the namespace in emacs and either eval the changed function or the whole buffer. If other namespaces need to be reloaded mount is my friend.

Database manual house keeping

Give a database to users and you will soon need to do some housekeeping. Merging double entries etc. If you need to just delete something you could do that with a seperate connection to the same database. The drawback here being, that all helper functions you wrote to keep your data sane will need to be copied from your project to the second admin project. So you just connect to the live system, and query for data with emacs c-u c-x c-e and the data gets inserted into your buffer. From there you massage it and check it and transact or retract as needed. An editor connected nrepl into production is perfect for this kind of work.

Database development of housekeeping functions

When I develop database accessing functions I normaly query and pull as described above, but I do not let them change the database. Instead of transact I write println, or log/debug. Of course I develop these functions locally, not on the production system but when I am confident enough to change the println to transact I use sesman to switch my buffer to the production system with the println version and I get a printout what my new function does to the actual, real, life production database data. It does not transact, mind you, it just prints.

Step two, of course, is to save functions like these into the code, to add to the repertoire of manual house keeping functions above.

Safety first

  • One of the first functions you might write is spit-db-export-to-disk.
  • You should make special backups before you touch anything.
  • Checkout ZFS Snapshots. Snapshots in the Filesystem are not backups, if the Disks crash they are gone, but you can do a lot more things with a lot more confidence if you can always roll back. That's another reason I love FreeBSD.

The Report Case

Fast iteration

I just take the report as an example because I use fulcro and deliver an SPA to users. This is, of course, the same case as the server serving a classical html page instead of the SPA basepage. Lets say a user needs a report.

That means for example:

  • having a route in the ring stack
  • parsing the route for parameters
  • collecting the data, propably through db queries and pull
  • transforming the data
  • piping the data through some sort of template or build a representation
  • add a cover for the report
  • create plots for graphical representations
  • build a pdf
  • stream the pdf via ring

You just get rough requirements from the user. Not as a lack of due dilligence, but as good practice, because only after a few iterations will both you and the user know, what they really needed. And when it is done, you normally see that you could not have guessed the solution beforehand.

So just get it roughly working, use git to push to production. Apart from you and the user(s) who is your 'partner in development' no one has to know about the link.

Really fast iterations and feedback for fine tuning

Now iterate with the user until it satisfies all stakeholders. You can push new features to production but there is no need to restart the application and cause downtime. Structure your Code so the report is in one namespace and reload this from your emacs.

This way iterations become really fast. You have the collaborating user on the phone. The user used the link and has the pdf open in the browser. You decide there needs to be a total somewhere in the report. You change that in the editor, eval with c-x c-e and tell the user to press F5. The user sees the new report.

And the Names of x needs to be fat: edit, c-x c-e, F5.

No, that is too fat take that back …

Welcome to agile development.

This is so tremendously valuable because the user sees the report with live production data. The data you thought up to test with will always be different in subtle ways, except when you work on production data too, which you can't copy if it is people related, because this would violate data protection laws.

Some Code

I worked with an ueberjar at first. Since the project contains the server and SPA code, the ueberjar pulls in the whole clojurescript ecosystem including google closure etc. So every single push to production was about 70 MB. Now I just git push to the host of the jail. Since I do not like or trust containers I put the production application in its own FreeBSD jail, behind a firewall and reverse proxy, unreachable from the outside internet.

Also watching the log etc. does not require ssh into the jail, which is not reachable on the network layer from outside. ssh into the FreeBSD host is enough.

The remote Cider nREPL

The documentation for embedding the nrepl-server into production, https://docs.cider.mx/cider-nrepl/usage.html, mentions bug #447, which bit me a lot.

Then I realised that clj -m nrepl-cmdline is already provided. Thank you to everyone who implemented this! This can just be added to the project, so it accepts an optional nrepl port.

So after I start my server with mount, I call nrepl.cmdline/-main and pass through the command line options my app got called with. That means my own app now takes the additional command line args of nrepl-cmdline.

Extra deps in deps.edn

{:prod     {:extra-deps {com.fulcrologic/fulcro {:mvn/version "3.1.8"}
                         nrepl                  {:mvn/version "0.6.0"}
                         cider/cider-nrepl      {:mvn/version "0.24.0"}}}}

A line added to the fulcro template

(ns net.markusgraf.hhh.server-main
  (:require
   [mount.core :as mount]
   net.markusgraf.hhh.server.http-server
   [nrepl.cmdline :as nrepl])
  (:gen-class))

;; This is a separate file for production.
;; We control the server in dev mode from src/dev/user.clj
(defn -main [& args]
  (mount/start-with-args {:config "config/prod.edn"})
  (when args
    (apply nrepl/-main args)
    #_(nrepl/-main "-p" "6666"
                   "--middleware" "[cider.nrepl/cider-middleware]")))

Starting production

I start the app via a shell script which calls

/user/local/bin/clojure -Aprod -J-Dconfig=xxx -m net.markusgraf.hhh.server-main \
    -p 6666 --middleware '["cider.nrepl/cider-middleware"]'

We now have a production backend that accepts nREPL connecitons.

Make sure this only binds to localhost or you just created one hell of a vulnerability. Make sure you can not access that port from the outside. Ideally the only machines that can reach your machine ip/ipv6-wise are the reverse proxy and the ssh jump host. That's why I like FreeBSD and its jails so much. I can manage a jail, that is itself not reachable, from the host.

Now enable sshd.

SSH port redirect

The ssh -L option redirects a port. So on the developer machine run:

ssh -L 6666:localhost-of-jail:6666 url.jail.com

The localhost-of-jail is used as seen from within the jail. You can use a container instead or a 'real' host.

The line means any connection to 6666 on the developer machine gets forwarded to port 6666 on the jail.

To not be prompted for your ssh key on every connect use ssh-agent. To use a jump host, add ProxyJump host to .ssh/config.

Emacs, Cider and Sesman

So, in a cider-enabled buffer, press c-c c-x c j localhost 6666 which calls cider-connect-clj and you should connect to your production backend.

To switch sesman sessions I find sesman-link-with-buffer or c-c c-s b the most useful.

Wrapping up

Functional programming, functional datastructures, reloadable workflow, the clojure ecosystem and the tooling are super powerful.

This is as awsome as it is terrifying!

With the right (functional) disciplines it can be mastered, the setup itself is actually quite straightforward because the hard work has already been done.

This is, of course, thanks to:

  • Cider, nREPL, sesman, emacs with Boshidar Batsov, Dan Sutton et. al.
  • Fulcro, pathom with Tony Kay, Wilker Lucio et. al. Thank you Tony for stellar educational material!
  • Clojure, Clojurescript, ring, mount, bidi, clj-pdf, datahike/script, datomic, shadow-clj with all the heros of the ecosystem.
  • FreeBSD, Jails, ZFS, PF (firewall), h2o (revrse proxy), openssh, openjdk etc.

Afterthoughts

Context

This is, of course, only feasible within context. We are talking about a really small system here. A FreeBSD real metal dedicated host with good connectivity can be leased from 30€/month if you go for the used machines. This is way more than enough for small inhouse apps. In this case there is a handfull of users that use an internal system to scratch an itch, and I have an excellent and direct connection with the desicion maker and can talk domain concerns while fixing the report.

Perspective

If the programmer, the consultant and the sales guy are the same person this becomes a sales tool. If you look at this from a sales perspective the customer can watch the system change into what he wants before his very eyes.