Editor remote REPL into production

When you start serious work you use an editor, in my case emacs, to write the code. The editor, in my case the awesome Cider package, does things like code highlighting, completion etc. And in some languages like C, Shell, Java, Python it has a button to run your code, possibly compiling and linking it in the process. In Clojure/Clojurescript that happens so rarely, that I don't even know what the command for that would be. When I load a project it happens automatically on startup.

Once.

No second time.

But I edit my code and have commands to evaluate the current expression, or the current buffer. To know the current value of a variable in the running application I just put the cursor there and eval. To update a function, I eval…

So called object oriented languages do not lend themselves to this workflow, because like hiding eastereggs in the garden for the children to find, object oriented code often hides its state in different objects in the code. If you tried to reload such an object you would loose the bit of state hidden in it. Functional programming has many functions that get applied to data and your state is in the data, not in the code. With your stateless functions and objects you can reload them in a running application without messing up your state. Of course there is limits to that. Mastery is to know where that limit lies.

On the server backend I can connect to a database, eg. datomic, datascript or datahike, that is immutable and behaves like a big functional datastructure. This behaves like being inside and part of the application, which in the case of datascript it is, while datahike keeps it in memory but also persists it to disk in the background. Datomic is the closed source database that brought us this api. These databases lend themselves to serve as graph databases and are perfectly suited to the above mentioned style.

The webserver part is build from a composable functional stack like ring. So it, too, fits the bill.

Then we need something to make the ring-stack as a whole remountable. I use mount.

I use fulcro and pathom with this whole setup copied from the fulcro template.

Until now I have not described anything new to a clojurian. This is one of the ways to develop an application. What I will only mention as a sidenote in this post, is that in development I run two REPLs, one into the development backend on my laptop and one into the browser frontend. I have the same functionality when connecting to the browser and to the jvm. I compile or reload per buffer or per function and can share code between the backend and frontend. But this is for another post. And we have not touched on the subject for this post jet.

The world changes, so do databases and their schemas. To do schema updates I have a namespace that contains the database schema. Then I wrote a function update-schema that does not blindly install this but reads the schema from the db, diffs my schema and the installed one and then installs the missing parts. This function can run on startup. But guess what… It is perfectly save to run on a live application. So if I changed the frontend code and added an entity type to the schema I can:

1. Call all users and tell them to keep their fingers of for 5 minutes while I restart or
2. Call the function in a running application.

The danger of course being that the code the application runs from and the installed schema get out of sync. But this is not how my work flowed. I write a new feature in development and test it. Then I push the code into production. So it is now, before the restart, that the application is out of sync with its code. It is just that instead of restarting the application I navigate to the namespace in emacs and either eval the changed function or the whole buffer. If other namespaces need to be reloaded mount is my friend.

Give a database to users and you will soon need to do some housekeeping. Merging double entries etc. If you need to just delete something you could do that with a seperate connection to the same database. The drawback here being, that all helper functions you wrote to keep your data sane will need to be copied from your project to the second admin project. So you just connect to the live system, and query for data with emacs c-u c-x c-e and the data gets inserted into your buffer. From there you massage it and check it and transact or retract as needed. An editor connected nrepl into production is perfect for this kind of work.

When I develop database accessing functions I normaly query and pull as described above, but I do not let them change the database. Instead of transact I write println, or log/debug. Of course I develop these functions locally, not on the production system but when I am confident enough to change the println to transact I use sesman to switch my buffer to the production system with the println version and I get a printout what my new function does to the actual, real, life production database data. It does not transact, mind you, it just prints.

Step two, of course, is to save functions like these into the code, to add to the repertoire of manual house keeping functions above.

• One of the first functions you might write is spit-db-export-to-disk.
• You should make special backups before you touch anything.
• Checkout ZFS Snapshots. Snapshots in the Filesystem are not backups, if the Disks crash they are gone, but you can do a lot more things with a lot more confidence if you can always roll back. That's another reason I love FreeBSD.

I just take the report as an example because I use fulcro and deliver an SPA to users. This is, of course, the same case as the server serving a classical html page instead of the SPA basepage. Lets say a user needs a report.

That means for example:

• having a route in the ring stack
• parsing the route for parameters
• collecting the data, propably through db queries and pull
• transforming the data
• piping the data through some sort of template or build a representation
• add a cover for the report
• create plots for graphical representations
• build a pdf
• stream the pdf via ring

You just get rough requirements from the user. Not as a lack of due dilligence, but as good practice, because only after a few iterations will both you and the user know, what they really needed. And when it is done, you normally see that you could not have guessed the solution beforehand.

So just get it roughly working, use git to push to production. Apart from you and the user(s) who is your 'partner in development' no one has to know about the link.

Now iterate with the user until it satisfies all stakeholders. You can push new features to production but there is no need to restart the application and cause downtime. Structure your Code so the report is in one namespace and reload this from your emacs.

This way iterations become really fast. You have the collaborating user on the phone. The user used the link and has the pdf open in the browser. You decide there needs to be a total somewhere in the report. You change that in the editor, eval with c-x c-e and tell the user to press F5. The user sees the new report.

And the Names of x needs to be fat: edit, c-x c-e, F5.

No, that is too fat take that back …

Welcome to agile development.

This is so tremendously valuable because the user sees the report with live production data. The data you thought up to test with will always be different in subtle ways, except when you work on production data too, which you can't copy if it is people related, because this would violate data protection laws.

The documentation for embedding the nrepl-server into production, https://docs.cider.mx/cider-nrepl/usage.html, mentions bug #447, which bit me a lot.

Then I realised that clj -m nrepl-cmdline is already provided. Thank you to everyone who implemented this! This can just be added to the project, so it accepts an optional nrepl port.

So after I start my server with mount, I call nrepl.cmdline/-main and pass through the command line options my app got called with. That means my own app now takes the additional command line args of nrepl-cmdline.

I start the app via a shell script which calls

/user/local/bin/clojure -Aprod -J-Dconfig=xxx \
    -m net.markusgraf.hhh.server-main \
    -p 6666 --middleware '["cider.nrepl/cider-middleware"]'

We now have a production backend that accepts nREPL connecitons.

Now enable sshd.

The ssh -L option redirects a port. So on the developer machine run:

ssh -L 6666:localhost-of-jail:6666 url.jail.com

The localhost-of-jail is used as seen from within the jail. You can use a container instead or a 'real' host.

The line means any connection to 6666 on the developer machine gets forwarded to port 6666 on the jail.

To not be prompted for your ssh key on every connect use ssh-agent. To use a jump host, add ProxyJump host to .ssh/config.

So, in a cider-enabled buffer, press c-c c-x c j localhost 6666 which calls cider-connect-clj and you should connect to your production backend.

To switch sesman sessions I find sesman-link-with-buffer or c-c c-s b the most useful.

This is, of course, only feasible within context. We are talking about a really small system here. A FreeBSD real metal dedicated host with good connectivity can be leased from 30€/month if you go for the used machines. This is way more than enough for small inhouse apps. In this case there is a handfull of users that use an internal system to scratch an itch, and I have an excellent and direct connection with the desicion maker and can talk domain concerns while fixing the report.

If the programmer, the consultant and the sales guy are the same person this becomes a sales tool. If you look at this from a sales perspective the customer can watch the system change into what he wants before his very eyes.