The Big Macro

Since the domain is very deeply nested and entangled, the more domain cross references got presented, the more the app got useful, the bigger the pulled trees. Loading highly complex data from the server means the data travels un-normalized with many duplicated nodes. Also 95 % of what gets pulled gets pulled in the next load again, and again … and again. So I introduced caching. Users started to open many windows at once so when does the cache get stale? Windows needed to inform each other about updates and when a different user updated data the server needed to push. So the server kept subscriptions to know who to push to when something updated. This got messy and entangled.

So I am in the next iteration now. I tuned my data model so I could get rid of pathom and pull directly from datalevin on the server. This sped up pulls by a factor of 15.

So I have a very big cache that simply caches everything. A service worker sets up a shared worker if supported, a dedicated worker if shared worker support is missing (chrome on android). Since datalevin is a fork of datascript a bit of tweaking of transit-writers can copy a whole filtered db from the server into the client in about 10 seconds. This gets cached by the webworker. Once up, the webworker opens a websocket connection to the server.

For pulling and queries fulcro now uses a remote that pulls from the worker, not the server. The db-code in the server and worker for pull and query are identical differing in only in the requiere

(ns database
(:require #?(:clj  [datalevin.core :as d]
             :cljs [datascript.core :as d])))

Why is this different from pulling from the server? Because we now pull from cache, so we can uptimise for many small pulls instead of having to optimize for aggregated caching pulls.

Server push gets enriched by the result of the datalevin transaction

[[#datascript/datom [1 :person/firstname "Peter"]]

The server pushes to the worker, which transacts the datoms and pushes the original mutation to connected windows and tabs.

Mutations from the browser could get send to the server directly but to have a good offline experinece fulcro browser uses the worker remote and hands them off to the worker which transacts as the server would do but uses datascripts with-db to keep pending mutations apart from transactions confirmed/pushed by the server.

The worker answers the mutation from the UI and forwards it to the server.

The server authoratively processes the mutation, transacts to the authorative database, adds the datoms from the transaction to the mutation and pushes it to all connected workers, including the one that forwarded the mutation from its UI.

This enriched mutation is timestamped and receives a serial number and gets persisted to disk (zfs which snapshots every few minutes to another hot standby machine). The serial numbers turn this into an ordered mutation-log.

The worker transacts the data and removes the corresponding optimistic update from the db-with-database of pending transactions.

So mutations start in the browser, get passed to the worker, from there to the server, from there to all connected workers, from these to all windows or tabs. Server push is the extended life of a mutation. Mutations start and end in fulcro in the browser.

Whenever the worker looses the websocket connection it can just ask for a replay of mutations since the last seen serial id. Also, on push, the worker will realise if it missed a mutation and can, ask for a replay.

If or when the server db gets lost it can be reconstructed from any snapshot by replaying the mutations too.

The UI uses Fulcro's defmutation which is patche to elide the db-action. The browser-client should not know about datascript or datalevin.

The macro gets parsed as by Fulcros's defmutation but everything except the db-action is elided. The db-action is executed in the context of datalevin, persisting to disk, timestamping and distributing the push.

The macro gets parsed as by Fulcros's defmutation but everything except the db-action is elided. The db-action is executed in the context of datascript and the housekeeping of the with-db-database. This throws away anything not db related so browser stuff does not get included in the advanced build.

The easiest would be if there were reader conditionals per module. Then the three macros would be implemented in three seperate namespaces like client.defmutation, worker.defmutation and server.defmutation. The ns declaration of net/markusgraf/hhh/domain/assignment/model.cljc would be

(ns net.markusgraf.hhh.domain.assignment.model
 (:require #?(:client [client.defmutation :refer [defmutation]]
              :worker [worker.defmutation :refer [defmutation]]
              :clj    [server.defmutation :refer [defmutation]])))

If the build environment supports separate paths per build then namespaces by the same name coud exist in different paths like

• src/server/defmutation
• src/client/defmutation

I can imagine that this would confuse dead code elimination since two modules would see two different versions of the same namespace in the same build. In this scenario the three implementations of (defmacro defmutation ...) would live in three sepearte files with identical namespaces and names.

All these macros are implemented as

(defmacro
  ^{:doc "some"} defmutation
  [& args]
  (defmutation* &env args))

So if there were some way to know at compile time what module and build we are in this could branch to three separate implementations of defmutation*.

All the actions on the server, the worker and the UI close over the same params and env. The model stays together.

I regularily reorganize or regroup files into different directories and namespaces. I often miss something when doing this.

This keeps all mutation code in its proper namespace, regardless of whether it runs on the server, worker or browser.

An alternative would be to keep Fulcro's defmutation unpatched in the browser with just the push-action added and have a separate def-db-mutation in a separate file and namespace. It could not live in the same namespace and file since it would not get elided when not used and would pull the database into the browser and the browser code into the webworker. This would require the def-db-mutation macros to do namespace translation on the mutations between the db namespaces

I need to be careful what I require.

On the backend defmutation would eliminat the ui actions of the mutation, but the namespaces would still be pulled. So the jar would grow by unused ui code. I can live with this.