#flatdb - a file-backed JSON store

import "flatdb.j" as flatdb;

A small JSON document store: load a file once into a value-semantic handle, query and edit it in memory through JSON Pointer, and write it back with a crash-atomic whole-file replace. Built from json (the data) and fs (the file); runs on either binary.

#What it is - and isn't

flatdb is not a database engine. Honestly, it is crash-atomic snapshotting of small data:

• Atomicity - whole-file, via a temp file + rename. A reader ever sees the whole old file or the whole new one, never a torn write.
• Consistency - application-level (you decide what's valid).
• Isolation - optimistic, with an opt-in lock. Each writer reads the whole file; save fails loudly if it changed since open (no silent lost update), and flatdb.update adds a cross-process advisory lock so concurrent writers serialize with no loss. No multi-key transactions.
• Durability - the rename is atomic, but flush-to-disk is OS-buffered.

For a real database, reach for a client over net (e.g. redis), not this. flatdb is the "embed a small store" need - config, a cache you can read, a benchmark history, a little app's saved state - where a single human-readable JSON file is exactly right.

#Handle, not a connection

A module holds no mutable state and spawn deep-copies scope, so a store can't be a shared open connection - it's a value you hold:

export def struct DB { path as string, data as json.Value, version as string };

Reading verbs leave the DB untouched; writing verbs return a fresh DB (thread it through, the same shape lists / maps / json use); save is the only side effect.

#Surface

value is any JSON value - a json.Value. Build objects and lists with json.map() / json.list() (then json.set / json.append into them), and scalars with json.decode (json.decode("42"), json.decode("\"hi\"")). Addressing is JSON Pointer, identical to json's.

#Example

use io;
use json;
import "flatdb.j" as flatdb;

def db as flatdb.DB init flatdb.open("state.json");   # empty on first run
$db = flatdb.set($db, "/runs", json.list());

def rec as json.Value init json.map();
$rec = json.set($rec, "/cpu", "Ryzen 5 7600X3D");
$rec = json.set($rec, "/ms", 118);
$db = flatdb.append($db, "/runs", $rec);

flatdb.save($db);                                     # atomic replace

def store as flatdb.DB init flatdb.open("state.json");
io.printf("%d runs; first on %s\n",
    flatdb.length($store, "/runs"),
    json.asString(flatdb.get($store, "/runs/0/cpu")));

A runnable version is examples/modules/flatdb_demo.j.

#Read-only from a URL or a string

flatdb.openString(text) loads a store from an in-memory JSON string instead of a file. flatdb stays transport-agnostic - it never imports http / net, so it remains fs-only and builds on both binaries - and you bring the bytes. To read a database published at a URL:

use io;
use json;
import "http.j" as http;
import "flatdb.j" as flatdb;

def resp as http.Response init http.get("https://example.com/config.json", {});
def db as flatdb.DB init flatdb.openString($resp.body);   # read-only
io.printf("theme: %s\n", json.asString(flatdb.get($db, "/ui/theme"), ""));

The result has no backing path, so every reader works and the mutating verbs still return fresh in-memory copies, but save throws (kind: "flatdb") - there is nowhere local to write. To persist it (or fork any store to a new file), use saveAs:

def db as flatdb.DB init flatdb.openString($resp.body);
def edited as flatdb.DB init flatdb.set($db, "/ui/theme", json.decode("\"dark\""));
def local as flatdb.DB init flatdb.saveAs($edited, "config.local.json");
flatdb.save($local);   # now writable - $local is bound to config.local.json

saveAs(db, path) writes to path and returns a fresh DB bound to that path; the db you passed in is unchanged (value semantics, like set / append / remove). So after a saveAs you hold both - which one is "current" for the next save is simply the handle you keep: reassign ($db = flatdb.saveAs($db, path);) to make the new file current, or keep the old handle to keep writing the original file. There is no hidden "active file" state; a DB is a value.

Decoding untrusted input is safe against a nesting bomb: json.decode caps container depth (a deeply-nested payload is a catchable error, not a fatal stack overflow), and JSON has no entity/alias expansion to amplify size - so a malicious URL can't crash or blow up the process. Bound the download itself with http's body cap (64 MiB by default; pass an explicit maxBytes to http.requestWith for more or less).

#Concurrent writers

Crash-atomic is not the same as write-safe. Two writers that each open, edit different parts, and save would clobber one of the two changes (last rename wins). flatdb closes that two ways:

save fails loudly on a conflict. open records a token of the file's on-disk state; save re-checks it and throws a flatdb error if the file moved since - so a lost update can never be silent. Catch it and re-open, or retry:

try {
    flatdb.save($db);
} catch (e) {
    # someone else wrote first - re-open, re-apply, retry
}

(This is best-effort - it rests on the filesystem's mtime granularity and has a tiny check-then-rename window - so it makes loss loud, not impossible.)

flatdb.update makes it impossible. It brackets the whole read-modify-write in a cross-process advisory lock (built on fs.writeNew, an atomic exclusive create), so concurrent writers serialize and no update is lost. transform is a func(db as DB) -> DB:

func addVisit(db as flatdb.DB) {
    def n as int init 0;
    if (flatdb.has($db, "/visits")) { $n = json.asInt(flatdb.get($db, "/visits")); }
    return flatdb.set($db, "/visits", json.decode(convert.toString($n + 1)));
}
flatdb.update("state.json", addVisit);   # locked read -> edit -> save -> release

The lock coordinates across separate processes (a CLI and a server, two CLI runs) as well as spawned tasks - a symlink-free lock file created with fs.writeNew, whose holder id and timestamp let a lock leaked by a crashed writer be broken after a stale window (120 s). Waiters that find the same stale lock break it one at a time, under an exclusive .lock.break file, and remove the lock only while it still holds the holder they judged stale - so a lock a faster waiter has already re-taken is never removed. A waiter gives up after 5 s with a flatdb error naming the current holder. A lock file that cannot be created at all - a missing directory, a read-only mount, a data directory owned by another user - has no holder to wait for, so lock / update fail at once with the create's own error (cannot create the lock file ...: permission denied) instead of reporting contention after the wait (a dangling symbolic link left at the lock path is named as such). flatdb.lock / flatdb.unlock are the raw primitives for a multi-step transaction, but prefer update, which releases on every exit path.

A note on save returning a DB: save returns the document rebound to the just-written file, so if you save the same handle more than once, thread the return - $db = flatdb.save($db); - or the next save sees the file it wrote as a conflict.

#Many readers: load once, share the parsed tree

An opened store is a parsed json.Value tree, which takes from about the file's size (text-heavy data) up to roughly 15 times it (many small values) in memory. So a server should not open the store in every request handler: N requests in flight would hold N parsed copies. Open it once and share it:

• Read-only for the server's lifetime - a top-level def is enough. def store as flatdb.DB init flatdb.open("registry.json"); and handlers read $store. Passing the DB around, or capturing it in a spawn, copies only the small struct around the tree; the tree itself is immutable and shared.
• Updated while serving - publish each version through a kv store, which holds an opaque value without copying it, and rebuild the DB around it where you need one:

use kv;
import "flatdb.j" as flatdb;

def cache as kv.Store init kv.open();
def first as flatdb.DB init flatdb.open("registry.json");
kv.setValue($cache, "registry", $first.data, 0);

# Readers (any handler / task): the current tree, no parse, no copy.
func current() {
    return flatdb.DB{ path: "registry.json", data: kv.getValue($cache, "registry"), version: "" };
}

# The writer: update, save, and publish the new tree.
func publish(db as flatdb.DB) {
    def saved as flatdb.DB init flatdb.save($db);
    kv.setValue($cache, "registry", $saved.data, 0);
}

A reader DB built this way is for reading: its empty version makes a save of it fail the conflict check, so writes go through a DB from flatdb.open (or flatdb.update). See kv values.

#Atomic save, in detail

save writes the encoded document to a sibling path + ".tmp" and then fs.renames it over the target. On POSIX the rename is atomic, so a concurrent reader never sees a half-written file. If the process dies mid-save (temp written, rename not reached), the original file is untouched - only a stray .tmp remains, which the next save overwrites. Durability past the rename is the OS's call (there is no fsync today).

#See also

• json - the value model and write surface flatdb layers over.
• fs - the file I/O (readString / writeString / rename) behind open / save.
• redis - a real store, over the network, when you outgrow a single file.