Transactions & MVCC

Atomic generated transactions, optimistic concurrent writers, a multi-process write coordinator, and group commit for bulk loads — built as three strict-superset tiers over the append-only engine.

Terse Detailed

ForgeDB gives you an atomic transaction boundary and a way to grow from one writer to several writer processes on a single machine. It is built in three tiers over the existing append-only engine, each adding capability without changing the on-disk format — no per-row version columns, no rewrite of stored data.

Tier 1 — generated transactions#

Wrap a group of writes in db.transaction(|tx| … ). Everything staged inside becomes visible together at commit, or not at all. If the closure returns an error, every touched row is rolled back to where it started.

db.transaction(|tx| {
    let user_id = tx.insert_user(User { /* … */ })?;
    tx.insert_post(Post { author: user_id, /* … */ })?;
    Ok(())
})?;
// Either both rows are visible, or neither is.

A transaction can span several models and still commit all-or-nothing across them.

FK visibility within a transaction

A foreign key validates against rows visible to the transaction — committed rows plus rows staged earlier in the same transaction (read-your-writes). So creating a parent and then its child in one transaction is valid, as long as the parent is staged first. (Split across separate transactions, the child's parent must already be committed.)

Tier 2 — optimistic concurrent writers#

Many transactions can do their work at the same time; only the final commit happens one at a time. At that point ForgeDB checks whether two writers touched the same row. If they did, the later one rolls back and retries. Nothing is stored for this check — it lives in memory only, so Tier 2 adds nothing to the on-disk format.

Tier 3 — multi-process writers (single machine)#

Tier 3 lets several separate processes write to the same data directory on one machine. You run a coordinator process:

forgedb coordinate ./data

Each writer connects to it instead of taking the directory lock itself:

// Connect-first; no DirLock held by the client.
let db = Database::connect("./data".into(), "/path/to/coordinator.sock")?;

The coordinator hands out the write turn and never touches your data — the actual column writes stay in each process's generated code. Each writer also picks up other writers' committed rows automatically.

If no coordinator is running, connect fails with CoordinatorUnavailable (HTTP 503). A process that uses the coordinator and a plain single-process writer cannot run against the same directory at the same time.

Single machine only. Tier 3 coordinates processes on one host. Writers spread across multiple machines are a separate future product, not this tier.

Group commit — fast bulk loads#

Each single insert normally waits for its own disk flush, and that flush dominates bulk loads. Put the rows inside a transaction and the whole batch pays one flush instead of one per row.

// One fsync barrier for the whole batch, not one per row.
db.transaction(|tx| {
    for record in batch {
        tx.insert_post(record)?;
    }
    Ok(())
})?;

This stays crash-safe: staged rows are invisible until commit, and a crash before commit loses only data that was never committed. The ordinary single-write path is unchanged and keeps its per-write flush.

Prefer moderate batches over one giant transaction

Once the fsync barrier is removed, a single very large transaction's commit-time reindex begins to dominate. Moderate batch sizes (on the order of a thousand rows per transaction) generally load faster than one enormous transaction.

Limits#

Concurrent prepare, serialized commit. Writers prepare in parallel but commit one at a time, and that serialized step includes the committing writer's disk flush. Truly parallel appends would need a larger change to the storage layout.
Single machine. Tier 3 runs on one host, over Unix sockets. Multi-machine writers are a separate future product.
Group commit removes the per-row flush; it does not reduce the cost of rebuilding indexes at commit time for one very large transaction.

See Durability & crash safety for the single-writer directory lock these tiers build on.

Tier 1 — generated transactions#

Tier 2 — optimistic concurrent writers#

Tier 3 — multi-process writers (single machine)#

Group commit — fast bulk loads#

Limits#

Search documentation