Control plane, data plane, durable storage: structuring a managed search engine in Rust

Almost every app starts with the same chore. Before a user can do anything, you build a way to know who they are. An email field, a password hash, an OAuth redirect, a confirmation link, a session cookie, a forgot-password flow. It is real work, and none of it is the thing you actually set out to build.if statements, and move on. That produces software that is wrong in exactly the ways that matter, because the literal text and the way a rule is actually enforced routinely diverge.

SSH has handled this since the 1990s, and almost nobody builds on it.

  create request
       |
       v
  Control plane   .  Postgres          does it exist? who owns it? quota
       |
       |  commit first (cheap, authoritative)
       v
  Data plane      .  tantivy           the inverted index, BM25, local disk
       |
       |  materialize second (expensive, can fail)
       v
  Durability      .  object storage    authoritative copy, survives the node

  create request
       |
       v
  Control plane   .  Postgres          does it exist? who owns it? quota
       |
       |  commit first (cheap, authoritative)
       v
  Data plane      .  tantivy           the inverted index, BM25, local disk
       |
       |  materialize second (expensive, can fail)
       v
  Durability      .  object storage    authoritative copy, survives the node

  create request
       |
       v
  Control plane   .  Postgres          does it exist? who owns it? quota
       |
       |  commit first (cheap, authoritative)
       v
  Data plane      .  tantivy           the inverted index, BM25, local disk
       |
       |  materialize second (expensive, can fail)
       v
  Durability      .  object storage    authoritative copy, survives the node

I kept noticing one-off “ssh into this” demos that used a sliver of this idea and then rebuilt everything else from scratch: presence, broadcasting, storage, rate limiting. So I pulled the common parts into a small Go framework called wharf. Writing apps on top of it changed how I think about a whole category of small, social, terminal-native software.

This is the case for it.invariants — statements that must hold for any input, derived from the regulation itself:

CREATE TABLE indexes (
    id         TEXT PRIMARY KEY,   -- a uuid, the real identifier
    org_id     TEXT NOT NULL,      -- the owning tenant
    name       TEXT NOT NULL,      -- a human label, not unique
    settings   JSONB NOT NULL,     -- field config, stored as an opaque blob
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX indexes_org_idx ON indexes (org_id)

CREATE TABLE indexes (
    id         TEXT PRIMARY KEY,   -- a uuid, the real identifier
    org_id     TEXT NOT NULL,      -- the owning tenant
    name       TEXT NOT NULL,      -- a human label, not unique
    settings   JSONB NOT NULL,     -- field config, stored as an opaque blob
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX indexes_org_idx ON indexes (org_id)

CREATE TABLE indexes (
    id         TEXT PRIMARY KEY,   -- a uuid, the real identifier
    org_id     TEXT NOT NULL,      -- the owning tenant
    name       TEXT NOT NULL,      -- a human label, not unique
    settings   JSONB NOT NULL,     -- field config, stored as an opaque blob
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX indexes_org_idx ON indexes (org_id)

The core move is to treat the public key fingerprint as the primary key for everything.

A fingerprint is not friendly to look at, so wharf derives a stable handle and color from it: the same key always becomes the same brave-otter in the same shade. Two strangers who have never signed up for anything now have persistent, distinct identities for as long as they keep their key. That is enough to build the features a normal app needs an accounts table for: read and unread state, bookmarks, a “continue where you left off” that just works, a per-user history that is simply there when they reconnect tomorrow.

What you get is an account that is anonymous and persistent at the same time. You never learn the person’s email, which means you are not holding data you have to protect, but you can still recognize them perfectly across visits. For a lot of small tools, that is the exact tradeoff you want.rapid or gopter in Go, Hypothesis in Python — generates large numbers of structured random inputs and checks each property against them. When it finds a violation, it shrinks the failure to the smallest input that still breaks, handing you a minimal reproduction instead of a haystack.

The whole identity layer is this: read the key off the connection, hash it into a handle, use the fingerprint as a key into a store. No tables to design, no flows to build.

A room is a single goroutine

Multiplayer is the other half. Once several people are connected, you need to broadcast to everyone in a space and keep an accurate list of who is present. The naive version is a shared map of members guarded by a mutex, and it is a reliable source of races as soon as people join and leave while messages are flying.know the map and keep it true.

wharf models each room as an actor. One goroutine owns the member set. Joining, leaving, and broadcasting are all messages sent to that goroutine over channels. Because a single goroutine is the only thing that ever touches the member map, there are no locks anywhere and no data races by construction. Concurrency becomes message passing instead of shared memory, which is the part of Go that actually holds up under pressure.derived from the regulation on adversarially generated input, the surface where it can still be wrong against a private corpus is small — and when a diagnostic does come back, the failing property localises it fast. You are not hoping you guessed the test cases right. You are proving the rules hold everywhere.

pub async fn create(&self, org_id: &str, settings: IndexSettings) -> Result<String> {
    validate(&settings)?;          // name length, language, vector dims
    let id = new_uuid();

    // 1. Control plane commits first. This insert is also the quota gate.
    if !self.catalog.insert(&id, org_id, &settings).await? {
        bail!("index quota reached");
    }

    // 2. Data plane materializes second.
    let index = match FtsIndex::open(&self.dir_for(&id), settings.clone()) {
        Ok(index) => index,
        Err(e) => {
            // Opening the on-disk index failed. Roll back the catalog row
            // so we never leave a record pointing at nothing.
            self.catalog.delete(&id).await?;
            return Err(e);
        }
    };

    // 3. Publish the open handle so queries can find it.
    self.live.write().insert(id.clone(), Entry { org_id: org_id.into(), index });
    Ok(id)
}

pub async fn create(&self, org_id: &str, settings: IndexSettings) -> Result<String> {
    validate(&settings)?;          // name length, language, vector dims
    let id = new_uuid();

    // 1. Control plane commits first. This insert is also the quota gate.
    if !self.catalog.insert(&id, org_id, &settings).await? {
        bail!("index quota reached");
    }

    // 2. Data plane materializes second.
    let index = match FtsIndex::open(&self.dir_for(&id), settings.clone()) {
        Ok(index) => index,
        Err(e) => {
            // Opening the on-disk index failed. Roll back the catalog row
            // so we never leave a record pointing at nothing.
            self.catalog.delete(&id).await?;
            return Err(e);
        }
    };

    // 3. Publish the open handle so queries can find it.
    self.live.write().insert(id.clone(), Entry { org_id: org_id.into(), index });
    Ok(id)
}

pub async fn create(&self, org_id: &str, settings: IndexSettings) -> Result<String> {
    validate(&settings)?;          // name length, language, vector dims
    let id = new_uuid();

    // 1. Control plane commits first. This insert is also the quota gate.
    if !self.catalog.insert(&id, org_id, &settings).await? {
        bail!("index quota reached");
    }

    // 2. Data plane materializes second.
    let index = match FtsIndex::open(&self.dir_for(&id), settings.clone()) {
        Ok(index) => index,
        Err(e) => {
            // Opening the on-disk index failed. Roll back the catalog row
            // so we never leave a record pointing at nothing.
            self.catalog.delete(&id).await?;
            return Err(e);
        }
    };

    // 3. Publish the open handle so queries can find it.
    self.live.write().insert(id.clone(), Entry { org_id: org_id.into(), index });
    Ok(id)
}

There is a nice second-order effect. Because the room is already the thing serializing every mutation, it is the natural home for shared state. wharf has stateful rooms, where the room holds a value and a reducer, hands each newcomer a snapshot the moment they join, and broadcasts the new state after every action. A live poll, a shared counter, a tiny game board: all of them become a reducer and a render, with the hard concurrency already solved one layer down.get in touch. header that declares the drawing units. it lies constantly. i had a file that declared “metres” while every coordinate was clearly millimetres. trust it and your apartment renders a thousand times too big.

Persistence as a seam, not a dependencytell me about it.

Per-user state needs somewhere to live, and this is where it would have been easy to make a mistake.

The tempting move is to pick a database and wire it in. The problem is that a framework which imports a Postgres driver forces that driver on everyone, including the person who only wanted to keep things in memory for a weekend toy. So persistence in wharf is a small interface, five methods, with the in-memory implementation living in the core and nothing else. Every real backend is its own Go module: SQLite, Postgres, Redis, and an embedded single-file option. Importing the core pulls in no database at all. Importing an adapter pulls in exactly one.
ezdxf.recover gives up, i do not: a dxf is at heart a stream of (group code, value) pairs, so i drop to a raw tag scanner that walks the text and pulls out every LINE, LWPOLYLINE, CIRCLE and TEXT it can find, geometry only, no object model. you lose elegance, you keep the floor plan. and a converter that emits latin-1 diagnostics into a utf-8 pipe cannot blow up the job either (capture bytes, decode with errors="replace", a real crash i hit on 7 of my first 35 test files).

count = SELECT count(*) ...        // says 4 of 5 used
if

count = SELECT count(*) ...        // says 4 of 5 used
if

count = SELECT count(*) ...        // says 4 of 5 used
if

This is the kind of decision that looks like fuss for a toy and turns out to be the difference between a library people can adopt and one they have to fork. The cost of getting it right was an afternoon. The cost of getting it wrong is every future user inheriting your dependency choices.processing and updates the run’s display label to match the row’s ID. If the run crashes anywhere downstream, the row stays, with the step name and stack trace attached. Nothing disappears between systems.

The bug that explained the designget in touch.

INSERT INTO indexes (id, org_id, name, settings)
SELECT $1, $2, $3, $4
WHERE (SELECT count(*) FROM indexes WHERE org_id = $2)
    < (SELECT max_indexes FROM organizations WHERE id = $2)

INSERT INTO indexes (id, org_id, name, settings)
SELECT $1, $2, $3, $4
WHERE (SELECT count(*) FROM indexes WHERE org_id = $2)
    < (SELECT max_indexes FROM organizations WHERE id = $2)

INSERT INTO indexes (id, org_id, name, settings)
SELECT $1, $2, $3, $4
WHERE (SELECT count(*) FROM indexes WHERE org_id = $2)
    < (SELECT max_indexes FROM organizations WHERE id = $2)

Early on, presence was occasionally wrong. When two people connected, one of them would sometimes not see the other arrive, even though both were clearly in the room. Broadcasts worked. Identity worked. Only the ordering of who joined when was off, and only sometimes.WAND or MAUER, english ones write A-WALL or WALLS, and everyone abbreviates differently. so classification is humble substring matching over a multilingual hint list:

The cause was timing. Before a session starts rendering, the terminal layer asks the client a question about its background color and waits for the answer. On a real terminal that reply comes back in milliseconds. On a client that never answers, the code waits for the timeout, and in my first version that wait happened before the session joined its room. So a slow or silent client would join late, after someone who connected after it, and the join and leave events arrived out of order.

The fix was to join the room before negotiating anything about rendering, so presence never waits on a cosmetic question to the terminal. It is a small change, but it only became obvious once I stopped assuming that “connected” and “in the room” happen at the same instant. The lesson generalized. The moment you decouple identity from rendering, a class of ordering bugs disappears, and the whole system gets easier to reason about.0, a classic), the pipeline falls back to “every segment that is not obviously a door, window, text or dimension” and attaches a warning saying results are approximate. honesty over confidence.

I keep this story in the writeup because the framework is more trustworthy with the scar visible than without it.

What it is good for, and what it is notField1: [empty]. The visible content lives in form widgets, not page content; the two are stored separately inside the file, and a parser that does not know to look for widget annotations returns the empty shell.

wharf is not a chat library, even though chat is the obvious demo. The primitives are identity, presence, rooms, shared state, and storage, and those fit a lot of things. The examples I shipped are a collaborative drawing canvas where you watch other people’s cursors move, a keyless chat room, and a live poll. The same shapes cover guestbooks, small multiplayer games, status boards, internal tools your team reaches with the keys they already have, and a personal assistant you talk to in your terminal.

There is one honest limitation worth stating plainly. SSH is pull-only. Nobody re-runs a command on a Tuesday for no reason, so an SSH app cannot tap someone on the shoulder the way an email or a push notification can. The reading experience is excellent and the retention is near zero unless you add a way back in. In practice that means pairing the SSH front door with something that can reach out, a small web mirror for discovery, or a notification when there is an actual reason to return. The terminal is the experience. It is not the reminder.

Why botherSystemError, UnicodeDecodeError, or RuntimeError from inside the widget enumeration call. The detection catches all three and uses them as signal.

def needs_flattening_safe(doc: fitz.Document) -> bool:
    try:
        return needs_flattening(doc)
    except (SystemError, UnicodeDecodeError, RuntimeError):
        return True  # crash on enumeration = file definitely needs flattening

def needs_flattening_safe(doc: fitz.Document) -> bool:
    try:
        return needs_flattening(doc)
    except (SystemError, UnicodeDecodeError, RuntimeError):
        return True  # crash on enumeration = file definitely needs flattening

def needs_flattening_safe(doc: fitz.Document) -> bool:
    try:
        return needs_flattening(doc)
    except (SystemError, UnicodeDecodeError, RuntimeError):
        return True  # crash on enumeration = file definitely needs flattening

A large amount of the software we build spends its first few weeks rebuilding identity, and a large amount of that machinery exists only because the web has no built-in notion of who is on the other end. SSH does. It has had a clean, verified, cryptographic answer to “who is this” for decades, sitting one port over from where everyone is already working. wharf is a small bet that this answer is more useful than we treat it, and that there is a category of fast, personal, multiplayer terminal apps waiting on the other side of taking it seriously.

It is open source under the MIT license. If you want to read the actor model, the adapter split, or the demos, the repository is at github.com/doganarif/wharf. If you build something odd with it, I would like to see it.

Site surveys arrive looking like real PDFs. They are not. There is no text layer. A naive parser returns an empty string and reports success. The pipeline detects scans before any extraction attempt with a text-density heuristic.is the door’s position, stored as one number, center_mm along the wall. openings with the same type and size (rounded to 50 mm) share a schedule mark, which is how D1, D2, W1 get assigned exactly the way a human drafter would.

every dimension nobody drew (storey height 3000, door height 2100, window sill 900) is a stated construction default, appended to the model’s warnings so the ui can show “assumed” instead of pretending the file said so.

The threshold is set above zero so mixed documents (a scanned cover sheet stapled to typed pages, common in tender packs) route correctly rather than failing the binary check on page one. OCR output is written back to the same storage key so reprocessing skips the OCR step on a second attempt.

Structurally corrupt

Print-to-PDF drivers occasionally emit documents with one or more pages containing empty content streams. The extraction service signals permanent failure and refuses the whole file. The pipeline does not give up. It scans every page for zero-length content streams, removes the offending pages, rebuilds the file, and retries.

If there is one rule worth keeping from this, it is the ordering. Commit the cheap, authoritative fact first. Materialize the expensive thing second. Roll back the first if the second fails.

A forty-page drawing set with two corrupt pages becomes a thirty-eight-page document that succeeds.

The general principle here is worth naming: do not try to predict which inputs are corrupt. Let the service tell you, then repair surgically. The same principle returns later for rate limits.plus one extra polygon: the outer boundary of the whole building, traced from the outside. my first heuristic dropped any face bigger than some fraction of the bounding box. worked on every rectangular test building, then an l-shaped floor plan arrived. the outer face of an l-shape is nowhere near its bounding box area, so it slipped under the threshold and appeared in the ui as a giant phantom room covering the entire footprint. the correct rule is topological, not metric: in a connected arrangement the unbounded face encloses everything else, so it is always the single largest face. drop the max, keep the rest. no threshold to tune, no shape that breaks it.

Oversizedget in touch.

When you ask a search service to “create an index,” it reads like a single write to a single place. While building a managed search engine in Rust this year, I started treating that one word as three separate facts, each owned by a different system: the record that an index exists, the index itself, and a durable copy for when the machine holding it dies.

Pulling those apart changed how I reason about failure and scaling, and it sharpened my sense of what a relational database is actually good for. Here is the structure I settled on, and the path a single create request takes through it.

  create request
       |
       v
  Control plane   .  Postgres          does it exist? who owns it? quota
       |
       |  commit first (cheap, authoritative)
       v
  Data plane      .  tantivy           the inverted index, BM25, local disk
       |
       |  materialize second (expensive, can fail)
       v
  Durability      .  object storage    authoritative copy, survives the node

  create request
       |
       v
  Control plane   .  Postgres          does it exist? who owns it? quota
       |
       |  commit first (cheap, authoritative)
       v
  Data plane      .  tantivy           the inverted index, BM25, local disk
       |
       |  materialize second (expensive, can fail)
       v
  Durability      .  object storage    authoritative copy, survives the node

  create request
       |
       v
  Control plane   .  Postgres          does it exist? who owns it? quota
       |
       |  commit first (cheap, authoritative)
       v
  Data plane      .  tantivy           the inverted index, BM25, local disk
       |
       |  materialize second (expensive, can fail)
       v
  Durability      .  object storage    authoritative copy, survives the node

Three planes, one job each

The control plane is Postgres. It holds metadata and nothing else: which indexes exist, which tenant owns each one, the field configuration, and the quota. The table is tiny.

CREATE TABLE indexes (
    id         TEXT PRIMARY KEY,   -- a uuid, the real identifier
    org_id     TEXT NOT NULL,      -- the owning tenant
    name       TEXT NOT NULL,      -- a human label, not unique
    settings   JSONB NOT NULL,     -- field config, stored as an opaque blob
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX indexes_org_idx ON indexes (org_id)

CREATE TABLE indexes (
    id         TEXT PRIMARY KEY,   -- a uuid, the real identifier
    org_id     TEXT NOT NULL,      -- the owning tenant
    name       TEXT NOT NULL,      -- a human label, not unique
    settings   JSONB NOT NULL,     -- field config, stored as an opaque blob
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX indexes_org_idx ON indexes (org_id)

CREATE TABLE indexes (
    id         TEXT PRIMARY KEY,   -- a uuid, the real identifier
    org_id     TEXT NOT NULL,      -- the owning tenant
    name       TEXT NOT NULL,      -- a human label, not unique
    settings   JSONB NOT NULL,     -- field config, stored as an opaque blob
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX indexes_org_idx ON indexes (org_id)

Look at what is missing. There is no document text and no tsvector. The settings column is JSONB, but I never query into it. It gets read back whole and parsed in the application. Postgres knows that an index exists and who owns it. It has no idea how to search one.

That indexes_org_idx is worth an aside, because the word “index” trips people up here. It is a plain B-tree that speeds up “list the indexes for this tenant.” It has nothing to do with full-text search. Two unrelated meanings of the same word, sitting in the same paragraph.

The data plane is tantivy, a Lucene-class search library for Rust. The real inverted index lives here, on local disk, memory-mapped. Document text, the BM25 postings, the term dictionary, the stored originals: all on disk in the data plane, none of it in Postgres.

Durability is object storage, an S3-compatible bucket. I treat the local tantivy files as a working copy. The authoritative durable copy lives in the bucket. If the box holding the working copy vanishes, the next boot rebuilds it from object storage.

One more piece sits in memory: a map from index id to its open tantivy handle, so a query does not reopen files on every request. That map is not a source of truth. It is reconstructed from the control plane at startup.

Walking a create through all three

The HTTP handler does almost nothing interesting. It checks that the API key has write permission and belongs to a tenant, then hands off to the index manager. The ordering inside the manager is where the design lives.

pub async fn create(&self, org_id: &str, settings: IndexSettings) -> Result<String> {
    validate(&settings)?;          // name length, language, vector dims
    let id = new_uuid();

    // 1. Control plane commits first. This insert is also the quota gate.
    if !self.catalog.insert(&id, org_id, &settings).await? {
        bail!("index quota reached");
    }

    // 2. Data plane materializes second.
    let index = match FtsIndex::open(&self.dir_for(&id), settings.clone()) {
        Ok(index) => index,
        Err(e) => {
            // Opening the on-disk index failed. Roll back the catalog row
            // so we never leave a record pointing at nothing.
            self.catalog.delete(&id).await?;
            return Err(e);
        }
    };

    // 3. Publish the open handle so queries can find it.
    self.live.write().insert(id.clone(), Entry { org_id: org_id.into(), index });
    Ok(id)
}

pub async fn create(&self, org_id: &str, settings: IndexSettings) -> Result<String> {
    validate(&settings)?;          // name length, language, vector dims
    let id = new_uuid();

    // 1. Control plane commits first. This insert is also the quota gate.
    if !self.catalog.insert(&id, org_id, &settings).await? {
        bail!("index quota reached");
    }

    // 2. Data plane materializes second.
    let index = match FtsIndex::open(&self.dir_for(&id), settings.clone()) {
        Ok(index) => index,
        Err(e) => {
            // Opening the on-disk index failed. Roll back the catalog row
            // so we never leave a record pointing at nothing.
            self.catalog.delete(&id).await?;
            return Err(e);
        }
    };

    // 3. Publish the open handle so queries can find it.
    self.live.write().insert(id.clone(), Entry { org_id: org_id.into(), index });
    Ok(id)
}

pub async fn create(&self, org_id: &str, settings: IndexSettings) -> Result<String> {
    validate(&settings)?;          // name length, language, vector dims
    let id = new_uuid();

    // 1. Control plane commits first. This insert is also the quota gate.
    if !self.catalog.insert(&id, org_id, &settings).await? {
        bail!("index quota reached");
    }

    // 2. Data plane materializes second.
    let index = match FtsIndex::open(&self.dir_for(&id), settings.clone()) {
        Ok(index) => index,
        Err(e) => {
            // Opening the on-disk index failed. Roll back the catalog row
            // so we never leave a record pointing at nothing.
            self.catalog.delete(&id).await?;
            return Err(e);
        }
    };

    // 3. Publish the open handle so queries can find it.
    self.live.write().insert(id.clone(), Entry { org_id: org_id.into(), index });
    Ok(id)
}

The shape here is simple to say: commit the cheap authoritative thing first, materialize the expensive thing second, and undo the first if the second fails. The catalog row is a few bytes in a transactional database. Opening a tantivy index touches the filesystem and can fail for boring reasons like no disk, bad permissions, or a corrupt directory. Writing the row first means the quota is enforced atomically. Rolling it back on failure avoids the worst state in this whole system: a row in Postgres that claims an index exists with no data behind it.

After the manager returns, the handler does one more thing. It syncs the new index directory to object storage, and if that sync fails it returns 503 instead of 200. I would rather tell the caller “this did not durably succeed” than report success for something that only exists on one disk.

The quota check is a single statement

The quota gate looks small, and getting it wrong is a classic race. The naive version reads the count, compares it to the limit, then inserts:

count = SELECT count(*) ...        // says 4 of 5 used
if

count = SELECT count(*) ...        // says 4 of 5 used
if

count = SELECT count(*) ...        // says 4 of 5 used
if

Run two creates at the same moment and both read “4 of 5,” both pass the check, and both insert. Now the tenant has 6 indexes on a limit of 5. The check and the write were two steps with a gap between them, and concurrency lives in that gap.

The fix is to make the check and the insert the same statement, so the database serializes them:

INSERT INTO indexes (id, org_id, name, settings)
SELECT $1, $2, $3, $4
WHERE (SELECT count(*) FROM indexes WHERE org_id = $2)
    < (SELECT max_indexes FROM organizations WHERE id = $2)

INSERT INTO indexes (id, org_id, name, settings)
SELECT $1, $2, $3, $4
WHERE (SELECT count(*) FROM indexes WHERE org_id = $2)
    < (SELECT max_indexes FROM organizations WHERE id = $2)

INSERT INTO indexes (id, org_id, name, settings)
SELECT $1, $2, $3, $4
WHERE (SELECT count(*) FROM indexes WHERE org_id = $2)
    < (SELECT max_indexes FROM organizations WHERE id = $2)

If the tenant is at the limit, the WHERE is false, no row goes in, and rows_affected() comes back as 0. The application reads that zero as “quota reached.” There is no window for two creates to both win, because there is no separate read to race against.

Why split metadata from the index at all

The split costs something. Every create now writes to two systems, and I have to keep them consistent. The payoff shows up in three places.

Search traffic never touches the control-plane database. A burst of queries hits the in-process tantivy reader and the local disk. It does not compete for connections with the database that authenticates requests and enforces quotas. The thing under the most load and the thing that must stay correct are physically different systems, and they scale on different axes.

Failure has a clear blast radius. Losing the data plane is annoying but recoverable, since the durable copy is in object storage and the catalog still knows what should exist. Losing the control plane is the real outage, which is exactly why I keep it small, transactional, and boring. A few kilobytes per index in Postgres is cheap insurance for the gigabytes of index data it describes.

Boot becomes a reconciliation. On startup the engine reads every row from the catalog and, for each one, opens the local copy or restores it from object storage if the local copy is missing. Then it sweeps the bucket for leftover data that has no matching catalog row, which is how a delete that got interrupted halfway gets cleaned up. The control plane is the list of what should exist, and startup makes reality match the list.

What a field actually turns into

One detail from the data plane is worth showing, because it explains why search and a database want different things from the same string. When I create a text field that is both searchable and filterable, it becomes two physical fields in tantivy:

title       ->  analyzed field   (tokenized, lowercased, stemmed)   for BM25 matching
title__kw   ->  keyword field    (the raw string, untouched)        for

title       ->  analyzed field   (tokenized, lowercased, stemmed)   for BM25 matching
title__kw   ->  keyword field    (the raw string, untouched)        for

title       ->  analyzed field   (tokenized, lowercased, stemmed)   for BM25 matching
title__kw   ->  keyword field    (the raw string, untouched)        for

Full-text matching wants “Running” to find “runs,” so it tokenizes and stems. An exact filter on a brand name wants “ACME” to mean exactly “ACME,” with no stemming and no folding. The same input needs opposite treatment depending on the question, so it gets stored twice, once for each job. Vectors are a third case. They never enter the tantivy schema at all; each vector field gets its own approximate-nearest-neighbor store on disk, beside the text index but separate from it.

What it costs to run this way

The architecture has real rough edges, and I would rather name them than pretend otherwise.

Writes commit synchronously. Each batch fsyncs and reloads the reader so documents are searchable the instant the call returns. That is good for correctness and bad for tiny frequent writes, so bulk loads have to batch thousands of documents per request or throughput falls off a cliff.

Cold starts pay for the durability. After a deploy, an index restores from object storage and pages in from disk, so the first handful of queries run slower until the cache warms. Warm and cold are genuinely different latency regimes.

A single index lives on a single node. There is no sharding of one index across machines yet. That is fine for tens of millions of documents on one large server and wrong for trillions. It is a deliberate ceiling, and worth stating plainly rather than hiding.

The takeaway

If there is one rule worth keeping from this, it is the ordering. Commit the cheap, authoritative fact first. Materialize the expensive thing second. Roll back the first if the second fails.

The record that an index exists belongs in a small transactional database. The index itself belongs in a purpose-built search library on fast local disk. The durable copy belongs in object storage. Wire a create through the three in that order and you get independent scaling and clean recovery for very little extra code, while the database goes back to doing the one thing it is best at: being the boring, correct source of truth for what exists.

I build backend systems with this kind of separation: control planes, search, and correctness under concurrency. If that is the problem in front of you, get in touch.