System Requirements Document for rules-ingestion-pipeline

1. Introduction

The rules-ingestion-pipeline project supplies the missing automated ingestion path for an already-built, already-tested rule validation and sealing backend. The existing backend — a Supabase PostgreSQL database in schema sovereign plus Python gate scripts running on Replit free tier — validates small verified Python functions ("rules") with formal pre/postconditions through a 6-gate pipeline (G1–G6) and permanently admits them via a single Postgres function sovereign.finalize_candidate(candidate_id, revision).

Today every rule candidate is created manually: the operator writes the Python function by hand, inserts it into rule_candidates via SQL, runs the gate scripts by hand, and calls finalize_candidate by hand. There is no automated path from a raw source to a candidate.

This document specifies the automated ingestion pipeline that closes that gap: raw source in (PDF textbook/paper, YouTube video URL, or plain text course excerpt), extracted text, durably chunked with source location metadata, LLM-proposed rule candidate with a Hoare-style contract, insertion into rule_candidates, automatic invocation of engine_main.py's evaluate(), and automatic finalize_candidate on all-6-gates-PASS. Failed candidates remain permanently recorded and are never deleted.

The audience is the single human actor — a non-technical solo operator working from a mobile phone only, with zero budget, on Supabase free tier and Replit free tier. The document is written so that this operator can build, run, and monitor the pipeline incrementally from a phone, and so that any engineer assisting them can implement it without redesigning the existing backend.

2. System Overview

2a. Product Interpretation and Delivery Boundary

What is being built. A first-party ingestion pipeline that turns raw sources into validated rule candidates and, when all six gates pass, into sealed sovereign rules. The pipeline is the only new product surface; the existing validation and sealing backend is treated as a fixed, external, already-tested dependency that must not be redesigned.

What is not being built. The existing sovereign schema tables (rule_candidates, rule_gate_receipts, sovereign_rules, policy_manifests, policy_bounds, audit_events), the sovereign.finalize_candidate Postgres function, and the Python gate scripts (engine_main.py, engine_g2.py, engine_g3.py, engine_g5.py, engine_g6.py, engine_seal.py, supa_client.py) are already built and tested. They are consumed as-is. No new gate, no new sealing path, no new policy mechanism, and no replacement of finalize_candidate is introduced.

Delivery and access ownership. The pipeline is a first-party, application-owned system with a single human actor. Because the operator must privately own and resume durable, actor-specific ingestion state (submitted sources, chunk queues, proposal queues, candidate processing state) across Replit container sleep and restarts, application-owned identity is required. First use establishes identity anonymously (self-service enrollment); returning use verifies identity before any protected ingestion or pipeline control is available. The entry interaction that establishes access is itself anonymous; protected ingestion and pipeline controls are unavailable until identity is established. No differentiated permissions, roles, or visibility rules are introduced — the operator is the sole human actor and has full control over their own ingestion state.

Current vs. future boundary. Everything in this document is current. No future-horizon capabilities are specified by the source. The pipeline must not require any paid service, VPS, or persistent server, and must not rely on local disk or in-memory state as the source of truth at any step.

Durable state. Postgres is the durable source of truth at every step. All new pipeline state (sources, chunks, queue entries, LLM proposals, candidate processing state, provider retry/fallback state) lives in Postgres tables. Replit containers may sleep, restart, or lose local disk at any time without losing in-progress work.

Validation strength. LLM calls go through a free-tier-compatible provider (e.g., OpenRouter free models) with explicit fallback handling if a model becomes unavailable or rate-limited. Fallback must never silently degrade validation strength: if no acceptable model is available, the affected work item is parked in an explicit provider-retry/fallback state rather than being processed with a weaker model or skipped.

Code safety. LLM-generated Python is never executed with real inputs during validation. It is only parsed and analyzed (AST) by the existing gate pipeline. No arbitrary code execution of LLM-generated code is allowed outside a strictly bounded, isolated process.

Runtime services and infrastructure. The pipeline is delivered as a small set of runnable services, and new pipeline APIs and domain modules are added to the existing shared backend rather than to a new process:

• backend — the shared FastAPI process that hosts the application APIs and domain modules. The ingestion pipeline's routes and modules (source submission, chunk and proposal inspection, candidate and sealed-rule inspection, provider state, pipeline control, retry control, and the operator-console API) are added to this existing backend, reusing its process, port, dependencies, and deployment. No separate container or ingress route is introduced for the pipeline API.
• database — MariaDB for application-owned persistent data (operator identity and the application-side records the console needs). This is separate from the sovereign Supabase PostgreSQL database, which remains the durable source of truth for all pipeline and rule state.
• frontend — the React application UI that renders the anonymous Ingestion Entry state and the protected Ingestion Operator Console.
• litellm — the LiteLLM gateway through which the backend makes its model-provider calls. All LLM proposal calls are routed through this gateway so that provider selection, fallback attempts, and rate-limit state are handled in one place.
• postgres — a PostgreSQL instance that supports LiteLLM. It is separate from application-owned storage and is not the pipeline's durable source of truth; the sovereign Supabase PostgreSQL database remains the durable source of truth at every step.

These services add no product requirements and do not change the access decisions above. The pipeline worker that claims queue work and drives extraction, proposal, evaluation, and sealing runs inside the shared backend process, triggered by the scheduled trigger or the operator's manual trigger; it is not a separate always-on daemon, because no guaranteed always-on process is available on Replit free tier.

2b. Source Content Inventory

Not applicable. No reference directive in the authoritative source declares a content_source; the source material is the user-authored requirement thread and the already-built backend description, both of which are preserved as requirements rather than as a content inventory.

2c. Page Content and Component Coverage

The pipeline is delivered as a headless, background-automated system with a single first-party operator surface. The page contract for this project is a single operator console page plus an anonymous entry state, both owned by the first-party application.

Ingestion Operator Console

The single first-party operator surface. It is the working context for submitting raw sources, monitoring extraction and chunking, monitoring LLM proposal generation, monitoring candidate evaluation and sealing, and inspecting the durable state of every source, chunk, proposal, and candidate. It is protected: it is only reachable after identity is established.

• Information and state
  • List of submitted sources with: source kind (PDF / YouTube URL / plain text excerpt), original reference (file name, URL, or excerpt label), submission timestamp, current pipeline stage (submitted / extracting / chunking / proposing / evaluating / sealing / done / failed / parked), and per-source counts of chunks, proposals, candidates, sealed rules, and failures.
  • Per-source detail: ordered chunk list with source location metadata (page number for PDFs, timestamp for YouTube, excerpt offset for plain text), chunk text preview, chunk status (pending / claimed / proposed / failed / parked), and the LLM proposal (or proposals) generated from each chunk.
  • Per-candidate detail: candidate_id, revision, proposal, code_artifact, hoare_triple, domain_claim, domain_type, concept_maturity, digest, created_at, and the six gate receipts (gate_id G1–G6, result PASS/FAIL/INDETERMINATE/NOT_RUN, measured_values, policy_digest, input_digest, checker_version, execution_verified).
  • Per-sealed-rule detail: rule_id, seal_id, title, code_artifact, hoare_triple, active, activated_at.
  • Provider state: current LLM provider/model in use, per-model availability, rate-limit state, and the list of work items parked in provider-retry/fallback state.
  • Pipeline control state: whether the pipeline is currently running, last successful run timestamp, and the next scheduled run.
• Primary actions

• Submit a new raw source: choose source kind (PDF upload, YouTube video URL, or plain text excerpt), provide the source reference, and confirm submission. Submission writes the source row and its initial queue state to Postgres before any extraction begins.
• Start or resume a pipeline run: trigger the pipeline worker for the current operator's pending work. This is the operator's manual trigger; the same work is also reachable via the scheduled trigger.
• Inspect a source, chunk, proposal, candidate, or sealed rule in detail.
• Retry a parked work item (provider-retry/fallback state) once a provider is available again.
• Acknowledge and dismiss a failed work item from the active view without deleting it (failed candidates and failed chunks remain permanently recorded).
• Supporting actions
  • Filter and sort sources, chunks, proposals, candidates, and sealed rules by stage, status, source kind, and timestamp.
  • Copy a candidate_id, rule_id, or seal_id for reference.
  • View the raw extracted text of a chunk and the exact prompt sent to the LLM for that chunk.
• Domain entities
  • Source (raw input: PDF, YouTube URL, or plain text excerpt).
  • Chunk (bounded, ordered piece of extracted text with source location metadata).
  • Queue entry (pending chunk or pending LLM proposal work item).
  • LLM proposal (the model's returned function plus Hoare-style contract, before insertion).
  • Rule candidate (rule_candidates row).
  • Gate receipt (rule_gate_receipts row).
  • Sealed rule (sovereign_rules row).
  • Provider state (model availability, rate-limit state, parked work items).

• Component responsibilities
  • Source submission form: collects source kind and reference, validates the reference shape (URL for YouTube, file for PDF, non-empty text for excerpt), and writes the source row and initial queue state to Postgres.
  • Source list and detail view: renders the durable state of every source and its downstream chunks, proposals, candidates, and sealed rules.
  • Chunk inspector: renders ordered chunks with source location metadata and chunk text preview.
  • Proposal inspector: renders the LLM proposal, the exact prompt used, and the model that produced it.
  • Candidate inspector: renders the rule_candidates row and its six gate receipts.
  • Sealed rule inspector: renders the sovereign_rules row.
  • Provider state panel: renders current model, availability, rate-limit state, and parked work items.
  • Pipeline control: triggers a pipeline run and shows last-run and next-run state.
  • Retry control: re-enqueues a parked work item.
• States
  • Loading: skeleton rows for sources, chunks, proposals, candidates, and sealed rules while the durable state is fetched.
  • Empty: no sources submitted yet — a single call to action to submit the first raw source; no chunks, proposals, candidates, or sealed rules yet — explanatory empty states tied to the parent source's stage.
  • Success: a source shows its full downstream chain (chunks → proposals → candidates → sealed rules) with per-stage counts; a sealed rule shows its rule_id, seal_id, title, and activated_at.
  • Error: a source, chunk, proposal, or candidate in a failed state shows the failure reason (extraction failure, chunking failure, LLM malformed/oversized/unparseable output, gate FAIL/INDETERMINATE/NOT_RUN, finalize_candidate rejection) and the durable record is preserved.

• Recovery: a parked work item (provider unavailable or rate-limited) shows the parked reason and a retry control; retrying re-enqueues the work item without duplicating already-completed work.

Ingestion Entry (anonymous)

The anonymous pre-identity entry state for the operator console. It is the only surface reachable before identity is established. It exists because the protected operator console cannot own the interaction that establishes access to itself.

• Information and state
  • A short explanation of what the pipeline does and that the operator console is protected.
  • The two entry paths: first-use self-service enrollment, and returning verification.
• Primary actions
  • First-use self-service enrollment: establish the operator's identity for the pipeline.
  • Returning verification: verify the operator's identity to resume their durable ingestion state.
• Supporting actions
  • Read the short explanation of the pipeline's purpose and the zero-budget free-tier operating context.
• Domain entities
  • Operator identity (application-owned, bound to the operator's durable ingestion state).
• Component responsibilities
  • Entry explanation block.
  • Enrollment form (first use).
  • Verification form (returning use).
• States
  • Loading: entry explanation and forms render immediately; no durable state is fetched before identity is established.
  • Empty: first-use state shows the enrollment form; returning state shows the verification form.
  • Success: identity established or verified; the operator is taken to the protected operator console.
  • Error: enrollment or verification failure shows a clear message and a retry path; no protected ingestion state is exposed.
  • Recovery: a failed enrollment or verification can be retried without losing any previously established identity or durable ingestion state.

3. Functional Requirements

Each requirement is a distinct story point with provenance, lifecycle facts, and observable acceptance. Provenance is explicit (stated in the authoritative user requirement thread), basic_default (accepted default), or required_inference (indispensable inferred mechanics).

3.1 Source Submission

FR-1 — Submit a raw source. As the Solo Operator, I should be able to submit a raw source of one of three kinds — a PDF textbook/paper, a YouTube video URL, or a plain text excerpt from a course — so that the pipeline can turn it into validated rule candidates.

• Provenance: explicit.
• Trigger/input: operator selects a source kind and provides the source reference (PDF file, YouTube URL, or plain text excerpt).
• Observable result: a durable source row is written to Postgres with the source kind, reference, submission timestamp, and initial pipeline stage; the source appears in the operator console.
• Access state: protected — requires established identity.
• Failure/recovery: an invalid reference (malformed URL, unreadable PDF, empty excerpt) is rejected at submission with a clear message; no partial source row is left in an inconsistent state.
• Continuation: the source is enqueued for extraction.

FR-2 — Durable source state. As the Solo Operator, I should have every submitted source recorded durably in Postgres before any extraction begins, so that a Replit container sleep or restart never loses a submitted source.

• Provenance: explicit (Postgres must be the durable source of truth at every step).
• Trigger/input: source submission.
• Observable result: the source row and its initial queue state are committed to Postgres before any extraction work starts.
• Access state: protected.
• Failure/recovery: if the commit fails, submission fails cleanly and the operator can retry.
• Continuation: extraction proceeds only after the durable commit.

3.2 Extraction

FR-3 — Extract text from a PDF. As the Solo Operator, I should have the pipeline extract text content from a submitted PDF textbook/paper so that its content can be chunked and proposed as rules.

• Provenance: explicit.
• Trigger/input: a submitted PDF source in the extraction stage.
• Observable result: extracted text is written durably to Postgres, associated with the source, with page-level location metadata preserved.
• Access state: protected.
• Failure/recovery: extraction failure (corrupt PDF, unsupported encoding, memory/time limit on Replit free tier) is recorded durably against the source with a failure reason; the source is not deleted and can be retried.
• Continuation: the extracted text is enqueued for chunking.

FR-4 — Extract text from a YouTube video. As the Solo Operator, I should have the pipeline extract transcript/captions from a submitted YouTube video URL so that its content can be chunked and proposed as rules.

• Provenance: explicit.
• Trigger/input: a submitted YouTube URL source in the extraction stage.
• Observable result: extracted transcript text is written durably to Postgres, associated with the source, with timestamp-level location metadata preserved.
• Access state: protected.
• Failure/recovery: extraction failure (no captions available, video unavailable, rate limit) is recorded durably against the source with a failure reason; the source is not deleted and can be retried.
• Continuation: the extracted transcript is enqueued for chunking.

FR-5 — Accept a plain text excerpt. As the Solo Operator, I should be able to submit a plain text excerpt from a course directly, so that short course material can be processed without a PDF or video.

• Provenance: explicit.
• Trigger/input: operator pastes or uploads a plain text excerpt.
• Observable result: the excerpt text is written durably to Postgres as the source's extracted text, with excerpt-offset location metadata.
• Access state: protected.
• Failure/recovery: an empty or unreadable excerpt is rejected at submission.
• Continuation: the excerpt is enqueued for chunking.

3.3 Chunking

FR-6 — Chunk extracted text into bounded, ordered pieces. As the Solo Operator, I should have extracted text split into bounded, ordered pieces so that each piece can be sent to the LLM as a self-contained proposal unit.

• Provenance: explicit.
• Trigger/input: extracted text for a source.
• Observable result: an ordered list of chunks, each with a bounded size and a stable order index, written durably to Postgres.
• Access state: protected.
• Failure/recovery: chunking failure is recorded durably against the source; the source is not deleted and can be retried.
• Continuation: each chunk is enqueued as a pending proposal work item.

FR-7 — Preserve source location metadata on every chunk. As the Solo Operator, I should have every chunk carry source location metadata (page number for PDFs, timestamp for YouTube, excerpt offset for plain text) so that every proposed rule can be traced back to its exact source location.

• Provenance: explicit.
• Trigger/input: chunking.
• Observable result: each chunk row includes its source location metadata alongside its text and order index.
• Access state: protected.
• Failure/recovery: a chunk without location metadata is not enqueued; the source is marked failed with a clear reason.
• Continuation: the chunk is enqueued for proposal.

FR-8 — Durable chunk storage. As the Solo Operator, I should have every chunk stored durably in Postgres so that work is never lost across Replit container sleep or restart.

• Provenance: explicit.
• Trigger/input: chunking.
• Observable result: all chunks for a source are committed to Postgres before any proposal work starts.
• Access state: protected.
• Failure/recovery: a partial chunk commit is rolled back; the source can be re-chunked without duplicating already-committed chunks.
• Continuation: proposal work proceeds only after the durable commit.

3.4 Rule Proposal

FR-9 — Send a chunk to the LLM for rule proposal. As the Solo Operator, I should have the pipeline send a chunk (or a set of chunks) to an LLM asking it to propose ONE small, pure, self-contained Python function that encodes a provable fact/rule from that text, along with a Hoare-style precondition/postcondition contract.

• Provenance: explicit.
• Trigger/input: a pending chunk work item.
• Observable result: the LLM's response is captured durably in Postgres as a proposal record associated with the chunk.
• Access state: protected.
• Failure/recovery: provider unavailability or rate limiting parks the work item in an explicit provider-retry/fallback state; the work item is not lost and is not processed with a weaker model.
• Continuation: the proposal is validated before insertion.

FR-10 — Enforce the proposal shape. As the Solo Operator, I should have the LLM asked for a function that is small, pure, self-contained, with a single entry point, no imports, no side effects, 120–2500 characters, and loop-free preferred, so that proposals are compatible with the existing gate pipeline.

• Provenance: explicit.
• Trigger/input: the proposal prompt.
• Observable result: the prompt explicitly states each of these constraints, and the returned proposal is checked against them before insertion.
• Access state: protected.
• Failure/recovery: a proposal that violates any constraint is handled per FR-13.
• Continuation: a conforming proposal proceeds to insertion.

FR-11 — Require a Hoare-style contract. As the Solo Operator, I should have the LLM return a Hoare-style precondition/postcondition contract alongside the function, so that the existing gate pipeline can validate the rule against its contract.

• Provenance: explicit.
• Trigger/input: the proposal prompt.
• Observable result: the proposal record includes both the function code and the Hoare-style contract.
• Access state: protected.
• Failure/recovery: a proposal missing a contract is handled per FR-13.
• Continuation: a proposal with a contract proceeds to insertion.

FR-12 — Strict parseable output format. As the Solo Operator, I should have the LLM prompt require a strict, parseable output format so that the pipeline can reliably extract the function and contract without ambiguity.

• Provenance: explicit (question 4 asks for an exact prompt structure yielding a strict parseable format).
• Trigger/input: the proposal prompt.
• Observable result: the prompt specifies the exact output structure, and the pipeline parses the response against that structure.
• Access state: protected.
• Failure/recovery: an unparseable response is handled per FR-13.
• Continuation: a parseable response proceeds to insertion.

FR-13 — Detect malformed, oversized, or unparseable LLM output before insertion. As the Solo Operator, I should have the pipeline detect malformed, oversized, or unparseable LLM output before it is inserted as a candidate, so that the existing gate pipeline is not polluted with proposals that cannot be evaluated.

• Provenance: explicit (question 5 asks whether to detect before insertion or insert everything and let the gate pipeline reject it).
• Trigger/input: an LLM response.
• Observable result: the response is checked for parseability, size bounds (120–2500 characters), single entry point, no imports, no side effects, and presence of a Hoare-style contract; a response that fails any check is recorded durably as a rejected proposal with the failure reason and is not inserted into rule_candidates.
• Access state: protected.
• Failure/recovery: the rejected proposal is preserved durably; the chunk is marked failed and the pipeline moves to the next chunk.
• Continuation: the next pending chunk is processed.

FR-14 — Provider fallback without weakening validation. As the Solo Operator, I should have LLM calls go through a free-tier-compatible provider (e.g., OpenRouter free models) with explicit fallback handling if a model becomes unavailable or rate-limited, without silently degrading validation strength.

• Provenance: explicit.
• Trigger/input: an LLM call.
• Observable result: the pipeline records the provider and model used for each proposal; if the primary model is unavailable or rate-limited, the pipeline attempts an explicit fallback model; if no acceptable model is available, the work item is parked in an explicit provider-retry/fallback state.
• Access state: protected.
• Failure/recovery: parked work items are visible in the operator console with the parked reason and a retry control.
• Continuation: a parked work item is retried when a provider is available again.

3.5 Candidate Insertion

FR-15 — Insert the LLM proposal as a new rule_candidates row. As the Solo Operator, I should have the LLM's proposal inserted as a new row in rule_candidates respecting the exact existing schema (candidate_id, revision, proposal, code_artifact, hoare_triple, domain_claim, domain_type, concept_maturity, digest, created_at), so that the existing gate pipeline can evaluate it.

• Provenance: explicit.
• Trigger/input: a validated proposal.
• Observable result: a new immutable rule_candidates row is written with all required fields populated, including a computed digest.
• Access state: protected.
• Failure/recovery: an insertion failure is recorded durably; the proposal is preserved and can be retried.
• Continuation: the new candidate is enqueued for evaluation.

FR-16 — Respect rule_candidates immutability. As the Solo Operator, I should have rule_candidates rows remain immutable once inserted (trigger-enforced), so that the existing backend's integrity guarantees are preserved.

• Provenance: explicit.
• Trigger/input: candidate insertion.
• Observable result: the pipeline never updates or deletes a rule_candidates row; corrections are made by inserting a new revision.
• Access state: protected.
• Failure/recovery: an attempted update or delete is rejected by the existing trigger; the pipeline treats this as a hard error and does not retry the mutation.
• Continuation: the pipeline proceeds with the immutable row as-is.

3.6 Automatic Evaluation and Sealing

FR-17 — Automatically trigger engine_main.py's evaluate() on each new candidate. As the Solo Operator, I should have the pipeline automatically trigger engine_main.py's evaluate() on each newly inserted candidate, so that I no longer run the gate scripts by hand.

• Provenance: explicit.
• Trigger/input: a newly inserted rule_candidates row.
• Observable result: evaluate(candidate) runs the existing gates G1–G6 and posts gate receipts to rule_gate_receipts.
• Access state: protected.
• Failure/recovery: an evaluation failure is recorded durably; the candidate remains permanently recorded and can be re-evaluated.
• Continuation: the candidate's gate receipts determine the next step.

FR-18 — Automatically call finalize_candidate when all 6 gates pass. As the Solo Operator, I should have the pipeline automatically call finalize_candidate when all 6 gates pass, so that sealed rules are produced without manual SQL or manual finalize calls.

• Provenance: explicit.
• Trigger/input: a candidate whose six gate receipts are all PASS against the current approved policy_digest.
• Observable result: sovereign.finalize_candidate(candidate_id, revision) is called; on success it returns the new rule_id and a sovereign_rules row is written.
• Access state: protected.
• Failure/recovery: a finalize_candidate rejection (duplicate code/title, policy mismatch, advisory lock contention) is recorded durably; the candidate remains permanently recorded.
• Continuation: the pipeline moves to the next chunk.

FR-19 — Leave failed candidates as-is and move on. As the Solo Operator, I should have the pipeline leave a candidate as-is (already permanently recorded, never deleted) when any gate fails, and move to the next chunk, so that failed candidates are preserved for inspection and the pipeline keeps making progress.

• Provenance: explicit.
• Trigger/input: a candidate with any gate receipt that is FAIL, INDETERMINATE, or NOT_RUN.
• Observable result: the candidate and its receipts remain durably recorded; the pipeline advances to the next pending chunk.
• Access state: protected.
• Failure/recovery: no deletion or mutation of the failed candidate occurs.
• Continuation: the next pending chunk is processed.

FR-20 — Preserve the existing validation and sealing paths. As the Solo Operator, I should have the existing engine evaluation and finalize_candidate flow remain the only validation and sealing paths, so that the pipeline never introduces an alternative validation or sealing mechanism.

• Provenance: required_inference (indispensable to preserve the explicit constraint that finalize_candidate is the ONLY path to seal a rule).
• Trigger/input: any candidate evaluation or sealing.
• Observable result: all evaluation goes through engine_main.py's evaluate(); all sealing goes through sovereign.finalize_candidate.
• Access state: protected.
• Failure/recovery: any attempt to bypass these paths is treated as a hard error.
• Continuation: the pipeline proceeds only through the existing paths.

3.7 Durable Queue and Resume

FR-21 — Postgres-only job queue. As the Solo Operator, I should have the job queue implemented using only Postgres tables (no Redis, no external queue service) to track pending chunks and pending LLM proposals, so that a crashed or sleeping worker can resume safely without duplicating work or losing a chunk.

• Provenance: explicit.
• Trigger/input: chunking and proposal work.
• Observable result: pending chunks and pending LLM proposals are represented as durable rows in Postgres queue tables with explicit status, claim, and lease fields.
• Access state: protected.
• Failure/recovery: a crashed or sleeping worker's claimed work items are reclaimed after their lease expires; no work item is lost or duplicated.
• Continuation: the resumed worker picks up pending work items in order.

FR-22 — Survive Replit container sleep/restart. As the Solo Operator, I should have the pipeline survive Replit container sleep/restart without losing in-progress work, with nothing relying on local disk or in-memory state as the source of truth.

• Provenance: explicit.
• Trigger/input: any pipeline stage.
• Observable result: all pipeline state is read from and written to Postgres; no stage depends on local disk or in-memory state for correctness.
• Access state: protected.
• Failure/recovery: after a restart, the pipeline resumes from the durable state in Postgres.
• Continuation: the pipeline continues from the last durable state.

FR-23 — Idempotent work claiming. As the Solo Operator, I should have work claiming be idempotent so that a worker that wakes up after a sleep does not duplicate work.

• Provenance: required_inference (indispensable to satisfy the explicit "without duplicating work" requirement).
• Trigger/input: a worker claiming a pending work item.
• Observable result: a work item is claimed atomically (e.g., via a conditional update or advisory lock) so that only one worker processes it at a time.
• Access state: protected.
• Failure/recovery: a claim that fails because the item is already claimed is treated as a no-op; the worker moves to the next item.
• Continuation: the worker processes the claimed item.

3.8 Operator Console and Access

FR-24 — First-use self-service enrollment. As the Solo Operator, I should be able to establish my identity on first use through self-service enrollment, so that I can privately own and resume my durable ingestion state.

• Provenance: required_inference (indispensable to make the accepted journey executable; the operator must privately own durable actor-specific state).
• Trigger/input: first visit to the anonymous entry state.
• Observable result: an application-owned operator identity is established and bound to the operator's durable ingestion state.
• Access state: anonymous entry state.
• Failure/recovery: enrollment failure shows a clear message and a retry path; no protected ingestion state is exposed.
• Continuation: the operator is taken to the protected operator console.

FR-25 — Returning verification. As the Solo Operator, I should be able to verify my identity on return, so that I can resume my durable ingestion state.

• Provenance: required_inference (indispensable to make the accepted journey executable).
• Trigger/input: a returning visit to the anonymous entry state.
• Observable result: the operator's identity is verified and their durable ingestion state is made available.
• Access state: anonymous entry state.
• Failure/recovery: verification failure shows a clear message and a retry path; no protected ingestion state is exposed.
• Continuation: the operator is taken to the protected operator console.

FR-26 — Protected operator console. As the Solo Operator, I should have the operator console protected so that my durable ingestion state is only reachable after identity is established.

• Provenance: required_inference (indispensable to protect durable actor-specific state).
• Trigger/input: any attempt to reach the operator console.
• Observable result: the operator console is only reachable after identity is established; the anonymous entry state is the only surface reachable before identity is established.
• Access state: protected.
• Failure/recovery: an unauthenticated attempt is redirected to the anonymous entry state.
• Continuation: after identity is established, the operator reaches the operator console.

FR-27 — Inspect durable pipeline state. As the Solo Operator, I should be able to inspect the durable state of every source, chunk, proposal, candidate, and sealed rule from the operator console, so that I can monitor the pipeline from my phone.

• Provenance: explicit (the operator must monitor the pipeline from a mobile phone).
• Trigger/input: operator opens a source, chunk, proposal, candidate, or sealed rule.
• Observable result: the operator console renders the durable state, including per-stage counts, failure reasons, and provider state.
• Access state: protected.
• Failure/recovery: a fetch failure shows a clear error and a retry path.
• Continuation: the operator can act on the inspected state (retry, dismiss, or continue monitoring).

FR-28 — Retry a parked work item. As the Solo Operator, I should be able to retry a work item parked in provider-retry/fallback state once a provider is available again, so that no work is permanently stuck.

• Provenance: required_inference (indispensable to make the explicit provider fallback requirement usable).
• Trigger/input: operator selects a parked work item and retries it.
• Observable result: the work item is re-enqueued without duplicating already-completed work.
• Access state: protected.
• Failure/recovery: a retry that fails again re-parks the work item with an updated reason.
• Continuation: the work item is processed when a provider is available.

FR-29 — No paid service, VPS, or persistent server. As the Solo Operator, I should have the pipeline require no paid service, VPS, or persistent server, so that it operates within my zero-budget constraint.

• Provenance: explicit.
• Trigger/input: any pipeline stage.
• Observable result: the pipeline runs entirely on Supabase free tier and Replit free tier.
• Access state: protected.
• Failure/recovery: any stage that would require a paid service is not implemented.
• Continuation: the pipeline continues within free-tier limits.

FR-30 — No arbitrary code execution of LLM-generated code. As the Solo Operator, I should have no arbitrary code execution of LLM-generated code outside a strictly bounded, isolated process, with the generated Python only parsed/analyzed (AST) and never executed with real inputs during validation.

• Provenance: explicit.
• Trigger/input: any LLM-generated code.
• Observable result: the generated code is only parsed and analyzed by the existing gate pipeline; it is never executed with real inputs during validation.
• Access state: protected.
• Failure/recovery: any attempt to execute generated code is treated as a hard error.
• Continuation: the pipeline proceeds with AST-only analysis.

4. User Personas

Solo Operator

Product context. The Solo Operator is the sole human actor for rules-ingestion-pipeline. They are non-technical and currently work from a mobile phone only, with no computer available. They already operate a working backend (Supabase PostgreSQL in schema sovereign, Python gate scripts on Replit free tier) that validates and seals rules through a 6-gate pipeline. Their recurring problem is that every rule candidate is created manually: they write the Python function by hand, insert it into rule_candidates via SQL, run the gate scripts by hand, and call finalize_candidate by hand. They need the automated ingestion pipeline to replace that manual work.

Primary goal. Submit a raw source (PDF textbook/paper, YouTube video URL, or plain text course excerpt) and have the pipeline turn it into validated rule candidates and, when all six gates pass, into sealed sovereign rules — without manual SQL insertion, manual gate runs, or manual finalize calls — all within zero-budget free-tier limits and from a mobile phone.

Distinct accepted responsibilities.

• Submit raw sources of three kinds: PDF textbook/paper, YouTube video URL, and plain text course excerpt.
• Monitor the pipeline's durable state: sources, chunks, proposals, candidates, gate receipts, and sealed rules.
• Inspect failed candidates and rejected proposals, which are permanently recorded and never deleted.
• Retry work items parked in provider-retry/fallback state when a provider is available again.
• Keep the automated path from raw source to validated rule candidate running within zero-budget free-tier limits.

Relevant inputs and decisions.

• Inputs: PDF files, YouTube video URLs, plain text excerpts, and the operator's own monitoring and retry decisions.
• Decisions: which source to submit next; whether to retry a parked work item; whether to inspect a failed candidate or move on.

Interactions with other accepted participants. The Solo Operator is the only human actor. The pipeline interacts with external providers (the LLM provider, e.g., OpenRouter free models) and with the existing backend (Supabase Postgres, the Python gate scripts, and finalize_candidate). These are non-persona actors; the operator does not interact with them directly except through the operator console.

Observable success. A source the operator submitted becomes one or more sealed sovereign rules without manual SQL insertion, manual gate runs, or manual finalize calls; failed candidates remain permanently recorded and inspectable; the pipeline survives Replit container sleep and restart without losing in-progress work; and no paid service, VPS, or persistent server is required.

Source-backed constraints. Zero budget; Supabase free tier (pauses after 1 week inactivity, no automatic backups); Replit free tier (containers sleep after inactivity, local disk is NOT persistent, no guaranteed always-on process); non-technical solo operator working from a mobile phone only; no arbitrary code execution of LLM-generated code outside a strictly bounded, isolated process; LLM calls through a free-tier-compatible provider with explicit fallback handling without silently degrading validation strength; Postgres as the durable source of truth at every step; no paid service, VPS, or persistent server.

5. Core User Flows

Flow 1 — First-use enrollment and first source submission

1. The Solo Operator opens the pipeline on their phone and lands on the anonymous Ingestion Entry state.
2. The operator reads the short explanation of the pipeline and chooses first-use self-service enrollment.
3. The operator completes enrollment; an application-owned operator identity is established and bound to their durable ingestion state.
4. The operator is taken to the protected Ingestion Operator Console.
5. The operator submits a raw source: they choose a source kind (PDF, YouTube URL, or plain text excerpt) and provide the source reference.
6. The pipeline writes the source row and its initial queue state durably to Postgres before any extraction begins.
7. The source appears in the operator console with its current pipeline stage.
8. Failure/recovery: if the reference is invalid (malformed URL, unreadable PDF, empty excerpt), submission is rejected with a clear message and no partial source row is left in an inconsistent state; the operator can correct and resubmit.
9. Continuation: the source is enqueued for extraction.

Flow 2 — Returning verification and pipeline monitoring

1. The Solo Operator returns to the pipeline on their phone and lands on the anonymous Ingestion Entry state.
2. The operator chooses returning verification and verifies their identity.
3. The operator is taken to the protected Ingestion Operator Console, where their durable ingestion state is available.
4. The operator inspects sources, chunks, proposals, candidates, gate receipts, and sealed rules.
5. Failure/recovery: if verification fails, a clear message and a retry path are shown; no protected ingestion state is exposed.
6. Continuation: the operator can act on the inspected state (retry a parked work item, dismiss a failed work item from the active view, or continue monitoring).

Flow 3 — PDF source to sealed rule

1. The Solo Operator submits a PDF textbook/paper from the Ingestion Operator Console.
2. The pipeline writes the source row durably to Postgres and enqueues it for extraction.
3. The pipeline extracts text from the PDF, preserving page-level location metadata, and writes the extracted text durably to Postgres.
4. The pipeline chunks the extracted text into bounded, ordered pieces, each carrying its page number, and writes the chunks durably to Postgres.
5. Each chunk is enqueued as a pending proposal work item in the Postgres-only job queue.
6. A worker claims a pending chunk idempotently and sends it to the LLM provider with the strict proposal prompt.
7. The LLM returns a single small, pure, self-contained Python function (single entry point, no imports, no side effects, 120–2500 characters, loop-free preferred) plus a Hoare-style precondition/postcondition contract in the strict parseable format.
8. The pipeline validates the response for parseability, size bounds, single entry point, no imports, no side effects, and presence of a Hoare-style contract.
9. If the response is valid, the pipeline inserts it as a new immutable rule_candidates row with all required fields populated, including a computed digest.
10. The pipeline automatically triggers engine_main.py's evaluate() on the new candidate; the existing gates G1–G6 run and post gate receipts to rule_gate_receipts.
11. If all six gate receipts are PASS against the current approved policy_digest, the pipeline automatically calls sovereign.finalize_candidate(candidate_id, revision); on success it returns the new rule_id and a sovereign_rules row is written.

12. If any gate receipt is FAIL, INDETERMINATE, or NOT_RUN, the candidate is left as-is (permanently recorded, never deleted) and the pipeline moves to the next chunk.
13. Failure/recovery: if the LLM provider is unavailable or rate-limited, the work item is parked in an explicit provider-retry/fallback state; the operator can retry it from the operator console once a provider is available again. If the LLM response is malformed, oversized, or unparseable, it is recorded durably as a rejected proposal with the failure reason and is not inserted into rule_candidates; the chunk is marked failed and the pipeline moves to the next chunk.
14. Continuation: the pipeline processes the next pending chunk; the operator can inspect the sealed rule, the failed candidates, and the rejected proposals in the operator console.

Flow 4 — YouTube source to sealed rule

1. The Solo Operator submits a YouTube video URL from the Ingestion Operator Console.
2. The pipeline writes the source row durably to Postgres and enqueues it for extraction.
3. The pipeline extracts the transcript/captions, preserving timestamp-level location metadata, and writes the extracted transcript durably to Postgres.
4. The pipeline chunks the transcript into bounded, ordered pieces, each carrying its timestamp, and writes the chunks durably to Postgres.
5. Steps 5–14 of Flow 3 apply identically, with timestamp metadata in place of page numbers.
6. Failure/recovery: if the video has no captions, is unavailable, or is rate-limited, extraction failure is recorded durably against the source with a failure reason; the source is not deleted and can be retried.

Flow 5 — Plain text excerpt to sealed rule

1. The Solo Operator submits a plain text excerpt from a course from the Ingestion Operator Console.
2. The pipeline writes the excerpt text durably to Postgres as the source's extracted text, with excerpt-offset location metadata.
3. The pipeline chunks the excerpt into bounded, ordered pieces, each carrying its excerpt offset, and writes the chunks durably to Postgres.
4. Steps 5–14 of Flow 3 apply identically, with excerpt offsets in place of page numbers.
5. Failure/recovery: an empty or unreadable excerpt is rejected at submission.

Flow 6 — Provider fallback and parked work recovery

1. A worker claims a pending chunk and attempts an LLM call.
2. The primary free-tier-compatible model is unavailable or rate-limited.
3. The pipeline attempts an explicit fallback model.
4. If the fallback model is available, the proposal proceeds through the normal validation and insertion path (Flow 3, steps 8–14).
5. If no acceptable model is available, the work item is parked in an explicit provider-retry/fallback state and is visible in the operator console with the parked reason.
6. The Solo Operator later opens the Ingestion Operator Console, sees the parked work item, and retries it.
7. The work item is re-enqueued without duplicating already-completed work.
8. Failure/recovery: a retry that fails again re-parks the work item with an updated reason.
9. Continuation: the work item is processed when a provider is available.

Flow 7 — Replit container sleep and resume

1. A worker is mid-pipeline (for example, it has claimed a chunk and is waiting on an LLM response) when the Replit container sleeps.
2. All pipeline state is already durable in Postgres; nothing depends on local disk or in-memory state for correctness.
3. The container restarts (via the scheduled trigger or the operator's manual trigger).
4. The worker reads pending work items from Postgres and reclaims any work item whose lease has expired.
5. Work claiming is idempotent, so no work item is duplicated.
6. Failure/recovery: a work item that was mid-flight is resumed from its last durable state.
7. Continuation: the pipeline continues from the last durable state.

Flow 8 — Inspecting a failed candidate

1. The Solo Operator opens the Ingestion Operator Console and filters candidates by failed status.
2. The operator opens a failed candidate and inspects its rule_candidates row and its six gate receipts, including measured_values, policy_digest, input_digest, checker_version, and execution_verified.
3. The operator sees the failure reason (gate FAIL, INDETERMINATE, or NOT_RUN, or a finalize_candidate rejection such as duplicate code/title or policy mismatch).
4. The candidate remains permanently recorded and is never deleted.
5. Continuation: the operator dismisses the failed candidate from the active view and continues monitoring the pipeline.

6. Visuals Colors and Theme

No CREATIVE DIRECTION block was supplied, and the authoritative source specifies no colors, fonts, or brand. The following is a coherent, accessible default for a phone-first, non-technical operator console. It is labeled as a default and is not product behavior.

[Default — not specified by user]

• Mode: light mode primary, with a dark mode variant.
• Colors (light mode):
  • Background: #F7F5F0 (warm off-white)
  • Surface: #FFFFFF
  • Surface muted: #EFEBE3
  • Border: #D9D3C7
  • Text primary: #1F1B16
  • Text secondary: #5C554A
  • Accent (primary action): #1F6F5C (deep teal-green)
  • Accent hover: #185A4B
  • Success (PASS, sealed): #2E7D4F
  • Warning (INDETERMINATE, parked): #B26A00
  • Error (FAIL, rejected): #B3261E
  • Info (NOT_RUN, pending): #3A5A8C
• Colors (dark mode):
  • Background: #14120F
  • Surface: #1E1B17
  • Surface muted: #2A2621
  • Border: #3A352E
  • Text primary: #F2EEE7
  • Text secondary: #B8B0A3
  • Accent: #4FB39A

• Success: #5FBF85
• Warning: #E0A24A
• Error: #E5736C
• Info: #7FA3D6
• Fonts: headings in "IBM Plex Sans", system-ui, sans-serif; body in "IBM Plex Sans", system-ui, sans-serif; code and identifiers in "IBM Plex Mono", ui-monospace, monospace.
• Type scale: 12 / 14 / 16 / 20 / 24 / 32 px, with 16 px body and 24–32 px section headings.
• Radius and shape: 8 px radius for cards and inputs, 999 px for status pills, 4 px for code blocks.
• Spacing rhythm: 4 / 8 / 12 / 16 / 24 / 32 px; 16 px page gutter on mobile.
• Imagery style: none required; the console is data-dense and text-first, with status pills and monospace identifiers carrying the visual weight.

7. Signature Design Concept

The public entry is the anonymous Ingestion Entry state. Its signature concept is a "source-to-seal" vertical trace: a single, phone-width vertical line that runs from a source icon at the top, through three labeled nodes (Extract, Chunk, Propose), to a candidate node, and finally to a sealed-rule node at the bottom. Each node is a small, tappable card showing the stage name and a one-line description of what happens there. The line is drawn in the accent color (#1F6F5C) and the nodes use the status colors (info, warning, success) to preview the pipeline's real states. The two entry actions — first-use enrollment and returning verification — sit as two clearly separated buttons below the trace. The concept recomposes only accepted content (the pipeline's stages and the two entry paths); it introduces no new behavior, page, or destination. It is implementable as a single responsive column with CSS-drawn connectors and no imagery.

8. Interaction Model & Motion Direction

No CREATIVE DIRECTION block was supplied. The tempo is chosen for the product's audience: a non-technical solo operator on a mobile phone, often on a slow connection, who needs to read durable state clearly and act without distraction. The appropriate tempo is restrained.

Interaction Model: Static Motion Tempo: restrained Hero Dimensionality: flat

Landing Hero Motion Brief

• Focal subject: the "source-to-seal" vertical trace on the anonymous Ingestion Entry state — a single vertical line connecting the pipeline's stages from source to sealed rule.
• Input → transformation → outcome thesis: as the operator scrolls or as the page loads, the trace's nodes fade in sequentially from top to bottom (source → extract → chunk → propose → candidate → sealed rule), transforming a static diagram into a readable narrative of what the pipeline does; the outcome is that the operator understands the pipeline's stages before choosing an entry path.
• Motion vocabulary: short opacity fades (150–250 ms) with a small upward translate (4–8 px), staggered by 60–80 ms per node; no parallax, no scale, no rotation.
• Composed first frame: the trace's first node (source) is fully visible, the remaining nodes are at 0 opacity, and the two entry buttons are fully visible and immediately tappable.
• Reduced-motion state: all nodes render at full opacity with no translate and no stagger; the trace is fully visible on first paint.

9. Non-Functional Requirements

• NFR-1 — Zero budget. The pipeline must work on Supabase free tier and Replit free tier and must not require any paid service, VPS, or persistent server. Provenance: explicit. Rationale: the operator's hard zero-budget constraint.
• NFR-2 — Supabase free-tier behavior. The pipeline must tolerate Supabase free-tier pausing after 1 week of inactivity and the absence of automatic backups. Provenance: explicit. Rationale: stated free-tier behavior. Mitigation: the operator's scheduled or manual trigger keeps the project active; durable state is the operator's responsibility given no automatic backups.
• NFR-3 — Replit free-tier behavior. The pipeline must tolerate Replit containers sleeping after inactivity, non-persistent local disk, and the absence of a guaranteed always-on process. Provenance: explicit. Rationale: stated free-tier behavior.
• NFR-4 — Postgres as durable source of truth. Nothing may rely on local disk or in-memory state as the source of truth; Postgres must be the durable source of truth at every step. Provenance: explicit.
• NFR-5 — No arbitrary code execution. LLM-generated Python must only be parsed/analyzed (AST) and never executed with real inputs during validation. Provenance: explicit.
• NFR-6 — Provider fallback without weakening validation. LLM calls must go through a free-tier-compatible provider (e.g., OpenRouter free models) with explicit fallback handling if a model becomes unavailable or rate-limited, without silently degrading validation strength. Provenance: explicit.
• NFR-7 — REST-only Supabase access from Python. Python must talk to Supabase exclusively via REST using a service_role key; no direct Postgres connection from Python. Provenance: explicit.

• NFR-8 — Immutability and append-only guarantees. rule_candidates rows are immutable once inserted (trigger-enforced); rule_gate_receipts are append-only with no update/delete allowed; failed candidates are never deleted. Provenance: explicit.
• NFR-9 — Single sealing path. sovereign.finalize_candidate(candidate_id, revision) is the ONLY path to seal a rule; it is idempotent, atomic, checks all 6 gate receipts are PASS against the current approved policy_digest, acquires an advisory lock, rejects exact code/title duplicates against currently active rules, writes the audit event, and returns the new rule_id. Provenance: explicit.
• NFR-10 — Mobile-first operator console. The operator console must be usable from a mobile phone, since the operator has no computer available. Provenance: explicit.
• NFR-11 — No redesign of existing components. The existing sovereign schema tables, the finalize_candidate function, and the Python gate scripts must not be redesigned. Provenance: explicit.
• NFR-12 — No external queue service. The job queue must be implemented using only Postgres tables; no Redis and no external queue service. Provenance: explicit.

10. Tech Stack

Source-specified choices are preserved exactly. Defaults are labeled.

• Database (durable source of truth): Supabase PostgreSQL, schema sovereign, with the existing tables rule_candidates, rule_gate_receipts, sovereign_rules, policy_manifests, policy_bounds, audit_events, and the existing function sovereign.finalize_candidate. New pipeline tables are added in the same schema (see Assumptions and Constraints for the proposed new tables). Provenance: explicit.
• Application-owned database: MariaDB, holding application-owned persistent data (operator identity and the application-side records the console needs). It is separate from the sovereign Supabase PostgreSQL database, which remains the durable source of truth for all pipeline and rule state. [Default — not specified by user]
• Backend service: a shared FastAPI process hosting the application APIs and domain modules. The pipeline's routes and modules are added to this existing backend, reusing its process, port, dependencies, and deployment; no separate container or ingress route is introduced for the pipeline API. [Default — not specified by user]
• Frontend: a React application UI rendering the anonymous Ingestion Entry state and the protected Ingestion Operator Console. [Default — not specified by user]
• LLM gateway: LiteLLM, through which the backend makes all model-provider calls, so that provider selection, explicit fallback attempts, and rate-limit state are handled in one place. [Default — not specified by user]
• Gateway-support database: a PostgreSQL instance supporting LiteLLM. It is separate from application-owned storage and is not the pipeline's durable source of truth. [Default — not specified by user]

• Python runtime: Replit free tier, running the existing gate scripts (engine_main.py, engine_g2.py, engine_g3.py, engine_g5.py, engine_g6.py, engine_seal.py, supa_client.py) plus new pipeline scripts. Provenance: explicit.
• Supabase access from Python: REST exclusively, using a service_role key via supa_client.py; no direct Postgres connection from Python. Provenance: explicit.
• LLM provider: a free-tier-compatible provider such as OpenRouter free models, with explicit fallback handling, reached through the LiteLLM gateway. Provenance: explicit.
• PDF text extraction: pypdf (pure-Python, low memory, no native dependencies) as the primary choice, with pdfminer.six as a fallback for PDFs that pypdf cannot extract. [Default — not specified by user]
• YouTube transcript extraction: youtube-transcript-api (pure-Python, no API key, low memory) as the primary choice. [Default — not specified by user]
• HTTP client: httpx (async-capable, low memory) for LLM provider calls and Supabase REST calls. [Default — not specified by user]
• Scheduling/triggering: a free scheduled trigger (e.g., a Supabase pg_cron job or a free external cron service such as cron-job.org) that calls the Replit pipeline entry point, plus the operator's manual trigger from the operator console. [Default — not specified by user]
• Operator console: a mobile-first React web UI served by the shared FastAPI backend on the Replit container, using the existing Python runtime. [Default — not specified by user]
• Containerization: not required for this project; the pipeline runs on Replit free tier and Supabase free tier. The runnable services are the shared backend (FastAPI), the application-owned MariaDB database, the React frontend, the LiteLLM gateway, and the PostgreSQL instance supporting LiteLLM. [Default — not specified by user]

11. Assumptions and Constraints

Assumptions

• A-1. The existing backend (Supabase schema sovereign, the finalize_candidate function, and the Python gate scripts) is fully built and tested and is consumed as-is. [Assumption — source-stated]
• A-2. The operator is the sole human actor and has full control over their own ingestion state; no differentiated permissions, roles, or visibility rules are needed. [Assumption — source-stated]
• A-3. The operator's phone has a modern mobile browser capable of running the operator console. [Assumption — default]
• A-4. A free scheduled trigger (e.g., a Supabase pg_cron job or a free external cron service) is available to wake the Replit container periodically. [Assumption — default]
• A-5. The LLM provider's free tier is sufficient for the operator's expected volume, and explicit fallback handling covers rate limiting and model unavailability. [Assumption — source-stated]

Proposed New Tables (Concrete Schema Proposals)

These are concrete schema proposals for the new pipeline state. They live in the sovereign schema alongside the existing tables and do not modify any existing table.

• sovereign.ingestion_sources — one row per submitted raw source.
  • source_id (uuid, primary key)
  • operator_id (uuid, references the operator identity)
  • source_kind (text, one of pdf, youtube, text)
  • source_reference (text: file name, URL, or excerpt label)
  • extracted_text (text, nullable until extraction completes)
  • stage (text: submitted, extracting, chunking, proposing, evaluating, sealing, done, failed, parked)
  • failure_reason (text, nullable)
  • created_at (timestamptz)
  • updated_at (timestamptz)
• sovereign.ingestion_chunks — one row per bounded, ordered chunk.
  • chunk_id (uuid, primary key)
  • source_id (uuid, references ingestion_sources)
  • order_index (integer, stable order within the source)
  • chunk_text (text)
  • location_metadata (jsonb: {page: n} for PDFs, {timestamp: "hh:mm:ss"} for YouTube, {offset: n} for plain text)
  • created_at (timestamptz)
• sovereign.ingestion_queue — the Postgres-only job queue for pending chunks and pending LLM proposals.
  • queue_id (uuid, primary key)
  • work_kind (text: chunk_proposal, candidate_evaluation, candidate_sealing)
  • source_id (uuid, nullable)

• chunk_id (uuid, nullable)
• candidate_id (uuid, nullable)
• revision (integer, nullable)
• status (text: pending, claimed, done, failed, parked)
• claimed_by (text, nullable: worker identifier)
• claimed_at (timestamptz, nullable)
• lease_expires_at (timestamptz, nullable)
• attempt_count (integer, default 0)
• last_error (text, nullable)
• created_at (timestamptz)
• updated_at (timestamptz)
• sovereign.ingestion_proposals — one row per LLM proposal, before and after insertion into rule_candidates.
  • proposal_id (uuid, primary key)
  • chunk_id (uuid, references ingestion_chunks)
  • provider (text)
  • model (text)
  • prompt_text (text)
  • raw_response (text)
  • parsed_code_artifact (text, nullable)
  • parsed_hoare_triple (text, nullable)
  • validation_result (text: valid, malformed, oversized, unparseable, missing_contract)
  • validation_reason (text, nullable)
  • candidate_id (uuid, nullable, references rule_candidates once inserted)
  • created_at (timestamptz)
• sovereign.ingestion_provider_state — one row per provider/model availability observation.
  • provider_state_id (uuid, primary key)
  • provider (text)
  • model (text)

• available (boolean)
• rate_limited_until (timestamptz, nullable)
• observed_at (timestamptz)

Constraints

• C-1. Zero budget: must work on Supabase free tier (pauses after 1 week inactivity, no automatic backups) and Replit free tier (containers sleep after inactivity, local disk is NOT persistent, no guaranteed always-on process). [Constraint — source-stated]
• C-2. The operator is a non-technical solo operator currently working from a mobile phone only, with no computer available. [Constraint — source-stated]
• C-3. No arbitrary code execution of LLM-generated code is allowed outside a strictly bounded, isolated process — the generated Python is only parsed/analyzed (AST), never executed with real inputs during validation. [Constraint — source-stated]
• C-4. LLM calls must go through a free-tier-compatible provider (e.g., OpenRouter free models) with explicit fallback handling if a model becomes unavailable or rate-limited, without silently degrading validation strength. [Constraint — source-stated]
• C-5. The pipeline must survive Replit container sleep/restart without losing in-progress work — nothing can rely on local disk or in-memory state as the source of truth; Postgres must be the durable source of truth at every step. [Constraint — source-stated]
• C-6. Must not require any paid service, VPS, or persistent server. [Constraint — source-stated]
• C-7. Existing components are already built and must not be redesigned: the sovereign schema tables, the finalize_candidate Postgres function as the ONLY path to seal a rule, and the Python gate scripts. [Constraint — source-stated]
• C-8. Python talks to Supabase exclusively via REST using a service_role key; no direct Postgres connection from Python. [Constraint — source-stated]

• C-9. rule_candidates rows are immutable once inserted (trigger-enforced); rule_gate_receipts are append-only with no update/delete allowed. [Constraint — source-stated]
• C-10. Failed candidates are never deleted — they remain permanently recorded. [Constraint — source-stated]

Concrete Build Order (Prioritized for Phone-Based Incremental Testing)

This build order answers question 6 of the authoritative source and is prioritized so that each step is a small, testable piece that can be built and verified from a phone.

1. First build step — durable source submission. Add the sovereign.ingestion_sources table and a minimal operator console page that submits a plain text excerpt and writes a durable source row. Verify from the phone that the row persists across a Replit container restart. This is the concrete first-build-step.
2. Chunking. Add the sovereign.ingestion_chunks table and chunk the extracted text into bounded, ordered pieces with location metadata. Verify that chunks persist durably and are ordered correctly.
3. Postgres-only job queue. Add the sovereign.ingestion_queue table with status, claim, and lease fields. Verify that a claimed work item is reclaimed after its lease expires and that claiming is idempotent.
4. PDF extraction. Add pypdf (with pdfminer.six fallback) and extract text from a small PDF. Verify page-level location metadata.
5. YouTube extraction. Add youtube-transcript-api and extract a transcript from a short video. Verify timestamp-level location metadata.
6. LLM proposal. Add the strict proposal prompt and the OpenRouter free-model call with explicit fallback handling. Verify that a valid proposal is returned and parsed.
7. Proposal validation. Add the pre-insertion checks for parseability, size bounds, single entry point, no imports, no side effects, and presence of a Hoare-style contract. Verify that malformed, oversized, and unparseable responses are recorded as rejected proposals and are not inserted.
8. Candidate insertion. Insert a validated proposal as a new immutable rule_candidates row with a computed digest. Verify immutability.

9. Automatic evaluation. Trigger engine_main.py's evaluate() on the new candidate and verify that gate receipts are posted to rule_gate_receipts.
10. Automatic sealing. Call sovereign.finalize_candidate when all six gate receipts are PASS and verify that a sovereign_rules row is written.
11. Failure handling. Verify that a candidate with any non-PASS gate receipt is left as-is and the pipeline moves to the next chunk.
12. Scheduled trigger. Add the free scheduled trigger (e.g., a Supabase pg_cron job or a free external cron service) that wakes the Replit container and runs the pipeline. Verify that the pipeline resumes from durable state after a container sleep.
13. Operator console polish. Add the source list, chunk inspector, proposal inspector, candidate inspector, sealed rule inspector, provider state panel, and retry control. Verify the full monitoring and retry flow from the phone.

12. Glossary

• Rule: a small verified Python function with formal pre/postconditions that encodes a provable fact or rule.
• Rule candidate: a row in sovereign.rule_candidates representing a proposed rule before sealing. Immutable once inserted (trigger-enforced).
• Gate: one of the six validation gates G1–G6 in the existing pipeline. G1 is AST parse/compile bounds check; G4 is AST density/complexity/length metrics; G2, G3, G5, and G6 are the remaining gate checks.
• Gate receipt: a row in sovereign.rule_gate_receipts recording a gate's result (PASS, FAIL, INDETERMINATE, or NOT_RUN) with measured_values, policy_digest, input_digest, checker_version, and execution_verified. Append-only.
• Sealed rule: a row in sovereign.sovereign_rules produced by finalize_candidate, with rule_id, seal_id, title, code_artifact, hoare_triple, active, and activated_at.
• finalize_candidate: the single Postgres function sovereign.finalize_candidate(candidate_id, revision) that is the ONLY path to seal a rule. Idempotent, atomic, checks all 6 gate receipts are PASS against the current approved policy_digest, acquires an advisory lock, rejects exact code/title duplicates against currently active rules, writes the audit event, and returns the new rule_id.
• Policy manifest: a row in sovereign.policy_manifests with manifest_sha256, approved, and approved_at. Versioned approved policy.
• Policy bounds: rows in sovereign.policy_bounds with manifest_version, gate, metric, min_v, and max_v. Numeric thresholds per gate.
• Audit event: a row in sovereign.audit_events with event_id, event_type, event_payload, prev_hash, and event_hash. Hash-chained append-only log.
• Chunk: a bounded, ordered piece of extracted text with source location metadata (page number, timestamp, or excerpt offset).

• Proposal: the LLM's returned function plus Hoare-style contract, before insertion into rule_candidates.
• Provider fallback: the explicit handling of LLM provider unavailability or rate limiting, without silently degrading validation strength.
• Parked work item: a work item in explicit provider-retry/fallback state, visible in the operator console with a retry control.
• Solo Operator: the single human actor for this project — a non-technical operator working from a mobile phone only, with zero budget, on Supabase free tier and Replit free tier.
• Ingestion Operator Console: the protected first-party operator surface for submitting sources, monitoring the pipeline, and inspecting durable state.
• Ingestion Entry: the anonymous pre-identity entry state for the operator console, offering first-use self-service enrollment and returning verification.