# ad2v — full documentation > ad2v (Agentic Development to Visualization) compares a repository with its spec-driven development (SDD) specs and writes one SQLite file (`.ad2v`) that describes both sides as a single graph: a zoomable map, PR diffs, and a read-only MCP server. Version 0.7.0; install with `pip install ad2v`; Apache-2.0. Website: https://ad2v.dev · Source: https://github.com/shanglai/ad2v · PyPI: https://pypi.org/project/ad2v/ · Short index: https://ad2v.dev/llms.txt --- # Overview and CLI # ad2v: agent development to viz **Comprehension debt** is what builds up when agents merge code faster than people can understand it. ad2v compares a **repository** with its **SDD specs** (spec-driven development) and writes **one SQLite file**, `*.ad2v`, that describes both sides as a single graph. The file also records how the spec items and the code pair up. Readers of that file: - a **viewer**: a zoomable isometric 2.5D map (like C4, with more levels) in dark and light themes. Glyph shape = node kind. Line style = status: spec-only items are dotted, implemented solid, verified glowing (dark) or haloed (light), drift carries a warning badge, orphan code is dashed, and code in a language no active extractor reads is thin, dotted and hollow with a "∅" badge (`not_evaluable`). A **Legend** button explains every status. Workflows replay step by step with a detail slider that follows the zoom. An **Architecture** tab shows a deterministic C4 view built from the code, the specs and the infrastructure (Terraform plan JSON or `.tf`): which service runs which container, env vars the code reads that no deployment provides, resources nobody uses, per environment. - **any MCP host's model** (Claude, ChatGPT, VS Code, …) through `ad2v mcp`: read-only tools that find components and explain them, with answers that point at node ids. Hosts with MCP Apps show the map inline in the conversation. Like mermaid.js, ad2v is an open **format** plus reference implementations. License: Apache-2.0. Current version: **0.7.0** ([PyPI](https://pypi.org/project/ad2v/): `pip install ad2v`). Code is read by **language extractors** and frameworks are recognized by **framework packs**, both plug-ins behind a stable public API (`ad2v.api`, api 1.0, entry-point groups `ad2v.extractors` / `ad2v.frameworks`). The built-in Python extractor and Python framework pack use the same API; third-party plug-ins are opt-in ([plug-in guide](https://github.com/shanglai/ad2v/blob/main/docs/plugins.md)). ## Status of the series | phase | scope | status | |---|---|---| | P1 | format spec v0.1, SQLite schema, IR, deterministic extractors (Python code, Gherkin / OpenAPI / Markdown specs), static workflows, canonical dump | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P1.md)) | | P2 | spec↔code pairing, single-epoch statuses, structural drift, roll-ups, risk severity, visual tokens (light + dark), format v0.2 | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P2.md)) | | P3 | content-addressed multi-epoch storage, parse cache, git history, epoch diff, lineage, `stale`, entry-point recognizers, orphan policy, format v0.3 | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P3.md)) | | P4 | bring-your-own-model LLM layer (informative only): semantic judge, link suggestions, summaries, narrations, prose-spec extraction, evaluation; format v0.4 | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P4.md)) | | P5 | three.js viewer: deterministic layout, LOD map, status styling, replay, epoch diff; `ad2v view` | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P5.md), [demo](https://github.com/shanglai/ad2v/blob/main/docs/reports/P5-demo.html)) | | P6 | MCP server (stdlib, protocol 2026-07-28) + the map as an MCP App; deterministic retrieval | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P6.md), [setup](https://github.com/shanglai/ad2v/blob/main/docs/mcp.md)) | | P6.1 | real-repo readiness: route recall (factories, `add_api_route`, Starlette, Flask), spec discovery, spec-kit / Kiro / custom presets, `ad2v specs` bootstrap, replay camera, opaque "dim the rest" | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P6.1.md)) | | P7 | verified quickstart, settings prefixes, retrieval v2, the viewer's own chat agent (BYOM, `ad2v view --chat`), thinking per task, schema digests, judge roll-up, viewer UX v2, format v0.5 | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P7.md), [quickstart](https://github.com/shanglai/ad2v/blob/main/docs/quickstart.md)) | | P8 | extractor plug-in API (`ad2v.api`, entry points, opt-in, conformance kit, out-of-tree examples), blind retrieval eval harness, packaging (not published), CI workflows, security review, sanitization, dogfood procedure | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P8.md), [plug-ins](https://github.com/shanglai/ad2v/blob/main/docs/plugins.md), [security](https://github.com/shanglai/ad2v/blob/main/docs/security.md)) | | P8.1 | `not_evaluable`: a status and legend for code in a language no active extractor reads; format v0.6 | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P8.md) §13, ADR-0066) | | P9 | brownfield onboarding: `ad2v init`, Spanish and English team formats (requirement tables, user stories, Spanish EARS and Gherkin), titles, non-functional requirements, issue 95, the spec side of the map, wider default excludes, fresh-clone install fix | done ([report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P9.md), ADR-0067..0072) | | **P10** | infrastructure layer: Terraform plan JSON (primary) and static `.tf` (python-hcl2), secrets allowlist, the env-var bridge (deploys / wired / unwired / orphan, per environment), a deterministic C4 view (Architecture tab, Structurizr export, `ad2v_architecture`), zero-config warning, test-name prefixes from the specs; format v0.7 | **done** ([report](https://github.com/shanglai/ad2v/blob/main/REPORT.md), [infra](https://github.com/shanglai/ad2v/blob/main/docs/infra.md), ADR-0073..0081) | Extraction makes **no LLM calls**. Every row it writes is *normative* (or *volatile*: deterministic line numbers): the same inputs (code, specs, rules, manifest, test results) give the same normative dump on any platform. The optional LLM layer (`ad2v llm`) writes only *informative* rows with full provenance, never touches an epoch id, and a model can only *suggest*: a human confirms by editing committed files (`ad2v.yaml`, spec files). ## Quickstart Requires Python ≥ 3.11. Install from PyPI (the wheel includes the viewer and the MCP App): ```bash pip install ad2v ``` To work on ad2v itself, install from a clone instead (`uv sync --extra dev` works too): ```bash python -m venv .venv ``` ```bash .venv/Scripts/python -m pip install -e ".[dev]" ``` (On macOS/Linux, use `.venv/bin/python`. A source checkout needs the viewer built to run `ad2v view`; see [Development](#development).) Then, from a clone, extract the bundled fixture, with its test results: ```bash ad2v extract --repo fixtures/payments-recon --spec fixtures/payments-recon/specs --test-results fixtures/payments-recon/test-results/junit.xml --out recon.ad2v ``` ```bash ad2v status recon.ad2v --side spec ``` To follow a repository over time, extract git history into one file (one epoch per commit whose inputs changed) and compare epochs: ```bash ad2v history --repo . --spec specs --commits HEAD~20..HEAD --into project.ad2v ``` ```bash ad2v diff project.ad2v --from parent --to head --format md ``` Open the map in a browser. The server binds 127.0.0.1 only and never writes the file: ```bash ad2v view recon.ad2v ``` To browse history in the viewer, materialize the epochs you want to see first: ```bash ad2v materialize project.ad2v --epochs last:5 ``` A self-contained demo (the fixture's 8-epoch history, with demo LLM rows from the offline `fake` provider) opens from disk: [`docs/reports/P5-demo.html`](https://github.com/shanglai/ad2v/blob/main/docs/reports/P5-demo.html). Ask about the map from Claude (or any MCP host). Add this server to the host's configuration; see [docs/mcp.md](https://github.com/shanglai/ad2v/blob/main/docs/mcp.md) for Claude Desktop, Claude Code and VS Code: ```bash python -m ad2v mcp --root path/to/maps ``` ## CLI ```text ad2v init --repo [--write [--force]] [--format yaml|json] [--check ] # propose a manifest ad2v extract --repo [--spec ...] [--no-spec-autodetect] [--manifest ad2v.yaml] [--test-results junit.xml ...] (--out project.ad2v | --into project.ad2v) [--no-cache] [--infra-plan [ENV=]plan.json ...] [--infra-env NAME] ad2v c4 project.ad2v [--env NAME] [--format structurizr|json] [--out FILE] # the C4 view (P10) ad2v history --repo --commits --into project.ad2v [--spec ...] [--manifest ] [--test-results ...] [--no-cache] ad2v diff project.ad2v [--from ] [--to ] [--format table|json|md] ad2v log project.ad2v [--format table|json] ad2v pin project.ad2v [--label v1.2] ad2v unpin project.ad2v ad2v prune project.ad2v --keep-last N [--keep-pinned] ad2v materialize project.ad2v (--epoch | --epochs last:N) # last N + pinned ad2v layout project.ad2v [--algorithm nestgrid-v1|nestgrid-v2] # v2: experimental shelf packing ad2v view project.ad2v [--port 0] [--no-browser] [--chat --manifest ad2v.yaml [--chat-budget-usd 0.25]] # chat with your own model ad2v mcp (--file project.ad2v | --root ) [--http [--port N]] [--no-warm] # MCP server, read-only ad2v status project.ad2v [--epoch ] [--side spec|code] [--status stale] [--all] [--format table|json] ad2v inspect project.ad2v [--epoch ] [--node ] [--kind loop] [--top-risk N] ad2v dump project.ad2v [--epoch ] [--normative-only] [--out project.jsonl] ad2v validate project.ad2v [--epoch ] ad2v preview-tokens --out tokens.html ``` Without `--spec`, specs are found from `spec_root` in ad2v.yaml, else auto-detected (`specs/`, `spec/`, `docs/specs/`, `.kiro/specs/`, `sdd/`, `docs/sdd/`, a root `openapi.yaml`); `extract` prints what it used. Markdown specs may follow the ad2v convention, spec-kit, Kiro, a requirements table, user stories (Spanish or English), or your own regexes ([convention §2, §6–7](https://github.com/shanglai/ad2v/blob/main/docs/spec/spec-markdown-convention.md)). **Existing project?** `ad2v init --repo ` reads the repository and runs nothing. It proposes an `ad2v.yaml`: spec folders, containers and their languages, test-name prefixes, non-functional prefixes, and route prefixes left empty for you to fill in. It writes only with `--write` ([quickstart §2b](https://github.com/shanglai/ad2v/blob/main/docs/quickstart.md)). An `` is an epoch id, `head`, `parent`, a pin label, or a commit SHA prefix (≥ 7 chars). LLM layer (bring your own model; `fake` for offline use, `vertex`, or any OpenAI-compatible server): ```text ad2v llm plan project.ad2v [--manifest ad2v.yaml] [--tasks judge,suggest,summarize,narrate,extract-prose] [--epoch head|all] [--count-tokens] ad2v llm run project.ad2v --max-cost-usd 0.50 --yes [--manifest ad2v.yaml] [--tasks ...] [--no-llm-cache] [--spec-root specs] [--mode call_depth --workflow ] ad2v llm eval project.ad2v --expected expected-judgments.yaml ad2v llm show-prose project.ad2v ad2v suggestions list|accept|reject project.ad2v [ --manifest ad2v.yaml] ad2v specs bootstrap --from docs/ [--plan | --yes --max-cost-usd 0.50] [--manifest ad2v.yaml] ad2v specs review # proposals: source, model, overlap, staleness, id clashes ad2v specs approve [--to specs/sub] [--force-stale] ``` - `plan` lists the calls still needed after cache hits, with token and USD estimates; `run` refuses to start without a budget and `--yes`, and stops before exceeding it. Results are cached by the exact input, so re-running is free. - The judge's verdicts appear in `node_status_effective` (an LLM `drift` can only upgrade implemented/verified; nothing the model says can clear a problem). `suggestions accept` appends the pair to `pairing.links` in ad2v.yaml; the next extract makes it a normative `manifest_link`. - Prose-spec extraction writes a proposal to `specs/.ad2v-extracted/`; moving it into the spec tree is the approval. - Repository content is sent to the model as fenced, untrusted data; outputs are strict JSON checked deterministically (cited lines must exist, echoed injections are rejected). - `extract --out` writes a fresh single-epoch file; `--into` appends a new head epoch, reusing the file's parse cache (the summary shows the hit rate), or reports "no changes" when no input changed. - `history` exports each commit with `git archive` (never checks out, never runs code) and appends one epoch per commit. - `diff` reports added/removed/renamed nodes (renames follow lineage), status transitions (including `stale`), risk changes, spec changes and workflow changes. `--format md` fits a PR comment (≤ 60 lines); see [`examples/github-action/ad2v-diff.yml`](https://github.com/shanglai/ad2v/blob/main/examples/github-action/ad2v-diff.yml). - `status` lists effective statuses (`drift`, `needs_review`, `missing`, `orphan`, `stale`, `verified`, `implemented`, `not_evaluable`, `specified`) with their reasons. `not_evaluable` (format v0.6): specified and built in a language no active extractor reads (TypeScript, Java, Go, …); annotate such code with `// ad2v: REQ-001` comments so the pairing still shows. `extract` prints which languages were not read. - `prune` keeps the last N epochs (and pinned ones), garbage-collects and vacuums. - `dump --normative-only` is the determinism contract: sorted JSONL, UTF-8, LF on every OS. - `validate` checks the format version (0.4 to 0.6), schema objects (no triggers, no foreign tables or views), text sizes and JSON depth, foreign keys and node references, version keys, the epoch chain, glyph and status coverage, the IR JSON Schema, and that ids, link ids, Merkle hashes and roll-ups recompute. - `view` serves the built viewer on 127.0.0.1 (GET/HEAD only, strict CSP). The viewer is read-only and never calls an LLM. Actions that change state are shown as CLI commands to copy. It shows materialized epochs only. `extract` and `materialize` also write the deterministic layout (`nestgrid-v1`, format spec §12), which keeps boxes in place across epochs. - `extractors list` shows the installed extractor and framework-pack plug-ins (built-in, enabled, disabled); `extractor check --fixture ` runs the conformance kit. `extract` and `history` print each active plug-in with its version and rules hash; `--extractor ` enables an installed third-party plug-in for one run (or list it in `ad2v.yaml` under `extractors.enabled`), `--strict` makes a failing plug-in fatal. - `eval blind` scores a private retrieval question set (path in `AD2V_BLIND_EVAL`) and prints aggregates only ([blind eval](https://github.com/shanglai/ad2v/blob/main/docs/blind-eval.md)). - `preview-tokens` writes one self-contained HTML page: every glyph × every status, dark and light (see [`docs/reports/P2-tokens.html`](https://github.com/shanglai/ad2v/blob/main/docs/reports/P2-tokens.html)). The extractor only **reads** the repository and ingests test-result files. It never imports or runs repository code or tests. ## Documents - [Format spec v0.7](https://github.com/shanglai/ad2v/blob/main/docs/spec/ad2v-format-v0.7.md) (normative: the product; §11 the LLM layer, §12 layout and viewer, §13 MCP, retrieval and chat, §14 the infra layer) - [Glyph grammar v0.2](https://github.com/shanglai/ad2v/blob/main/docs/spec/glyph-grammar-v0.2.md) and [visual tokens v0.2](https://github.com/shanglai/ad2v/blob/main/docs/spec/visual-tokens-v0.2.md) - [Spec Markdown convention](https://github.com/shanglai/ad2v/blob/main/docs/spec/spec-markdown-convention.md) - [DDL](https://github.com/shanglai/ad2v/blob/main/schema/ad2v-v0.7.sql), [IR JSON Schema](https://github.com/shanglai/ad2v/blob/main/schema/ir-v0.7.schema.json), [themes](https://github.com/shanglai/ad2v/blob/main/schema/themes) - [ADRs](https://github.com/shanglai/ad2v/blob/main/docs/adr) and [dependencies and licenses](https://github.com/shanglai/ad2v/blob/main/docs/dependencies.md) - [REPORT.md](https://github.com/shanglai/ad2v/blob/main/REPORT.md) (P10), [infra layer](https://github.com/shanglai/ad2v/blob/main/docs/infra.md), [P9 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P9.md), [P8 + P8.1 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P8.md), [P7 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P7.md), [plug-in guide](https://github.com/shanglai/ad2v/blob/main/docs/plugins.md), [security](https://github.com/shanglai/ad2v/blob/main/docs/security.md), [blind eval](https://github.com/shanglai/ad2v/blob/main/docs/blind-eval.md), [dogfood procedure](https://github.com/shanglai/ad2v/blob/main/docs/dogfood.md), [releasing (not for now)](https://github.com/shanglai/ad2v/blob/main/docs/RELEASING.md), [quickstart](https://github.com/shanglai/ad2v/blob/main/docs/quickstart.md), [MCP setup](https://github.com/shanglai/ad2v/blob/main/docs/mcp.md), [P6.1 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P6.1.md), [P6 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P6.md), [P5 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P5.md), [P4 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P4.md), [P3 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P3.md), [P2 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P2.md), [P1 report](https://github.com/shanglai/ad2v/blob/main/docs/reports/P1.md), [status](https://github.com/shanglai/ad2v/blob/main/status.md), [issues](https://github.com/shanglai/ad2v/blob/main/issues.md), [pending](https://github.com/shanglai/ad2v/blob/main/pending.md) ## Optional manifest (`ad2v.yaml`) ```yaml system: {name: payments-recon} containers: - {name: api, paths: [api], kind: service, replicas: 2} - {name: worker, paths: [worker], kind: worker} exclude: [migrations] # added to the default excludes behavior: {emit_all: false} # true: emit loop/retry/... nodes even without risk flags workflows: {max_depth: 12, max_steps: 5000} spec_ids: {test_name_prefixes: [REQ, CMP]} pairing: components: {CMP-002: worker} # spec component -> container or component path_prefix_map: {/api: ""} # route normalization, both sides case_insensitive: false orphan_policy: auto # auto (orphans only once specs exist) | strict | off links: [] # human-confirmed pairs (accepted suggestions): manifest_link rejected: [] # rejected suggestions, never proposed again path_constants: {"settings.API_V1_STR": "/api/v1"} # values for unresolvable route prefixes resolve_settings_defaults: false # true: read literal pydantic Settings defaults (warned) rules_overlay: rules-overlay.yaml # deep-merged over the built-in rules (changes rules_hash) extractors: enabled: [] # third-party extractor / framework-pack plug-ins to run (opt-in) llm: # informative only: never in config_hash or an epoch id provider: vertex # fake | vertex | openai_compatible project: my-gcp-project location: global model: gemini-3.8-flash pricing: {input_per_mtok: 0.75, output_per_mtok: 3.75} budget_usd: 1.00 output_language: auto # chat answers in the question's language tasks: # thinking per task: off | low | default | token budget (Gemini 2.5) judge: {thinking: default} summarize: {thinking: low} chat: {thinking: low, max_output_tokens: 2048} ``` Without a manifest, each top-level directory that has `pyproject.toml`, `setup.py` or `Dockerfile` is a container. ## Development ```bash .venv/Scripts/python -m pytest --cov=ad2v ``` ```bash .venv/Scripts/ruff check . ``` CI (`.github/workflows/ci.yml`) runs the tests, determinism (LF vs CRLF+BOM, across systems), both license gates, the viewer checks and builds, and the plug-in conformance kit; while the repository is private, Windows and macOS run only on a manual dispatch. The viewer (Node ≥ 20) lives in `viewer/`: ```bash cd viewer && npm ci && npm run build ``` `npm run build` type-checks, bundles into `viewer/dist` (which `ad2v view` serves and the wheel ships) and runs the license gate. `npm run build:demo` regenerates the test fixtures and `docs/reports/P5-demo.html`. `npm test` runs the unit tests (Vitest) and `npm run lint` runs ESLint. `npm run e2e` (after `build:demo`) runs the Playwright tour over the demo and writes `docs/reports/P5-screens/`. It needs `npx playwright install chromium` once. The MCP layer (P6): `pytest -m mcp` (repository root) runs the Python protocol and tool tests, including the retrieval eval. `npm run test:mcp` (in `viewer/`) builds the MCP App (`npm run build:app` → `viewer/app-dist/app.html`, which the wheel ships). It then runs the parity test (the viewer's data over the real stdio server), protocol conformance with the MCP Inspector (via `npx`; not a dependency), and the App e2e in a test host (`docs/reports/P6-screens/`). The retrieval eval also runs standalone: `python -m ad2v.mcp.evaluate fixtures/payments-recon/expected-questions.yaml [--tune]`. After editing `schema/ad2v-v0.7.sql`, regenerate the spec's column table with `python -m ad2v.ddl`. A test fails if you forget. --- # Quickstart: step by step on your own project # ad2v quickstart: step by step on your own project Every local command here was run against the `payments-recon` fixture, and `tests/test_quickstart.py` runs them again on each test run. Commands are shown for Windows PowerShell. Where the POSIX shell (macOS, Linux) differs, the variant follows. Every command accepts `--help`. --- ## 0. Where things go ``` my-project/ ← your repo, as-is (no code changes required) ├── src/… ← Python code (ad2v extracts Python only) ├── tests/… ← pytest tests (optional: named test_req_001_…) ├── specs/ ← the SDD, inside the repo and versioned with the code │ ├── openapi.yaml ← endpoints │ ├── features/*.feature ← Gherkin scenarios │ └── components.md ← Markdown (ad2v convention, spec-kit, Kiro or your own) ├── ad2v.yaml ← optional manifest (containers, pairing, LLM config) └── .gitignore ← add: .ad2v/ C:\ad2v-maps\my-project.ad2v ← the output file, outside the repo (or in a gitignored .ad2v/) ``` **Why the SDD goes inside the repo.** `ad2v history` reads the specs from *each commit*. If the specs live outside the repo, ad2v can't see how they changed, and `stale` ("the spec changed after the code") never fires. **Why the `.ad2v` file stays out of git.** It stores code snippets and LLM outputs. Treat it like the source code itself. `ad2v specs bootstrap` also keeps its cache in `/.ad2v/`, so add `.ad2v/` to `.gitignore`. **The repo as-is:** ad2v only reads the repo and never runs your code. Everything in steps 3 and 4 is optional; each item makes the map more accurate. --- ## 1. Install ad2v (once) From PyPI, in any virtual environment (the wheel includes the viewer and the MCP App): ```powershell pip install ad2v ad2v --version ``` The steps below use the fixtures in the source repository, so clone it too (`git clone https://github.com/shanglai/ad2v.git`). To work on ad2v itself, install from the clone: ```powershell cd C:\path\to\ad2v python -m venv .venv .\.venv\Scripts\Activate.ps1 pip install -e . cd viewer; npm ci; npm run build; npm run build:app; cd .. ad2v --version ``` POSIX: `source .venv/bin/activate`, and `(cd viewer && npm ci && npm run build && npm run build:app)`. Node is only needed to build the viewer and the MCP App from source. `ad2v` itself is pure Python. --- ## 2. Try the fixture first (5 min) ```powershell cd viewer; npm run fixtures; cd .. ad2v view viewer\tests\fixtures\demo.ad2v ``` In the browser: 1. **Zoom:** wheel to zoom, double-click to enter a box, `Esc` to go up, `/` to search. 2. **Status:** enter `api` and find the red dashed orphan and the risky loop in `matching.py`. 3. **AI drift:** click **Normative status** in the toolbar to switch to the effective status, and look at REQ-003's drift with the AI badge. 4. **Replay:** pick `POST /v1/reconciliations` in **Replay a workflow…**, then step through it and move the detail slider. **Dim** greys out everything but the step. 5. **History:** in the bottom epoch strip, select epoch 3 and press **Compare with parent**. --- ## 2b. Existing project? Start with `ad2v init` On a repository that already has code and specs, let `ad2v init` look first. It reads files only: it runs nothing, calls no model and writes nothing unless you pass `--write`. ```powershell ad2v init --repo C:\path\to\my-project ``` It prints what it found: - the spec folders, with a score and the files per format; - the prose no preset reads; - the spec id prefixes; - one container per project manifest, with its language and whether ad2v can read it; - the vendored folders it drops; - SQL, infrastructure, and test results or a pytest configuration; - the route prefixes it cannot resolve. Then it prints a proposed `ad2v.yaml`. Save the proposal outside the repository while you try it: ```powershell ad2v init --repo C:\path\to\my-project --write C:\ad2v-maps\my-project.yaml ``` Open that file and fill in any empty value: `pairing.path_constants: {"settings.API_PREFIX": ""}` means your routers are mounted under a setting whose value ad2v will not guess (for example `/api/v1`). An empty value is ignored, so the file works as written. Then: ```powershell ad2v extract --repo C:\path\to\my-project --manifest C:\ad2v-maps\my-project.yaml --out C:\ad2v-maps\my-project.ad2v ad2v init --repo C:\path\to\my-project --check C:\ad2v-maps\my-project.yaml # later: what changed since ``` `--check` exits 1 and lists the differences: - new or vanished spec folders and containers; - excludes that match nothing; - prefixes your tests use that are missing from the manifest; - empty path constants. `--format json` prints the findings for tools. TypeScript, Java and other languages without an active extractor are shown as `not_evaluable`, not as missing (step 7, **Legend**). --- ## 3. Your SDD format ad2v reads specs deterministically. Use what you already have, mix them freely, and let ad2v find them: without `--spec` it uses `spec_root` from `ad2v.yaml`, otherwise the first of `specs/`, `spec/`, `docs/specs/`, `.kiro/specs/`, `sdd/`, `docs/sdd/` that holds a spec, plus a root `openapi.yaml`. `extract` prints the roots it used. Spanish and English are both read: EARS (`Cuando …, el sistema deberá …`), Gherkin with `# language: es`, and the team formats below. **OpenAPI** (`specs/openapi.yaml`): endpoints pair with FastAPI, Flask and Starlette routes by method and path. `x-ad2v-refs: [REQ-001]` ties an endpoint to a requirement. **Gherkin** (`specs/features/*.feature`): tag scenarios with `@REQ-001`. **ad2v convention** (`specs/components.md`). No `ad2v.yaml` needed: ```markdown ## Component: Reconciliation API {#CMP-001} - [REQ-001] When an operator starts a run, the system shall fetch all bank statements for the date. - [REQ-002] If amounts differ by more than one cent, then the system shall mark the pair unmatched. ``` **spec-kit** (`specs/001-payments/spec.md`, as spec-kit writes it). Ids become `001-payments:FR-001`. A test named `test_payments_fr_001_…` links to it. No `ad2v.yaml` needed: ```markdown ### Functional Requirements - **FR-001**: System MUST record every refund with its reason. - **FR-002**: System MUST reject refunds above the original amount. ``` **Kiro** (`.kiro/specs/order-export/requirements.md`). Ids become `ORDEREXPORT-1`, and each acceptance criterion becomes `ORDEREXPORT-1.2`: ```markdown ### Requirement 1: Export orders **User Story:** As an accountant, I want a CSV export, so that I can reconcile. 1. WHEN an export is requested THEN the Exporter SHALL write one row per order ``` ```yaml spec_ids: test_name_prefixes: [REQ, ORDEREXPORT] # so test_orderexport_1_… links ``` **Your own Markdown** (custom regexes): ```markdown ## Invoices REQ: BILL-1 - When an invoice is paid, the system shall send a receipt. REQ: BILL-2 - The system shall keep invoices for seven years. ``` ```yaml spec_root: requirements spec_markdown: - glob: "requirements/*.md" preset: custom component: '^## (?P.+?)\s*$' requirement: '^REQ: (?P[A-Za-z]+-[0-9]+) - (?P.+?)\s*$' ``` **A requirements table** (the `table` preset, picked by shape; no `ad2v.yaml` needed). Headers are matched in Spanish or English, ignoring case and accents. A `Componente` column pairs each row with the container of that name. `RNF-…` ids (or a `Tipo` cell saying *No funcional*) are non-functional: counted apart and badged NFR in the viewer. ```markdown ## Requerimientos funcionales | ID | Título | Descripción | Prioridad | Componente | |----|--------|-------------|-----------|------------| | RF-001 | Iniciar conciliación | Cuando el operador inicia una corrida, el sistema deberá cargar los movimientos. | Alta | Backend | ``` **User stories** (the `stories` preset). Each criterion becomes `HU-001.1`, `HU-001.2`, …: ```markdown ## HU-001 Cargar un extracto Como operador, quiero cargar el extracto del banco, para conciliarlo con el libro mayor. Criterios de aceptación: 1. Dado un archivo CSV válido, cuando lo cargo, entonces veo la cantidad de movimientos. ``` ```yaml spec_ids: test_name_prefixes: [REQ, RF, HU] # so test_rf_002_… links to RF-002 (ad2v init proposes this) ``` Requirements get a short title: a `Título` column, a story heading or `- [REQ-001] (Título) …` wins. Otherwise ad2v derives one from the text. The map labels them `RF-001 · Iniciar conciliación`. **Free prose** (design notes, Word exports): bootstrap it. `--from` paths are relative to `--repo`: ```powershell ad2v specs bootstrap --repo C:\path\to\my-project --from docs --plan # free: lists files, estimates the cost ad2v specs bootstrap --repo C:\path\to\my-project --from docs --yes --max-cost-usd 0.20 --manifest C:\path\to\my-project\ad2v.yaml ad2v specs review --repo C:\path\to\my-project # overlap with existing specs, staleness ad2v specs approve --repo C:\path\to\my-project notes.md # moves it into specs/: now it counts ``` Proposals use `REQ-X…` ids so they never clash with yours. Nothing counts until you approve it. The `--yes` run calls your model (step 8). Details: [spec Markdown convention §6–8](https://github.com/shanglai/ad2v/blob/main/docs/spec/spec-markdown-convention.md). --- ## 4. Optional: make the links explicit (recommended) Without these, ad2v still pairs by route and by name, but explicit links are more precise. - **Annotations in code:** `# ad2v: REQ-001` on the line above a function, or a docstring line `ad2v: REQ-001`. - **Test names:** `test_req_001_starts_run` counts as a linked test. - **Test results:** these are what mark requirements **verified**: `pytest --junitxml=junit.xml`. - **`ad2v.yaml`** at the repo root is read automatically (pass `--manifest` only if it lives elsewhere): ```yaml containers: # otherwise inferred from pyproject.toml / Dockerfile dirs - {name: api, paths: [src/api], kind: service, replicas: 2} - {name: worker, paths: [src/worker], kind: worker} pairing: components: {CMP-001: api} # spec component → container/component path_prefix_map: {"/v1": ""} # if the spec paths lack a mount prefix orphan_policy: auto # auto | strict | off path_constants: {"settings.API_V1_STR": "/api/v1"} # FastAPI prefixes from settings (see below) # rules_overlay: ad2v-rules.yaml # custom risk rules / severity weights ``` **Route prefixes from settings.** If your routers are mounted under `settings.API_V1_STR` (or any setting), ad2v can't know the value: those routes link as `needs_review` until you either map the expression in `pairing.path_constants` (above), or set `pairing.resolve_settings_defaults: true` to use the literal default in your pydantic `Settings` class (the links then carry a warning that the environment may override it). --- ## 5. First extraction ```powershell mkdir C:\ad2v-maps -Force ad2v extract --repo C:\path\to\my-project --test-results C:\path\to\my-project\junit.xml --out C:\ad2v-maps\my-project.ad2v ``` POSIX: `mkdir -p ~/ad2v-maps` and `ad2v extract --repo ~/code/my-project --test-results ~/code/my-project/junit.xml --out ~/ad2v-maps/my-project.ad2v`. Add `--spec ` (repeatable) to override the spec discovery. Drop `--test-results` if you have none. The summary shows the spec roots used, the ports, the risky behaviors, the status counts and any unresolved spec references. ```powershell ad2v status C:\ad2v-maps\my-project.ad2v --status missing ad2v status C:\ad2v-maps\my-project.ad2v --status orphan ad2v inspect C:\ad2v-maps\my-project.ad2v --top-risk 10 ad2v validate C:\ad2v-maps\my-project.ad2v ``` **Sanity check.** Before trusting the map, read the top risks and the orphans, and ask yourself "Is that true?" --- ## 5b. Optional: infrastructure (Terraform) If the repository has Terraform, ad2v can tell which service runs which container, which env vars the code reads that no deployment provides, and which resources nobody uses. It shows all of this in an **Architecture** tab (a C4 view). ad2v reads files only: it never runs `terraform`. 1. Point it at the Terraform root modules (`ad2v init` proposes this): ```yaml infra_root: [infra] ``` That alone reads the `.tf` files statically, in every epoch and in `history`. 2. For the deployed shape per environment, produce a plan JSON **yourself**. This runs the providers and needs credentials and state; keep the file out of the repository: ```powershell terraform init; terraform plan -refresh=false -out tfplan; terraform show -json tfplan > C:\ad2v-maps\plan-prod.json ``` 3. Pass one plan per environment: ```powershell ad2v extract --repo C:\path\to\my-project --out C:\ad2v-maps\my-project.ad2v --infra-plan prod=C:\ad2v-maps\plan-prod.json ad2v c4 C:\ad2v-maps\my-project.ad2v --env prod --format structurizr > C:\ad2v-maps\workspace.dsl ``` The summary prints `Infra (prod): …`. In the viewer, **Architecture** shows the deployment units, the code containers they run, datastores, queues and buckets, with each one's infra status: - deployed, undeployed (dashed); - unwired env var (⚠); - orphan resource (dotted). **Infra badges** puts the same marks on the map. Only names of env vars are stored, never values; details in [docs/infra.md](https://github.com/shanglai/ad2v/blob/main/docs/infra.md). --- ## 6. Add history (the "what changed while I wasn't looking" view) ```powershell ad2v history --repo C:\path\to\my-project --commits HEAD~20..HEAD --into C:\ad2v-maps\my-project.ad2v ad2v log C:\ad2v-maps\my-project.ad2v ad2v diff C:\ad2v-maps\my-project.ad2v --format md ad2v materialize C:\ad2v-maps\my-project.ad2v --epochs last:5 ``` - `--commits` is a git range. `HEAD~20..HEAD` needs at least 21 commits. Use `--commits HEAD` to take the whole history of a young repository. - Specs are found in each commit as in step 5. In `history`, `--spec` and `--manifest` are *relative to the repo*, because they're read from each commit. - `materialize` prepares epochs for browsing in the viewer. After each new commit or agent session, append one epoch (it reports "no changes" when nothing changed): ```powershell ad2v extract --repo C:\path\to\my-project --into C:\ad2v-maps\my-project.ad2v ``` --- ## 7. Look at it ```powershell ad2v view C:\ad2v-maps\my-project.ad2v ``` This serves on 127.0.0.1 only and opens your browser (`--no-browser` prints the URL instead). Nothing is uploaded. **Chat with your own model about the map** (optional; costs money, see step 8 for `llm:`): ```powershell ad2v view C:\ad2v-maps\my-project.ad2v --chat --manifest C:\path\to\my-project\ad2v.yaml --chat-budget-usd 0.25 ``` The model runs in the `ad2v view` process with your credentials; the page gets a one-time token in its URL. The panel shows the budget meter and an estimate before each question; answers cite map nodes (click to fly there) and the map follows with **Follow** on. The chat can't change anything: for that it gives you the CLI command. With `provider: fake` it works offline for a dry run. --- ## 8. Optional: LLM layer (judge, suggestions, summaries, narrations). Costs money. **Authenticate once.** No API key is needed: ```powershell gcloud auth application-default login ``` **Add to `ad2v.yaml`:** ```yaml llm: provider: vertex project: your-gcp-project location: global model: gemini-3.8-flash output_language: en # or es-MX pricing: {input_per_mtok: 0.75, output_per_mtok: 3.75} # check current Vertex pricing budget_usd: 1.00 summaries: {scope: structural} tasks: # thinking per task (low cost ~60% less per call in P7) summarize: {thinking: low} narrate: {thinking: low} ``` **Always plan first, and read the USD estimate.** `plan` makes no calls: ```powershell ad2v llm plan C:\ad2v-maps\my-project.ad2v --manifest C:\path\to\my-project\ad2v.yaml ``` With the model's default thinking, real output has run about **2× the static plan** (reasoning tokens); with `thinking: low` it ran **below** the plan in P7. The plan re-calibrates after the first call of a run. **Then run within a hard cap:** ```powershell ad2v llm run C:\ad2v-maps\my-project.ad2v --manifest C:\path\to\my-project\ad2v.yaml --tasks judge,suggest --max-cost-usd 0.50 --yes ``` Start with `judge,suggest`: they're cheap and only need specs. `summarize,narrate` scale with the repo size, so run them later and on purpose. Every call is labeled `project=ad2v`. **Review the suggestions.** A human confirms them, and the decision is written to `ad2v.yaml`: ```powershell ad2v suggestions list C:\ad2v-maps\my-project.ad2v --open ad2v suggestions accept C:\ad2v-maps\my-project.ad2v --manifest C:\path\to\my-project\ad2v.yaml ad2v suggestions reject C:\ad2v-maps\my-project.ad2v --manifest C:\path\to\my-project\ad2v.yaml ``` Then re-extract with `--into`, and accepted links become normative. --- ## 9. Use it from Claude Desktop (MCP + the map inside the chat) Put your `.ad2v` files in one folder (e.g. `C:\ad2v-maps`). Then edit `%APPDATA%\Claude\claude_desktop_config.json` (macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`): ```json { "mcpServers": { "ad2v": { "command": "C:\\path\\to\\ad2v\\.venv\\Scripts\\python.exe", "args": ["-m", "ad2v", "mcp", "--root", "C:\\ad2v-maps"] } } } ``` POSIX: `"command": "/path/to/ad2v/.venv/bin/python"`. The same server runs by hand as `ad2v mcp --root C:\ad2v-maps` (stdio; add `--http` for Streamable HTTP on 127.0.0.1). Restart Claude Desktop and try: - "Using ad2v, give me an overview of my-project and show it on the map." - "Which requirements are missing or drifting? Show me where." - "Replay how
works, step by step." - "What changed in the last epoch? Highlight it." The server is read-only. It never runs the LLM and never edits files; for those actions it hands you the CLI command. More hosts: [docs/mcp.md](https://github.com/shanglai/ad2v/blob/main/docs/mcp.md). --- ## 10. Limits - **Code extraction is Python only.** Code in other languages (TypeScript, Java, Go, …) is listed but not read: it shows as `not_evaluable` (thin, dotted, hollow, "∅" badge; see **Legend** in the viewer). Add `// ad2v: REQ-001` comments there so the spec item still points at the right file. - **Spec formats:** OpenAPI 3, Gherkin, the ad2v convention, spec-kit `spec.md`, Kiro `requirements.md`, requirement tables, user stories, custom regexes. Not yet parsed: Kiro `design.md`/`tasks.md`, spec-kit `plan.md`/`tasks.md`. - **Infra is Terraform only** (plan JSON or `.tf`), GCP and AWS core types; Azure, Pulumi, CloudFormation and Kubernetes manifests are detected but not read. No NFR-vs-infra checks yet. - **Route prefixes from settings** (`settings.API_V1_STR`) are unresolved unless mapped (step 4); those routes link as `needs_review` by their path tail. - **`verified` needs a JUnit file.** Without one, the best status is `implemented`. - **Judge accuracy** was measured on 11 labeled links. Treat the "AI" drift badges as leads, not verdicts. --- # MCP hosts # ad2v over MCP: set up a host `ad2v mcp` is a read-only MCP server over your `.ad2v` files. The model gets tools to find components (`ad2v_locate`, `ad2v_search`), explain them (`ad2v_get_node`, `ad2v_links`, `ad2v_workflow`, `ad2v_diff`, …), and point at them on the map (`ad2v_show`). Hosts that support **MCP Apps** (Claude, ChatGPT, VS Code, …) render the map inline in the conversation. - Protocol: MCP **2026-07-28**; older hosts that open with `initialize` (2025-11-25, 2025-06-18, 2025-03-26) work too. MCP Apps extension **2026-01-26** (ADR-0039, ADR-0040). - Read-only: files are opened with SQLite `mode=ro`. There are no write tools. Accepting suggestions, running the LLM layer, and materializing or pruning epochs stay CLI steps; tools return the exact command instead. - No LLM calls and no network: the server only reads local files. The map app declares no network origins. Prepare a file first (see the README). The map needs a materialized epoch; for history, run `ad2v materialize project.ad2v --epochs last:5`. ## Serving files ```text python -m ad2v mcp --root # every *.ad2v under (listed by ad2v_files) python -m ad2v mcp --file # exactly one file python -m ad2v mcp --root --http [--port 8765] # Streamable HTTP on 127.0.0.1 only ``` With `--root`, a `file` argument is resolved strictly inside the root. Absolute paths, `..`, symlinks leaving the root and non-`.ad2v` files are refused. ## Claude Desktop Edit `claude_desktop_config.json`: - Windows: `%APPDATA%\Claude\claude_desktop_config.json` - macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` Use the Python **inside the virtualenv where ad2v is installed**, with absolute paths. Windows: ```json { "mcpServers": { "ad2v": { "command": "C:\\Users\\you\\projects\\ad2v\\.venv\\Scripts\\python.exe", "args": ["-m", "ad2v", "mcp", "--root", "C:\\Users\\you\\projects\\maps"] } } } ``` macOS / Linux: ```json { "mcpServers": { "ad2v": { "command": "/Users/you/projects/ad2v/.venv/bin/python", "args": ["-m", "ad2v", "mcp", "--root", "/Users/you/projects/maps"] } } } ``` Restart Claude Desktop, then ask for example: "Using ad2v, where are bank statements fetched? Show me on the map." ## Claude Code ```bash claude mcp add ad2v -- /path/to/ad2v/.venv/bin/python -m ad2v mcp --root /path/to/maps ``` Add `--scope project` to share it through `.mcp.json` in the repository. Check it with `claude mcp list`. ## VS Code `.vscode/mcp.json` in the workspace (or the user-level MCP configuration): ```json { "servers": { "ad2v": { "type": "stdio", "command": "${workspaceFolder}/.venv/bin/python", "args": ["-m", "ad2v", "mcp", "--root", "${workspaceFolder}"] } } } ``` On Windows use `${workspaceFolder}\\.venv\\Scripts\\python.exe`. ## ChatGPT and other hosts that need a remote HTTPS server `ad2v mcp --http` binds **127.0.0.1 only**. It validates `Origin` and `Host` against DNS rebinding and has no authentication. It is meant for local testing and for local hosts that speak HTTP. A host that only connects to remote HTTPS servers (ChatGPT connectors, for example) cannot reach it. Exposing it would need a tunnel or a deployment. **That is out of scope, and this project does not set one up.** Before doing it yourself, weigh the risk: the server gives read access to your code map, which includes names, paths, spec text, risks and LLM summaries of your repository. Checklist for later: - [ ] Put an authenticating reverse proxy in front (the MCP authorization spec, OAuth 2.1), never the bare server. - [ ] Serve only a dedicated `--root` holding the maps you mean to share. - [ ] Keep `Origin` validation; add the public origin to the allowed list in code, not `*`. - [ ] TLS end to end; rate limits; access logs. - [ ] Remember that a host may cache tool results (`cacheScope`) and send them to its model provider. ## Without an MCP host: the viewer's own chat (P7) `ad2v view --chat --manifest ad2v.yaml` answers questions in the viewer's panel with your own model (the `llm:` section). It calls the same agent tools in-process (all of them except `ad2v_files` and `ad2v_llm_plan`), with the same fenced untrusted text and no write actions. See the [quickstart](https://github.com/shanglai/ad2v/blob/main/docs/quickstart.md#7-look-at-it). P7 also changed two tool behaviors: `ad2v_status` lists ports and spec endpoints first, then containers/components, then the rest; `ad2v_locate` finds spec ids written in the question (`FR-001`, `001-payments:FR-001`, `1.2`) and propagates scores along spec↔code links for requirement questions. ## Troubleshooting - **stdout is for the protocol only.** On stdio every byte on stdout must be a JSON-RPC message. ad2v writes all diagnostics to **stderr**. If you wrap the command in a script, don't `echo` anything to stdout. - **Windows paths.** In JSON, backslashes are doubled (`C:\\Users\\…`), or use forward slashes. Use absolute paths: hosts start servers from their own working directory. - **The wrong Python.** `ModuleNotFoundError: ad2v` means the host started a Python without ad2v. Point `command` at the virtualenv's interpreter (`.venv\Scripts\python.exe` or `.venv/bin/python`), or at the `ad2v` executable next to it (`args: ["mcp", …]`). - **"not materialized".** The tool result carries the command. Run it, for example `ad2v materialize project.ad2v --epochs last:5`. - **The map shows "not built".** In a source checkout, run `npm ci && npm run build:app` in `viewer/`. Wheels include the app. - **Nothing inline.** The host may not support MCP Apps. `ad2v_show` still lists the components, and `ad2v view ` opens the full map locally. - **Logs.** Claude Desktop keeps server stderr in its MCP log folder (`%APPDATA%\Claude\logs` or `~/Library/Logs/Claude`). Look for `[ad2v-mcp]` lines. - **Inspect it yourself.** Put the same `mcpServers` block in a file and run `npx -y @modelcontextprotocol/inspector@2.8.0 --cli --config servers.json --server ad2v --method tools/list`, or run it without `--cli` for the web UI. `viewer/scripts/conformance.mjs` does this for every tool. --- # Infrastructure layer # The infra layer (P10) ad2v maps spec ↔ code. With infra it can also answer: - "Which Cloud Run service runs this component?" - "The code reads `DATABASE_URL`: does any deployment provide it?" - "What does the architecture look like, and which requirements does each part carry?" Infra is read deterministically and never executed. It feeds two things: the env-var bridge between code and infra, and a C4 view in its own tab (**Architecture**). ## 1. Inputs | input | when | what ad2v does | |---|---|---| | plan JSON (`--infra-plan [ENV=]plan.json`, repeatable) | you produce it | reads `planned_values`, `configuration`, `resource_changes` (actions, unknowns) | | `.tf` files (`infra_root: [infra]` in `ad2v.yaml`) | every epoch, also in `history` | parses them statically (python-hcl2); nothing is evaluated | **ad2v never runs `terraform`, `tofu`, `terragrunt` or providers.** `plan` runs providers and `external` data sources (arbitrary programs) and needs credentials and state. You produce the plan yourself; `ad2v init` prints the command: ```bash terraform init && terraform plan -refresh=false -out tfplan && terraform show -json tfplan > plan.json ``` Keep `plan.json` out of the repository. Plan versions: `format_version` 1.x (Terraform ≥ 1.1, OpenTofu). Key order, whitespace, BOM and the timestamp never change the epoch. ```bash ad2v extract --repo . --manifest ad2v.yaml --out app.ad2v --infra-plan prod=plan-prod.json --infra-plan dev=plan-dev.json ``` An epoch extracted without a plan carries the previous epoch's plan forward. A change in the plans or the `.tf` files alone gives a new epoch with `change_kind: infra`. **Plan and static together.** For the same environment the plan wins. The static read only fills gaps: resources the plan's configuration does not declare at all, which happens when the plan is older than the `.tf`. Those nodes are marked `gap_fill`, and the summary says "filled from .tf". ## 2. Resource types `src/ad2v/infra/infra.yaml` maps provider types to kinds. It is part of the epoch rules, so it is covered by `rules_hash`. | kind | GCP | AWS | |---|---|---| | `deployment_unit` | Cloud Run services (v1, v2) and jobs, Cloud Functions v2 | Lambda, ECS service (env from its task definition), Batch job definition, App Runner | | `managed_datastore` | Cloud SQL, Firestore, BigQuery dataset, Memorystore | RDS instance and cluster, DynamoDB, ElastiCache | | `managed_queue` | Pub/Sub topic and subscription | SQS, SNS | | `bucket` | GCS (a `website` block makes it a static site) | S3 (+ `aws_s3_bucket_website_configuration`) | | `secret_store` | Secret Manager secret (names only) | Secrets Manager secret | | `network` | VPC network, VPC access connector | | | `scheduler` | Cloud Scheduler job | EventBridge rule | | `infra_other` | anything else (kept and counted) | | Azure and Kubernetes manifests are not read yet (`pending.md`). ## 3. Secrets policy The policy (ADR-0076): - Only an allowlist of attributes per type is stored: names, locations, scaling, the image repository without tag or digest, engine and tier, encryption flags as booleans, schedules. - Env vars keep their **names** and a **source** (`reference` to a resource, `secret`, `literal`, `unknown`), never the value. - Anything under `sensitive_values` is dropped, and `variables`, `prior_state` and output values are never read. - Module sources become `remote: true|false`, never the URL. A test plants more than 15 fake secrets in plans and `.tf` files. It checks that none appears in the `.ad2v` file's bytes or in its dump. ## 4. Static `.tf` limits - `count` / `for_each` are not expanded: one node, `multiplicity: unresolved`. - Local modules are followed one level. Deeper calls are reported, and remote modules are recorded but not read. - Variables stay unresolved, and their defaults are never stored. - Only literals are known; `locals` are followed for references. - `dynamic` blocks and `jsonencode(...)` are opaque. - Files over 4 MB are skipped, and a file that does not parse is reported, never fatal. - One static environment per infra root: `default` for one root, otherwise the folder's name. ## 5. The env-var bridge **Code side.** The env var names the code reads are collected: - `os.environ[...]`, `os.getenv(...)`, `os.environ.get(...)`; - pydantic `BaseSettings` fields, with `env_prefix`, `Field(alias=…)` and `validation_alias`. They are stored as `attrs.env_reads`, only in epochs with infra input. **`deploys`** (deployment unit → code container), per environment, first match wins: 1. the manifest: `infra: {deploys: {"google_cloud_run_v2_service.api": backend}}`; 2. a name match: the unit's name or its image repository equals the container name, or ends or starts with it at a `-` (`conciliacion-backend` ↔ `backend`); 3. env-var overlap: the container's reads are a subset of the unit's env, with ≥ 2 shared names and one best candidate. Several candidates at one level are **ambiguous**: no edge, and `ad2v init` proposes `infra.deploys` placeholders. One unit may deploy several containers, and units with the same image may deploy one container. A website bucket can deploy a container too (a static site). **`provides_env` / `wires`.** An env var read in a deployed container is `wired` when its unit sets it, with an edge to the resource the value comes from when known. Otherwise it is `unwired`, naming the variable and the unit. ## 6. Statuses: a separate dimension Infra findings never change the ten spec↔code statuses (ADR-0078). They live in `infra_status`, per environment: | applies to | statuses | |---|---| | code containers | `deployed`, `undeployed`, `ambiguous` | | code nodes reading env vars (in a deployed container) | `wired`, `unwired` | | infra resources | `in_use`, `infra_orphan` | | everything, without infra input | `unknown` (no rows: nothing is claimed) | On the map, **Infra badges** (toolbar) prefix labels: ☁ deployed, ∅☁ undeployed, ⚠ unwired, ?☁ ambiguous, ∅ orphan. Node cards list the status per environment. ## 7. The C4 view (Architecture tab) The view is derived from the epoch's graph, never drawn by hand (ADR-0079): - **Context level:** the system, persons (user-story actors) and external systems. - **Container level:** - deployment units, with the code containers they run; - static sites; - datastores, queues, buckets and schedulers; - code containers that nothing deploys. - **Relations:** code calls, reads, writes and publishes aggregated to containers; `wires` (env var names on hover); persons → the containers their stories pair with. Each container shows: - the roll-up of its requirements' statuses; - its infra status (line style + color); - unwired env vars; - its NFRs ("NFR (checked against infra in a later phase)"). Click an element for its card; **open in map** jumps to the container. Selecting on the map highlights the element. The **Env** selector switches environments. The layout is deterministic and keeps each element's slot across epochs. ```bash ad2v c4 app.ad2v --env prod --format structurizr > workspace.dsl ``` ```bash ad2v c4 app.ad2v --env prod --format json ``` ## 8. MCP - `ad2v_architecture`: elements, relations and findings per environment. - `ad2v_show` with intent `c4` (and `env`): opens the Architecture view in the MCP App. - `ad2v_overview`: reports infra counts and findings. - `ad2v_locate` understands infra words: "which service runs the backend", "unwired", "orphan bucket". ## 9. Per environment Every finding is computed per environment. A container can be deployed in prod and not in dev, and an env var can be wired in one and missing in the other. Plans name their environment. Without plans, each infra root is one environment. --- # Plug-ins # Writing ad2v plug-ins: extractors and framework packs ad2v reads code through **language extractors** and recognizes frameworks through **framework packs**. Both are plug-ins behind a stable public API (`ad2v.api`, api_version **1.0**), so they can live in their own packages and repositories. The built-in Python extractor (`python`) and the built-in Python framework pack (`python-frameworks`) use the same API. Decisions: ADR-0063 (API and compatibility), ADR-0064 (discovery, opt-in, isolation), ADR-0065 (identity in the epoch). Security model: `docs/security.md` §6. ## 1. Trust model: plug-ins are trusted code A plug-in runs **in-process with your privileges**. ad2v does not sandbox it. Therefore: - Only the built-ins are active by default. Installing a package never activates it. - You enable a plug-in per repository in `ad2v.yaml`, or per run: ```yaml extractors: enabled: [routes-dsl, bottle] ``` ```bash ad2v extract --repo . --out x.ad2v --extractor routes-dsl ``` - `extract` and `history` never import disabled plug-ins. `ad2v extractors list` shows what is installed without importing third-party code (`--inspect` imports it to show versions). - A third-party plug-in that raises or exceeds `--plugin-timeout` (default 600 s) is reported, recorded in the epoch (`plugin_failures`) and skipped; `--strict` makes that fatal. - Install plug-ins only from sources you trust, as you would any Python package. ## 2. Registering a plug-in ```toml # your pyproject.toml [project.entry-points."ad2v.extractors"] routes-dsl = "ad2v_routes_dsl:RoutesDslExtractor" [project.entry-points."ad2v.frameworks"] bottle = "ad2v_bottle_pack:BottlePack" ``` The entry point name must equal the plug-in's `name` (`[a-z][a-z0-9-]*`). A name already used by a built-in, or by another installed distribution, is refused. ## 3. The interfaces Import only from `ad2v.api`. ```python class Extractor(Protocol): api_version: str # "1.0" name: str version: str # bump whenever output can change: it enters the epoch id languages: tuple[str, ...] file_patterns: tuple[str, ...] # e.g. ("*.routes",), matched on file name or repo path rules_files: tuple[Path, ...] # YAML; their canonical hash enters the epoch id node_kinds: frozenset[str] # what you emit, besides system/container/component edge_kinds: frozenset[str] # besides contains def extract(self, ctx: ExtractContext) -> ExtractOutput: ... class FrameworkPack(Protocol): api_version: str name: str version: str attaches_to: tuple[str, ...] # extractor names, e.g. ("python",) frameworks: tuple[str, ...] file_patterns: tuple[str, ...] # extra inputs your hook reads (e.g. "*.sql"), or () rules_files: tuple[Path, ...] # fragments merged into the extractor's rules node_kinds: frozenset[str] edge_kinds: frozenset[str] def post_extract(self, ctx: FrameworkContext) -> None: ... ``` `ExtractContext`: `graph` (add `Node`s and edges; it already holds the system node), `layout` (`container_for(path)`, `parent_for_module(path)`), `files` (a `FileAccessor`: iterate `SourceFile`s with normalized `text` and `content_hash`; `read_text(path)` for one file; anything outside the set raises `FileAccessError`), `config`, `rules`, `cache`, `warn(message)`. `ExtractOutput`: `facts` (`CodeFacts`; fill `files_read`, and `entry_ports` if you emit ports with handlers), `cache_entries`, `timings`, `stats`. Rules for a good plug-in (checked by the conformance kit): - deterministic: no wall clock, randomness, environment or file-system order in the output; line endings and BOMs must not matter (the accessor already normalizes them); - node ids come from `(side, kind, qualified_name)`: never put line numbers in a qualified name; - read files only through `ctx.files`; no network, no subprocesses, no writes; - emit only the kinds you declare; use the IR kinds (`ad2v.api.NODE_KINDS`, `EDGE_KINDS`). To make routes pair with OpenAPI / Markdown endpoint specs, emit `http_route` nodes named `" "` with qualified name `"http:: "`, parent = the container, and `attrs` `{method, path, framework, handler}`, plus an `exposes` edge from the container (see the example). ## 4. Framework packs for the built-in Python extractor The Python extractor reads its recognizers from its rules, so a pack contributes recognizers as a rules **fragment**: mappings merge, lists are unioned (built-in items first, then packs in name order), scalars replace. The recognizer vocabulary (from `python-frameworks.yaml`): | key | meaning | |---|---| | `ports.http.app_constructors` | callables whose result is an app/router (FastAPI, Flask, Starlette, …) | | `ports.http.method_decorators` / `route_decorators` | `@app.get(...)` / `@app.route(...)` | | `ports.http.include_methods`, `prefix_kwargs` | router inclusion and prefixes | | `ports.http.add_route_methods`, `route_list_constructors`, `route_constructors`, `mount_constructors`, `mount_methods` | imperative and list-style registration | | `ports.http.settings_bases` | classes whose literal defaults the opt-in settings resolver reads | | `ports.mcp.server_constructors`, `tool_decorators` | MCP tool registration | | `ports.cli.typer_constructors`, `typer_decorators`, `click_decorators`, `click_group_decorators` | CLIs | | `ports.events` | subscriber callbacks (`constructor`, `method`, `callback_kwarg`, …) | | `ports.entry_points.celery` / `apscheduler` / `schedule` | jobs and schedulers | | `resources.` | datastore / queue / external service / LLM client libraries (in `python.yaml`) | Example (`examples/plugins/bottle-framework-pack/src/ad2v_bottle_pack/bottle.yaml`): ```yaml ports: http: app_constructors: [bottle.Bottle] ``` A pack may also define a `post_extract` hook; it runs after the extractor, on its graph, and may add nodes, edges and attrs (the built-in pack attaches schema digests this way). A pack cannot add new syntax-tree algorithms to the Python extractor in api 1.0 (ADR-0063, "what did not fit"). ## 5. Checking a plug-in ```bash ad2v extractors list ad2v extractor check routes-dsl --fixture path/to/small-repo --json ``` From Python (no pytest needed): ```python from pathlib import Path from ad2v.testing.conformance import check result = check("routes-dsl", Path("tests/fixture")) assert result["passed"], result ``` Checks: `api_handshake`, `runs_clean`, `determinism` (two runs, CRLF+BOM copy), `id_stability` (blank lines prepended to your files plus a new empty file), `schema` (`ad2v validate` of a real extract), `file_access` (no file opened outside your set while your code runs), `side_effects` (no network, subprocess or file write while your code runs), `declared_kinds`. ## 6. Compatibility policy (api 1.0) - `MAJOR.MINOR`. Minor versions only add (optional fields and attributes, kinds, helpers). - Breaking (new major): removing or renaming anything in `ad2v.api`, changing a signature or the meaning of a field, the pack-merge rules, the accessor contract or the identity inputs, or making an optional attribute required. - The core accepts plug-ins of its own major (with a minor not newer than its own) and of the previous major (with a deprecation note). Deprecated elements keep working for at least one minor release and until the next major. ## 7. Examples `examples/plugins/` holds two out-of-tree packages: a toy routes-DSL extractor and a Bottle framework pack. See `examples/plugins/README.md` for install steps. --- # Security model # ad2v threat model Scope: ad2v 0.7.0.dev0 (P8). ad2v reads a repository and its specs and writes one SQLite file; a viewer, a chat and an MCP server read that file on the user's machine. Nothing runs repository code. Everything listens on 127.0.0.1 only. The optional LLM layer is bring-your-own-model and spends the user's money. Severity scale (ad2v's own context, not CVSS): **high**: code execution or data exfiltration without the user's deliberate action; **medium**: spending money, disclosure of local data or integrity loss, triggered by opening untrusted input; **low**: denial of service of a local tool, or an issue that needs the user to opt into something unsafe; **info**: hardening. Reporting: `SECURITY.md`. Findings of the P8 review: §9. ## 1. A malicious `.ad2v` file Threat: someone sends a crafted file. Opening it in `ad2v view`, the MCP App or `ad2v mcp` must not run script, write files or exfiltrate; `ad2v validate` must say it is not conformant. | attack | mitigation | test | |---|---|---| | XSS in any rendered text (names, qualified names, paths, attrs, summaries, narrations, evidence, rationales, labels, meta) | the viewer never uses `innerHTML` (DOM `textContent` only; the chat Markdown renderer builds nodes and links only to node ids); labels are drawn on a canvas; strict CSP in `ad2v view` (`script-src 'self' 'wasm-unsafe-eval'`, no inline script, `object-src`, `base-uri`, `form-action`, `frame-ancestors` all `'none'`); the MCP App runs in the host's sandboxed iframe and makes no requests | `viewer/e2e/hostile.spec.ts`: payloads in every free-text column and JSON leaf, in both viewer modes; no script runs, no element is injected, payloads appear as text, no page error | | huge strings | the viewer truncates labels; `validate` flags any text value over 1 MiB (source blobs exempt) | e2e: a 100,000-character name renders; `tests/test_security.py` (2 MiB name rejected) | | deep nesting | `validate` flags JSON nested deeper than 64 levels (scanned without parsing) and survives a `RecursionError` | `test_security.py` (5,000 levels) | | wrong format version | readers accept 0.4/0.5 only; writers 0.5 (0.4 upgraded) | `test_security.py` (`9.9` rejected) | | extra tables, views, triggers; a built-in view redefined | `validate` lists every trigger, unknown object and changed view; readers open `mode=ro` (+ `query_only` in MCP), so triggers cannot fire; writers (`extract --into`, `llm run`, …) refuse a file with triggers or altered views; the MCP server refuses to serve it; every connection sets `PRAGMA trusted_schema = OFF` | `test_security.py`: read-only insert fails, node count unchanged, `open_rw` and `extract --into` refuse, MCP refuses | Residual risk: **low**. The standalone viewer runs SQL (sql.js) over the whole file in the page; a file whose views were redefined to be pathological (recursive CTEs) can hang that browser tab. The viewer does not compare the schema with the format's before querying (the MCP server and the Python tools do). ## 2. The local server (`ad2v view`) | item | mitigation | test | |---|---|---| | exposure | binds 127.0.0.1 only, random port by default | `test_view.py` | | methods | GET/HEAD for files; POST only on `/api/chat/*` with `--chat`; PUT/DELETE/PATCH/OPTIONS refused, so no CORS preflight ever succeeds | `test_view.py`, `test_security.py` | | path traversal | static paths resolved and contained in the viewer directory; the map is served only at one fixed path | `test_security.py` (`..`, `%2e%2e`, `%2f`) | | DNS rebinding | **P8:** every request must carry `Host: 127.0.0.1:` or `localhost:` (it was checked only for chat before) | `test_security.py` | | headers | CSP (§1), `X-Content-Type-Options: nosniff`, `Cache-Control: no-store`, **P8:** `Referrer-Policy: no-referrer` | `test_security.py` | Residual risk: **low**: any local process can read the map over the port, like any local file. ## 3. The chat endpoint (spends money) `ad2v view --chat` (ADR-0056/0057): a per-launch token from `secrets` in the URL fragment (never sent in a request line, never logged), echoed in `X-Ad2v-Token`; `Host` and `Origin` checks; JSON only with a 16 KiB body cap; no CORS; a session budget (`--chat-budget-usd`, default $0.25) and a per-turn estimate; ≤ 6 model steps per turn; cancel stops before the next model call; the provider needs a manifest with pricing. Tests: `tests/test_chat.py` (security rejections, budget mid-turn, iteration cap, cancel). Residual risk: **low**: a local process that reads the browser's URL (or the terminal output) can spend up to the session budget. ## 4. Prompt injection through repository or spec content All repository and spec text sent to a model is fenced (``, content-derived ids, closing tags neutralized), instruction-like phrases are flagged, outputs are schema-validated, and LLM results are informative only (never in an epoch id, never a normative status). Chat answers must cite node ids returned by the turn's own tool calls (hallucinated citations dropped and flagged); the chat and MCP have **no write tools**: a request to change something gets the copyable CLI command (ADR-0029, ADR-0042, ADR-0056). Tests: `test_llm.py`, `test_chat.py` (injection echoed is flagged, write requests get the command only), `test_mcp.py` (untrusted text wrapped). Residual risk: **medium** (inherent): a poisoned repository can still bias a summary, a judgment or a chat answer; it cannot write, run code or leave the session budget. ## 5. The MCP server (`ad2v mcp`) stdio by default; Streamable HTTP only on 127.0.0.1 with `Origin`/`Host` validation; `--file` or `--root` with strict containment (absolute paths, `..`, symlinks out of the root refused; messages do not echo resolved paths); SQLite opened `mode=ro` + `query_only`; **P8:** files with triggers or altered views are refused; 13 read-only agent tools; repository text in results wrapped as `untrusted_text`. Tests: `test_mcp.py` (origin/host/methods, traversal, read-only connections, untrusted text), `test_security.py`. Residual risk: **low**. ## 6. Third-party extractor and framework plug-ins Plug-ins are **trusted code**: they run in-process with the user's privileges (ADR-0063..0065, `docs/plugins.md`). | threat | mitigation | residual | |---|---|---| | code execution by merely installing a package | opt-in: only built-ins are active; others need `ad2v.yaml` `extractors.enabled` or `--extractor`; `extract` never imports a disabled plug-in; `extractors list` does not import them without `--inspect`; names that shadow a built-in or are registered twice are refused | a plug-in the user enables can do anything the user can (**accepted; documented**) | | reading outside the repository | the core pre-reads exactly the claimed files and hands them over a `FileAccessor`; the conformance kit fails a plug-in that opens other files (audit hook) or asks the accessor for them | not enforced at run time (**low**: trusted code) | | network access, subprocesses, writes | conformance kit `side_effects` check (audit hook on socket connect/resolve/bind, subprocess, write-mode opens) | not enforced at run time (**low**) | | nondeterminism poisoning epochs | plug-in identity (name, version, api, rules hash) is part of the epoch rules and id; the kit checks two runs, a CRLF+BOM copy and id stability | a plug-in that is nondeterministic only on some inputs passes the kit (**low**) | | crash or hang | third-party plug-ins run with exception isolation and a timeout (`--plugin-timeout`, default 600 s); a failure is printed, recorded in `epochs.attrs.plugin_failures` and skipped; `--strict` makes it fatal; a hook that fails cannot leave partial changes (it runs on a copy) | a timed-out thread keeps running until the process exits (**low**) | Tests: `tests/test_plugins.py` (opt-in, shadowing, failures, timeouts, strict, kit catching undeclared kinds, nondeterminism, accessor misuse, real file opens, network, subprocess, writes). ## 7. Path traversal in CLI arguments and repository-supplied paths Paths the **user** types (`--repo`, `--spec`, `--manifest`, `--out`, `--test-results`) are the user's choice. Paths that come from **the repository itself** must not reach outside it, because a map may be shared: | path | mitigation | test | |---|---|---| | `ad2v.yaml` `spec_root` | **P8:** must resolve inside the repository (before: `../../x` was read into the map) | `test_security.py` | | auto-detected `specs/` etc. | **P8:** a symlink that points outside the repository is not a spec root | `test_security.py` (skipped on Windows without symlink privilege) | | `ad2v.yaml` `rules_overlay` | **P8:** must stay inside the manifest's directory | `test_security.py` | | `specs bootstrap --from` | repository-relative, contained (ADR-0052) | `test_bootstrap.py` | | `specs approve ` | only files directly in `/.ad2v-extracted/`; **P8:** `--to` must be inside the repository | `test_security.py` | | `mcp --root` | strict containment (§5) | `test_mcp.py` | | file walking | never follows symlinks (`os.walk(followlinks=False)`, symlinked files skipped) | `test_determinism_and_cli.py` | | `history` | `git archive` into a temporary directory; never checks out or runs code | `test_history.py` | ## 8. Supply chain - Runtime closure (Python): 13 dependencies, all MIT except `typing_extensions` (PSF-2.0) and, since P10, `regex` (Apache-2.0 AND CNRI-Python, via python-hcl2); no `certifi`. `scripts/check_python_licenses.py` gates it in CI; npm has its own gate (`viewer/scripts/check-licenses.mjs`). - `pip-audit` 2.10.1 (run from a throwaway virtual environment, outside the tree) on the runtime closure and on the dev environment: **no known vulnerabilities**. - `npm audit`: **0** vulnerabilities in runtime dependencies; **3 high** in the dev/build chain, all one advisory: `braces` ≤ 3.0.3 (GHSA-vfj7-8cjw-p6xm, stack exhaustion on deeply nested brace patterns) via `micromatch` via `vite-plugin-singlefile` (used only to build the single-file demo and the MCP App). No patched `braces` exists; npm's only "fix" downgrades the plug-in to 0.9.0, a breaking change. The patterns come from our own build configuration, so it is not exploitable (**low**, open). - CI actions pinned by full commit SHA; workflows have `contents: read`; the release workflow is disabled (`docs/RELEASING.md`). - Tools used but not depended on (build, twine, pip-audit, the MCP Inspector) run outside the tree (`docs/dependencies.md`). ## 8b. Infrastructure inputs (P10) A plan JSON and `.tf` files are untrusted input: a repository (or whoever hands you a plan) controls them. | risk | defense | test | |---|---|---| | running repository code | ad2v never runs `terraform`, `tofu`, `terragrunt` or providers; it reads the JSON the human produced and parses `.tf` statically (nothing evaluated) | by construction; `docs/infra.md` | | secrets in plans and tfvars | allowlist per resource type; env var values reduced to a source; `sensitive_values`, `variables`, `prior_state` and outputs never read; module URLs reduced to `remote: true` (ADR-0076) | `test_infra.py::test_planted_secrets_never_reach_the_file_or_its_dump` | | huge files | plan ≤ 256 MB and ≤ 200,000 resources (refused beyond); `.tf` ≤ 4 MB per file, ≤ 2,000 per root (skipped and reported); strings stored ≤ 200 characters | `test_size_limits_for_hostile_inputs` | | deep nesting | JSON nesting errors are refused cleanly (`InfraInputError`, exit 2); trees are walked with depth bounds (16 for modules, 32 for HCL bodies, 8 for reference resolution) | `test_unsupported_and_hostile_plans` | | a `.tf` that crashes the parser | each file is parsed alone; a failure is a warning, never fatal | `test_static_limits` | | reading outside the repository | `infra_root` must be relative and stay inside the repository; local module sources that climb out are not followed | `test_manifest_rejects_infra_roots_outside_the_repo`, `test_static_limits` | | hostile names in the C4 view, MCP or chat | element names are stored as text, drawn as SVG text nodes (never HTML), wrapped as untrusted text by the MCP tools | by construction (DOM text only); MCP: `clean_name` | ## 9. Findings of the P8 review | # | issue | surface | severity | status | |---|---|---|---|---| | S1 | `ad2v.yaml` `spec_root` could point outside the repository, so a repository could make ad2v read local files into a map the user then shares | §7 | medium | fixed | | S2 | `ad2v.yaml` `rules_overlay` could point outside the repository (read into the stored rules) | §7 | medium | fixed | | S3 | an auto-detected `specs/` symlink could point outside the repository | §7 | medium | fixed | | S4 | `ad2v view` checked `Host` only on chat endpoints; a DNS-rebinding page could read the served map | §2 | medium | fixed | | S5 | writers (`extract --into`, `llm run`) would fire triggers embedded in a hostile file | §1 | medium | fixed (writers refuse; `trusted_schema = OFF`) | | S6 | `validate` did not report extra tables, views or triggers, huge strings or deep nesting (deep JSON crashed it) | §1 | low | fixed | | S7 | the MCP server served files with redefined views (wrong answers, errors) | §1, §5 | low | fixed (refused) | | S8 | `specs approve --to` accepted a destination outside the repository | §7 | low | fixed | | S9 | CSP lacked `object-src`, `base-uri`, `form-action`, `frame-ancestors`; no `Referrer-Policy` | §2 | info | fixed | | S10 | the conformance kit did not detect network, subprocess or write side effects | §6 | low | fixed (`side_effects` check) | | S11 | the `ad2v.chat` package was missing from the wheel (P7) | packaging | info | fixed | | S12 | plug-ins are not sandboxed (in-process, full privileges) | §6 | low | open (by design; opt-in and documented) | | S13 | the standalone viewer can be hung by pathological SQL views in a hostile file | §1 | low | open | | S14 | `braces` advisory in the build-only dependency chain | §8 | low | open (no fix available) | | S15 | prompt injection can bias informative LLM output | §4 | medium | open (inherent; informative only, no write tools) | | S16 | a timed-out plug-in thread keeps running until the process exits | §6 | low | open | | S17 | (P10) plan JSON and `.tf` are a new untrusted input: secrets, size, nesting, paths | §8b | medium | fixed (allowlist, limits, containment; tests) | | S18 | (P10) a hostile plan can still make a map claim a deployment that does not exist (it is the plan's word) | §8b | low | open (by design: the plan is the human's input) | | S19 | (P10) python-hcl2/lark parse untrusted `.tf`; a parser bug could hang the process on a crafted file | §8b, §8 | low | open (size limits; per-file isolation of errors, not of time) | --- # GitHub Action example ```yaml # EXAMPLE ONLY: a GitHub Actions workflow that posts an ad2v epoch diff as a pull-request comment. # It is not wired to any repository and uses no secrets beyond the job's own GITHUB_TOKEN. # Copy it to .github/workflows/ad2v-diff.yml in your repository and adapt the paths. # # What it does: # 1. builds a two-epoch .ad2v file: the PR's merge base, then the PR head (`ad2v history`), # 2. renders `ad2v diff --format md` (at most ~60 lines), # 3. creates or updates a single PR comment with it. # ad2v only reads the code and ingests test-result files; it never runs repository code. name: ad2v diff on: pull_request: types: [opened, synchronize, reopened] permissions: contents: read pull-requests: write # to post the comment jobs: ad2v-diff: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 # history is needed for `git archive` of the base commit - uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install ad2v run: pip install "ad2v==0.7.0" # pin the version for reproducible diffs # Optional: produce JUnit results so links can be `verified` (paths are examples). # - run: pip install -e . && pytest --junitxml=test-results/junit.xml - name: Build epochs (merge base, then PR head) env: BASE_SHA: ${{ github.event.pull_request.base.sha }} HEAD_SHA: ${{ github.event.pull_request.head.sha }} run: | MERGE_BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA") ad2v history --repo . --spec specs --test-results test-results/junit.xml \ --commits "$MERGE_BASE^!" --into pr.ad2v ad2v history --repo . --spec specs --test-results test-results/junit.xml \ --commits "$MERGE_BASE..$HEAD_SHA" --into pr.ad2v # For long PRs, compare the merge base with the head directly: ad2v diff pr.ad2v --from "$MERGE_BASE" --to head --format md > ad2v-diff.md cat ad2v-diff.md - name: Post or update the PR comment uses: actions/github-script@v7 with: script: | const fs = require('fs'); const body = '\n' + fs.readFileSync('ad2v-diff.md', 'utf8'); const { owner, repo } = context.repo; const issue_number = context.payload.pull_request.number; const comments = await github.paginate(github.rest.issues.listComments, { owner, repo, issue_number }); const mine = comments.find(c => c.body && c.body.startsWith('')); if (mine) { await github.rest.issues.updateComment({ owner, repo, comment_id: mine.id, body }); } else { await github.rest.issues.createComment({ owner, repo, issue_number, body }); } - name: Keep the .ad2v file as an artifact (optional) uses: actions/upload-artifact@v4 with: name: ad2v path: pr.ad2v ``` --- # Spec Markdown convention # ad2v spec Markdown convention, v0.1 Status: **draft, normative for extractors**. Reference implementation: `src/ad2v/extract/spec/markdown.py`. Most SDD specs are Markdown. ad2v extracts only what the text marks explicitly, deterministically. Everything else stays as prose, and the extractor does **not** interpret it. Semantic extraction from prose is an LLM task. It is planned for P3 as a *frozen* step (its output stored with provenance and reused), and it is never part of the deterministic extractor. ## 1. Components ```markdown ## Component: Reconciliation API {#CMP-001} ``` - A level-2 heading that starts with `Component:` declares a `spec_component`. - `{#ID}` gives its spec ID, which is also its qualified name. IDs match `[A-Z][A-Z0-9]*-[0-9]+`. - If no ID is given, the qualified name is `::`. This is allowed but discouraged, because renaming the heading changes its identity. - The component section runs until the next level-1 or level-2 heading. Non-requirement text inside the section is stored in `attrs.text`. ## 2. Requirements (EARS) ```markdown - [REQ-012] When , the shall . ``` - A list item (`-` or `*`) that starts with `[ID]` declares a `requirement`. Its qualified name is the ID. - A requirement inside a component section is a child of that component. Otherwise it is a child of the file's `spec_doc`. - The text is matched against the EARS templates, in this order. The first match wins, and keywords are case-insensitive: | `attrs.ears` | template | captured attrs | |---|---|---| | `unwanted` | `If , then the shall .` | trigger, system, response | | `event` | `[While , ]When , the shall .` | precondition?, trigger, system, response | | `state` | `While , the shall .` | precondition, system, response | | `optional` | `Where , the shall .` | precondition, system, response | | `ubiquitous` | `The shall .` | system, response | | `null` | anything else | (only `text`) | A requirement that doesn't match any template is still a requirement, with `ears: null`. **Spanish** (P9, ADR-0069). The same five forms, with *deberá(n)*, *debe(n)* or *tendrá que*, are tried after the English ones. A leading article on the system is dropped: | `attrs.ears` | plantilla | |---|---| | `unwanted` | `Si , [entonces ]el deberá .` | | `event` | `[Mientras , ]Cuando , el deberá .` | | `state` | `Mientras , el deberá .` | | `optional` | `Donde , el deberá .` | | `ubiquitous` | `El deberá .` | **Titles.** `- [REQ-012] (Exportar a CSV) When …` gives the requirement the title in parentheses. Without one, ad2v derives up to six words from the response. Titles are display only: the map reads `REQ-012 · Exportar a CSV`. ## 3. Everything else - The file becomes a `spec_doc` node (qualified name = path relative to the spec root). Text that is neither a component heading, a requirement line, nor inside a component section is concatenated into `attrs.prose`. - Fenced code blocks (```` ``` ````) are never parsed for components or requirements. - Level-3+ headings do not end a component section. ## 4. Referencing specs from code See format spec §3.6. In short: `# ad2v: REQ-012` comments, `ad2v: REQ-012` docstring lines, and test names such as `test_req_012_*`. ## 5. Other spec formats - **Gherkin** (`*.feature`): Feature → `spec_component` (`::`), Rule → prefix of its scenarios' names, Scenario / Scenario Outline / Background → `scenario`, steps → `scenario_step` (`::step[n]`). Tags that match the spec ID pattern (`@REQ-012`) are *references* (`spec_ref` + `annotated_as`), not definitions (ADR-0005). - **OpenAPI 3.x** (any `.yaml`/`.yml`/`.json` whose top-level `openapi` starts with `3`): each operation becomes a `spec_endpoint` with qualified name `METHOD /path`. The optional operation extension `x-ad2v-refs: [ID, …]` adds references. ## 6. SDD presets (P6.1, ADR-0048) The ad2v convention above is one of four Markdown **presets**. A file is read with exactly one. | preset | selected when | components | requirements | ids | |---|---|---|---|---| | `ad2v` | `## Component: … {#CMP-n}` or an EARS `REQ-n` line | `## Component:` | `- **REQ-n** …` | as written | | `spec-kit` | `spec.md` with a Functional Requirements heading and `**FR-` lines (github/spec-kit v1.1.0 template) | the feature folder | `**FR-001**: System MUST …`, `**SC-001**` (success criteria) | `:FR-001` | | `kiro` | `.kiro/specs//requirements.md` | the feature | `### Requirement N[: Title]` with `**User Story:**`; numbered acceptance criteria (EARS) as children | `-N`, `-N.M` | | `custom` | a `spec_markdown` rule with `preset: custom` | `component` regex (optional; group `name`) | `requirement` regex (required; groups `id` and `text`) | as captured | | `table` (P9) | a Markdown table whose header has an id column and a text column | the heading above the table; a `Componente`/`Component` cell → a sub-component | one per row whose id fits the grammar | as written | | `stories` (P9) | `HU-n`/`US-n` headings or items with "Como … quiero …" / "As a … I want …" | the heading above | the story; each acceptance criterion a child | `HU-001`, `HU-001.N` | Selection: the first `spec_markdown` rule in `ad2v.yaml` whose glob matches wins; otherwise auto-detection in the order kiro, spec-kit, ad2v, table, stories. ```yaml spec_markdown: - glob: "docs/requirements/*.md" preset: custom requirement: '^- \[(?PBIL-[0-9]+)\] (?P.+)$' - glob: ".kiro/specs/payments/requirements.md" preset: kiro id_prefix: PAY # instead of PAYMENTS ``` - **spec-kit**: user stories become `scenario` nodes. Their `**Given**`/`**When**`/`**Then**` clauses become steps. `[NEEDS CLARIFICATION: …]` sets `attrs.needs_clarification`. - **Kiro**: the prefix is the feature folder slug, upper-case alphanumerics of up to 12 characters (`order-export` → `ORDEREXPORT`, `001-auth` → `K001AUTH`). If two features share a prefix, the second (in path order) gets `2`, then `3`. Each criterion's EARS pattern is parsed (`attrs.ears`: `event`, `state`, `optional`, `unwanted`, `ubiquitous`). - **custom**: lines that look like ids but do not match are listed in the doc node's `attrs.skipped_ids`. - Tests cite namespaced ids by name (`test_fr_001_*`). Words before the prefix narrow the namespace: `test_refunds_fr_001_*` → `002-refunds:FR-001`. An ambiguous reference is `needs_review`. - **table**: header names are matched without regard to case or accents: - id: `ID`, `Id`, `Clave`, `Identificador`, `Código`, `Req`; - text: `Descripción`, `Requerimiento`, `Requisito`, `Description`, `Requirement`; - optional: `Título`/`Nombre`/`Title`, `Prioridad`/`Priority`, `Tipo`/`Type`, `Componente`/`Módulo`/`Component`. `\|` and pipes inside code spans are text, and `
` is a space. A row continued on the next line is merged and noted in `table_notes`. Rows whose id does not fit are listed in `skipped_ids`, repeated ids in `duplicate_ids`. A `Tipo` of *No funcional* / *Non-functional* / *NFR* / *Calidad* marks the row non-functional. - **stories**: "Como , quiero , para " (or "As a …, I want …, so that …"). The criteria list ("Criterios de aceptación" / "Acceptance criteria") can be numbered, bulleted or written as Dado/Cuando/Entonces (Given/When/Then), on one line or several. Each criterion is a child requirement with its `gwt` triple or EARS form. - **Non-functional**: ids with a prefix in `spec_ids.nonfunctional_prefixes` (default `RNF`, `NFR`, `NF`) are `requirement_type: nonfunctional`. They are counted apart and badged NFR. - Spanish Gherkin (`# language: es`, `Característica` / `Escenario` / `Dado` / `Cuando` / `Entonces`) is read like English. - Not parsed: Kiro `design.md`/`tasks.md`, spec-kit `plan.md`/`tasks.md`. ## 7. Where specs are found (ADR-0047) `--spec` (repeatable) beats `spec_root` in `ad2v.yaml` (a string or a list), which beats auto-detection: the first of `specs/`, `spec/`, `docs/specs/`, `.kiro/specs/`, `sdd/`, `docs/sdd/` that holds a recognized file, plus a root `openapi.{yaml,yml,json}`. `--no-spec-autodetect` turns that off. Code under a spec root is never extracted as code. `ad2v init` (ADR-0071) also finds spec folders with other names, by content, and proposes them as `spec_root`. ## 8. No convention at all: bootstrap from prose (ADR-0049) `ad2v specs bootstrap --from docs/ --plan` lists prose files with no recognized structure and estimates an LLM pass. `--yes --max-cost-usd N` writes proposals in this convention, with `-X` ids, to `/.ad2v-extracted/`. `ad2v specs review` compares them with existing specs. `ad2v specs approve ` moves one into the spec tree; nothing is normative before that. --- # Glyph grammar # ad2v glyph grammar, v0.2 Changes from v0.1: the `spec_root` kind (§3), and the status channel is now defined in [`visual-tokens-v0.2.md`](https://github.com/shanglai/ad2v/blob/main/docs/spec/visual-tokens-v0.2.md) (§7). Status: **draft, normative**. Reference implementation: `src/ad2v/glyphs.py`. A glyph is how a node looks. The grammar makes it computable without an LLM, identical across viewers, and learnable: a reader who knows the ten shapes can read any ad2v map. ## 1. Encoding channels | channel | encodes | source | decided in | |---|---|---|---| | **shape** (base shape + variant) | node **kind** | `nodes.kind` (+ container kind) | this document | | **modifiers** | node **attributes** | graph structure + attrs | this document | | **line style / fill / emphasis / color** | effective **status** | `node_status.status` | [`visual-tokens-v0.2.md`](https://github.com/shanglai/ad2v/blob/main/docs/spec/visual-tokens-v0.2.md) | | **size** | **weight** | `glyphs.size_weight` | this document | **Never overload a channel.** A risk is a modifier (badge), not a color. A status is a line style, not a shape. An API gateway is a gateway with N port axles, not a new shape. ## 2. Base shapes (closed set of 10) | base shape | used for | |---|---| | `cylinder` | datastore | | `tube` | queue | | `trapezoid` | ui; containers whose manifest kind is `gateway`, `api_gateway`, `ui` or `frontend` | | `rounded_box` | system, container, component, module, class, function, transaction | | `hexagon` | llm_endpoint, llm_call | | `diamond` | decision/branching behavior: fan_out | | `ring` | loop, retry | | `port_notch` | http_route, mcp_tool, cli_command, event_consumer. Drawn attached to the parent's face. | | `folded_sheet` | every spec kind | | `cloud` | external_service | ## 3. Exact kind → (base shape, variant) mapping | kind | base shape | variant | drawing note | |---|---|---|---| | system | rounded_box | `system` | largest box, thin wall | | container | rounded_box | `container` (or trapezoid `gateway`) | thick wall, tall | | component | rounded_box | `component` | medium | | module | rounded_box | `module` | flat tile | | class | rounded_box | `class` | **small** box, 0.5× module height | | function | rounded_box | `function` | **smallest** box, 0.25× module height, flat | | transaction | rounded_box | `transaction` | bracketed outline around its children | | http_route | port_notch | `http_route` | notch label = `METHOD path` | | mcp_tool | port_notch | `mcp_tool` | notch with tool glyph | | cli_command | port_notch | `cli_command` | notch with prompt glyph | | event_consumer | port_notch | `event_consumer` | inward notch | | datastore | cylinder | `datastore` | | | queue | tube | `queue` | horizontal | | external_service | cloud | `external_service` | | | llm_endpoint | hexagon | `llm_endpoint` | | | llm_call | hexagon | `llm_call` | small, flat | | ui | trapezoid | `ui` | | | loop | ring | `loop` | | | retry | ring | `retry` | ring with arrowhead | | fan_out | diamond | `fan_out` | | | spec_root … spec_endpoint | folded_sheet | = kind | spec_root largest (stacked sheets), scenario_step smallest | ## 4. Modifiers `glyphs.modifiers_json` always contains all five keys: | modifier | type | rule | drawn as | |---|---|---|---| | `ports` | int | number of outgoing `exposes` edges | notches/axles on the glyph | | `replicas` | int ≥ 1 | containers: `attrs.replicas` (manifest), others 1 | stacked shadow copies (max 3 drawn) | | `has_loop_risk` | bool | the node's `attrs.risk` ∩ {`io_in_loop`, `llm_in_loop`, `nested_loop_io`, `unbounded_retry`} ≠ ∅, **or any descendant has it** | warning badge | | `has_llm` | bool | kind is llm_endpoint/llm_call, or the node has an outgoing `invokes_llm` edge, or any descendant does | hexagon badge | | `external` | bool | kind is external_service, or the node has an outgoing `invokes_external` edge, or any descendant does | cloud badge | Modifiers roll up the `contains` hierarchy, so a collapsed container still tells you it hides an N+1 loop. ## 5. Level of detail (LOD) The same glyph at three detail levels. **The shape never changes across zoom.** | LOD | shows | |---|---| | 0 silhouette | base shape + variant proportions only | | 1 modifiers | + modifier badges, port notches, replica shadows, name label | | 2 open interior | + children laid out inside (only if `has_interior = 1`) | ## 6. Size `size_weight` (int ≥ 1): for resources (`datastore`, `queue`, `external_service`, `llm_endpoint`, `ui`), it is the sum of the weights of incoming non-hierarchy edges (how much the resource is used). For every other node, it is the line count of its span (`span_end - span_start + 1`). Nodes without a span use the sum over their children. Viewers SHOULD map size with a sub-linear scale (e.g. `sqrt`). ## 7. Status channel The status channel (line style, fill, decoration, weight, and color as reinforcement) is defined in [`visual-tokens-v0.2.md`](https://github.com/shanglai/ad2v/blob/main/docs/spec/visual-tokens-v0.2.md). Its source is `node_status.status` (one effective status per node, format spec §6.4). At far zoom, a collapsed glyph MAY show a small stacked bar of `status_rollup.counts_json` under it. That bar is a modifier and never changes the glyph's own line style. ## 8. Palette constraints (informative) Color is reserved for status (§7). Kinds are distinguished by shape only, so the map stays readable in grayscale and for color-blind users. Dark and light themes are defined in `schema/themes/`. --- # Format specification v0.7 (normative) # The ad2v file format, version 0.7 Status: **draft, normative**. The key words MUST, MUST NOT, SHOULD, and MAY are used as in RFC 2119. An `.ad2v` file is a single SQLite 3 database. It describes a software system from two sides, the **code** in a repository and the **specs** (spec-driven development documents), as one graph of nodes and edges. It also holds glyphs for rendering, workflows for replay, and the deterministic **pairing** of spec items with code: links, per-node statuses and roll-ups. Since v0.3 a file holds a **history**: a chain of epochs (one per extraction) with content-addressed deduplication, rename lineage between consecutive epochs, and the `stale` status. Viewers, agents (via MCP) and diff tools read it. Producers ("extractors") write it. Companion documents: - [`glyph-grammar-v0.2.md`](https://github.com/shanglai/ad2v/blob/main/docs/spec/glyph-grammar-v0.2.md): how nodes become shapes (normative, unchanged in v0.3 and v0.4). - [`visual-tokens-v0.2.md`](https://github.com/shanglai/ad2v/blob/main/docs/spec/visual-tokens-v0.2.md): how statuses become line styles and colors (status → style normative, palettes informative; unchanged in v0.3 and v0.4). - [`spec-markdown-convention.md`](https://github.com/shanglai/ad2v/blob/main/docs/spec/spec-markdown-convention.md): the Markdown spec convention (normative for extractors). - [`../../schema/ad2v-v0.7.sql`](https://github.com/shanglai/ad2v/blob/main/schema/ad2v-v0.7.sql): DDL (normative). - [`../../schema/ir-v0.7.schema.json`](https://github.com/shanglai/ad2v/blob/main/schema/ir-v0.7.schema.json): JSON Schema of the IR (normative). ### Changes from v0.6 (P10, ADR-0073..0081) - **A third side, `infra`** (§14): kinds `deployment_unit`, `managed_datastore`, `managed_queue`, `bucket`, `secret_store`, `network`, `scheduler`, `infra_other`, `infra_module`; `node_versions.side` CHECK includes `infra`. Infra nodes have glyphs and `node_status = none`, but no `layout` row (they are drawn in the C4 view). - **Edge kinds** `references`, `wires`, `deploys`, `provides_env` (§14.3). - **Epochs**: column `infra_hash` (`''` when the epoch has no infra input; it enters the epoch id only when set) and `change_kind` value `infra` (§5.1). - **Derived tables** `infra_status` (§14.4), `c4_elements`, `c4_relations` (§14.5); presentation table `c4_layout` (kept per epoch like `layout`). - **Rules**: the epoch rules gain the key `infra` (the typing and allowlist of `src/ad2v/infra/infra.yaml`), covered by `rules_hash`. - **Config**: `infra_root`, `infra.deploys`; `config_json.effective.derived_test_prefixes` when the specs add test-name prefixes (ADR-0073). - **Upgrade**: a v0.6 file is upgraded in place when opened for writing (`epochs` and `node_versions` rebuilt, rows copied with `infra_hash = ''`, views recreated, new tables created); v0.4 and v0.5 files chain through. Readers accept 0.4–0.7. - Without infra input nothing infra-related is written except `c4_*` rows of the code-only view (`env = ''`); normative dumps of such epochs are otherwise unchanged (ADR-0077 §6). ### Additions within v0.6 (P9, ADR-0068..0072) No DDL change and no version bump; v0.6 readers need no change. - **Spec attributes.** `attrs.title` on spec items (explicit or derived, display only); `attrs.requirement_type = "nonfunctional"` (by id prefix, `spec_ids.nonfunctional_prefixes`, or a table type cell). Table rows add `priority`, `type` and `component`. Stories add `actor`, `action` and `benefit`. Criteria add `gwt: {given, when, then}`. Steps add `keyword_type`. `spec_doc` nodes may carry `skipped_ids`, `duplicate_ids` and `table_notes`. - **Spanish EARS** fills the same `attrs.ears` values (§3.5, convention §2). - **Evidence kinds** (§6.2): `component_out_of_scope` (`component`, `languages`) on a requirement under a spec component paired with unread code; `test_in_unread_language` on a link from a test title in an unextracted module. `spec_ref_sources` values may be `test_title`. - **Statuses** (§6.4): a `scenario_step` is always `none` (reason `step`). A `scenario` adds one `via_spec_ref` reason per (referenced requirement, link status). - **Layout** (§12.1): `scenario_step` nodes get no `layout` row (`OFF_MAP_KINDS`). Files written before P9 keep their rows. - **Excludes** (§10): the default list grows (ADR-0068). `migrations/` and `alembic/` are code-only excludes: their `*.sql` stays in the walk. ### Changes from v0.5 (P8.1, ADR-0066) - New status **`not_evaluable`** (links and nodes, §6.2, §6.4): specified and implemented in a language no active extractor reads. The CHECK constraints of `links.status` and `node_status.status` include it; a v0.5 file is upgraded in place when opened for writing (both tables rebuilt, rows copied unchanged); v0.4 files go 0.4 → 0.5 → 0.6. Readers accept 0.4, 0.5 and 0.6. - **Language scope** (§4.1): code files in a language no active extractor reads become `module` nodes with `attrs.extracted = false` and `attrs.language`; they are epoch inputs, and the scope table is stored in the epoch rules under `scope`. - Containers are also inferred from `package.json`, `go.mod`, `Cargo.toml`, `pom.xml` and `build.gradle(.kts)` (§4). - Visual tokens: `status.not_evaluable`, `badge.scope`, decoration `scope_badge` (visual tokens). ### Changes from v0.4 v0.5 (P7) changes only informative objects and adds normative attributes. Epoch ids of repositories without `*.sql` files and every normative table keep their shape. - `llm_results.task` also allows `chat` (§13.3); new table `judge_rollups`; the view `node_status_effective` reads it (§11.4). - `datastore` nodes may carry `attrs.schema_digest`, and `http_route` nodes `attrs.schema_models` (§3.9). `*.sql` files are epoch inputs (they enter `repo_tree_hash`). ### Additions within v0.5 (P8, ADR-0065) No DDL change; readers of v0.5 need no change. - **Plug-in identity.** The stored epoch rules (`rules_versions.rules_json`) gain a key `plugins`: one record per active extractor or framework pack (`kind`, `name`, `version`, `api_version`, `rules_hash` of its own rules files, `attaches_to` for packs, `effective_rules_hash` for extractors), sorted by kind and name. Because it is part of the rules, it is covered by `rules_hash` and therefore by the epoch id; a changed, added or removed plug-in yields a new epoch with `change_kind` `rules`. Third-party extractors' effective rules are stored under `plugin_rules.`. Epochs written before P8 lack the key; the first P8 extraction of an unchanged tree therefore appends a new epoch (`rules`). - **Inputs.** `repo_tree_hash` covers every file claimed by an active plug-in's `file_patterns` (the built-ins claim `*.py` and `*.sql`, exactly the P7 inputs). - **Parse cache key.** `parse_cache.extractor_version` is `@[:]` (`python@1.0.0:python-local-3`) and `parse_cache.rules_hash` is that extractor's `effective_rules_hash` (§10.1). - `epochs.attrs.plugin_failures` (N, only when non-empty): third-party plug-ins that raised or timed out and were skipped (`name`, `kind`, `version`, `error`). - `http_route` nodes may carry `attrs.prefix_sources` (§6.1, `pairing.path_constants` and `pairing.resolve_settings_defaults`). - Readers accept 0.4 and 0.5. A writer upgrades a 0.4 file in place, in one transaction (ADR-0055). ### Changes from v0.3 v0.4 adds the **informative LLM layer** (§11) and nothing normative changes shape. `ad2v validate` rejects other format versions (re-extract; there is no migration). | area | change | |---|---| | storage | new `source_blobs` (file texts by content hash, class I), `llm_results` (content-addressed LLM cache), `llm_runs`, `llm_run_results`, `link_judgments`, `link_suggestions`, `spec_extractions`; `summaries` and `narrations` gain `result_id` (§11.3) | | views | `node_status_effective`: normative status plus LLM overlays that can only upgrade implemented/verified to `drift` (§11.3) | | pairing | new deterministic method `manifest_link` from `pairing.links` (human-confirmed suggestions); `pairing.rejected` (§6.1, §11.5) | | spec IDs | the grammar accepts a reserved `-X` range (`REQ-X001`) for LLM-proposed items; `.ad2v-extracted/` is never extracted (§3.6, §11.8) | | history | a callee rename no longer changes its callers (`call_masked_hash`, `call_sites`, evidence `callee_renamed`, §6.7, ADR-0026) | | config | the `llm:` section of ad2v.yaml has its own `llm_config_hash` and never enters `config_hash` or an epoch id (§11.1) | ### Changes from v0.2 v0.3 is not backward compatible with v0.2 files, and there is no migration. `ad2v validate` rejects other format versions with a message to re-extract. The logical model (nodes, edges, derived tables) keeps its v0.2 shape: readers that query the `nodes`, `edges`, `unresolved_refs` and `test_results` **views** and the derived tables see the same columns as in v0.2. | area | change | |---|---| | storage | base facts are content-addressed `*_versions` rows referenced per epoch by `epoch_*` membership rows; `nodes`, `edges`, `unresolved_refs`, `test_results` become views of the v0.2 shape (§5, ADR-0017) | | column classes | new class **V** (volatile): span lines and edge line lists. Deterministic, but ignored by `ad2v diff` and excluded from version keys (§1, §5.2) | | epochs | new `change_kind` (N), `pinned`, `label`, `materialized` (I). `seq` is UNIQUE. `rules_hash`/`config_hash` reference the stored rules and manifest (§5.1) | | derived data | glyphs, workflows, links, statuses exist only for **materialized** epochs (head + pinned); others are computed on demand (§5.3) | | history | new tables `id_lineage` (renames, §6.6), `link_state` (per-epoch link status incl. `stale`, §6.7), `code_fact_versions` (workflow inputs), `parse_cache` (§10.1) | | statuses | `stale` is produced (§6.7). Evidence kinds `spec_changed_since`, `code_changed_since`, `code_changed_unverified`, `retested`, `reannotated`, `fingerprint_mismatch`, `both_changed`, `descendants_changed`, `lineage_ambiguous` | | annotations | `# ad2v: REQ-012@` records which spec text the code was checked against (§3.6, §6.7) | | entry points | `cli_command` for `if __name__ == "__main__":` blocks; `event_consumer` for Celery tasks and beat, APScheduler, `schedule`; `attrs.trigger` on every port. Nested Click commands are named by their group path (§3.3, §3.5, ADR-0021) | | pairing | `pairing.orphan_policy` (`auto` / `strict` / `off`), part of `config_hash`; `meta.spec_coverage` (§6.2, ADR-0022) | | narrations | `mode` (containment / call_depth) is part of the primary key (§5.4) | | dump | `--epoch `; views are dumped, physical storage tables are not (§8) | | tools | `extract --into`, `history`, `diff`, `log`, `pin`/`unpin`, `prune`, `materialize` (§10) | --- ## 1. Principles 1. **Column classes.** Every table and column is **N** (normative: deterministic; the same inputs produce the same value in every conforming producer), **I** (informative: produced by an LLM, by runtime observation, by the wall clock, or file-management state such as pins), or **V** (volatile: deterministic, but moves with unrelated edits, like line numbers). Informative rows that come from an LLM MUST carry `model`, `model_version`, `prompt_hash`, `input_hash` and `confidence`. The DDL enforces this with `CHECK` constraints. V columns are part of the normative dump but are ignored when comparing epochs (§8.2) and never enter a version key. 2. **Stable identity.** A node's `id` depends on what the node *is* (side, kind, qualified name), never on its content. Change is tracked separately by `content_hash` and `subtree_hash`, and renames across epochs by lineage (§6.6). 3. **Cross-platform determinism.** Paths use `/`. Content is hashed after LF normalization and BOM stripping. All iteration is sorted. Timestamps never enter hashes. 4. **One file, canonical dump.** Two producers agree when their normative canonical dumps (§8) are byte-identical. SQLite file bytes are *not* compared. 5. **Relative levels.** There are no global levels (L1/L2/L3). Depth is the depth in the `contains` hierarchy and MAY differ between branches. `level_hint` is presentation metadata only. 6. **Knowledge never grants capability.** Producers read the repository. They MUST NOT import, execute or evaluate repository code. Test results are *ingested* from files; producers never run tests. `ad2v history` reads commits with `git archive`; it never checks out or runs anything. 7. **Additive evolution.** Later versions MAY add tables, columns, node kinds, edge kinds, attrs keys and risk flags. They MUST NOT change the meaning of an existing column or value. 8. **History is a pure function of the inputs.** An epoch's logical content (its view rows and derived rows) MUST equal what a fresh single-epoch extraction of the same inputs produces, except for the history-dependent parts that are defined in terms of the parent epoch: `id_lineage`, `link_state` and the link and node statuses they change (§6.6, §6.7), and the layout (§12.1). ## 2. Normalization and hashing (normative) | Rule | Definition | |---|---| | Text decoding | Strip one leading UTF-8 BOM (`EF BB BF`), decode UTF-8 with U+FFFD replacement, then replace `\r\n` and lone `\r` with `\n`. | | Paths | Relative to the repo root (code side) or the spec root (spec side); `/` separators; no leading `./`. The root itself is `""`. | | `sha256_hex(s)` | Lowercase hex SHA-256 of the UTF-8 bytes of `s`. | | Canonical JSON | `json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)`. Every `*_json` column holds canonical JSON. | | Node id | `"n_" + sha256_hex(side + "|" + kind + "|" + qualified_name)[:20]` | | Workflow id | `"w_" + sha256_hex(side + "|" + entry_node_id + "|" + source)[:20]` | | Link id | `"l_" + sha256_hex((spec_node_id or "") + "|" + (code_node_id or "") + "|" + method)[:20]` | | Epoch id | `"e_" + sha256_hex(join("|", [parent_epoch_id or "", repo_tree_hash, spec_tree_hash, rules_hash, config_hash, test_results_hash]))[:20]` | | Version key | `"v_" + sha256_hex(canonical_json(normative payload))[:20]`. Node payload: every N field of §3.1 (no span lines). Edge payload: `src`, `kind`, `dst`, `weight`, `provenance`, `attrs` without `lines`. Other versions: the row's N columns in DDL order. | | `content_hash` | For nodes with a source span: `sha256_hex(normalized text of lines span_start_line..span_end_line)` (lines joined with `\n`). For files (modules, `spec_doc`): the whole normalized file. For nodes without text (system, `spec_root`, container, component, resources, ports): `sha256_hex(canonical_json({"kind","name","attrs"}))`. | | `body_hash` | Functions and classes: `content_hash` of the span with the defined name masked in the `def`/`class` line. Used as the rename-invariant **fingerprint** of §6.6–6.7; other nodes use `content_hash`. | | `subtree_hash` | `sha256_hex(join("\n", [content_hash] + [":" for children sorted by id]))`, where children are nodes whose `parent_id` is this node. | | Tree hashes | `repo_tree_hash`, `spec_tree_hash`, `test_results_hash` = `sha256_hex(join("\n", ["\t" sorted by path]))` over the files read (test results: file paths relative to the repo if inside it, else the file name). | | `rules_hash` | `sha256_hex(canonical_json(merged rules))`: the built-in Python extractor's rules, the enabled framework packs' fragments pack-merged into them (maps merge, lists are unioned), the manifest's `rules_overlay` deep-merged (maps merge, lists and scalars replace), plus the `plugins` identity records and any third-party `plugin_rules` (P8, ADR-0065). Comments and formatting do not count; content does. | | `config_hash` | `sha256_hex(canonical_json({"effective": {"orphan_policy": …}, "manifest": parsed ad2v.yaml or {}}))`. Effective settings whose default is not visible in the manifest are included, so changing a default changes the hash. | Every input that can change the output (code, specs, rules, manifest, test results) enters the epoch id. Because the parent id enters it too, the same inputs appended after different histories give different epoch ids. **Test results carry forward.** When an extraction gets no test-result files, the epoch inherits the parent's test results and `test_results_hash`. Results are **fresh** in an epoch when files were given and their hash differs from the parent's (§6.7). ## 3. The IR The IR is the logical model of one epoch. `schema/ir-v0.7.schema.json` defines its JSON form, and the views and tables in §5 store it. ### 3.1 Node | field | class | meaning | |---|---|---| | `id` | N | stable id (§2) | | `side` | N | `code` or `spec` | | `kind` | N | closed enum (§3.2) | | `name` | N | display name | | `qualified_name` | N | canonical name, unique per (side, kind) (§3.3) | | `parent_id` | N | parent in the `contains` (code) / `declares` (spec) hierarchy; NULL for roots | | `path` | N | source file (repo- or spec-root-relative) | | `span_start_line`, `span_end_line` | V | source location (1-based, inclusive). Spec nodes point into spec files. | | `content_hash`, `subtree_hash` | N | §2 | | `level_hint` | N | optional hint: `system`, `container`, `component`, `code`, `port`, `resource`, `behavior`, `spec` | | `attrs` | N | kind-specific JSON object (§3.5) | | `spec_ref` | N | sorted list of explicit spec IDs attached to this node (§3.6) | ### 3.2 Node kinds (closed enum, unchanged from v0.2) | group | kinds | |---|---| | Structural | `system`, `container` (deployable/service), `component` (package/directory), `module` (source file), `class`, `function` (incl. methods and nested functions) | | Ports | `http_route`, `mcp_tool`, `cli_command`, `event_consumer` | | Resources | `datastore`, `queue`, `external_service`, `llm_endpoint`, `ui` | | Behavior | `loop`, `retry`, `fan_out`, `transaction`, `llm_call` | | Spec | `spec_root`, `spec_doc`, `spec_component`, `requirement`, `scenario`, `scenario_step`, `spec_endpoint` | Spec kinds have `side = spec`. All other kinds have `side = code`. Each side has exactly one root: `system` (code) and `spec_root` (spec, written when specs are extracted). Resources are children of `system`, and every `spec_doc` is a child of `spec_root`. ### 3.3 Qualified names | kind | qualified_name | example | |---|---|---| | `system` | the system name (manifest `system.name`, else repo directory name) | `payments-recon` | | `spec_root` | the system name (the id differs from `system` because side and kind differ) | `payments-recon` | | `container` | `container:` | `container:api` | | `component` | directory path | `api/recon_api` | | `module` | file path | `api/recon_api/matching.py` | | `class` | `::` (nested: `Outer.Inner`) | `api/recon_api/db.py::Run` | | `function` | `::`; methods `::.`; nested `::.` | `api/recon_api/matching.py::match_batch` | | `http_route` | `http:: ` | `http:api:GET /v1/reconciliations/{id}` | | `mcp_tool` | `mcp::` | `mcp:api:reconciliation_summary` | | `cli_command` | `cli::`: the command name, prefixed by the names of the nested Click groups above it (not the root group) and separated by spaces | `cli:worker:run-nightly`, `cli:dispatch:plugins list` | | `cli_command` (main block) | `cli::__main__:` | `cli:worker:__main__:worker/recon_worker/cli.py` | | `event_consumer` | `event::` for subscriptions; `event:::` for scheduled and task entry points (§3.5) | `event:worker:celery_beat:worker/tasks.py::nightly` | | resources | `:` (§3.7) | `datastore:env:RECON_DB_URL` | | behavior | `@#[~n]`, see below | `api/recon_api/matching.py::match_batch@loop#81dc075c` | | `spec_doc` | spec file path | `components.md` | | `spec_component`, `requirement` | the explicit spec ID when present (`CMP-001`, `REQ-012`), else `::` | `REQ-012` | | `scenario` | `::::` (`#2`, `#3`… appended to duplicates) | `features/recon.feature::Statement reconciliation::Start a run` | | `scenario_step` | `::step[]`, 1-based | `...::Start a run::step[2]` | | `spec_endpoint` | ` ` as written in the spec (without the `servers` base path) | `POST /reconciliations` | **Behavior anchors (content-anchored identity, ADR-0011).** The anchor is `sha256_hex(normalized header)[:8]`, where the header is whitespace-normalized (runs of whitespace → one space, trimmed) and chosen per construct: | construct | header text | |---|---| | `for` loop | the iterable expression (`transactions`) | | `while` loop | the condition | | comprehension | the iterable of its first `for` clause | | retry / transaction decorator | the whole decorator (`@retry(wait=...)`) | | `Retrying(...)` call, `llm_call` | the call expression | | `asyncio.gather`-style fan-out | the argument list (`(*[f(x) for x in xs])`) | | executor `map` / `submit` fan-out | the call expression | | `with X.begin():` transaction | the with-item expression | When two candidates of the same kind in the same enclosing scope have identical headers, the second one gets `~2`, the third `~3`, and so on, in source order. Anchors count all candidates, including the ones not emitted, so changing `behavior.emit_all` never renames a node. The normalized header is stored in `attrs.header`; lineage uses it to follow a behavior whose header was edited (§6.6). **Spec IDs are the qualified names of spec nodes.** ### 3.4 Edges `(src, kind, dst)` is unique within an epoch. Repeated occurrences increase `weight`. `attrs.lines` (sorted, deduplicated; class V) records the source lines of occurrences on the `src` side when known. | kind | from → to | meaning | |---|---|---| | `contains` | code parent → code child | hierarchy (mirrors `parent_id`) | | `declares` | spec parent → spec child | spec hierarchy (mirrors `parent_id`) | | `imports` | module → module | repo-internal import | | `calls` | function/module/port → function/class/module | resolved call; port → handler (a main-block port calls its module) | | `reads`, `writes` | function/module → `datastore` | access classified by method / SQL verb; unknown access defaults to `reads` with `attrs.access = "default"` | | `publishes`, `consumes` | function/module → `queue` | | | `exposes` | container → port | the container serves the port | | `invokes_external` | function/module → `external_service` | | | `invokes_llm` | function/module → `llm_endpoint` | | | `annotated_as` | any node → spec node | explicit spec ID reference (§3.6). Also spec → spec (a Gherkin scenario tagged `@REQ-012`), see ADR-0005. Code → spec edges are the input of the `annotation` pairing method (§6.1). | `provenance` is `deterministic` for every edge a v0.3+ extractor writes. ### 3.5 Attributes (well-known keys) | kind | keys | |---|---| | `container` | `container_kind` (manifest `kind`), `replicas`, `paths`, `declared` (bool) | | `function` | `async` (bool), `decorators` (list of dotted names), `is_test` (bool), `params` (names in order, without `self`/`cls`), `var_kwargs` (bool: has `**kwargs`), `body_hash` (§2), `spec_ref_fingerprints` (§3.6) | | `class` | `body_hash`, `spec_ref_fingerprints` | | `http_route` | `method`, `path`, `framework`, `handler` (qname) | | `mcp_tool` | `tool_name`, `framework`, `handler` | | `cli_command` | `command`, `framework`, `handler`, `trigger` (`click`, `main`), `group` (Click group qname, nested commands) | | `event_consumer` | `framework`, `handler`, `source` (subscription name if resolvable), `trigger` (see below) | | resources | `libraries`, `resource_name`, `name_source` (`literal`, `env`, `unknown`) | | behavior | `risk` (sorted list of risk flags), `construct`, `library`, `touches`, `header` (§3.3), `severity`, `severity_rank`, `severity_reach` (§3.8) | | `requirement` | `text`, `ears` (`event`, `state`, `unwanted`, `optional`, `ubiquitous`, or `null`), `trigger`, `system`, `response`, `precondition` | | `scenario` | `keyword`, `tags`, `background` (bool), `examples` (count), `rule` | | `scenario_step` | `keyword`, `keyword_type`, `text` | | `spec_endpoint` | `method`, `path`, `operation_id`, `summary`, `tags`, `base_path` (path of `servers[0].url`), `path_params`, `query_params`, `required_query` (parameter names; local `$ref`s resolved) | | `spec_doc` | `format` (`markdown`, `gherkin`, `openapi`), `prose` (Markdown only: unparsed text), `base_path` (OpenAPI) | **Entry points (ADR-0021).** Besides HTTP routes, MCP tools, Typer/Click commands and queue subscriptions, the rules (`ports.entry_points` in `rules/python.yaml`) recognize: | construct | port | `trigger` | |---|---|---| | `if __name__ == "__main__":` (either operand order) | `cli_command`, handler = the module, span = the `if` block | `main` | | `@app.task`, `@shared_task` (Celery) | `event_consumer` | `celery_task` | | `app.conf.beat_schedule = {…: {"task": ""}}` | `event_consumer` for the resolved task function | `celery_beat` | | `scheduler.add_job(fn, …)`, `@scheduler.scheduled_job(…)` (APScheduler) | `event_consumer` | `apscheduler` | | `schedule.every(…)….do(fn)` | `event_consumer` | `schedule` | | Click commands and groups | `cli_command` | `click` | Entry points start workflows (§7), so they raise the `reach` of the behaviors they lead to (§3.8). **Risk flags** (`attrs.risk`): `io_in_loop`, `llm_in_loop`, `unbounded_retry`, `fan_out`, `nested_loop_io`, `slow_io_in_transaction` (ADR-0006). By default, behavior nodes are emitted only when they have at least one risk flag. Set `behavior.emit_all: true` to emit all. ### 3.6 Explicit spec references A spec ID matches `(?:[a-z0-9][a-z0-9-]*:)?[A-Z][A-Z0-9]*-X?[0-9]+(?:\.[0-9]+)?` (P6.1, ADR-0048): - an optional **namespace** (`001-payments:FR-001`): spec-kit ids are namespaced by their feature folder, so two features can both define `FR-001`; - an optional **sub-id** (`ORDEREXPORT-1.2`): a Kiro acceptance criterion, a child of `ORDEREXPORT-1`; - the `-X` range is reserved for LLM-proposed items (§11.8). A reference without a namespace resolves to the single namespace that defines it. If several do, the edge carries `attrs.ambiguous` and the link is `needs_review` with evidence `ambiguous_spec_ref` (candidates listed). The default test-name prefixes are `REQ`, `CMP`, `FR`. Explicit references come from: 1. a comment `# ad2v: ID[, ID…]` inside a function/class span, or directly above a definition or its decorators (contiguous comment block); a comment elsewhere attaches to the module; 2. a docstring line `ad2v: ID[, ID…]`; 3. the name of a test function (`test_*` in `test_*.py` / `*_test.py`) that contains `_` for a configured prefix (default `REQ`, `CMP`): `test_req_012_x` → `REQ-012`; 4. Gherkin tags `@ID` on features and scenarios; 5. OpenAPI operation extension `x-ad2v-refs: [ID, …]`. In comments and docstrings an ID MAY carry a **spec fingerprint**, `ID@`, where `fp8` is the first 8 hex digits of the spec node's `content_hash` the author checked the code against (`# ad2v: REQ-012@1a2b3c4d`). The fingerprints are recorded in the node's `attrs.spec_ref_fingerprints` (`{ID: fp8}`), and `spec_ref` holds the bare ID. `ad2v inspect FILE --node REQ-012` shows the spec node's current `content_hash`. Every reference is recorded in the node's `spec_ref`. When a spec node has qualified name `ID`, an `annotated_as` edge is written. Otherwise a row is written to `unresolved_refs` (`reason = 'not_found'`), and the node gets the status `needs_review` (§6.4). ### 3.7 Resource naming A resource is named by the first match, in order: 1. a string literal that contains `://`, with credentials, query and fragment stripped. Datastores keep `scheme://host[:port][/path]`. External services, queues and LLM endpoints keep `scheme://host[:port]`. For f-strings, only the literal prefix before the first `{...}` is used; 2. an environment-variable read (`os.environ[...]`, `os.getenv(...)`, `os.environ.get(...)`), giving `env:` (a default value is ignored); 3. for libraries whose rule sets `name_from_literal` (queues), a plain string literal; 4. otherwise `:unknown`. Arguments are searched first in the rule's `name_kwargs` (for which any literal counts, e.g. `model=`), then in positional arguments. Constants are followed through module scope and imports. A client built from another client inherits its resource (`sessionmaker(bind=engine)`). A touch on a client whose resource is still `unknown` may be named from the call's own arguments (`publisher.publish(TOPIC, ...)`). **What counts as a touch** (an IO edge): a call to a method of a client bound to a resource, unless the rule lists the method in `pure_methods` (for SQLAlchemy this includes the query builders `select`, `join`, `filter`, `where`, …); or a library-level call, unless the rule lists it as a constructor or `pure`. For datastore libraries, library-level calls do not count (`direct_calls_touch: false`) unless listed in `touch_functions`: `sqlalchemy.inspect()` is not IO, while `session.execute()` is. ### 3.9 Datastore schema digests (normative, v0.5, ADR-0059) A producer MAY attach `attrs.schema_digest = {"tables": [...]}` to `datastore` nodes, derived from the repository text only: SQLAlchemy declarative classes (`__tablename__`, `Column`/`mapped_column`, `UniqueConstraint`/unique `Index` in `__table_args__`) and `CREATE TABLE` statements in `*.sql` files. Each table is `{table, source: "sqlalchemy" | "sql", path, class?, columns: [{name, type, pk?, unique?, not_null?, fk?}], unique: [[column, ...], ...]}`, sorted by table, source and path. A model attaches to every datastore read or written by its module or by a module that imports it; SQL DDL attaches to the only datastore whose `libraries` include a SQL library; otherwise a synthetic `datastore::models` node under the code system holds the tables. `http_route` nodes MAY carry `attrs.schema_models = [{role: "request" | "response", class, fields: [{name, type}]}]` for Pydantic models used by their handler. ### 3.8 Risk severity (normative, ADR-0014) Every behavior node gets `attrs.severity` (float), `attrs.severity_reach` (int) and `attrs.severity_rank` (int, 1 = worst): ``` severity = round((max base over the node's risk flags + sum of nesting bonuses) * reach_factor, 4) reach = number of distinct code-side workflows whose steps include the node reach_factor = unreachable if reach = 0 = min(cap, reachable + per_extra_workflow * (reach - 1)) otherwise rank = position in the order (-severity, node id), starting at 1 ``` The default weights are `base` = {llm_in_loop 5, unbounded_retry 4, io_in_loop 3, slow_io_in_transaction 3, fan_out 2}, `nesting_bonus` = {nested_loop_io 2}, and `reach` = {unreachable 1.0, reachable 1.5, per_extra_workflow 0.1, cap 2.5}. They live in the rules (`severity:`), so they are covered by `rules_hash` and an overlay can change them. ## 4. Containers If a manifest declares `containers`, a file belongs to the container with the longest path prefix that matches its path. Without declared containers, every top-level directory that contains `pyproject.toml`, `setup.py` or `Dockerfile` (since v0.6 also `package.json`, `go.mod`, `Cargo.toml`, `pom.xml`, `build.gradle` or `build.gradle.kts`) is a container named after the directory. Files that match no container go to a fallback container named after the system, with `paths: ["."]`. It is created only if such files exist. When no container is found, this fallback is the repo root. ### 4.1 Language scope (v0.6, ADR-0066) A producer only evaluates code its extractors read. Every other file whose extension is in the scope table (TypeScript, JavaScript, Vue, Svelte, Java, Kotlin, Scala, Go, Rust, C#, F#, Ruby, PHP, Swift, Objective-C, Dart, Elixir, C, C++; `*.d.ts` excluded), that no active extractor or framework pack claims and that is outside the excludes and the spec roots, MUST be recorded as a `module` node (side `code`, qualified name = path, `content_hash` of the normalized file) with `attrs.extracted = false` and `attrs.language`. Its content enters `repo_tree_hash`. `ad2v: ` markers in comments (`//`, `#`, `/*`, `*`, `--`, ` **meta** | column | class | note | |---|---|---| | `key` | N | | | `value` | N/I | (the row's `class`) | | `class` | N | | **epochs** | column | class | note | |---|---|---| | `epoch_id` | N | e_ + sha256(parent\|repo\|spec\|rules\|config\|test_results hashes)[:20] | | `parent_epoch_id` | N | NULL only for the first kept epoch | | `seq` | N | 1, 2, 3... in append order | | `repo_commit` | N | git commit the epoch was extracted from, if known | | `repo_tree_hash` | N | | | `spec_tree_hash` | N | | | `rules_hash` | N | | | `config_hash` | N | | | `test_results_hash` | N | tree hash of the normalized test-result files in effect | | `change_kind` | N | which input hashes changed vs the parent | | `pinned` | I | file-management state: kept by prune, stays materialized | | `label` | I | file-management state (e.g. a release tag) | | `materialized` | I | file-management state: derived rows present | | `attrs_json` | N | config echo, pairing warnings, unmatched tests, cache stats | | `created_at` | I | excluded from canonical dump | | `infra_hash` | N | v0.7: tree hash of the infra inputs (plans, .tf); '' when none | **rules_versions** | column | class | note | |---|---|---| | `rules_hash` | N | | | `rules_json` | N | merged, canonicalized rules | **config_versions** | column | class | note | |---|---|---| | `config_hash` | N | | | `config_json` | N | {"manifest": ..., "effective": {...}} | **epoch_materialized** | column | class | note | |---|---|---| | `epoch_id` | I | | | `table_name` | I | | **node_versions** | column | class | note | |---|---|---| | `vid` | I | storage surrogate, file-local | | `version_key` | N | v_ + sha256(canonical normative payload)[:20] | | `node_id` | N | n_ + sha256(side\|kind\|qualified_name)[:20] | | `side` | N | infra: v0.7 | | `kind` | N | closed enum, see spec §3.2 | | `name` | N | | | `qualified_name` | N | | | `parent_id` | N | NULL for roots | | `path` | N | repo- or spec-root-relative, forward slashes | | `content_hash` | N | | | `subtree_hash` | N | | | `level_hint` | N | optional presentation hint | | `attrs_json` | N | | | `spec_ref_json` | N | sorted list of explicit spec IDs | **epoch_nodes** | column | class | note | |---|---|---| | `seq` | N | the epoch | | `vid` | N | | | `span_start_line` | V | 1-based, inclusive | | `span_end_line` | V | 1-based, inclusive | **edge_versions** | column | class | note | |---|---|---| | `vid` | I | storage surrogate, file-local | | `version_key` | N | v_ + sha256(canonical normative payload)[:20] | | `src` | N | | | `dst` | N | | | `kind` | N | closed enum, see spec §3.4 | | `weight` | N | occurrence count | | `attrs_json` | N | without the volatile `lines` key | | `provenance` | N | | | `model` | I | required when provenance='llm' | | `model_version` | I | | | `prompt_hash` | I | | | `input_hash` | I | | | `confidence` | I | | **epoch_edges** | column | class | note | |---|---|---| | `seq` | N | the epoch | | `vid` | N | | | `lines_json` | V | attrs.lines (source lines of occurrences) | **unresolved_ref_versions** | column | class | note | |---|---|---| | `vid` | I | storage surrogate, file-local | | `version_key` | N | | | `src` | N | node carrying the reference | | `ref` | N | the referenced identifier, verbatim | | `ref_kind` | N | 'spec_id' | | `reason` | N | 'not_found' | **epoch_unresolved_refs** | column | class | note | |---|---|---| | `seq` | N | the epoch | | `vid` | N | | **test_result_versions** | column | class | note | |---|---|---| | `vid` | I | storage surrogate, file-local | | `version_key` | N | | | `test_node_id` | N | | | `outcome` | N | | | `source_file` | N | result file name (repo-relative if inside the repo) | | `duration_ms` | N | as recorded in the input file | **epoch_test_results** | column | class | note | |---|---|---| | `seq` | N | the epoch | | `vid` | N | | **code_fact_versions** | column | class | note | |---|---|---| | `vid` | I | storage surrogate, file-local | | `version_key` | N | | | `fact_kind` | N | | | `subject_id` | N | function/module id ('events','wrappers') or port id ('entry') | | `payload_json` | N | | **epoch_code_facts** | column | class | note | |---|---|---| | `seq` | N | the epoch | | `vid` | N | | **id_lineage** | column | class | note | |---|---|---| | `epoch_id` | N | the newer epoch | | `old_id` | N | node id in the parent epoch | | `new_id` | N | node id in this epoch | | `method` | N | | | `score` | N | 1.0, or the header Jaccard similarity | **link_state** | column | class | note | |---|---|---| | `epoch_id` | N | | | `link_id` | N | | | `spec_node_id` | N | | | `code_node_id` | N | | | `method` | N | | | `status` | N | final link status in this epoch | | `history_json` | N | history evidence (spec_changed_since, ...) | | `agreed_epoch_id` | N | last epoch where the link was implemented/verified | | `agreed_spec_hash` | N | spec content_hash at that agreement | | `agreed_code_hash` | N | code content_hash at that agreement | **parse_cache** | column | class | note | |---|---|---| | `path` | I | repo-relative path | | `content_hash` | I | | | `rules_hash` | I | | | `extractor_version` | I | | | `payload_blob` | I | | **glyphs** | column | class | note | |---|---|---| | `epoch_id` | N | | | `node_id` | N | | | `base_shape` | N | glyph grammar §2 | | `variant` | N | glyph grammar §3 | | `modifiers_json` | N | glyph grammar §4 | | `size_weight` | N | glyph grammar §6 | | `has_interior` | N | 0/1, LOD 2 availability | **workflows** | column | class | note | |---|---|---| | `epoch_id` | N | | | `workflow_id` | N | w_ + sha256(side\|entry_node_id\|source)[:20] | | `name` | N | | | `side` | N | | | `entry_node_id` | N | | | `scenario_id` | N | spec-side workflows: the scenario node | | `source` | N | | | `attrs_json` | N | e.g. truncated, max_depth | | `model` | I | required when source='llm_inferred' | | `model_version` | I | | | `prompt_hash` | I | | | `input_hash` | I | | | `confidence` | I | | **workflow_steps** | column | class | note | |---|---|---| | `epoch_id` | N | | | `workflow_id` | N | | | `seq` | N | 0-based, contiguous | | `node_ids_json` | N | nodes lit together (parallel steps share a row) | | `depth` | N | call depth from the entry (entry = 0) | **workflow_step_nodes** | column | class | note | |---|---|---| | `epoch_id` | N | | | `workflow_id` | N | | | `seq` | N | | | `node_id` | N | | **links** | column | class | note | |---|---|---| | `epoch_id` | N | | | `link_id` | N | l_ + sha256(spec_node_id\|code_node_id\|method)[:20] | | `spec_node_id` | N | NULL for orphan code | | `code_node_id` | N | NULL for missing/specified spec items | | `method` | N | how the pair was found; 'none' for unpaired rows | | `status` | N/I | (I when provenance='llm') | | `provenance` | N | | | `evidence_json` | N/I | | | `confidence` | N/I | 1.0 for deterministic links | | `model` | I | | | `model_version` | I | | | `prompt_hash` | I | | | `input_hash` | I | | | `created_at` | I | excluded from canonical dump | **node_status** | column | class | note | |---|---|---| | `epoch_id` | N | | | `node_id` | N | | | `status` | N | | | `reasons_json` | N | {precedence: [...], reasons: [...]} (spec §6.4) | **status_rollup** | column | class | note | |---|---|---| | `epoch_id` | N | | | `node_id` | N | | | `counts_json` | N | {status: count}, zero counts omitted | **infra_status** | column | class | note | |---|---|---| | `epoch_id` | N | | | `node_id` | N | | | `env` | N | infra environment ('default', 'prod', ...) | | `status` | N | | | `reasons_json` | N | [{kind, ...}] sorted | **c4_elements** | column | class | note | |---|---|---| | `epoch_id` | N | | | `env` | N | | | `element_id` | N | c4_ + sha256(kind\|key)[:20] | | `level` | N | | | `kind` | N | person, system, external_system, deployment_unit, ... | | `name` | N | | | `node_ids_json` | N | the map nodes it stands for (click-through) | | `attrs_json` | N | coverage roll-up, infra status, NFRs, ... | **c4_relations** | column | class | note | |---|---|---| | `epoch_id` | N | | | `env` | N | | | `src` | N | element_id | | `dst` | N | element_id | | `kind` | N | uses, calls, reads, writes, publishes, wires, references, ... | | `attrs_json` | N | weight, env var names (wires) | **c4_layout** | column | class | note | |---|---|---| | `epoch_id` | N | | | `env` | N | | | `element_id` | N | | | `row` | N | persons, system, units, data, external | | `slot` | N | position in its row (kept while the element exists) | | `x` | N | | | `y` | N | | | `w` | N | | | `h` | N | | **summaries** | column | class | note | |---|---|---| | `epoch_id` | N | | | `node_id` | N | | | `length` | N | | | `text` | I | | | `model` | I | | | `model_version` | I | | | `prompt_hash` | I | | | `input_hash` | I | | | `confidence` | I | | | `created_at` | I | | | `result_id` | I | the cached LLM result (NULL for label) | **narrations** | column | class | note | |---|---|---| | `epoch_id` | N | | | `workflow_id` | N | | | `mode` | N | projection mode | | `detail_level` | N | projection level (workflows.project) | | `span_start` | N | first fine step seq (inclusive) | | `span_end` | N | last fine step seq (inclusive) | | `text` | I | | | `model` | I | | | `model_version` | I | | | `prompt_hash` | I | | | `input_hash` | I | | | `confidence` | I | | | `created_at` | I | | | `result_id` | I | the cached LLM result | **layout** | column | class | note | |---|---|---| | `epoch_id` | N | | | `node_id` | N | | | `lod` | N | | | `algorithm` | N | name@version of the deterministic layout | | `x` | N | | | `y` | N | | | `z` | N | | | `w` | N | | | `h` | N | | | `d` | N | | **layout_grids** | column | class | note | |---|---|---| | `epoch_id` | N | | | `node_id` | N | the parent node | | `algorithm` | N | e.g. nestgrid-v1 | | `grid_json` | N | {"cols": [...], "rows": [...], "slots": {child: [row, col]}} | **tours** | column | class | note | |---|---|---| | `epoch_id` | N | | | `tour_id` | N | | | `title` | N/I | | | `source` | N | | | `workflow_id` | N | optional: tour that replays a workflow | | `model` | I | | | `model_version` | I | | | `prompt_hash` | I | | | `input_hash` | I | | | `confidence` | I | | | `created_at` | I | | **tour_stops** | column | class | note | |---|---|---| | `epoch_id` | N | | | `tour_id` | N | | | `seq` | N | | | `node_id` | N | camera target | | `camera_json` | N | {lod, yaw, pitch, zoom} | | `text` | I | | **source_blobs** | column | class | note | |---|---|---| | `content_hash` | I | | | `text_blob` | I | | **llm_results** | column | class | note | |---|---|---| | `result_id` | I | r_ + sha256(cache key \| ordinal)[:20] | | `seq` | I | insertion order | | `task` | I | | | `input_hash` | I | sha256(canonical JSON of the exact slice) | | `prompt_hash` | I | sha256(normalized prompt template) | | `provider` | I | | | `model` | I | | | `model_version` | I | as reported by the provider | | `params_hash` | I | sha256(canonical generation params) | | `subject` | I | what was asked about (link id, node id, ...) | | `output_json` | I | the parsed output (NULL if not JSON) | | `valid` | I | 0/1: passed schema + deterministic checks | | `validation_errors` | I | JSON list | | `usage_json` | I | {"input_tokens","output_tokens"} | | `cost_usd_est` | I | from llm.pricing (NULL if unpriced) | | `latency_ms` | I | | | `created_at` | I | | **llm_runs** | column | class | note | |---|---|---| | `run_id` | I | u_ + sha256(started_at\|tasks\|epoch)[:20] | | `started_at` | I | | | `ended_at` | I | | | `epoch_id` | I | | | `provider` | I | | | `model` | I | | | `model_version` | I | last reported by the provider | | `llm_config_hash` | I | | | `llm_config_json` | I | effective llm config (no secrets) | | `tasks_json` | I | | | `calls` | I | provider calls made | | `cache_hits` | I | | | `input_tokens` | I | | | `output_tokens` | I | | | `cost_usd_est` | I | | | `max_cost_usd` | I | | | `aborted` | I | 1: stopped before exceeding the budget | | `notes_json` | I | | **llm_run_results** | column | class | note | |---|---|---| | `run_id` | I | | | `result_id` | I | | | `cached` | I | 1: served from the cache | **link_judgments** | column | class | note | |---|---|---| | `epoch_id` | I | | | `link_id` | I | | | `result_id` | I | | | `verdict` | I | | | `confidence` | I | | | `rationale` | I | | | `evidence_json` | I | | | `valid` | I | | | `overlay` | I | | **link_suggestions** | column | class | note | |---|---|---| | `epoch_id` | I | | | `suggestion_id` | I | s_ + sha256(spec_node_id\|code_node_id)[:20] | | `spec_node_id` | I | | | `code_node_id` | I | | | `candidate_rank` | I | 1-based BM25 rank | | `lexical_score` | I | BM25 score (deterministic) | | `result_id` | I | | | `verdict` | I | | | `confidence` | I | | | `rationale` | I | | | `state` | I | | **spec_extractions** | column | class | note | |---|---|---| | `epoch_id` | I | | | `spec_doc_node_id` | I | | | `result_id` | I | | | `output_path` | I | spec-root-relative | | `state` | I | | **judge_rollups** | column | class | note | |---|---|---| | `epoch_id` | I | | | `node_id` | I | a container, component or spec_component | | `verdict` | I | | | `confidence` | I | of the judgment that set the verdict | | `counts_json` | I | {"conforms": n, "partial": n, ...} (confident ones) | | `link_ids_json` | I | the judged descendant links that count | | `min_confidence` | I | the threshold used |