Corpus pipeline
The pipeline separates authoritative, large inputs from the compact projections needed at runtime. Rebuilding has two stages: canonical corpus construction, then public query-artifact generation.
Source locations
The raw Sanskrit repository is expected at /Users/amrit/SanskritRepo/11-sanskrit in the current local setup. It contains the parsed text material and supporting folders including 00-concepts, 00-dhatus, and 00-lemma-index.
Scripts do not need that absolute path hard-coded. Their source-dependent locations can be overridden with environment variables:
| Variable | Used by |
|---|---|
FRACTALDHARMA_RAW_DIR | text and bundle construction scripts |
FRACTALDHARMA_WIKI_DIR | concept, lemma concordance, and description builders |
Set these when building on another machine or in CI. Verify the directory contents before rebuilding; an incorrect source root can create a syntactically valid but incomplete corpus.
Canonical data
The working canonical data lives under static/data. Important inputs include texts.json, texts/*.json, text descriptions, lemmas, dictionaries, dhātus, root-word links, occurrences, concordance data, concepts, and sūtras.
These are not runtime assets. They are deliberately excluded from the deployed asset directory because they are too large and too broad for normal page requests.
Full rebuild order
pnpm data:rebuild executes the corpus stages in dependency order:
build-texts.jssplit-large-texts.jspatch-texts-data.mjsbuild-occurrences.jsbuild-bundles.jsbuild-lemma-concordance.mjsbuild-concept-graph.mjsbuild-text-descriptions.mjsbuild-query-artifacts.mjs
Use this after raw-source or analytical-data changes. Use pnpm data:build when canonical static/data is already correct and only the public projections need regeneration.
Text notes
Human-readable text notes are Markdown files in static-runtime/data/generated/v1/text-notes. A text page fetches the note named after its slug and renders only the intended sections. Text notes are a content layer; they do not replace parsed reader data.
Rebuild checks
After a rebuild, inspect representative texts, verify expected slugs in texts/index.json, and run pnpm test:data. Check especially for nested raw source directories, changed reference conventions, and large split texts.