Parser and Translation Tree

Use this runbook when changing sentence extraction, marker placement, Markdown translation files, migration, sync, or output structure audits.

Pipeline

compact template
  -> expanded template with translation markers
  -> translator-facing translation tree
  -> translated expanded template
  -> packaged DSW document template

The key goal is to give translators complete, editable units without changing the executable Jinja behavior of the original template.

What Belongs Where

  • Template-specific Jinja rewrite logic belongs under _template_transform/science_europe*.py.

  • Generic marker, branch, and inline conditional behavior belongs under _template_transform/.

  • Translation Markdown format and metadata belongs under _translation_tree/.

  • XLIFF exchange belongs under _translation_tree/xliff.py; keep it as a thin adapter around the canonical Markdown tree.

  • Cross-version translation reuse belongs in _translation_tree/merge.py and translation_repository/.

  • Target-language output cleanup belongs in _translation_tree/output_polish.py.

  • Named source-preview regression guards belong in _translation_tree/source_quality_rules.py; they must not mutate executable template source.

Do not fix broken sentence boundaries by hand-editing generated translation.md files. Fix the transform/export logic and regenerate.

Adding an Upstream Rewrite

Use a profile rule only when generic Jinja/HTML parsing cannot preserve a complete translation unit.

  1. Confirm the template identity in template.json. Science Europe-only rules belong in science_europe_balanced_rules.py or science_europe_unbalanced_rules.py, not a generic parser module.

  2. Add the smallest reversible replacement to a named ReversibleReplacementGroup. Give it a stable group_id and a rationale that explains the upstream shape it handles.

  3. Use the balanced phase when every generated branch is syntactically valid; use the unbalanced phase only when the original source crosses an HTML/Jinja boundary.

  4. Add forward, reverse, and wrong-template-identity tests. A demo:sample template must not receive a Science Europe rewrite.

  5. Run make transform followed by make explain-transform. Inspect the profile, rule IDs, files, and match counts. The same evidence is stored in .transform/manifest.json under rewrite_trace.

  6. Complete the parser-change checklist below and render regression before accepting the rule.

If executable source is correct but a sentence preview exposes a known broken fragment, add a named SourceFragmentRule instead. This is an audit guard, not a substitute for fixing extraction.

Translation Unit Quality Rules

Good units:

  • contain a complete sentence, heading, paragraph, list item, or branch-complete sentence group

  • allow placeholders to be reordered

  • hide machine metadata below the editable text

  • avoid raw Jinja in translator-editable blocks

Bad units:

  • split a sentence around an if or elif

  • combine unrelated independent sentences into a Cartesian explosion

  • expose branch separators such as / as something translators must preserve

  • require translators to understand Jinja scope to translate safely

Static strings inside Jinja are editable only when the parser can show that they feed rendered output. For example, a string-list initializer is exported when the same collection is later rendered through |join; an otherwise identical internal list remains machine code. Keep this data-flow check in jinja_literals.py and its marker placement in template_transform.py. Do not replace it with variable-name, template-version, or source-sentence special cases.

Changing Parsing Logic

  1. Add or update transform/export tests.

  2. Run:

    make check

  3. Inspect which profile rules fired:

    make transform
make explain-transform

  4. Build clean upstream artifacts:

    make build-upstream-artifacts

  5. Generate a fresh tree and merge existing translations exactly:

    make export-fresh-translation-tree
make merge-translation-tree

  6. Inspect generated outline.md, merge-report.json, and representative translation.md files. The outline is regenerated from the current translation blocks, so its checked counts should match the filled, updated, and preserved translations reported by the merge step.

  7. If the Markdown translation format changed, run an XLIFF export/import round trip and confirm it does not rewrite machine metadata unexpectedly.

  8. Sync translations and render a demo PDF.

  9. Confirm no translator-facing files contain raw Jinja or unnatural fragments.

Synchronization After Parser Changes

Parser changes can invalidate old translation paths and hashes. The synchronization policy is intentionally conservative:

  • exact source hash and executable-placeholder matches may fill or update translations

  • non-exact matches keep their target state and require version-specific review

  • sentence-match suggestions should be treated as review hints, not automatic truth

When a parser change is broad, refresh public repository sync/v* branches with clean scaffold artifacts and inspect what became untranslated before asking translators to continue.

Audits to Trust

  • translation_tree.py audit catches unsafe translator-facing blocks.

  • translation_tree.py sync catches missing or changed placeholders.

  • translation_tree.py audit-output catches executable Jinja, HTML wiring, asset, link, or static source drift.

  • Render previews catch runtime DSW/TDK/template integration issues.

No single audit proves translation wording is good. Downstream translation repositories should review wording, glossary, and layout against their own QA checklist.

Known zh-Hant Output Concerns

  • Some upstream templates use |dot. In zh-Hant output, leftover |dot can create duplicate punctuation after Chinese sentences. The sync output polish step removes unsafe leftover |dot behavior.

  • Upstream sentence lists may use join(" ") for English prose. The zh-Hant output polish joins variables named *sentences without that half-width gap; it also removes English trailing spaces after full-width punctuation inside those Jinja strings. Translators should keep normal full-width punctuation in each sentence.

  • CJK font behavior differs from the English template. Avoid layout changes unless PDF QA shows a structural problem.

  • Do not force Markdown list styling changes unless the rendered PDF clearly loses semantic list structure.