CLAUDE.md

1# About Quarkdown
2 
3This is the Quarkdown project. Quarkdown is a:
4- Turing-complete Markdown flavor, with a `.qd` standard file extension
5- Typesetting system, as an alternative to LaTeX, with high-quality typography and layout customization
6- Compiler, parser and renderer to:
7  - HTML
8  - PDF (via Puppeteer)
9  - Plain text
10- CLI tool
11 
12Quarkdown supports different document types, which can be set via the `.doctype {type}` function:
13- Plain documents (`plain`), suitable for notes, website, etc. Notion-like.
14- Paged documents (`paged`), suitable for books, articles, reports, etc. LaTeX-like.
15- Slides (`slides`), suitable for presentations.
16- Documentation (`docs`), suitable for technical documentation websites and wikis.
17 
18The Quarkdown flavor extends CommonMark and GFM with various features. The most notable one is *functions*:
19 
20- Inline function:
21  
22```markdown
23  Lorem ipsum .myfunction {arg1} param:{arg2} dolor sit amet.
24```
25 
26- Block function:
27```markdown
28  .myfunction {arg1} param:{arg2} 
29      arg3
30```
31 
32Quarkdown is dynamically typed, although types do live in the native Kotlin implementation of functions.
33 
34For a full function call syntax reference, see [here](docs/syntax-of-a-function-call.qd).
35 
36For any other information, see the [documentation](docs) and the [README](README.md).
37 
38# Making changes
39 
40## Guidelines
41 
42You are a senior software engineer with high expertise in handling complex codebases, compilers, and typesetting systems.
43You care about software quality, maintainability, and readability.
44Avoid repetitive code at all costs and strive for elegant solutions, abstracting common patterns into reusable components.
45Keep functions and classes small and focused on a single responsibility.
46It's possible to over-engineer when necessary to achieve high cohesion, low coupling, to anticipate future changes,
47leveraging design patterns, such as strategy and visitor (frequent in this codebase), and best practices.
48 
49Write medium-sized documentation comments for all public classes, methods, and properties,
50and also non-public ones when the logic is not straightforward. Update existing documentation when making changes to the codebase, both in code and [docs](docs), and make sure to keep it consistent with the style used in the project.
51 
52Aim for a test-driven development (TDD) approach when possible. Tests play an important role. See [Testing](#testing) below.
53 
54When creating new files, always add them to git via `git add`, and make sure to place them in the correct module and package, following the existing project structure.
55 
56## Overview
57 
58The project is structured as a multi-module Gradle project.
59 
60- To build, always run `./gradlew installDist` or `distZip` from the root folder. Never run `build`.
61- To test, run `./gradlew test`, optionally specifying a module, e.g., `:quarkdown-core:test`.
62- `./gradlew run` is acceptable. 
63 
64## Compiler
65 
66The main compiler, located in [quarkdown-core](quarkdown-core),
67along with rendering extensions, such as [quarkdown-html](quarkdown-html) and [quarkdown-plaintext](quarkdown-plaintext),
68the language server, located in [quarkdown-lsp](quarkdown-lsp),
69the CLI, located in [quarkdown-cli](quarkdown-cli),
70and other modules, is written in Kotlin with the Ktlint code style.
71Follow the code style used in the project, and make sure to run `./gradlew ktlintFormat` after making changes to ensure the code is properly formatted.
72 
73### Pipeline
74 
75The compiler is structured as a sequential pipeline (`pipeline` package).
76See `Pipeline-*` files in the [documentation](docs) to understand the different stages (`pipeline/stages` package).
77 
78### Context
79 
80`Context` is the most important interface in the compiler (`context` package). A context contains information about libraries, functions, 
81metadata, settings, and other data needed during compilation.
82 
83Each function call has a reference to the context it was parsed in.
84A context can be forked to create a child context with additional or overridden data.
85There are three forking methods, depending on the implementation, which affect the sandbox level:
86 
87- `SharedContext`: exchanges information bi-directionally. Changes made in the child context are reflected in the parent context, and vice versa,
88  allowing for full sharing of variables, functions and other declarations.
89- `ScopeContext`: like `SharedContext`, but the child context does not share new declarations (functions and variables) back to the parent context.
90  This is the behavior used within lambda blocks, such as in `.foreach`.
91 `SubdocumentContext`: no information is shared back to the main file's context, only inherited from it. This also applies to the document info (metadata, title, etc.),
92  This is the behavior used for subdocuments (see [Subdocuments](docs/subdocuments.qd)).
93 
94### Nodes
95 
96Nodes are defined in the `ast/base` or `ast/quarkdown` package, depending on whether they are from CommonMark/GFM or Quarkdown-specific.
97 
98Defining a new node involves:
99- Implementing `Node` or `NestableNode`, depending on whether the node can have children or not. Nodes should never be data classes, and `children` must always be the last property.
100- Implementing `override fun <T> accept(visitor: NodeVisitor<T>): T = visitor.visit(this)`
101- Adding a `visit` method to `NodeVisitor` and its implementations (`*Renderer`)
102- Adding lexing/parsing logic (very uncommon in the current state of the project) or, more commonly for non-GFM nodes,
103  defining a native function in the [standard library](#standard-library) that returns the node. See the `Layout` stdlib module for examples.
104 
105### Primitive functions
106 
107A *primitive function* is a stdlib function that backs a Markdown syntax element (`.heading` backs `#`, `.paragraph` backs paragraphs,
108`.link` backs `[label](url "title")` with an optional title, `.figure` backs standalone images, `.pagebreak` backs `<<<`, `.math` backs `$ ... $`, and so on).
109Because Markdown syntax and the primitive call produce the same AST node, extending the primitive via `.extend {name}` also applies to the corresponding Markdown syntax.
110 
111The full list of primitives is documented for end users at [`docs/primitives.qd`](docs/primitives.qd); keep it in sync whenever you add a new one.
112 
113The wiring lives in `TreeRewriteStage` / `AstRewriter`: after function call expansion, the rewriter walks the AST and, whenever it finds a
114`PrimitiveFunctionBackedNode` whose `backingFunctionName` has been extended, it wraps the node in a synthesized `FunctionCallNode`.
115No other wiring is needed.
116 
117Adding a new primitive-backed node involves two sides:
118 
119- **AST node** (in `quarkdown-core`, under `ast/base` or `ast/quarkdown`).
120  Make the node implement `PrimitiveFunctionBackedNode`:
121  - Set `backingFunctionName` to the primitive's Quarkdown name (matching the `@Name` on the stdlib function, if any).
122  - Implement `toFunctionCallArguments()` to materialize the node's properties as `FunctionCallArgument`s whose *names* must match the primitive's parameter names.
123  - For inline nodes (e.g. `MathSpan`, `Link`), override `isBackingCallBlock` to `false` so the inline output mapper is used during expansion.
124    Block nodes get the default `true` for free.
125  - If the node should be user-styleable, make it also implement `StylableNode` with a `style: NodeStyle` property.
126    Remember to carry it through any custom `copy(...)` helpers on the node.
127 
128- **stdlib primitive** (in [`quarkdown-stdlib/.../Primitives.kt`](quarkdown-stdlib/src/main/kotlin/com/quarkdown/stdlib/Primitives.kt)).
129  Declare a `@QFunction` returning the AST node wrapped as a value. Parameter names must match those emitted by `toFunctionCallArguments()`.
130  Include a KDoc example that uses `.extend {name}`, following the pattern already used by `.heading`, `.paragraph`, `.figure`, `.pagebreak`, `.math`, and `.link`.
131  If the node is styleable, accept `@Spread style: StyleOptions = StyleOptions.DEFAULT` and pass `style.toNodeStyle()` into the node.
132 
133- **Renderer** (for each rendering backend, e.g. [`quarkdown-html/.../QuarkdownHtmlNodeRenderer.kt`](quarkdown-html/src/main/kotlin/com/quarkdown/rendering/html/node/QuarkdownHtmlNodeRenderer.kt)).
134  If the node is styleable, the renderer's `visit(node)` must actually emit the style. In the HTML backend that means calling `style(node.style)` inside the tag builder block:
135 
136```kotlin
137  override fun visit(node: MyNode) =
138      buildTag("my-tag") {
139          +node.content
140          style(node.style)
141      }
142```
143 
144  Without this, the `style: NodeStyle` on the AST node is inert and `foreground:`, `background:`, `padding:`, etc. from `StyleOptions` will not reach the output.
145 
146Tests should cover both the primitive function itself (in an appropriate module test, e.g. `MathTest`), the extension mechanism
147(in `quarkdown-test/.../primitive/<Name>PrimitiveFunctionTest.kt`, mirroring the existing files there), and an e2e test that exercises the primitive in a real document if there are styling options involved (in `quarkdown-html/src/test/e2e/<category>/styling/`, see `paragraph/styling/` for an example).
148 
149### Function calls and scripting
150 
151The function call subsystem spans parsing, resolution, execution, and output mapping.
152 
153#### Parsing and refinement
154 
155Source code function calls (e.g. `.foo {x}::bar {y}`) are first extracted by the `FunctionCallWalker` (lexer-level)
156into `WalkedFunctionCall` structures. These are then refined by `FunctionCallRefiner` into `FunctionCallNode` AST nodes.
157 
158**Inline vs body arguments:** Inline arguments (inside `{...}`) are eagerly evaluated as expressions via `ValueFactory.safeExpression`,
159resolving nested function calls at parse time. Body arguments (indented blocks) are stored as raw `DynamicValue` strings
160for lazy evaluation by the consuming function. This distinction is critical: body arguments intentionally defer evaluation
161so that the receiving function can choose to use them as raw text, evaluate them as Markdown, or both.
162 
163**Chaining:** `FunctionCallRefiner` transforms the linked-list chain `.foo {x}::bar {y}` into a nested tree `bar(foo(x), y)`.
164 
165#### Resolution and execution
166 
167`FunctionCallNodeExpander` drives function call expansion during the `FunctionCallExpansionStage`:
168 
1691. Each `FunctionCallNode` carries the `Context` it was parsed in (`node.context`).
1702. Resolution: `node.context.resolveUnchecked(node)` finds the function by name and creates an `UncheckedFunctionCall`
171   with `context = this` (the resolving context). This context is accessible as `call.context` during execution.
1723. Execution: the function's `invoke(bindings, call)` runs and returns an `OutputValue`.
1734. Output mapping: the result is passed to a `NodeOutputValueVisitor` (block or inline), which converts it to an AST `Node`.
174 
175For `DynamicValue` results containing raw strings, the visitor calls `parseRaw`, which invokes `ValueFactory.blockMarkdown`
176or `ValueFactory.inlineMarkdown` to parse the string as Markdown with function expansion. The context used for `parseRaw`
177is the one held by the `FunctionCallNodeExpander`, which is the context passed to `ValueFactory.markdown` when the current
178parse cycle was initiated.
179 
180#### Custom functions and lambdas
181 
182Custom user-defined functions (`.function` in the `Flow` stdlib module) bridge Quarkdown scripting with the native function system:
183 
1841. **Definition:** `Flow.function()` creates a `SimpleFunction` and registers it in a `Library` prefixed with `__func__`.
185   The function's parameters are derived from the Lambda's explicit parameters.
186 
1872. **Lambda invocation (`Lambda.invokeDynamic`):**
188   - Forks from `parentContext` (the context where the Lambda was *defined*, not called).
189   - Registers lambda parameter functions via `createLambdaParametersLibrary`: each parameter becomes a zero-arg `SimpleFunction`
190     that returns `DynamicValue(argument.unwrappedValue)`.
191   - Propagates the calling context's libraries (when `callingContext` is provided), so that variable references
192     from the calling scope can be resolved within the lambda body.
193   - Calls the Lambda's `action(arguments, forkedContext)`, which typically runs `ValueFactory.eval(body, forkedContext)`.
194 
1953. **`ValueFactory.eval` and recursive resolution:** `eval` parses a raw string as an expression (via `safeExpression`),
196   evaluates it, and returns the result. When the result is a `DynamicValue` wrapping a single-line string different
197   from the input (indicating an intermediate, unresolved reference such as a lambda parameter holding `.y`), `eval`
198   recursively evaluates the result in the same context. Multi-line strings are excluded from recursion
199   as they represent raw Markdown body content intended for lazy evaluation.
200 
2014. **Variables:** `Flow.variable()` defines a variable as a function with an optional parameter, acting as both getter and setter.
202   Variable reassignment scans the context hierarchy upward to find the owning context.
203 
204#### Key files
205 
206| File                                                  | Role                                                                      |
207|-------------------------------------------------------|---------------------------------------------------------------------------|
208| `FunctionCallRefiner`                                 | Refines walked calls into `FunctionCallNode`s, handles chaining           |
209| `FunctionCallNodeExpander`                            | Expands function call nodes in the AST, maps outputs to nodes             |
210| `FunctionCallExpansionStage`                          | Pipeline stage that drives expansion                                      |
211| `Lambda`                                              | Parameterized action block with context forking and argument registration |
212| `ValueFactory.eval` / `safeExpression` / `expression` | Expression parsing and evaluation                                         |
213| `NodeOutputValueVisitor`                              | Converts function output values to AST nodes                              |
214| `Flow.kt` (`function`, `variable`)                    | Custom function and variable definition                                   |
215 
216## Standard library
217 
218The standard library is located in [quarkdown-stdlib](quarkdown-stdlib).
219It's a *native* library, meaning it's implemented in Kotlin.
220 
221The stdlib is organized into modules, each one with its own Kotlin source file,
222with a `QuarkdownModule` declaration, which exposes functions:
223 
224```kotlin
225val Layout: QuarkdownModule =
226    moduleOf(
227        ::container,
228        ::align,
229        ::center,
230        // ...
231    )
232```
233 
234The module should then be registered in [Stdlib](quarkdown-stdlib/src/main/kotlin/com/quarkdown/stdlib/Stdlib.kt).
235 
236By default, a function declared as `fun x(y: Type): ReturnType` in Kotlin
237is exposed to Quarkdown as a function call `.x y:{arg}` that returns a dynamic value.
238 
239Additionally, `@Name` can be used to rename functions and parameters.
240For instance, Quarkdown's standard uses lowercase, while Kotlin uses camelCase:
241 
242```kotlin
243@Name("myfunction")
244fun myFunction(
245    @Name("myparam") myParam: String
246): StringValue {
247    // ...
248}
249```
250 
251Native functions can also accept and return Quarkdown AST nodes directly,
252for example: `Paragraph(...).wrappedAsValue()`. `wrappedAsValue()` is available for many value types.
253 
254Functions must be documented thoroughly with KDoc comments,
255including examples of usage in Quarkdown syntax. All parameters and return types must be documented.
256 
257A `Context` parameter can be added to access context information during execution,
258by declaring it as the first parameter of the function, and marked as `@Injected`.
259This parameter is not exposed to Quarkdown and must not be documented.
260 
261## Quarkdoc
262 
263Quarkdoc is Quarkdown's documentation generation system, located in [quarkdoc](quarkdoc).
264It relies on Dokka v2 to generate documentation from KDoc comments in the Kotlin codebase,
265with custom extensions.
266 
267Quarkdoc's HTML output is bundled in the build, or can be generated separately via `./gradlew quarkdocGenerateAll`
268 
269When writing native functions, the following annotations are useful to document them properly:
270 
271- `@LikelyNamed`: indicates that a parameter is likely to be named rather than positional when called from Quarkdown.
272  For example, `.container width:{100}` instead of `.container {100}`.
273  Using `@Name` implies `@LikelyNamed`.
274 
275- `@LikelyBody`: indicates that a parameter is likely to be passed as a body block when called from Quarkdown.
276  Body parameters are always the last parameters of a function.
277 
278```markdown
279  .container width:{100}
280      This is the body content.
281```
282 
283- `@LikelyChained`: indicates that a function is likely to be used in a chained manner via the chain syntax
284  (see [Function call syntax](docs/syntax-of-a-function-call.qd#chaining-calls)).
285  For example, in `.myvar::uppercase`, `uppercase` is marked with `@LikelyChained`.
286 
287- `@OnlyForDocumentType`/`@NotForDocumentType`: indicates that a function is only available for, or not available for,
288  specific document types. An error is raised if the function is called in an incompatible document type.
289 
290## HTML front-end
291 
292The HTML rendering engine is located in [quarkdown-html](quarkdown-html).
293After the Kotlin extension renders the Quarkdown AST to HTML elements,
294the front-end TypeScript code takes care of interactivity and dynamic features,
295while SCSS files handle styling and layout.
296 
297Additionally, Puppeteer is used to generate PDF output from the HTML rendering,
298relying on the webserver, located in [quarkdown-server](quarkdown-server).
299 
300### Offline asset bundling
301 
302Rendered HTML documents are fully offline: every third-party asset (fonts, JS libraries, CSS, code highlighting, themes) is bundled into the Quarkdown installation and copied next to each generated document, instead of being fetched from a CDN at view time.
303 
304The bundling flow is centralized in [`quarkdown-html/build.gradle.kts`](quarkdown-html/build.gradle.kts), which produces the `build/install/` directory:
305 
3061. **`npmInstall`** pulls every runtime dependency declared in `quarkdown-html/package.json` (Bootstrap Icons, KaTeX, highlight.js, Mermaid, reveal.js, Paged.js, `@fontsource/*`, ...) into `quarkdown-html/node_modules/`.
3072. **`bundleHighlightJs`** pre-bundles `highlight.js/lib/common.js` into a single browser-ready IIFE via esbuild, since the npm package ships only as ES modules.
3083. **`bundleThirdParty`** copies a curated subset of `node_modules/` into `quarkdown-html/build/install/lib/<library>/`. To add a new third-party library, append a new `LibrarySpec` and add it to `package.json`.
3094. **`bundleTypeScript`** bundles and minifies the Quarkdown runtime TypeScript into `build/install/script/quarkdown.min.js` (+ source map) via esbuild.
3105. **`assembleThemes`** reshapes the `compileSass` output from `build/scss-compiled/` into the per-theme layout under `build/install/theme/` (see [Themes](#themes)).
3116. The root build's **`installLibLayout`** copies all of `build/install/` into `lib/html/` for both `installDist` and `assembleDevLib`, so a Quarkdown installation always carries the bundle alongside the JARs.
312 
313At render time, no library is read from the JAR classpath. The install layout is navigated via the `quarkdown-install-layout-navigator` module. Each post-renderer decides which libraries are active based on document type and AST attribute presence (`markCodePresence`, `markMathPresence`, `markMermaidDiagramPresence`), so unused libraries are never copied to the output.
314 
315When tests need the real bundle, the `test` task depends on `:assembleDevLib`, and `InstallLayout.get` will return it, mirroring the real installation layout.
316 
317### Themes
318 
319Quarkdown allows for a layout theme and a color theme to be selected independently, for more combination possibilities.
320 
321[scss](quarkdown-html/src/main/scss) is compiled by the `compileSass` Gradle task into `quarkdown-html/build/scss-compiled/`, then reshaped by `assembleThemes` into a per-theme directory layout under `quarkdown-html/build/install/theme/`, which ends up at `lib/html/theme/` in the installation:
322- `global.css`: global styles
323- `layout/<name>/<name>.css` (+ sibling asset folders from `<name>.json` `exports`)
324- `color/<name>/<name>.css`
325- `locale/<tag>/<tag>.css` (+ optional sibling assets, e.g. CJK fonts)
326 
327At render time, `ThemePostRendererResource` receives the `InstallLayout.Html.Themes` node and reads the active theme components from it, instead of the JAR classpath.
328 
329#### Shipping offline assets with a theme
330 
331A layout or color theme can ship sibling assets (e.g. fonts) that travel with its CSS into the offline distribution. To do so, add a JSON manifest next to the theme's `.scss` source, named after the theme (for example, `layout/beamer.json` next to `layout/beamer.scss`).
332 
333## Server
334 
335[quarkdown-server](quarkdown-server) is a Ktor-based web server that serves the HTML rendering and allows PDF generation via Puppeteer. The `/preview/<path>` endpoint, used in combination with the CLI's `--preview` and `--watch` options, serves the HTML through a double iframe buffer, allowing for live preview during editing.
336 
337## Testing
338 
339The project has high test coverage, with three types of tests:
340- Regular unit tests, located in each module's `src/test/kotlin` folder for Kotlin, and `__tests__` folders for TypeScript, 
341  which test individual components, classes, and functions in isolation.
342- Integration unit tests, located in [quarkdown-test](quarkdown-test/src/test/kotlin),
343  which test the compiler as a whole, by compiling Quarkdown source files into different output formats, mainly HTML.
344- End-to-end tests, located in [e2e](quarkdown-html/src/test/e2e), which test the HTML rendering engine in a real browser environment via Playwright,
345  ensuring HTML output, TypeScript runtime, and CSS styles work correctly together. CSS, in particular, is prone to visual issues that are hard to catch otherwise.
346  When adding or modifying an E2E test, run only the affected test file to speed up the feedback loop:
347```bash
348  cd quarkdown-html && npx playwright test path/to/test.spec.ts
349```
350 
351When making changes to the compiler or other modules, make sure to add or update tests accordingly.
352 
353### E2E test structure
354 
355Each E2E test lives in a directory under `quarkdown-html/src/test/e2e/` containing:
356- `main.qd`: the Quarkdown source document for the test.
357- `<test-name>.spec.ts`: the Playwright spec file.
358 
359The test framework (`quarkdown.ts`) provides a `suite(testDir)` factory that returns:
360- `test(name, fn, options?)`: defines a single test case. `options` supports `subpath` for subdocument navigation.
361- `testMatrix(name, docTypes, fn, options?)`: runs the same test across multiple document types (e.g. `["plain", "paged", "slides"]`),
362  creating separate test cases for each. The runner prepends `.doctype {type}` to the source automatically.
363  This is the only way to specify a document type; `test()` does not support `docType`.
364- `expect`: Playwright's `expect` for assertions.
365 
366The runner (`__util/runner.ts`) compiles the source via the CLI, navigates to the Quarkdown server, and waits for `window.isReady()`.
367Each test run gets a unique ID for parallel isolation.
368 
369Utility helpers in `__util/css.ts` provide computed style access.
370 
371## Documentation
372 
373Documentation files are located in the [docs](docs) folder, and are written in Quarkdown itself.
374 
375When making changes to the compiler or other modules, features or changes,
376make sure to also update the documentation accordingly, along with [CHANGELOG](CHANGELOG.md).
377The changelog follows the [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format,
378uses [Semantic Versioning](https://semver.org/), uses extensive description for each major change,
379with links to the corresponding documentation at `https://quarkdown.com/wiki/Page`.
380 
381When writing documentation and changelog entries, you're an expert technical writer who follows these guidelines:
382- Use American English spelling.
383- Use active voice.
384- Be concise and clear, but not at the cost of clarity. Avoid unnecessary jargon but also ambiguity.
385- Use consistent terminology. For example, always use "function call" instead of sometimes "function invocation".
386- Use a professional and friendly tone, and be as human as possible.
387  Avoid overly technical or robotic language.
388  Avoid en-dashes, em-dashes, and emojis.
389- Write for end users, not engineers. Describe what changed from the user's perspective
390  and what they can now do, rather than implementation details. Avoid internal terms or class names. Instead, describe the visible outcome:
391  what the user writes, what they see, and how it behaves.
392 
393To demo a source+output example, use functions defined in [`_Setup.qd`](docs/_setup.qd):
394- `.examplemirror` for showing both source code and rendered output side-by-side.
395  This is great for Quarkdown snippets that don't affect the overall document structure or style.
396- `.example` for showing the source code and a manual output, such as an image.
397 
398For new features not yet documented, create a new documentation file in the `docs` folder,
399using existing files as reference.
400 
401### Compiling the documentation
402 
403To compile it, run the following command from the `docs` folder via `gradlew run`:
404 
405```bash
406c main.qd --strict --allow all --clean
407```
408 
409This will generate the documentation website in `docs/output/Quarkdown-Wiki`.