parser-development

Parser Development

Use crates/biome_parser/CONTRIBUTING.md as the canonical parser guide. Read the section matching the current grammar, lexer, parse-rule, or recovery task.

Workflow

1. Inspect the language grammar and neighboring parse rules.
2. Add or adjust focused ok/ and error/ fixtures before changing recovery behavior.
3. Update the grammar when the typed tree shape changes.
4. Regenerate syntax and factory artifacts after .ungram changes.
5. Implement parser logic with explicit presence tests and bounded recovery.
6. Run the language parser's focused tests and inspect snapshots.

Grammar

Grammar files under xtask/codegen/ define typed syntax nodes and fields. Follow existing language naming:

• prefix nodes with the language name;
• name unions with Any;
• use Bogus nodes for recoverable invalid syntax;
• end list node names with List;
• represent lists as present, possibly empty nodes rather than optional fields.

After a grammar change, run:

just gen-grammar <lang>

This updates generated syntax nodes, syntax kinds, factories, macros, and language-specific mappings. Parser rules remain hand-written.

Presence Contract

A parse function returns Absent only when it has consumed no tokens. Test the first distinguishing token before calling start, bump, eat, expect, or another parser that can advance.

Use:

• expect for required tokens that should create a diagnostic when absent;
• eat for optional tokens;
• .ok() for optional nodes;
• .or_add_diagnostic(...) for required nodes;
• typed recovery for malformed nodes that should remain in the CST.

Do not use backtracking where a bounded lookahead or a more precise presence test can distinguish the syntax.

Error Recovery

Recovery must preserve following valid syntax and produce a bogus node permitted by the grammar at that position.

A recovery set normally includes the nearest relevant:

• list separator;
• list or block terminator;
• statement boundary;
• token that starts the next valid construct.

Do not copy a recovery set from an unrelated grammar position. Verify which tokens the caller expects after the failed parse.

Lists

Use the parser infrastructure's list traits instead of open-coded loops when their contract matches the grammar.

For separated lists, verify:

• the parser recognizes the enclosing terminator;
• separator handling agrees with trailing-separator grammar;
• a malformed element makes progress or stops;
• recovery cannot consume the enclosing terminator;
• an empty list still produces the required list node.

Lexer and Token Source

Follow the current Lexer and buffered token-source traits from biome_parser; do not copy an old trait implementation from a skill or issue.

• Use checked byte and character accessors supplied by the lexer infrastructure.
• Keep lexing context explicit where the same bytes have context-dependent meaning.
• Ensure every lexer path advances or returns EOF.
• Use &str lexer tests for valid multibyte characters and character-boundary/index behavior. Test malformed byte sequences only through an existing byte-oriented API.

Testing

Load testing-codegen for parser quick tests and snapshot mechanics.

• Use the crate's quick_test to inspect a CST while developing.
• Add persistent fixtures under the parser crate's current ok/ and error/ directories.
• A parser bug fix needs the smallest fixture that failed before the change.
• Recovery changes need malformed input followed by valid syntax to prove parsing resumes correctly.

Review Checklist

• Absent paths consume no input.
• Required nodes and tokens produce useful diagnostics.
• Recovery emits a grammar-valid bogus node.
• Every loop either advances or exits.
• List recovery stops before the enclosing boundary.
• The CST retains all source text, including malformed input.
• Grammar changes include generated syntax and factory artifacts.
• Valid and malformed fixtures exercise the changed path.

References

• Parser guide: crates/biome_parser/CONTRIBUTING.md
• Grammars: xtask/codegen/*.ungram
• Parser infrastructure: crates/biome_parser/src/
• Language implementations: crates/biome_*_parser/src/