diff --git a/IMPLEMENTATION_PLAN.md b/IMPLEMENTATION_PLAN.md index 08a54f7..16f7279 100644 --- a/IMPLEMENTATION_PLAN.md +++ b/IMPLEMENTATION_PLAN.md @@ -42,14 +42,26 @@ bulldoze priorities. - [x] Update SPEC.md §3 to enumerate `Arg.IsCwdAttribution`; cross-tfm note on `VerbChain.Joined` -### 3. BashLexer (SPEC §5) - -- [ ] Token kinds: WORD, QUOTED_STRING, OPERATOR, WHITESPACE, - CONTINUATION -- [ ] Quote handling (single literal, double with `\"`, `\\`, `\$`) -- [ ] Escape handling outside quotes -- [ ] Operator boundaries (no whitespace required) -- [ ] Heavy unit tests on tokenization +### 3. BashLexer + opaque-region scanner — PR 2, in progress + +- [x] `Internal/Lexing/OpaqueRegionScanner.cs` — shared, grammar-agnostic; + `Scan` for `(`/`)` style + `ScanSymmetric` for backtick; quote-aware, + escape-aware, nesting-aware +- [x] `Internal/Bash/Lexing/{BashLexer,BashToken,BashTokenKind}.cs` +- [x] Token kinds: Word, QuotedString, Operator, Whitespace, Continuation, + OpaqueSubstitution, UnparseableSentinel +- [x] Quote handling (single literal, double with `\"`, `\\`, `\$`, + `\` + newline) +- [x] Escape handling outside quotes +- [x] Operator boundaries (no whitespace required); `<<-` heredoc variant +- [x] `$(…)` and backticks → `OpaqueSubstitution` (locked interpretation #2) +- [x] `$((expr))` and `${var//pat/repl}` → `UnparseableSentinel` +- [x] Heredoc body skip per SPEC §4 +- [x] 78 lexer + scanner unit tests (combined with PR 1's 18 → 96/96 + passing) +- [x] SPEC §1 / §5 / §11 updated for token kinds + non-goal additions +- [x] OpenSpec change `v0.1-locked-interpretations` tasks.md updated + (Phase 2 marked [x]) ### 4. Verb tables (SPEC §6, data only) diff --git a/SPEC.md b/SPEC.md index 697093c..8ed8c0c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -48,6 +48,12 @@ command can consume it. - Variable expansion. We mark dynamic tokens, never resolve them. - Function definitions, here-docs body extraction, complex parameter expansion (`${var//pattern/replacement}`), arithmetic expansion. +- Command-substitution evaluation. `$(cmd)` and backtick `` `cmd` `` are + recognized at the lex level and collapsed into a single + `Kind=DynamicSkip, IsPath=false` arg per locked interpretation #2 (see + `openspec/changes/archive/.../v0.1-locked-interpretations`). The + surrounding clause stays parseable so hard-deny rules still fire on + visible parts. - Performance tuning beyond "fast enough to invoke per shell call without noticeable latency" (~1ms per typical input). @@ -359,14 +365,28 @@ quoted_string := single-quoted | double-quoted The lexer produces tokens consumed by the parser. Token kinds: - **WORD** — sequence of non-whitespace, non-operator, non-quote chars. - Example: `git`, `/etc/foo`, `--force`, `~/path`, `$VAR`. + Example: `git`, `/etc/foo`, `--force`, `~/path`, `$VAR`. Simple + parameter expansion `${VAR}` (no `//` slash) is absorbed into a Word + token; the resolver in §8 decides `Kind`. - **QUOTED_STRING** — single- or double-quoted string. The lexer strips the quote delimiters from the token value. Example: `"hello world"` becomes the token value `hello world`. - **OPERATOR** — `&&`, `||`, `;`, `|`, `>`, `>>`, `<`, `2>`, `2>>`, - `(`, `)`, `<<`. -- **WHITESPACE** — one or more spaces or tabs. Discarded after splitting. + `(`, `)`, `<<`, `<<-`. +- **WHITESPACE** — one or more spaces or tabs (or newlines outside a + heredoc body). Discarded after splitting. - **CONTINUATION** — `\` + `\n`. Treated as whitespace. +- **OPAQUE_SUBSTITUTION** — `$(cmd)` or backtick `` `cmd` ``. The full + substitution slice (including delimiters) becomes a single token. + Boundary tracking handles nested same-kind regions, nested quotes, + and `\X` escapes via a shared opaque-region scanner. The parser + consumes this token as `Arg{ Kind=DynamicSkip, IsPath=false, + Resolved=null }` per locked interpretation #2. +- **UNPARSEABLE_SENTINEL** — `$((expr))` arithmetic expansion or + `${var//pat/repl}` complex parameter expansion. The lexer skips past + the matching close (`))` or `}` respectively) and emits a sentinel + whose reason names the rejected construct. The parser consumes this + token by setting outer `ParsedCommand.IsUnparseable = true` (see §11). ### Quote handling @@ -759,7 +779,11 @@ Conditions that produce `IsUnparseable = true`: `then`, `fi`, `case`, `esac`). - Function definitions (`name() { ... }`). - Process substitution (`<(cmd)`, `>(cmd)`). -- Recursion depth exceeded on `bash -c` chains. +- Arithmetic expansion `$((expr))` (per §1 non-goal; lexer emits an + UNPARSEABLE_SENTINEL token; parser sets the outer flag). +- Complex parameter expansion `${var//pat/repl}` (per §1 non-goal; same + mechanism). +- Recursion depth exceeded on `bash -c` chains (>5 levels). Consumers (e.g. Netclaw's gate evaluator) route unparseable commands to a safe-fail path (prompt the user; offer only Once and Deny — no persistent diff --git a/openspec/changes/v0.1-locked-interpretations/tasks.md b/openspec/changes/v0.1-locked-interpretations/tasks.md index 7650fd8..d6dc342 100644 --- a/openspec/changes/v0.1-locked-interpretations/tasks.md +++ b/openspec/changes/v0.1-locked-interpretations/tasks.md @@ -20,36 +20,46 @@ the SPEC.md sections that get updated alongside the implementation. - [x] 1.6 Add both projects to `ShellSyntaxTree.slnx` - [x] 1.7 Bootstrap OpenSpec scaffolding (`openspec/`, change directories, this proposal/design/tasks/specs delta) -- [ ] 1.8 Update `SPEC.md` §2/§3 to enumerate `Arg.IsCwdAttribution`; +- [x] 1.8 Update `SPEC.md` §3 to enumerate `Arg.IsCwdAttribution`; annotate `VerbChain.Joined` to use `string.Join(" ", Tokens)` for cross-tfm compatibility (was `string.Join(' ', …)`, char overload missing on netstandard2.0) -- [ ] 1.9 Update `IMPLEMENTATION_PLAN.md` — mark PR 1 in-progress; +- [x] 1.9 Update `IMPLEMENTATION_PLAN.md` — mark PR 1 in-progress; reference this OpenSpec change -- [ ] 1.10 Update `TOOLING.md` to list installed OpenSpec skills under +- [x] 1.10 Update `TOOLING.md` to list installed OpenSpec skills under Helper Skills -- [ ] 1.11 Run `pwsh ./scripts/Add-FileHeaders.ps1`; verify with `-Verify` -- [ ] 1.12 `dotnet build -c Release` clean; `dotnet test -c Release` - all-green -- [ ] 1.13 Commit (signed) and push `pr1-bootstrap`; open PR with - `gh pr merge --auto --squash` against `dev` -- [ ] 1.14 On merge: archive this OpenSpec change to +- [x] 1.11 Run `pwsh ./scripts/Add-FileHeaders.ps1`; verify with `-Verify` +- [x] 1.12 `dotnet build -c Release` clean; `dotnet test -c Release` + all-green (18/18 tests passing) +- [x] 1.13 Commit (signed) and push `pr1-bootstrap`; opened PR #4 with + `gh pr merge --auto --squash` against `dev`. Auto-merged at + 2026-05-10T17:55:09Z (Linux 27s, Windows 1m19s). +- [ ] 1.14 Archive this OpenSpec change to `openspec/changes/archive/2026-05-10-v0.1-locked-interpretations/` + (deferred: not all interpretations have landed yet — change spans + PRs 1–6. Archive when PR 6 merges.) ## 2. PR 2 — Lexer + opaque-region scanner (interpretation #2) -- [ ] 2.1 Implement `Internal/Lexing/OpaqueRegionScanner.cs` — finds - balanced delimiters with quote-aware nesting; configurable open/close -- [ ] 2.2 Implement `Internal/Bash/Lexing/BashLexer.cs` per SPEC §5 -- [ ] 2.3 Recognize `$(...)` and `` ` `` regions via the scanner; emit - `OpaqueSubstitution` token -- [ ] 2.4 Recognize `$((` and `${var//` openers; emit `IsUnparseable` - sentinel -- [ ] 2.5 Update `SPEC.md` §1 (note that command substitution is marked - DynamicSkip), §5 (lex rules for opaque regions), §11 (add arithmetic - and complex-param-expansion to IsUnparseable triggers) -- [ ] 2.6 Open OpenSpec change `bash-lexer-opaque-regions` capturing the - §1/§5/§11 deltas in their final form +- [x] 2.1 Implement `Internal/Lexing/OpaqueRegionScanner.cs` — finds + balanced delimiters with quote-aware nesting; `Scan` for `(`/`)` + style and `ScanSymmetric` for backtick style; honors `\X` escapes, + single-quote literal preservation, double-quote escape table +- [x] 2.2 Implement `Internal/Bash/Lexing/BashLexer.cs` per SPEC §5; + single entry point `Tokenize(string) -> IReadOnlyList` +- [x] 2.3 Recognize `$(...)` and backtick `` `...` `` regions via the + scanner; emit `OpaqueSubstitution` token +- [x] 2.4 Recognize `$((` and `${var//` openers; emit + `UnparseableSentinel` token with reason naming the construct +- [x] 2.5 Update `SPEC.md` §1 non-goals (command substitution → DynamicSkip), + §5 (add OPAQUE_SUBSTITUTION + UNPARSEABLE_SENTINEL token kinds, add + `<<-` operator, simple `${VAR}` absorbed into Word, newlines outside + heredoc treated as Whitespace), §11 (add arithmetic + + complex-param-expansion to IsUnparseable conditions list) +- [ ] 2.6 Tests: 16 OpaqueRegionScanner tests + 62 BashLexer tests, all + green. Combined with PR 1's 18: 96/96 passing. +- [ ] 2.7 Run header script; verify; commit (signed) and push `pr2-lexer`; + open PR with `gh pr merge --auto --squash` ## 3. PR 3 — Verb tables + parser core diff --git a/src/ShellSyntaxTree/Internal/Bash/Lexing/BashLexer.cs b/src/ShellSyntaxTree/Internal/Bash/Lexing/BashLexer.cs new file mode 100644 index 0000000..2b90b47 --- /dev/null +++ b/src/ShellSyntaxTree/Internal/Bash/Lexing/BashLexer.cs @@ -0,0 +1,765 @@ +// ----------------------------------------------------------------------- +// +// Copyright (C) 2026 - 2026 Aaron Stannard +// +// ----------------------------------------------------------------------- +using System; +using System.Collections.Generic; +using System.Text; +using ShellSyntaxTree.Internal.Lexing; + +namespace ShellSyntaxTree.Internal.Bash.Lexing; + +/// +/// Tokenizer for the bash subset described in SPEC §4 / §5. The output +/// is a flat of +/// — the parser is responsible for grouping into clauses, verb chains, +/// args, and redirects. +/// +/// Design notes: +/// +/// The lexer never expands variables. $VAR and +/// ${VAR} stay literal inside a +/// token; the resolver in PR 4 decides what to do with them. +/// Opaque regions ($(…) and backtick `…`) are +/// consumed by and emitted as +/// a single token +/// per the v0.1 locked interpretation. +/// Constructs SPEC §1 calls non-goals — arithmetic expansion +/// $((…)) and complex parameter expansion +/// ${var//pat/repl} — emit a +/// . The parser +/// lifts that sentinel into ParsedCommand.IsUnparseable. +/// Heredoc bodies are dropped per SPEC §4. Only the +/// <</<<- operator and the delimiter +/// word make it into the token stream. +/// +/// +internal static class BashLexer +{ + /// + /// Tokenize per SPEC §5. Never throws on + /// non-null input; malformed regions surface as + /// tokens that the + /// parser must lift into ParsedCommand.IsUnparseable. + /// + internal static IReadOnlyList Tokenize(string input) + { + if (input is null) + { + throw new ArgumentNullException(nameof(input)); + } + + if (input.Length == 0) + { + return Array.Empty(); + } + + var tokens = new List(); + var src = input.AsSpan(); + var i = 0; + + while (i < src.Length) + { + var c = src[i]; + + // ---- whitespace ---- + if (c == ' ' || c == '\t') + { + var start = i; + while (i < src.Length && (src[i] == ' ' || src[i] == '\t')) + { + i++; + } + + tokens.Add(new BashToken( + BashTokenKind.Whitespace, "", null, start, i - start, null)); + continue; + } + + // ---- newline (treated as a sequence terminator analogous to ';') ---- + // Per SPEC §4, top-level newlines separate clauses just like ';'. + // Emitting them as a Whitespace token preserves source fidelity for + // the parser without requiring a dedicated token kind. + if (c == '\n' || c == '\r') + { + var start = i; + while (i < src.Length && (src[i] == '\n' || src[i] == '\r')) + { + i++; + } + + tokens.Add(new BashToken( + BashTokenKind.Whitespace, "", null, start, i - start, null)); + continue; + } + + // ---- backslash + newline = continuation (treat as whitespace) ---- + if (c == '\\' && i + 1 < src.Length && (src[i + 1] == '\n' || src[i + 1] == '\r')) + { + var start = i; + i += 2; + // Optionally consume a paired \r\n. + if (i - start == 2 && start + 1 < src.Length && src[start + 1] == '\r' + && i < src.Length && src[i] == '\n') + { + i++; + } + + tokens.Add(new BashToken( + BashTokenKind.Continuation, "", null, start, i - start, null)); + continue; + } + + // ---- operators (longer-match first) ---- + // Order matters: `&&` before `&`, `||` before `|`, `>>` before `>`, + // `2>>` before `2>`, `<<-` before `<<`, `<<` before `<`. We don't + // recognize a bare `&` in v0.1 (no background-job support; SPEC §1). + if (TryReadOperator(src, i, out var opLen, out var opText)) + { + var operatorTok = new BashToken( + BashTokenKind.Operator, "", opText, i, opLen, null); + tokens.Add(operatorTok); + i += opLen; + + // Heredoc handling: when we just emitted `<<` or `<<-`, the + // *next* word is the delimiter and the body that follows the + // first newline must be skipped per SPEC §4. + if (opText == "<<" || opText == "<<-") + { + i = ConsumeHeredoc(src, i, opText, tokens); + } + + continue; + } + + // ---- quoted strings ---- + if (c == '\'') + { + i = ReadSingleQuoted(src, i, tokens); + continue; + } + + if (c == '"') + { + i = ReadDoubleQuoted(src, i, tokens); + continue; + } + + // ---- $( ... ) command substitution / $((expr)) / ${var//...} ---- + if (c == '$' && i + 1 < src.Length) + { + var next = src[i + 1]; + if (next == '(') + { + // $(( -> arithmetic, unparseable. Detect before $(. + if (i + 2 < src.Length && src[i + 2] == '(') + { + i = ConsumeArithmetic(src, i, tokens); + continue; + } + + i = ConsumeCommandSubstitution(src, i, tokens); + continue; + } + + if (next == '{') + { + if (TryConsumeComplexParamExpansion(src, i, tokens, out var afterBrace)) + { + i = afterBrace; + continue; + } + + // Simple ${VAR}: fall through to word reader so the whole + // thing (and any adjacent text) becomes one Word token. + } + } + + // ---- backtick command substitution ---- + if (c == '`') + { + i = ConsumeBacktickSubstitution(src, i, tokens); + continue; + } + + // ---- word ---- + i = ReadWord(src, i, tokens); + } + + return tokens; + } + + // ---------------------------------------------------------------- operators + + private static bool TryReadOperator( + ReadOnlySpan src, int i, out int length, out string? text) + { + // Multi-char operators first. + if (i + 1 < src.Length) + { + var c0 = src[i]; + var c1 = src[i + 1]; + if (c0 == '&' && c1 == '&') { length = 2; text = "&&"; return true; } + if (c0 == '|' && c1 == '|') { length = 2; text = "||"; return true; } + if (c0 == '>' && c1 == '>') { length = 2; text = ">>"; return true; } + if (c0 == '<' && c1 == '<') + { + if (i + 2 < src.Length && src[i + 2] == '-') + { + length = 3; text = "<<-"; return true; + } + + length = 2; text = "<<"; return true; + } + + if (c0 == '2' && c1 == '>') + { + if (i + 2 < src.Length && src[i + 2] == '>') + { + length = 3; text = "2>>"; return true; + } + + length = 2; text = "2>"; return true; + } + } + + // Single-char operators. + switch (src[i]) + { + case ';': length = 1; text = ";"; return true; + case '|': length = 1; text = "|"; return true; + case '>': length = 1; text = ">"; return true; + case '<': length = 1; text = "<"; return true; + case '(': length = 1; text = "("; return true; + case ')': length = 1; text = ")"; return true; + default: + length = 0; text = null; return false; + } + } + + // ---------------------------------------------------------------- quoted + + private static int ReadSingleQuoted( + ReadOnlySpan src, int start, List tokens) + { + // Single quotes preserve bytes literally per SPEC §5 — no escape + // processing, no variable expansion. Find the next ' and we're done. + var i = start + 1; + while (i < src.Length && src[i] != '\'') + { + i++; + } + + if (i >= src.Length) + { + // Unbalanced — emit a sentinel covering the rest of the input + // and stop. The parser will lift this to ParsedCommand.IsUnparseable. + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + src.Slice(start).ToString(), + null, + start, + src.Length - start, + $"unbalanced quote at position {start}")); + return src.Length; + } + + // src[start] = opening ' + // src[i] = closing ' + // Strip the delimiters from the value per SPEC §5. + var inner = src.Slice(start + 1, i - start - 1).ToString(); + tokens.Add(new BashToken( + BashTokenKind.QuotedString, inner, null, start, (i - start) + 1, null)); + return i + 1; + } + + private static int ReadDoubleQuoted( + ReadOnlySpan src, int start, List tokens) + { + // Double quotes preserve whitespace but recognize \", \\, \$, and + // \\+newline as escape sequences (SPEC §5). Other backslashes are + // preserved literally. $VAR / ${VAR} are *not* expanded — kept literal. + var sb = new StringBuilder(); + var i = start + 1; + while (i < src.Length) + { + var c = src[i]; + if (c == '"') + { + tokens.Add(new BashToken( + BashTokenKind.QuotedString, sb.ToString(), null, + start, (i - start) + 1, null)); + return i + 1; + } + + if (c == '\\' && i + 1 < src.Length) + { + var n = src[i + 1]; + if (n == '"' || n == '\\' || n == '$' || n == '`') + { + sb.Append(n); + i += 2; + continue; + } + + if (n == '\n') + { + // Line continuation inside double quotes: drop both. + i += 2; + continue; + } + + // Other backslashes preserved literally per SPEC §5. + sb.Append(c); + i++; + continue; + } + + sb.Append(c); + i++; + } + + // Unbalanced double quote. + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + src.Slice(start).ToString(), + null, + start, + src.Length - start, + $"unbalanced quote at position {start}")); + return src.Length; + } + + // ---------------------------------------------------------------- substitutions + + private static int ConsumeCommandSubstitution( + ReadOnlySpan src, int start, List tokens) + { + // src[start] = '$', src[start+1] = '(' + var openParen = start + 1; + var scan = OpaqueRegionScanner.Scan(src, openParen, '(', ')'); + if (!scan.Closed) + { + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + src.Slice(start).ToString(), + null, + start, + src.Length - start, + "unbalanced '$(' command substitution")); + return src.Length; + } + + // EndIndex is the closing ')' (inclusive). The opaque region runs + // from start ($) through EndIndex (closing paren) inclusive. + var length = scan.EndIndex - start + 1; + tokens.Add(new BashToken( + BashTokenKind.OpaqueSubstitution, + src.Slice(start, length).ToString(), + null, + start, + length, + null)); + return start + length; + } + + private static int ConsumeBacktickSubstitution( + ReadOnlySpan src, int start, List tokens) + { + var scan = OpaqueRegionScanner.ScanSymmetric(src, start, '`'); + if (!scan.Closed) + { + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + src.Slice(start).ToString(), + null, + start, + src.Length - start, + "unbalanced backtick command substitution")); + return src.Length; + } + + var length = scan.EndIndex - start + 1; + tokens.Add(new BashToken( + BashTokenKind.OpaqueSubstitution, + src.Slice(start, length).ToString(), + null, + start, + length, + null)); + return start + length; + } + + private static int ConsumeArithmetic( + ReadOnlySpan src, int start, List tokens) + { + // src[start] = '$', src[start+1] = '(', src[start+2] = '(' — and we + // need to skip past matching '))'. Use the opaque scanner anchored + // at the outer '(' so we count the correct depth: the inner '(' is + // its own sub-region for the scanner. + var outerParen = start + 1; + var scan = OpaqueRegionScanner.Scan(src, outerParen, '(', ')'); + // We want to land *one* paren past EndIndex (the inner ))) close). + // OpaqueRegionScanner's depth counting closes when the *outer* paren + // balances; for $(( the inner '(' bumps depth back to 1 when it sees + // the first ')', then to 0 when it sees the second ')'. So EndIndex + // already points at the *second* ')' — i.e. the bash arithmetic close. + int endInclusive; + if (scan.Closed) + { + endInclusive = scan.EndIndex; + } + else + { + // Unterminated: still consume the rest so the lexer can move on, + // but mark the token's reason accordingly. + endInclusive = src.Length - 1; + } + + var length = endInclusive - start + 1; + var reason = scan.Closed + ? "arithmetic expansion '$((…))' not supported in v0.1" + : "unterminated arithmetic expansion '$((…))' (also not supported in v0.1)"; + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + src.Slice(start, length).ToString(), + null, + start, + length, + reason)); + return start + length; + } + + private static bool TryConsumeComplexParamExpansion( + ReadOnlySpan src, int start, List tokens, out int afterBrace) + { + // src[start] = '$', src[start+1] = '{'. We need to find the matching + // '}' and decide: simple ${VAR} -> false (let word reader take it); + // ${...//...} or any other "complex" form -> emit UnparseableSentinel. + // + // For v0.1 we treat the presence of a slash inside the braces as the + // single signal of "complex param expansion" (per the locked + // interpretation #2 in the OpenSpec change). Other operators inside + // ${...} (like ${X-default}, ${X#prefix}) fall through to the word + // reader; a future PR can tighten this if needed. + var openBrace = start + 1; + var scan = OpaqueRegionScanner.Scan(src, openBrace, '{', '}'); + if (!scan.Closed) + { + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + src.Slice(start).ToString(), + null, + start, + src.Length - start, + "unbalanced '${' parameter expansion")); + afterBrace = src.Length; + return true; + } + + var endInclusive = scan.EndIndex; + var bodyStart = openBrace + 1; + var bodyEnd = endInclusive; // exclusive of '}' + var hasSlash = false; + for (var k = bodyStart; k < bodyEnd; k++) + { + if (src[k] == '/') + { + hasSlash = true; + break; + } + } + + if (!hasSlash) + { + // Simple ${VAR} (or ${X-default} etc.). Caller will fall through + // to the word reader and absorb it as part of a Word token. + afterBrace = -1; + return false; + } + + var length = endInclusive - start + 1; + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + src.Slice(start, length).ToString(), + null, + start, + length, + "complex parameter expansion '${var//pat/repl}' not supported in v0.1")); + afterBrace = start + length; + return true; + } + + // ---------------------------------------------------------------- words + + private static int ReadWord( + ReadOnlySpan src, int start, List tokens) + { + // A word continues until we hit whitespace, an operator boundary, + // a newline, a quote, a backtick, or the start of an opaque region + // ($( or $(( or ${...//...} that we'd lift to a sentinel). We + // honor `\X` escapes by consuming both the backslash and X + // verbatim — the resulting Word value contains X literally, + // which preserves SPEC §5's "echo \$HOME → token whose value is + // $HOME" behavior. (We can't distinguish escaped-$ from + // literal-$ in the value alone; the parser doesn't need to — + // unescaped $VAR/${VAR} stayed in the source verbatim, escaped + // ones came out the same way after escape collapse, and the + // resolver classifies based on the original source positions if + // necessary. SPEC §5 explicitly says `echo \$HOME` produces a + // Literal token.) + var sb = new StringBuilder(); + var i = start; + while (i < src.Length) + { + var c = src[i]; + + // Stop conditions. + if (c == ' ' || c == '\t' || c == '\n' || c == '\r') break; + if (c == '\'' || c == '"' || c == '`') break; + if (IsOperatorStart(src, i)) break; + + // Backslash escapes the next character (outside quotes). + if (c == '\\') + { + if (i + 1 >= src.Length) + { + // Trailing lone backslash — preserve it as literal. + sb.Append('\\'); + i++; + break; + } + + var n = src[i + 1]; + if (n == '\n' || n == '\r') + { + // Continuation: terminate the current word; the outer + // loop will pick up the continuation token. + break; + } + + sb.Append(n); + i += 2; + continue; + } + + // $( and ${...//...} terminate the word — they emit their own + // tokens. Simple ${VAR}, $VAR, $$ etc. are absorbed. + if (c == '$' && i + 1 < src.Length) + { + var next = src[i + 1]; + if (next == '(') break; + if (next == '{') + { + // Decide complex vs simple by looking for a '/' in the body. + var openBrace = i + 1; + var scan = OpaqueRegionScanner.Scan(src, openBrace, '{', '}'); + if (!scan.Closed) break; // let outer loop emit the sentinel + + var bodyHasSlash = false; + for (var k = openBrace + 1; k < scan.EndIndex; k++) + { + if (src[k] == '/') { bodyHasSlash = true; break; } + } + + if (bodyHasSlash) break; + + // Simple form — absorb whole ${...} verbatim. + // StringBuilder.Append(ReadOnlySpan) is net6+ + // only; spell out the loop for netstandard2.0 parity. + var braceLen = scan.EndIndex - i + 1; + for (var k = 0; k < braceLen; k++) + { + sb.Append(src[i + k]); + } + + i += braceLen; + continue; + } + } + + sb.Append(c); + i++; + } + + if (sb.Length == 0) + { + // Defensive: caller should not invoke ReadWord on a position + // that produces no chars (would loop forever). Advance one + // char to make progress; this should not trigger in practice + // because the dispatch in Tokenize covers every printable + // case. + return start + 1; + } + + tokens.Add(new BashToken( + BashTokenKind.Word, sb.ToString(), null, start, i - start, null)); + return i; + } + + private static bool IsOperatorStart(ReadOnlySpan src, int i) + { + var c = src[i]; + switch (c) + { + case ';': + case '|': + case '<': + case '>': + case '(': + case ')': + return true; + case '&': + // Only `&&` is an operator in v0.1; bare `&` is unsupported + // background-job syntax. Treat `&` not followed by `&` as a + // word char to avoid silently splitting; consumers will see + // it in the Raw value. + return i + 1 < src.Length && src[i + 1] == '&'; + case '2': + // `2>` and `2>>` start with '2' — only treat them as operator + // starts when the immediate next char is '>'. + return i + 1 < src.Length && src[i + 1] == '>'; + default: + return false; + } + } + + // ---------------------------------------------------------------- heredoc + + private static int ConsumeHeredoc( + ReadOnlySpan src, int i, string opText, List tokens) + { + // Skip any whitespace between `<<` and the delimiter word. + while (i < src.Length && (src[i] == ' ' || src[i] == '\t')) + { + tokens.Add(new BashToken( + BashTokenKind.Whitespace, "", null, i, 1, null)); + i++; + } + + if (i >= src.Length) + { + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + "", + null, + i, + 0, + "heredoc operator '" + opText + "' missing delimiter")); + return i; + } + + // Read the delimiter as a word (no quote/escape unwrapping handling + // beyond what ReadWord does — bash supports `<<'EOF'` for + // unexpanded bodies, but we skip the body either way so the value + // doesn't matter). Capture the delimiter text from the freshly + // appended token. + var beforeDelim = tokens.Count; + var afterDelim = ReadWord(src, i, tokens); + if (tokens.Count == beforeDelim) + { + // ReadWord didn't produce a token (delimiter started with an + // operator/quote we don't unwrap here). Treat as malformed. + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + "", + null, + i, + 0, + "heredoc operator '" + opText + "' missing delimiter")); + return afterDelim; + } + + var delimToken = tokens[tokens.Count - 1]; + var delim = delimToken.Value; + + // Skip the heredoc body: from the next newline to the line that + // contains only `delim` (or, for `<<-`, optional leading tabs + + // delim). On unterminated body, emit a sentinel and stop. + var j = afterDelim; + // Find the first newline that opens the body. + while (j < src.Length && src[j] != '\n') + { + j++; + } + + if (j >= src.Length) + { + // No newline at all after the delimiter — unterminated heredoc. + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + "", + null, + afterDelim, + src.Length - afterDelim, + $"heredoc body not terminated (delimiter '{delim}' not found)")); + return src.Length; + } + + j++; // step past the opening newline; body now starts at j. + var bodyStart = j; + + var stripTabs = opText == "<<-"; + while (j <= src.Length) + { + // Read the next line: from j to the next '\n' or end-of-input. + var lineStart = j; + while (j < src.Length && src[j] != '\n') + { + j++; + } + + var lineEnd = j; // exclusive + + // For <<-, optional leading tabs are stripped before comparing. + var compareStart = lineStart; + if (stripTabs) + { + while (compareStart < lineEnd && src[compareStart] == '\t') + { + compareStart++; + } + } + + var lineSlice = src.Slice(compareStart, lineEnd - compareStart); + if (lineSlice.SequenceEqual(delim.AsSpan())) + { + // Found terminator. Skip the body silently — no token is + // emitted for the body itself — but emit a single + // Whitespace token covering the terminator's trailing + // newline (if any) so the parser still sees a clause + // boundary between the heredoc-using clause and whatever + // follows. Without this, `cmd <= src.Length) + { + // End of input without terminator. + tokens.Add(new BashToken( + BashTokenKind.UnparseableSentinel, + "", + null, + bodyStart, + src.Length - bodyStart, + $"heredoc body not terminated (delimiter '{delim}' not found)")); + return src.Length; + } + + j++; // step over the '\n'. + } + + return j; + } +} diff --git a/src/ShellSyntaxTree/Internal/Bash/Lexing/BashToken.cs b/src/ShellSyntaxTree/Internal/Bash/Lexing/BashToken.cs new file mode 100644 index 0000000..1bd000d --- /dev/null +++ b/src/ShellSyntaxTree/Internal/Bash/Lexing/BashToken.cs @@ -0,0 +1,39 @@ +// ----------------------------------------------------------------------- +// +// Copyright (C) 2026 - 2026 Aaron Stannard +// +// ----------------------------------------------------------------------- +namespace ShellSyntaxTree.Internal.Bash.Lexing; + +/// +/// One token emitted by . +/// +/// Token class — see . +/// +/// The token's logical content, with quote delimiters stripped for +/// per SPEC §5. For +/// this is the literal text after escape +/// processing. For this is +/// the full source slice including delimiters (e.g. $(echo foo)). +/// For this is the source +/// slice that triggered the sentinel. Empty string for kinds where it +/// carries no information (Operator, Whitespace, Continuation). +/// +/// For , +/// the literal operator text ("&&", "<<-", +/// etc.). Null for every other kind. +/// 0-based index into the original input where +/// this token starts. +/// Length of the original-input slice this +/// token covers, in chars. +/// For +/// , the human-readable +/// reason (e.g. "unbalanced quote at position 4"). Null for every +/// other kind. +internal readonly record struct BashToken( + BashTokenKind Kind, + string Value, + string? OperatorText, + int SourceStart, + int SourceLength, + string? UnparseableReason); diff --git a/src/ShellSyntaxTree/Internal/Bash/Lexing/BashTokenKind.cs b/src/ShellSyntaxTree/Internal/Bash/Lexing/BashTokenKind.cs new file mode 100644 index 0000000..3e13ecc --- /dev/null +++ b/src/ShellSyntaxTree/Internal/Bash/Lexing/BashTokenKind.cs @@ -0,0 +1,55 @@ +// ----------------------------------------------------------------------- +// +// Copyright (C) 2026 - 2026 Aaron Stannard +// +// ----------------------------------------------------------------------- +namespace ShellSyntaxTree.Internal.Bash.Lexing; + +/// +/// Token classes emitted by . See SPEC §5 for the +/// canonical definitions; the enum mirrors that table 1:1, plus two +/// additional kinds the parser uses to honor the locked interpretation +/// for opaque regions and for unparseable bash constructs. +/// +internal enum BashTokenKind +{ + /// Non-whitespace, non-operator, non-quote chars. Examples: + /// git, /etc/foo, --force, ~/path, + /// $VAR, ${VAR}. + Word, + + /// Single- or double-quoted string. The lexer strips the + /// outer quote delimiters from . + /// SPEC §5. + QuotedString, + + /// One of the bash operators recognized in v0.1: &&, + /// ||, ;, |, >, >>, + /// <, 2>, 2>>, (, ), + /// <<, <<-. The literal text is in + /// . + Operator, + + /// Run of spaces/tabs. Emitted for source-fidelity; the + /// parser typically filters these out. + Whitespace, + + /// Backslash + newline line continuation. Treated as + /// whitespace by the parser. SPEC §5. + Continuation, + + /// An opaque region — $(…) or backtick-quoted + /// `…`. The parser consumes one of these as a single + /// Arg{ Kind = DynamicSkip, IsPath = false } per the v0.1 + /// locked interpretation. contains the + /// region's full source slice (delimiters included). + OpaqueSubstitution, + + /// An unparseable construct that should set + /// ParsedCommand.IsUnparseable = true on the outer parsed + /// command — e.g. $((…)) arithmetic expansion, complex + /// parameter expansion ${var//pat/repl}, or an unbalanced + /// quote / unterminated heredoc. The reason text is in + /// . + UnparseableSentinel, +} diff --git a/src/ShellSyntaxTree/Internal/Lexing/OpaqueRegionScanner.cs b/src/ShellSyntaxTree/Internal/Lexing/OpaqueRegionScanner.cs new file mode 100644 index 0000000..a23dc0a --- /dev/null +++ b/src/ShellSyntaxTree/Internal/Lexing/OpaqueRegionScanner.cs @@ -0,0 +1,202 @@ +// ----------------------------------------------------------------------- +// +// Copyright (C) 2026 - 2026 Aaron Stannard +// +// ----------------------------------------------------------------------- +using System; + +namespace ShellSyntaxTree.Internal.Lexing; + +/// +/// Grammar-agnostic boundary scanner for "opaque regions" — input slices +/// that the lexer must skip over verbatim because their interior follows +/// rules the outer lexer does not model. In v0.1 this is bash's +/// $(…) command substitution and backtick `…`; in v0.2 it +/// will additionally serve PowerShell's $( … ) and @( … ). +/// +/// The scanner is intentionally permissive about interior content — it +/// does not parse the inside, it only finds the matching close. The +/// outer parser treats the whole region as a single +/// Arg{ Kind = DynamicSkip, IsPath = false } per the locked +/// interpretation in the v0.1 OpenSpec change +/// (proposal §"2. Command substitution + arithmetic + complex param +/// expansion"). +/// +internal static class OpaqueRegionScanner +{ + /// + /// Result of a scan. is the index of the + /// closing delimiter character (inclusive) when + /// is true; when is false the + /// scanner ran off the end of the input and + /// equals the input length. + /// + internal readonly record struct ScanResult(int EndIndex, bool Closed); + + /// + /// Find the matching close delimiter for an asymmetric opaque region + /// (e.g. (/)) starting at + /// — which must point at the opening delimiter character. Handles: + /// + /// + /// nested same-kind regions (depth tracking), + /// single- and double-quoted nested strings (delimiters + /// inside quotes don't count), + /// \X escapes outside of single quotes (the next char + /// is consumed verbatim). + /// + /// + internal static ScanResult Scan( + ReadOnlySpan input, + int startIndex, + char openChar, + char closeChar) + { + // Caller contract: startIndex points at the opening delimiter. + // We start scanning at startIndex+1 with depth=1 already counted. + if (startIndex < 0 || startIndex >= input.Length || input[startIndex] != openChar) + { + return new ScanResult(input.Length, false); + } + + var depth = 1; + var i = startIndex + 1; + while (i < input.Length) + { + var c = input[i]; + + // Backslash escapes the next character (outside single quotes). + // The single-quote branch below short-circuits before this code + // ever runs while inside '...'. + if (c == '\\' && i + 1 < input.Length) + { + i += 2; + continue; + } + + if (c == '\'') + { + // Single-quoted: bytes are literal, no escapes recognized. + i = SkipSingleQuoted(input, i + 1); + continue; + } + + if (c == '"') + { + i = SkipDoubleQuoted(input, i + 1); + continue; + } + + if (c == openChar) + { + depth++; + } + else if (c == closeChar) + { + depth--; + if (depth == 0) + { + return new ScanResult(i, true); + } + } + + i++; + } + + return new ScanResult(input.Length, false); + } + + /// + /// Variant for symmetric opaque regions (e.g. backtick-quoted command + /// substitution) where the open and close delimiters are the same + /// character. points at the opening + /// delimiter; the scan returns at the next unescaped occurrence of + /// . There is no nesting — the next + /// unescaped match wins. + /// + internal static ScanResult ScanSymmetric( + ReadOnlySpan input, + int startIndex, + char delimiter) + { + if (startIndex < 0 || startIndex >= input.Length || input[startIndex] != delimiter) + { + return new ScanResult(input.Length, false); + } + + var i = startIndex + 1; + while (i < input.Length) + { + var c = input[i]; + + if (c == '\\' && i + 1 < input.Length) + { + // Escape: skip backslash + next char verbatim. Bash treats + // \` inside a backtick context as a literal backtick — the + // skip is the right behavior either way. + i += 2; + continue; + } + + if (c == delimiter) + { + return new ScanResult(i, true); + } + + i++; + } + + return new ScanResult(input.Length, false); + } + + /// + /// Skip past a single-quoted string. points at + /// the first char after the opening quote. Returns the index of the + /// char after the closing quote, or input.Length if no close + /// was found. Single-quoted strings preserve bytes literally — no + /// escape processing per SPEC §5. + /// + private static int SkipSingleQuoted(ReadOnlySpan input, int i) + { + while (i < input.Length) + { + if (input[i] == '\'') + { + return i + 1; + } + + i++; + } + + return input.Length; + } + + /// + /// Skip past a double-quoted string. points at + /// the first char after the opening quote. Backslash escapes the + /// next character (covers \" in particular). Returns the + /// index of the char after the closing quote, or input.Length + /// if no close was found. + /// + private static int SkipDoubleQuoted(ReadOnlySpan input, int i) + { + while (i < input.Length) + { + var c = input[i]; + if (c == '\\' && i + 1 < input.Length) + { + i += 2; + continue; + } + + if (c == '"') + { + return i + 1; + } + + i++; + } + + return input.Length; + } +} diff --git a/src/ShellSyntaxTree/ShellSyntaxTree.csproj b/src/ShellSyntaxTree/ShellSyntaxTree.csproj index c646e0b..f3db7e6 100644 --- a/src/ShellSyntaxTree/ShellSyntaxTree.csproj +++ b/src/ShellSyntaxTree/ShellSyntaxTree.csproj @@ -11,4 +11,9 @@ true + + + + + diff --git a/tests/ShellSyntaxTree.Tests/Lexing/BashLexerTests.cs b/tests/ShellSyntaxTree.Tests/Lexing/BashLexerTests.cs new file mode 100644 index 0000000..471586c --- /dev/null +++ b/tests/ShellSyntaxTree.Tests/Lexing/BashLexerTests.cs @@ -0,0 +1,616 @@ +// ----------------------------------------------------------------------- +// +// Copyright (C) 2026 - 2026 Aaron Stannard +// +// ----------------------------------------------------------------------- +using System.Linq; +using ShellSyntaxTree.Internal.Bash.Lexing; +using Xunit; + +namespace ShellSyntaxTree.Tests.Lexing; + +/// +/// Unit tests for . Tests deliberately compare on +/// the meaningful (non-whitespace) tokens because whitespace is emitted +/// for source fidelity but is filtered by the parser. SPEC §5 is the +/// canonical contract; whenever a test pins a behavior, the rule it +/// pins is called out either inline or by the test name. +/// +public class BashLexerTests +{ + private static BashToken[] LexNonWs(string input) => + BashLexer.Tokenize(input) + .Where(t => t.Kind != BashTokenKind.Whitespace + && t.Kind != BashTokenKind.Continuation) + .ToArray(); + + // ------------------------------------------------------------ basic shapes + + [Fact] + public void Empty_input_returns_empty_token_list() + { + var tokens = BashLexer.Tokenize(""); + Assert.Empty(tokens); + } + + [Fact] + public void Single_word_lexes_as_one_word_token() + { + var tokens = LexNonWs("git"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.Word, t.Kind); + Assert.Equal("git", t.Value); + Assert.Equal(0, t.SourceStart); + Assert.Equal(3, t.SourceLength); + } + + [Fact] + public void Whitespace_separates_words() + { + var tokens = LexNonWs("git push"); + Assert.Equal(2, tokens.Length); + Assert.Equal("git", tokens[0].Value); + Assert.Equal("push", tokens[1].Value); + } + + [Fact] + public void Whitespace_token_is_emitted_between_words() + { + var tokens = BashLexer.Tokenize("a b"); + // Expected: Word("a"), Whitespace, Word("b") + Assert.Equal(3, tokens.Count); + Assert.Equal(BashTokenKind.Word, tokens[0].Kind); + Assert.Equal(BashTokenKind.Whitespace, tokens[1].Kind); + Assert.Equal(BashTokenKind.Word, tokens[2].Kind); + } + + // ------------------------------------------------------------ operators + + [Theory] + [InlineData("&&")] + [InlineData("||")] + [InlineData(";")] + [InlineData("|")] + [InlineData(">")] + [InlineData(">>")] + [InlineData("<")] + [InlineData("2>")] + [InlineData("2>>")] + [InlineData("(")] + [InlineData(")")] + public void Each_operator_lexes_in_isolation(string op) + { + var tokens = LexNonWs(op); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.Operator, t.Kind); + Assert.Equal(op, t.OperatorText); + } + + [Fact] + public void Operator_boundary_does_not_require_whitespace_and_if() + { + // cd /tmp&&ls + var tokens = LexNonWs("cd /tmp&&ls"); + Assert.Equal(4, tokens.Length); + Assert.Equal("cd", tokens[0].Value); + Assert.Equal("/tmp", tokens[1].Value); + Assert.Equal(BashTokenKind.Operator, tokens[2].Kind); + Assert.Equal("&&", tokens[2].OperatorText); + Assert.Equal("ls", tokens[3].Value); + } + + [Fact] + public void Operator_boundary_does_not_require_whitespace_pipe() + { + var tokens = LexNonWs("a|b"); + Assert.Equal(3, tokens.Length); + Assert.Equal("a", tokens[0].Value); + Assert.Equal("|", tokens[1].OperatorText); + Assert.Equal("b", tokens[2].Value); + } + + [Fact] + public void Operator_boundary_does_not_require_whitespace_semicolon() + { + var tokens = LexNonWs("a;b"); + Assert.Equal(3, tokens.Length); + Assert.Equal("a", tokens[0].Value); + Assert.Equal(";", tokens[1].OperatorText); + Assert.Equal("b", tokens[2].Value); + } + + [Fact] + public void Redirect_operator_attaches_to_filename_word() + { + var tokens = LexNonWs("cmd>file"); + Assert.Equal(3, tokens.Length); + Assert.Equal("cmd", tokens[0].Value); + Assert.Equal(">", tokens[1].OperatorText); + Assert.Equal("file", tokens[2].Value); + } + + [Fact] + public void Append_redirect_prefers_double_arrow() + { + // cmd >> log — must be `>>`, not `>` followed by `>` + var tokens = LexNonWs("cmd >> log"); + Assert.Equal(3, tokens.Length); + Assert.Equal(">>", tokens[1].OperatorText); + } + + [Fact] + public void Stderr_redirect_2gtgt_prefers_long_form() + { + var tokens = LexNonWs("cmd 2>> log"); + Assert.Equal(3, tokens.Length); + Assert.Equal("2>>", tokens[1].OperatorText); + } + + [Fact] + public void Heredoc_dash_is_recognized() + { + // cmd <<-EOF\n\tbody\n\tEOF\nrest + var input = "cmd <<-EOF\n\tbody\n\tEOF\nrest"; + var tokens = LexNonWs(input); + // Expected meaningful tokens: cmd, <<-, EOF, rest. + Assert.Equal(4, tokens.Length); + Assert.Equal("cmd", tokens[0].Value); + Assert.Equal("<<-", tokens[1].OperatorText); + Assert.Equal("EOF", tokens[2].Value); + Assert.Equal("rest", tokens[3].Value); + } + + // ------------------------------------------------------------ quoting + + [Fact] + public void Single_quoted_string_strips_delimiters() + { + var tokens = LexNonWs("'hello world'"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.QuotedString, t.Kind); + Assert.Equal("hello world", t.Value); + Assert.Equal(0, t.SourceStart); + Assert.Equal("'hello world'".Length, t.SourceLength); + } + + [Fact] + public void Single_quoted_preserves_dollar_literally() + { + // SPEC §5: single quotes preserve bytes literally. + var tokens = LexNonWs("'foo$bar'"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.QuotedString, t.Kind); + Assert.Equal("foo$bar", t.Value); + } + + [Fact] + public void Single_quoted_preserves_backslash_literally() + { + // No escape processing in single quotes. + var tokens = LexNonWs(@"'a\nb'"); + var t = Assert.Single(tokens); + Assert.Equal(@"a\nb", t.Value); + } + + [Fact] + public void Double_quoted_string_strips_delimiters() + { + var tokens = LexNonWs("\"hello world\""); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.QuotedString, t.Kind); + Assert.Equal("hello world", t.Value); + } + + [Fact] + public void Double_quoted_with_escaped_quote() + { + var tokens = LexNonWs("\"hello \\\"world\\\"\""); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.QuotedString, t.Kind); + Assert.Equal("hello \"world\"", t.Value); + } + + [Fact] + public void Double_quoted_with_escaped_backslash() + { + var tokens = LexNonWs("\"a\\\\b\""); + var t = Assert.Single(tokens); + Assert.Equal("a\\b", t.Value); + } + + [Fact] + public void Double_quoted_with_escaped_dollar() + { + var tokens = LexNonWs("\"\\$HOME\""); + var t = Assert.Single(tokens); + Assert.Equal("$HOME", t.Value); + } + + [Fact] + public void Double_quoted_preserves_unexpanded_env_var() + { + // SPEC §5: $VAR is recognized but NOT expanded by the lexer. + var tokens = LexNonWs("\"$HOME\""); + var t = Assert.Single(tokens); + Assert.Equal("$HOME", t.Value); + } + + // ------------------------------------------------------------ words / paths + + [Fact] + public void Tilde_is_part_of_word() + { + var tokens = LexNonWs("~/path"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.Word, t.Kind); + Assert.Equal("~/path", t.Value); + } + + [Fact] + public void Long_flag_is_a_word() + { + var tokens = LexNonWs("--force"); + var t = Assert.Single(tokens); + Assert.Equal("--force", t.Value); + } + + [Fact] + public void Absolute_path_with_slashes_is_a_word() + { + var tokens = LexNonWs("/etc/foo"); + var t = Assert.Single(tokens); + Assert.Equal("/etc/foo", t.Value); + } + + [Fact] + public void Continuation_collapses_to_whitespace() + { + // cmd \\\n foo -- meaningful tokens are [cmd, foo]. + var tokens = LexNonWs("cmd \\\nfoo"); + Assert.Equal(2, tokens.Length); + Assert.Equal("cmd", tokens[0].Value); + Assert.Equal("foo", tokens[1].Value); + } + + [Fact] + public void Backslash_escapes_dollar_outside_quotes() + { + // SPEC §5 `echo \$HOME` — produces a Word with value $HOME (the + // backslash is consumed; the parser treats this as Literal). + var tokens = LexNonWs("echo \\$HOME"); + Assert.Equal(2, tokens.Length); + Assert.Equal("echo", tokens[0].Value); + Assert.Equal("$HOME", tokens[1].Value); + Assert.Equal(BashTokenKind.Word, tokens[1].Kind); + } + + [Fact] + public void Backslash_escapes_space_outside_quotes() + { + // `cmd\ foo` is a single token "cmd foo". + var tokens = LexNonWs("cmd\\ foo"); + var t = Assert.Single(tokens); + Assert.Equal("cmd foo", t.Value); + } + + [Fact] + public void Simple_brace_expansion_stays_as_word() + { + var tokens = LexNonWs("cmd ${HOME}/path"); + Assert.Equal(2, tokens.Length); + Assert.Equal("cmd", tokens[0].Value); + Assert.Equal("${HOME}/path", tokens[1].Value); + Assert.Equal(BashTokenKind.Word, tokens[1].Kind); + } + + [Fact] + public void Bare_dollar_var_stays_as_word() + { + var tokens = LexNonWs("echo $HOME"); + Assert.Equal(2, tokens.Length); + Assert.Equal("$HOME", tokens[1].Value); + } + + // ------------------------------------------------------------ opaque regions + + [Fact] + public void Dollar_paren_command_substitution_is_one_opaque_token() + { + var tokens = LexNonWs("$(echo foo)"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.OpaqueSubstitution, t.Kind); + Assert.Equal("$(echo foo)", t.Value); + Assert.Equal(0, t.SourceStart); + Assert.Equal("$(echo foo)".Length, t.SourceLength); + } + + [Fact] + public void Nested_command_substitutions_are_one_opaque_token() + { + var tokens = LexNonWs("$(echo $(date))"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.OpaqueSubstitution, t.Kind); + Assert.Equal("$(echo $(date))", t.Value); + } + + [Fact] + public void Backtick_substitution_is_one_opaque_token() + { + var tokens = LexNonWs("`pgrep nginx`"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.OpaqueSubstitution, t.Kind); + Assert.Equal("`pgrep nginx`", t.Value); + } + + [Fact] + public void Mixed_word_and_command_substitution() + { + var tokens = LexNonWs("kill $(pgrep -f foo)"); + Assert.Equal(2, tokens.Length); + Assert.Equal(BashTokenKind.Word, tokens[0].Kind); + Assert.Equal("kill", tokens[0].Value); + Assert.Equal(BashTokenKind.OpaqueSubstitution, tokens[1].Kind); + Assert.Equal("$(pgrep -f foo)", tokens[1].Value); + } + + [Fact] + public void Command_substitution_with_quoted_close_paren_is_balanced() + { + // The ')' inside the double-quoted string must not close the region. + var tokens = LexNonWs("$(echo \"a)b\")"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.OpaqueSubstitution, t.Kind); + Assert.Equal("$(echo \"a)b\")", t.Value); + } + + // ------------------------------------------------------------ unparseables + + [Fact] + public void Arithmetic_expansion_emits_unparseable_sentinel() + { + var tokens = LexNonWs("$((1 + 2))"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.UnparseableSentinel, t.Kind); + Assert.Equal("$((1 + 2))", t.Value); + Assert.Contains("arithmetic", t.UnparseableReason); + } + + [Fact] + public void Complex_param_expansion_emits_unparseable_sentinel() + { + var tokens = LexNonWs("${PATH//:/\\n}"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.UnparseableSentinel, t.Kind); + Assert.Contains("complex parameter expansion", t.UnparseableReason); + } + + [Fact] + public void Unbalanced_double_quote_emits_unparseable_sentinel() + { + var tokens = LexNonWs("cmd \"foo"); + // First a Word("cmd"), then the sentinel. + Assert.Equal(2, tokens.Length); + Assert.Equal(BashTokenKind.Word, tokens[0].Kind); + Assert.Equal(BashTokenKind.UnparseableSentinel, tokens[1].Kind); + Assert.Contains("unbalanced quote", tokens[1].UnparseableReason); + } + + [Fact] + public void Unbalanced_single_quote_emits_unparseable_sentinel() + { + var tokens = LexNonWs("cmd 'foo"); + Assert.Equal(2, tokens.Length); + Assert.Equal(BashTokenKind.UnparseableSentinel, tokens[1].Kind); + Assert.Contains("unbalanced quote", tokens[1].UnparseableReason); + } + + [Fact] + public void Unbalanced_command_substitution_emits_unparseable_sentinel() + { + var tokens = LexNonWs("$(echo foo"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.UnparseableSentinel, t.Kind); + Assert.Contains("unbalanced", t.UnparseableReason); + } + + [Fact] + public void Unbalanced_backtick_emits_unparseable_sentinel() + { + var tokens = LexNonWs("`pgrep nginx"); + var t = Assert.Single(tokens); + Assert.Equal(BashTokenKind.UnparseableSentinel, t.Kind); + Assert.Contains("unbalanced", t.UnparseableReason); + } + + // ------------------------------------------------------------ heredoc + + [Fact] + public void Heredoc_body_is_skipped() + { + var input = "cmd <= 0); + Assert.True(restIdx > eofIdx); + var hasSeparator = false; + for (var i = eofIdx + 1; i < restIdx; i++) + { + if (tokens[i].Kind == BashTokenKind.Whitespace) { hasSeparator = true; break; } + } + + Assert.True(hasSeparator, "expected a Whitespace token between heredoc terminator and following clause"); + } + + // ------------------------------------------------------------ compound clause + + [Fact] + public void Compound_clause_lexes_all_pieces() + { + // cd /target && cmd file.txt + var tokens = LexNonWs("cd /target && cmd file.txt"); + Assert.Equal(5, tokens.Length); + Assert.Equal("cd", tokens[0].Value); + Assert.Equal("/target", tokens[1].Value); + Assert.Equal("&&", tokens[2].OperatorText); + Assert.Equal("cmd", tokens[3].Value); + Assert.Equal("file.txt", tokens[4].Value); + } + + [Fact] + public void Subshell_parens_are_individual_operators() + { + var tokens = LexNonWs("(cd /tmp && ls)"); + // ( cd /tmp && ls ) + Assert.Equal(6, tokens.Length); + Assert.Equal("(", tokens[0].OperatorText); + Assert.Equal("cd", tokens[1].Value); + Assert.Equal("/tmp", tokens[2].Value); + Assert.Equal("&&", tokens[3].OperatorText); + Assert.Equal("ls", tokens[4].Value); + Assert.Equal(")", tokens[5].OperatorText); + } + + [Fact] + public void Pipe_with_quoted_argument() + { + var tokens = LexNonWs("grep \"foo bar\" | wc -l"); + Assert.Equal(5, tokens.Length); + Assert.Equal("grep", tokens[0].Value); + Assert.Equal(BashTokenKind.QuotedString, tokens[1].Kind); + Assert.Equal("foo bar", tokens[1].Value); + Assert.Equal("|", tokens[2].OperatorText); + Assert.Equal("wc", tokens[3].Value); + Assert.Equal("-l", tokens[4].Value); + } + + // ------------------------------------------------------------ source positions + + [Fact] + public void Source_positions_are_populated_for_every_token() + { + const string input = "cd /tmp && ls"; + var tokens = BashLexer.Tokenize(input); + foreach (var t in tokens) + { + Assert.True(t.SourceStart >= 0); + Assert.True(t.SourceStart + t.SourceLength <= input.Length); + } + + // First Word starts at 0. + var firstWord = tokens.First(t => t.Kind == BashTokenKind.Word); + Assert.Equal(0, firstWord.SourceStart); + + // The `&&` operator starts where the source `&&` is. + var op = tokens.First(t => t.Kind == BashTokenKind.Operator); + Assert.Equal(input.IndexOf("&&"), op.SourceStart); + Assert.Equal(2, op.SourceLength); + } + + [Fact] + public void Word_source_length_includes_escape_sequence() + { + // The escape-collapsed Value is shorter than the source slice. + const string input = "\\$HOME"; + var tokens = LexNonWs(input); + var t = Assert.Single(tokens); + Assert.Equal("$HOME", t.Value); + Assert.Equal(0, t.SourceStart); + Assert.Equal(input.Length, t.SourceLength); + } + + [Fact] + public void Operator_text_is_set_exactly_for_each_kind() + { + // Sweep all operators in one input to lock the OperatorText shape. + var tokens = LexNonWs("a&&b||c;d|e>f>>g

i2>>j(k)"); + var ops = tokens.Where(t => t.Kind == BashTokenKind.Operator) + .Select(t => t.OperatorText).ToArray(); + Assert.Equal( + new[] { "&&", "||", ";", "|", ">", ">>", "<", "2>", "2>>", "(", ")" }, + ops); + } + + // ------------------------------------------------------------ misc + + [Fact] + public void Multiple_redirects_lex_independently() + { + var tokens = LexNonWs("cmd > out 2> err"); + Assert.Equal(5, tokens.Length); + Assert.Equal("cmd", tokens[0].Value); + Assert.Equal(">", tokens[1].OperatorText); + Assert.Equal("out", tokens[2].Value); + Assert.Equal("2>", tokens[3].OperatorText); + Assert.Equal("err", tokens[4].Value); + } + + [Fact] + public void Empty_quoted_strings_are_preserved() + { + var tokens = LexNonWs("cmd '' \"\""); + Assert.Equal(3, tokens.Length); + Assert.Equal(BashTokenKind.Word, tokens[0].Kind); + Assert.Equal(BashTokenKind.QuotedString, tokens[1].Kind); + Assert.Equal("", tokens[1].Value); + Assert.Equal(BashTokenKind.QuotedString, tokens[2].Kind); + Assert.Equal("", tokens[2].Value); + } + + [Fact] + public void Adjacent_word_and_quoted_string_lex_separately() + { + // bash actually concatenates these into one argument, but the + // lexer emits them as two adjacent tokens — concatenation is the + // parser's job. This test pins the lexer behavior. + var tokens = LexNonWs("cmd foo'bar'"); + Assert.Equal(3, tokens.Length); + Assert.Equal(BashTokenKind.Word, tokens[0].Kind); + Assert.Equal(BashTokenKind.Word, tokens[1].Kind); + Assert.Equal("foo", tokens[1].Value); + Assert.Equal(BashTokenKind.QuotedString, tokens[2].Kind); + Assert.Equal("bar", tokens[2].Value); + } + + [Fact] + public void Tokenize_throws_on_null_input() + { + Assert.Throws(() => BashLexer.Tokenize(null!)); + } +} diff --git a/tests/ShellSyntaxTree.Tests/Lexing/OpaqueRegionScannerTests.cs b/tests/ShellSyntaxTree.Tests/Lexing/OpaqueRegionScannerTests.cs new file mode 100644 index 0000000..3b0eab6 --- /dev/null +++ b/tests/ShellSyntaxTree.Tests/Lexing/OpaqueRegionScannerTests.cs @@ -0,0 +1,177 @@ +// ----------------------------------------------------------------------- +// +// Copyright (C) 2026 - 2026 Aaron Stannard +// +// ----------------------------------------------------------------------- +using System; +using ShellSyntaxTree.Internal.Lexing; +using Xunit; + +namespace ShellSyntaxTree.Tests.Lexing; + +/// +/// Unit tests for . The scanner is +/// grammar-agnostic; tests use bash-style inputs because that's what the +/// v0.1 lexer drives, but the assertions don't depend on bash semantics. +/// +public class OpaqueRegionScannerTests +{ + [Fact] + public void Scan_balanced_parens_returns_close_index() + { + // 0123456 + // (foo) + var input = "(foo)".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.True(result.Closed); + Assert.Equal(4, result.EndIndex); + } + + [Fact] + public void Scan_nested_parens_returns_outer_close() + { + // 0 1 + // 0123456789012345 + // (foo (bar) baz) + var input = "(foo (bar) baz)".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.True(result.Closed); + Assert.Equal(14, result.EndIndex); + } + + [Fact] + public void Scan_close_inside_double_quotes_is_ignored() + { + // (foo "ignored ) close" bar) + var input = "(foo \"ignored ) close\" bar)".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.True(result.Closed); + Assert.Equal(input.Length - 1, result.EndIndex); + Assert.Equal(')', input[result.EndIndex]); + } + + [Fact] + public void Scan_close_inside_single_quotes_is_ignored() + { + // Single quotes do NOT honor escapes, but they DO swallow ')'. + // (foo ')' bar) + var input = "(foo ')' bar)".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.True(result.Closed); + Assert.Equal(input.Length - 1, result.EndIndex); + } + + [Fact] + public void Scan_escaped_close_outside_quotes_is_skipped() + { + // (foo \) bar) + var input = "(foo \\) bar)".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.True(result.Closed); + Assert.Equal(input.Length - 1, result.EndIndex); + } + + [Fact] + public void Scan_escaped_open_outside_quotes_is_skipped() + { + // ( \( ) — the escaped '(' must not push depth. + var input = "( \\( )".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.True(result.Closed); + Assert.Equal(input.Length - 1, result.EndIndex); + } + + [Fact] + public void Scan_unclosed_returns_input_length_and_not_closed() + { + var input = "(foo bar baz".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.False(result.Closed); + Assert.Equal(input.Length, result.EndIndex); + } + + [Fact] + public void Scan_empty_body_closes_immediately() + { + var input = "()".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.True(result.Closed); + Assert.Equal(1, result.EndIndex); + } + + [Fact] + public void Scan_deep_nesting_finds_outer_close() + { + var input = "((((( inner )))))".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.True(result.Closed); + Assert.Equal(input.Length - 1, result.EndIndex); + } + + [Fact] + public void Scan_braces_returns_close_index() + { + // {a} + var input = "{a}".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '{', '}'); + Assert.True(result.Closed); + Assert.Equal(2, result.EndIndex); + } + + [Fact] + public void Scan_offset_start_index_uses_only_inner_region() + { + // foo (bar) baz — start at the '(' at index 4. + var input = "foo (bar) baz".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 4, '(', ')'); + Assert.True(result.Closed); + Assert.Equal(8, result.EndIndex); + } + + [Fact] + public void Scan_returns_unclosed_when_start_is_not_open_char() + { + var input = "foo )".AsSpan(); + var result = OpaqueRegionScanner.Scan(input, 0, '(', ')'); + Assert.False(result.Closed); + Assert.Equal(input.Length, result.EndIndex); + } + + [Fact] + public void ScanSymmetric_backtick_returns_close_index() + { + // `foo` + var input = "`foo`".AsSpan(); + var result = OpaqueRegionScanner.ScanSymmetric(input, 0, '`'); + Assert.True(result.Closed); + Assert.Equal(4, result.EndIndex); + } + + [Fact] + public void ScanSymmetric_escaped_delimiter_is_skipped() + { + // `foo \` bar` + var input = "`foo \\` bar`".AsSpan(); + var result = OpaqueRegionScanner.ScanSymmetric(input, 0, '`'); + Assert.True(result.Closed); + Assert.Equal(input.Length - 1, result.EndIndex); + } + + [Fact] + public void ScanSymmetric_unclosed_returns_input_length_and_not_closed() + { + var input = "`foo bar".AsSpan(); + var result = OpaqueRegionScanner.ScanSymmetric(input, 0, '`'); + Assert.False(result.Closed); + Assert.Equal(input.Length, result.EndIndex); + } + + [Fact] + public void ScanSymmetric_empty_body_closes_immediately() + { + var input = "``".AsSpan(); + var result = OpaqueRegionScanner.ScanSymmetric(input, 0, '`'); + Assert.True(result.Closed); + Assert.Equal(1, result.EndIndex); + } +}