Skip to content

Improve mobile terminal reliability and session tooling - #173

Open
Lint111 wants to merge 43 commits into
Ark0N:masterfrom
Lint111:agent/mobile-terminal-experience
Open

Improve mobile terminal reliability and session tooling#173
Lint111 wants to merge 43 commits into
Ark0N:masterfrom
Lint111:agent/mobile-terminal-experience

Conversation

@Lint111

@Lint111 Lint111 commented Jul 28, 2026

Copy link
Copy Markdown

Summary

  • Add opt-in mobile terminal navigation with swipe/up/down/Enter controls, safer edge-positioned Tab/Esc actions, dialog coexistence, and touch feedback.
  • Make mobile input durable across tab changes and browser lifecycle events, with multiline paste, delete/local-echo synchronization, mirrored input, and keyboard-aware layout fixes.
  • Add lossless bounded terminal streaming, compressed incremental transport, warm session caches, restart recovery, and immutable frame handoffs that hide Codex history replay during tab and keyboard transitions.
  • Add repository/worktree diff and response inspection views, inline case edit/delete actions, preserved session labels/work paths, and related mobile UX improvements.
  • Reduce hook noise while preserving background-command wakeups and integrate the branch with upstream web tabs, themes, and repository conventions.

Merge notes

Merged origin/master at 57b6be1. Conflicts in CLAUDE.md and styles.css retain upstream structure/theme work and document the combined mobile/webview invariants.

Validation

  • Lockfile, formatting, public assets, frontend syntax, typecheck, lint, and production build passed.
  • Mobile browser suites: 78 keyboard tests and 17 navigation/file-viewer tests passed.
  • CI-equivalent suite: 197 files passed, 1 skipped; 3,926 tests passed, 12 skipped.
  • Live Playwright capture verified immutable Claude-to-Codex and Codex keyboard open/close transitions against a 26 MB terminal history.

Lint111 added 30 commits July 25, 2026 22:22
- Replace the split extended-keyboard option with responsive Esc, arrows, Enter, and Tab controls for keyboard-hidden and keyboard-open layouts.
- Add swipe navigation, Up+Down Enter chords, best-effort volume-key input, and optional haptic/sound feedback.
- Keep the accessory layout keyboard-safe, phone-width contained, and hidden behind application modals.

Verified: 21 unit tests, 64 mobile browser tests, public asset/syntax checks, and npm run build.
- arbitrate shared PTY dimensions from explicit interaction and input
- force one desktop redraw when the terminal or session tab retakes control
- isolate mobile and viewport browser tests from host state
- distinguish editable prompt rows from TUI-owned readbacks and choices\n- keep content taps keyboard-free while preserving prompt focus and cursor positioning\n- cover readback collapse in unit and mobile browser tests
- add admin-only graceful shutdown with exact supervisor ownership checks

- add a keyboard-safe header confirmation flow with phone browser coverage
- keep multiline clipboard text inside one bracketed-paste frame across native, CJK, dialog, image fallback, and voice paths
- hold the first local draft until replay establishes the real prompt position
- keep bracketed multiline text on the direct PTY path when WebSocket delivery falls back to HTTP
- recognize Android input-only paste events and Enter races in the CJK field
- confirm multiline paste from Android insertText and line-break event bursts
- preserve ordinary Gboard word-space input and desktop autocomplete behavior
- cover full textarea, segmented, and non-paste input paths in browser tests
- clear xterm helper text after handled touch input
- turn Android delete mutations into one terminal Backspace each
- forward deletes when editable text already lives in the PTY
- keep transient IME candidates separate from committed draft text
- commit finalized composition atomically without duplicate words
- guard newer compositions from stale cleanup timers
- let the prompt-aware overlay own composition rendering
- keep xterm composition processing active but visually hidden
- cover native-helper and local-overlay coexistence
Commit the last Android composition when xterm omits its final data event. Suppress one delayed matching payload so continued typing does not duplicate the word.
Read insertText mutations from xterm's helper textarea when Android omits InputEvent.data. Cover continued typing after the first visible character.
Release stale composition state when Android resumes with an explicitly non-composing text mutation. Keep later submissions visible without reviving the abandoned candidate.
Route mobile-control Enter through local echo so committed and composing text is sent before Enter. Keep the next draft clean without focusing xterm or opening the keyboard.
@Lint111
Lint111 marked this pull request as ready for review July 28, 2026 10:28
Copilot AI review requested due to automatic review settings July 28, 2026 10:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

@Ark0N

Ark0N commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Thanks for the effort here, and for the audit document, that is a genuinely useful artifact. I deployed this branch to an isolated instance and tested it on a real phone before reviewing the code. Reporting what I found, starting with the blocker.

Blocking: the on-screen keyboard can no longer be opened

On a phone, tapping the terminal no longer raises the keyboard. There is no longer any way to type into a session. I hit this by hand first, then reduced it.

Reproduction, iPhone-class viewport (390x844), Claude-mode session, tap the middle of the terminal:

document.activeElement after tap typed text reaches input layer
master textarea.xterm-helper-textarea yes, pendingText: "hello"
this branch body no, pendingText: ""

Mechanism. _classifyMobileTerminalTap() returns content for the tap, and _handleMobileTerminalTap() then runs:

if (intent === 'content') {
  this._blurMobileTerminalInput();
}

so the tap explicitly blurs. Separately, touchstart runs ev.preventDefault() whenever touchStartIntent !== 'input', which cancels the compatibility click that would otherwise have focused xterm. Both routes to focus are therefore closed on the same gesture.

The intended escape hatch is that tapping the prompt row classifies as input and focuses. In practice that zone does not exist. Sampling _classifyMobileTerminalTap() at 21 points from the top of the terminal to the bottom on a Claude-mode session returns content at every single one:

{"content": 21}
  0%: content    25%: content    50%: content    75%: content   100%: content
  ... (all 21 sample points identical)

The reason is the early return:

if (!mouseTrackingOn && !this._sessionUsesServerMouseStrip()) return 'input';

For claude/codex/gemini sessions the server strips the mouse DECSETs, so _sessionUsesServerMouseStrip() is true and that input early-return never fires. Classification then falls through to prompt-row detection, which on a normal Claude screen resolves to content everywhere. The net effect is that the keyboard is unreachable for exactly the CLI modes Codeman is mostly used with.

_focusMobileTerminalInput() itself is fine, calling it directly from the console focuses the helper textarea correctly. The plumbing works; the classification and blur policy is what closes the door.

Worth adding a test that asserts a tap somewhere on a claude-mode terminal ends with document.activeElement === terminal.textarea. The current mobile tests assert the routing decisions, so they stay green while the user-visible outcome is broken.

Security: POST /api/system/shutdown is unauthenticated in the default configuration

The handler is gated with requireAdmin, but that resolves to:

return !isMultiUserMode() || getAuthUser(req).role === 'admin';

Multi-user mode is opt-in and off by default, so in the default single-user configuration requireAdmin returns true for every caller. The endpoint then runs systemctl --user stop codeman-web.service (or launchctl bootout), which stops the supervising unit, so the service does not come back on its own.

That matters because Codeman is commonly reached over a tunnel or tailnet without CODEMAN_PASSWORD set. In that setup any device that can reach the API can permanently stop the instance with a single unauthenticated POST, and recovery needs shell access to the host. A remote kill switch needs its own explicit authorization story, not the ambient admin check that is a no-op in the default mode.

The mobile header guard was changed rather than satisfied

test/mobile-header-buttons-policy.test.ts exists because header controls have repeatedly leaked onto the phone header. This branch edits the guard itself:

-const MOBILE_VISIBLE_ALLOWLIST = new Set<string>([]);
+const MOBILE_VISIBLE_ALLOWLIST = new Set<string>(['btn-instance-shutdown', 'btn-file-viewer']);

-const KNOWN_PHONE_HIDDEN = ['btn-settings', 'btn-lifecycle-log', 'btn-session-manager', 'btn-file-viewer'];
+const KNOWN_PHONE_HIDDEN = ['btn-settings', 'btn-lifecycle-log', 'btn-session-manager'];

The second hunk removes btn-file-viewer from the explicit lock whose own comment says it is there "so a future edit that removes a hide rule fails loudly (not silently)". The result is a shutdown button permanently in the phone header, one mis-tap from stopping the service, on the same small screen where the keyboard currently cannot be opened. Please revert the guard and hide the button on phones instead.

Scope

The PR title and summary describe opt-in mobile terminal navigation. The branch is 39 commits, 106 files, +19,572 / -1,890, large enough that GitHub refuses to render the diff. It includes at least two subsystems the description does not mention:

  • a git repository browser (src/git-repository-browser.ts, ~760 lines, three new endpoints, its own doc)
  • a remote shutdown facility (src/web/instance-shutdown.ts, a new port interface, an endpoint, a header button, a modal)

plus a working-directory reassignment endpoint, case inline edit and delete, terminal history paging, a warm terminal cache, PTY resize arbitration, roughly eight commits of IME and paste fixes, notification-noise changes, a new Bash-matcher hook written into users' settings.local.json, Codex animation toggles, and multiline draft persistence.

This is not reviewable as one change, and it is not releasable as one either: a regression in any part blocks all of it, which is what has happened here. Please split it. The mobile navigation pad, and the IME and paste fixes, are each plausibly mergeable on their own. The shutdown feature needs the authorization question answered first. The repository browser is a separate feature that deserves its own review.

Also, the summary calls the navigation controls opt-in, but mobileTerminalControlsEnabled and mobileControlHaptics both default to true, so they are on for everyone after upgrade.

What is good here

To be clear that this is a scope and correctness problem rather than a quality one:

  • tsc --noEmit, eslint, and prettier --check are all clean, and the full suite passes at 3926 tests.
  • The git browser's path handling is careful: absolute paths and NUL bytes rejected, relative() escape checks, a realpath double-check before reading, and spawn('git', [args]) with no shell.
  • PUT /api/sessions/:id/working-directory is correctly gated with findSessionOrFail, isWorkingDirAllowed, a remote and docker exclusion, and a stat check.
  • The zero-lag overlay was edited in packages/xterm-zerolag-input/src/, not the generated vendor bundle, which is the single-source rule most contributors get wrong.
  • test/opencode-resize.test.ts looked like deleted coverage but is a rename into terminal-viewport-resize.test.ts that keeps all six original cases and adds seven.
  • The new tests build their git fixtures in temp directories rather than the repo.

Status

CI has never run on this branch, it is sitting at action_required, so none of the above was caught automatically. The branch also now conflicts with master in four files: docs/architecture-invariants.md, src/web/public/keyboard-accessory.js, src/web/routes/file-routes.ts, src/web/routes/session-routes.ts.

Happy to look again once it is split and the keyboard regression is fixed. The navigation pad itself is a good idea, the Tab key in particular is a real gap in the current mobile toolbar.

@Lint111

Lint111 commented Jul 28, 2026

Copy link
Copy Markdown
Author

I see, you would prefer a more focused pr's 8n the future, make sense I just kind off went on every qa issue i found to make the mobile experience more convenient for me.

Ill go over the issues you reported

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants