Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions docs/architecture/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,15 @@ bytes. This keeps them testable and keeps each command file small.
spinner, no in-place updates. The scheduler invokes prosa this way, so
this is the default for the production install.

The fallback decision lives in `internal/cli/sync.go` and looks at whether
`stderr` is a TTY (not stdout — sync writes its summary to stdout).
The fallback decision lives in `internal/cli/sync.go` and requires both
`stdout` and `stderr` to be TTYs (`IsInteractive` in `internal/cli/term.go`).
Sync writes its summary to `stderr`, keeping `stdout` free for piped output.

The interactive branch owns the terminal for the duration of the frame, so
it swaps the *scoped* loggers of the pusher and of every importer
(`opts.Logger`) for one that tallies warn-level records; the count is
reported in the summary afterwards. The process-global `slog` default is
never touched.

## Store access

Expand Down
11 changes: 10 additions & 1 deletion docs/architecture/importers.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@ type SkipCache interface {
decide whether to no-op, parse it, classify its usage, and write into
the sink. `opts.Overwrite` bypasses the idempotency short-circuit and
the no_usage skip cache (used by `prosa sync --overwrite`).
`opts.Logger` is where this import's diagnostics go; resolve it with
`importerutil.Logger(opts)`, which falls back to `slog.Default()` when
the caller left it nil. The interactive sync path passes a counting
logger so warnings are tallied instead of written into its progress
frame.

The `Sink` is implemented by `internal/store` (locally) and by an in-memory
fake for tests. Importers never know about SQLite, Postgres, or the server.
Expand Down Expand Up @@ -227,7 +232,9 @@ The short version:
2. Create `internal/importers/<agent>/`. Mirror the shape of
`internal/importers/claudecode/`.
3. Implement `Walk` and `Import`. Be paranoid about malformed records and
partial files.
partial files: skip blank lines silently, and report malformed records
once per file with a count and the first offending line rather than
once per record.
4. Write parser tests (`parse_test.go`) covering representative records,
missing fields, malformed JSON, truncated files, sessions with no
turns.
Expand Down Expand Up @@ -265,3 +272,5 @@ Skill backing the reviewer:
- Special-case a single user. If a behavior is needed, document it in
`docs/sources/<agent>.md` and put it under a flag if it's optional.
- Mutate the agent's raw files in any way.
- Call package-level `slog`. Diagnostics go through
`importerutil.Logger(opts)` so the caller decides where they land.
5 changes: 5 additions & 0 deletions docs/cli/motion.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,11 @@ The command should finish with a non-zero exit status only when the requested
operation cannot complete. Per-file importer errors can be summarized at the
end while allowing the rest of the import to proceed.

Warn-level diagnostics are counted rather than printed while the frame is
live — an out-of-band write desyncs the in-place repaint — and the tally
lands in the final summary. Plain and `--json` mode stream them to `stderr`
as they happen.

## Cancellation

`ctrl+c` cancels the progress program and releases the terminal. After
Expand Down
12 changes: 11 additions & 1 deletion docs/cli/rendering-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,12 @@ Use `stderr` for operational context:
- cancellation notices;
- logs.

Between the first and last frame of an in-place progress render, nothing
writes to `stdout` or `stderr` outside the renderer: the frame repaints by
moving the cursor over its own previous output, so an out-of-band line
desyncs that accounting. Warn-level diagnostics raised in that window are
counted and reported in the summary instead.

For non-interactive automation, progress may stream to `stderr` while the final
result remains pipeable on `stdout`.

Expand Down Expand Up @@ -341,10 +347,14 @@ Live: imported N · skipped N · errors N
Legacy: imported N · skipped N · errors N (of N catalog rows)
Push: sent N · skipped N · errors N
Catch-up: sent N · skipped N · errors N (local L · remote R)
Warnings: N diagnostic logs suppressed in TTY; use `--verbose` to see them
Remote: server unavailable at <server>; local import is saved. Run `prosa sync` again when it is back.
```

`Legacy` only appears when `--legacy-bundle` was passed. `Push` and
`Warnings` appears in the TTY summary when the run raised warn-level
diagnostics — from the importers or from the catch-up phase — that the
in-place frame could not display. `Legacy` only appears when
`--legacy-bundle` was passed. `Push` and
`Catch-up` only appear when the device is logged in to a prosa-server
(i.e. when `~/.config/prosa/auth.json` exists). `Catch-up` is the
manifest-driven reconcile that makes the remote converge to the local
Expand Down
3 changes: 3 additions & 0 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,9 @@ they are enforced by reviewers (human or agent).
- **Error wrapping**: `fmt.Errorf("doing X: %w", err)`. No
`pkg/errors`-style ladders.
- **Logging**: `log/slog` with the default text handler in CLI commands.
Importers and the pusher write through a caller-supplied `*slog.Logger`
so the interactive sync path can scope them; `slog.SetDefault` is never
called at runtime.
- **Comments** are sparse, local, and surgical. Reach for one only to
document a behaviour that is not immediately obvious; otherwise let the
code speak for itself. Docstrings are allowed but stay short and avoid
Expand Down
4 changes: 4 additions & 0 deletions docs/sources/claude-code.md
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,10 @@ metadata-rich version with tool-call summaries.

## Notes for prosa importers

- Claude Code appends to the transcript while the session runs, so a torn
write occasionally leaves a line cut mid-token. Such lines are skipped,
as are blank lines; the file earns one warning carrying the count, the
first offending line, and the first parse error.
- Walk `*.jsonl` directly. `sessions-index.json` is a hint, never the
source of truth.
- Subagents are the majority of files in active workspaces. Include
Expand Down
4 changes: 4 additions & 0 deletions docs/sources/codex.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,10 @@ metadata and tool-call summaries; the recipe above is for raw inspection.

## Notes for prosa importers

- Codex appends to the rollout file while the session runs, so a torn
write occasionally leaves a line cut mid-token. Such lines are skipped,
as are blank lines; the file earns one warning carrying the count, the
first offending line, and the first parse error.
- Recent files use the envelope; older files may emit top-level
`message`, `reasoning`, `function_call`, `function_call_output`. Both
shapes must be projected.
Expand Down
22 changes: 17 additions & 5 deletions internal/cli/sync_run.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,8 @@ func runSyncInteractive(
}
updates := make(chan spinner.Update, len(work)*2+16)

// Scope log suppression to the pusher's logger, not the process-global slog
// default, so only reconcile chatter is silenced during Bubble Tea repaints.
var suppressedWarnings atomic.Int64
if push != nil {
push.logger = slog.New(warningCounterHandler{count: &suppressedWarnings})
}
opts = quietSyncLogging(opts, push, &suppressedWarnings)

go func() {
defer close(updates)
Expand Down Expand Up @@ -242,6 +238,22 @@ func runSyncPlain(ctx context.Context, work []syncJob, sink importer.Sink, push
return nil
}

// quietSyncLogging points the pusher and every importer at a logger that
// tallies warn-level records instead of writing them. Bubble Tea repaints
// by cursor-up over its own frame on stdout, so an unrelated stderr write
// mid-frame desyncs that line accounting and smears the output. The tally
// surfaces afterwards in the summary; the plain and JSON paths keep
// slog.Default(). Scoped per call — the process-global default is never
// swapped.
func quietSyncLogging(opts importer.ImportOptions, push *pusher, count *atomic.Int64) importer.ImportOptions {
quiet := slog.New(warningCounterHandler{count: count})
if push != nil {
push.logger = quiet
}
opts.Logger = quiet
return opts
}

// localItemErr returns what the spinner shows on a local-phase row.
// Import error wins; a pushFailed outcome surfaces the push error so a healthy
// import with a failed push isn't rendered as a clean check mark (issue #74).
Expand Down
40 changes: 40 additions & 0 deletions internal/cli/sync_slog_test.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
package cli

import (
"bytes"
"log/slog"
"strings"
"sync/atomic"
"testing"

"github.com/c3-oss/prosa/pkg/importer"
"github.com/stretchr/testify/require"
)

Expand Down Expand Up @@ -37,3 +39,41 @@ func TestSuppressedWarningsTextSingular(t *testing.T) {
require.True(t, strings.Contains(got, "1 diagnostic log suppressed"))
require.True(t, strings.Contains(got, "see it"))
}

// TestQuietSyncLoggingScopesImporterWarnings asserts the interactive path
// routes importer diagnostics into the same counter the pusher uses, so
// the summary's Warnings row covers both phases — and that nothing leaks
// onto the process-global slog default, which issue #154 took out of this
// code path for good.
func TestQuietSyncLoggingScopesImporterWarnings(t *testing.T) {
var global bytes.Buffer
prev := slog.Default()
slog.SetDefault(slog.New(slog.NewTextHandler(&global, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })

var count atomic.Int64
push := &pusher{}
got := quietSyncLogging(importer.ImportOptions{Overwrite: true, Profile: "mz"}, push, &count)

require.NotNil(t, got.Logger)
require.Same(t, got.Logger, push.logger, "one counter must feed both phases")
require.True(t, got.Overwrite, "the value copy must preserve the rest of opts")
require.Equal(t, "mz", got.Profile)

got.Logger.Warn("codex: malformed JSONL lines skipped", "count", 2)
push.log().Warn("reconcile: catching up")

require.Equal(t, int64(2), count.Load())
require.Empty(t, global.String(), "sync diagnostics must not reach the global slog default")
}

// TestQuietSyncLoggingWithoutPusher covers the server-less sync (no
// auth.json): importers still get the scoped logger.
func TestQuietSyncLoggingWithoutPusher(t *testing.T) {
var count atomic.Int64
got := quietSyncLogging(importer.ImportOptions{}, nil, &count)

require.NotNil(t, got.Logger)
got.Logger.Warn("counted")
require.Equal(t, int64(1), count.Load())
}
5 changes: 4 additions & 1 deletion internal/importers/claudecode/importer.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"github.com/c3-oss/prosa/internal/importers/importerutil"
"github.com/c3-oss/prosa/internal/paths"
"github.com/c3-oss/prosa/pkg/importer"
"github.com/c3-oss/prosa/pkg/session"
)

// Name is the agent identifier used in session rows and CLI output.
Expand Down Expand Up @@ -46,7 +47,9 @@ func (i *Importer) Import(ctx context.Context, jsonlPath string, sink importer.S
Opts: opts,
Hash: importerutil.HashAndSize,
PeekID: peekSessionID,
Parse: parseSession,
Parse: func(ctx context.Context, p string) (session.Session, []session.Turn, []session.ToolUsage, session.UsageState, error) {
return parseSession(ctx, p, importerutil.Logger(opts))
},
PreserveRaw: func(srcPath, sessionID string, startedAt time.Time) (string, error) {
return importerutil.PreserveRaw(Name, sessionID, ".jsonl", startedAt, srcPath)
},
Expand Down
36 changes: 36 additions & 0 deletions internal/importers/claudecode/importer_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"bytes"
"context"
"encoding/json"
"log/slog"
"os"
"path/filepath"
"strings"
Expand Down Expand Up @@ -812,3 +813,38 @@ func TestImportStripsStdoutWrapperAndANSI(t *testing.T) {
require.Equal(t, "user", turns[0].Role)
require.Equal(t, "now refactor the sync logic", turns[0].Content)
}

// TestImportAggregatesTornAndBlankLines mirrors the codex case: a torn
// append leaves a line cut mid-token, and the session must survive it
// with one warning for the whole file.
func TestImportAggregatesTornAndBlankLines(t *testing.T) {
ctx := context.Background()
t.Setenv("PROSA_HOME", filepath.Join(t.TempDir(), "prosa-home"))
src := writeFixtureSmall(t, t.TempDir())

clean, err := os.ReadFile(src)
require.NoError(t, err)

torn := []byte(`{"type":"user","sessionId":"` + fixtureSessionID + `","message":{"role":"use` + "\n" +
`{"type":"assistant","sessionId":"` + fixtureSessionID + "\n" +
"\n \t\n")
require.NoError(t, os.WriteFile(src, append(clean, torn...), 0o644))

var logs bytes.Buffer
sink := newSink()
res, err := New().Import(ctx, src, sink, importer.ImportOptions{
Logger: slog.New(slog.NewTextHandler(&logs, nil)),
})
require.NoError(t, err)
require.False(t, res.Skipped)

s := sink.Sessions[fixtureSessionID]
require.Equal(t, fixtureSessionID, s.ID)
require.NotNil(t, s.Usage)
require.Equal(t, int64(135), s.Usage.TotalTokens)
require.Len(t, sink.Turns[fixtureSessionID], 3)

require.Equal(t, 1, strings.Count(logs.String(), "level=WARN"))
require.Contains(t, logs.String(), "claude-code: malformed JSONL lines skipped")
require.Contains(t, logs.String(), "count=2")
}
27 changes: 22 additions & 5 deletions internal/importers/claudecode/parse.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package claudecode

import (
"bufio"
"bytes"
"context"
"encoding/json"
"errors"
Expand Down Expand Up @@ -93,7 +94,7 @@ func peekSessionID(path string) (string, error) {
// parseSession streams the JSONL once and returns the projected metadata
// plus a UsageState classifying whether the transcript carried any usage
// event (and if so, whether totals were positive).
func parseSession(ctx context.Context, path string) (session.Session, []session.Turn, []session.ToolUsage, session.UsageState, error) {
func parseSession(ctx context.Context, path string, log *slog.Logger) (session.Session, []session.Turn, []session.ToolUsage, session.UsageState, error) {
f, err := os.Open(path)
if err != nil {
return session.Session{}, nil, nil, session.UsageStateUnknown, err
Expand All @@ -114,6 +115,10 @@ func parseSession(ctx context.Context, path string) (session.Session, []session.
modelSet bool
firstPromptSet bool
line int

malformed int
malformedAt int
malformedErr error
)

// Subagent transcripts live under `<parent-uuid>/subagents/` and
Expand All @@ -135,10 +140,17 @@ func parseSession(ctx context.Context, path string) (session.Session, []session.
return session.Session{}, nil, nil, session.UsageStateUnknown, err
}

raw := sc.Bytes()
if len(bytes.TrimSpace(raw)) == 0 {
continue
}

var r rawRecord
if err := json.Unmarshal(sc.Bytes(), &r); err != nil {
slog.Warn("claude-code: malformed JSONL line skipped",
"path", path, "line", line, "err", err)
if err := json.Unmarshal(raw, &r); err != nil {
malformed++
if malformedAt == 0 {
malformedAt, malformedErr = line, err
}
continue
}

Expand Down Expand Up @@ -228,13 +240,18 @@ func parseSession(ctx context.Context, path string) (session.Session, []session.

if err := sc.Err(); err != nil {
if errors.Is(err, bufio.ErrTooLong) {
slog.Warn("claude-code: JSONL line exceeded 16 MiB scan buffer; partial session",
log.Warn("claude-code: JSONL line exceeded 16 MiB scan buffer; partial session",
"path", path, "line", line+1)
} else {
return session.Session{}, nil, nil, session.UsageStateUnknown, fmt.Errorf("scan %s: %w", path, err)
}
}

if malformed > 0 {
log.Warn("claude-code: malformed JSONL lines skipped",
"path", path, "count", malformed, "first_line", malformedAt, "err", malformedErr)
}

tools := make([]session.ToolUsage, 0, len(toolCounts))
for name, count := range toolCounts {
tools = append(tools, session.ToolUsage{Name: name, Count: count})
Expand Down
5 changes: 4 additions & 1 deletion internal/importers/codex/importer.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"github.com/c3-oss/prosa/internal/importers/importerutil"
"github.com/c3-oss/prosa/internal/paths"
"github.com/c3-oss/prosa/pkg/importer"
"github.com/c3-oss/prosa/pkg/session"
)

// Name is the agent identifier used in session rows and CLI output.
Expand Down Expand Up @@ -48,7 +49,9 @@ func (i *Importer) Import(ctx context.Context, jsonlPath string, sink importer.S
Opts: opts,
Hash: importerutil.HashAndSize,
PeekID: peekSessionID,
Parse: parseSession,
Parse: func(ctx context.Context, p string) (session.Session, []session.Turn, []session.ToolUsage, session.UsageState, error) {
return parseSession(ctx, p, importerutil.Logger(opts))
},
PreserveRaw: func(srcPath, sessionID string, startedAt time.Time) (string, error) {
return importerutil.PreserveRaw(Name, sessionID, ".jsonl", startedAt, srcPath)
},
Expand Down
Loading
Loading