UN-4008 [FEAT] Add the unstract CLI to run extractions and API deployments from the terminal

.github/workflows/ci.yml

@@ -0,0 +1,20 @@
+name: ci
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  # Offline by design: no network, no credentials, sub-second. Live round trips
+  # are a manual pre-release step, not a per-PR gate.
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv venv --python 3.12
+      - run: uv pip install -e '.[dev]'
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+      - run: uv run pytest -q

.gitignore

@@ -0,0 +1,7 @@
+.venv/
+__pycache__/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+dist/
+build/

README.md

@@ -0,0 +1,135 @@
+# unstract-cli
+
+`unstract` — one CLI for the Unstract suite: extract a document with
+LLMWhisperer, run it through a Document Studio API deployment, get structured
+JSON back. It also clones one organization's resources into another.
+
+```bash
+uv tool install git+https://github.com/Zipstack/unstract-cli
+unstract config init
+unstract config doctor
+```
+
+Or run it without installing: `uvx --from git+https://github.com/Zipstack/unstract-cli unstract --discover groups`.
+
+## Output
+
+`unstract` prints a table by default — in a terminal and in a pipe alike, so
+what you see while trying something is what a script sees running it.
+
+**Parsing anything? Pass `-o json`.** stdout then carries exactly one envelope,
+on success and on failure alike:
+
+```json
+{"ok": true, "data": {...}, "error": null, "meta": {"contract_version": 1}}
+```
+
+`-o json` output depends on nothing but the command and its arguments — not the
+terminal, not the config, not the environment. `-o raw` prints one field
+unwrapped, for piping a document's text somewhere else. Diagnostics, warnings
+and progress always go to stderr.
+
+Consuming the JSON: ignore fields you do not recognise, and refuse a
+`meta.contract_version` above the one you were written against. `unstract
+--discover full` publishes the whole contract alongside every command and flag.
+
+If a coding agent is driving (detected from the environment it sets), the
+*default* becomes json. `--agent yes|no` forces that either way, and an explicit
+`-o` always wins over both.
+
+Failures exit non-zero with a stable code. The codes are this CLI's own
+convention, not a service's — they are the `ExitCode` enum in
+`core/errors.py`, and `--discover full` publishes the table so a caller does not
+have to copy it:
+
+| Code | Meaning |
+|------|---------|
+| 0 | success |
+| 1 | generic failure |
+| 2 | usage error |
+| 3 | authentication failed |
+| 4 | not found |
+| 5 | validation failed |
+| 6 | rate limited |
+| 7 | timed out (the job handle is in the error payload — resume, do not resubmit) |
+| 8 | server error |
+| 9 | result already consumed (one-shot read; use `--save` next time) |
+| 10 | the result was read but could not be saved — it is in `error.details` |
+| 130 | interrupted (128 + SIGINT) — the user stopped it, not a failure |
+
+## Configuration
+
+`~/.unstract/config.toml`, or a project-local `.unstract.toml` found by upward
+search, or `$UNSTRACT_CONFIG`, or `--config`. Every setting resolves
+**flag > env > profile > built-in default**, and the CLI is fully usable with no
+config file at all. The flag tier is the connection options on each product
+group — `unstract docstudio --base-url … --org-id … deployment run …`, and
+`--base-url`/`--api-key` on `whisper` — which override the profile for that one
+invocation without writing anything.
+
+```toml
+default_profile = "cloud-us"
+
+[profiles.cloud-us.llmwhisperer]
+base_url = "https://llmwhisperer-api.us-central.unstract.com/api/v2"
+api_key = "env:LLMWHISPERER_API_KEY"
+
+[profiles.cloud-us.docstudio]
+base_url = "https://us-central.unstract.com"
+org_id = "org_ABC123"
+api_key = "env:UNSTRACT_DEPLOYMENT_KEY"
+
+[profiles.cloud-us.deployments.invoices]
+api_name = "invoice-parser"
+```
+
+One `api_key` on the `docstudio` block covers every alias under it: a key minted
+under **Settings → API Key Manager** authenticates every API deployment in the
+organisation, so an alias normally carries only its `api_name`. Give an alias its
+own `api_key` when its deployment has a separate key of its own.
+
+Get an LLMWhisperer key from the LLMWhisperer console; a deployment key is shown
+on the API deployment's own page in the Unstract UI, and an organisation-wide one
+under Settings → API Key Manager. `config init` also writes an
+`onprem-example` profile as a shape to copy for a self-hosted install — its host
+is a placeholder, and only the *active* profile is ever resolved.
+
+A credential can be written into the file literally, but `env:VAR_NAME`
+indirection is what `config init` writes and what the examples use: the file
+then records where a secret lives rather than the secret itself, and stays safe
+to copy or commit. Either way the file is created `0600`, and `config doctor`
+warns when its mode is wider than that.
+
+`unstract config doctor` reports where each setting resolved from — including
+whether an `env:` reference is actually set in the current process — without
+echoing any value. It exits non-zero when one of its own checks failed, so a
+setup script can branch on it.
+
+A project-local `.unstract.toml` **found by upward search** may not supply
+`api_key` or `base_url`. Those are ignored, with a warning; everything else in it
+— profile selection, `org_id`, deployment aliases — applies as usual. A checkout
+you did not write is not trusted to name the host your key is sent to. Name the
+file explicitly (`--config` or `$UNSTRACT_CONFIG`) and it is honoured in full.
+
+What that protects is the key and the host, not the routing: `org_id`,
+`api_name` and profile selection stay repo-controllable by design, so a
+project file can still decide *which* deployment a command runs against on a
+host you trust. Read one before you run inside a checkout you did not write.
+
+`clone` is the exception, and it is an operator command: a human moving one
+organisation's resources into another, holding two admin Platform keys. It is
+not part of the document-processing path the rest of this CLI wraps, so an agent
+serving a user request should not reach for it unasked. It talks to two
+deployments at once, which no single profile describes, so it takes both
+endpoints as flags and both keys from `UNSTRACT_SRC_PLATFORM_KEY` /
+`UNSTRACT_TGT_PLATFORM_KEY`. It exits 0 when nothing failed, which is not the
+same as everything having moved: oversize and unsupported documents are skipped
+by design, and `data.skipped` counts them.
+
+## Development
+
+```bash
+uv venv && uv pip install -e '.[dev]'
+uv run pytest   # offline; no network, no credentials
+uv run ruff check .
+```

pyproject.toml

@@ -0,0 +1,54 @@
+[project]
+name = "unstract-cli"
+version = "0.1.0"
+description = "Unified, LLM-friendly CLI for the Unstract suite of products"
+readme = "README.md"
+requires-python = ">=3.12"
+
+dependencies = [
+    # Pinned to a major: `--discover` reads the shape of
+    # `click.Parameter.to_info_dict()`.
+    "click>=8.1,<9",
+    # Writing the config file only; reading it uses the stdlib `tomllib`.
+    "tomli-w>=1.0",
+    # Pinned to a commit: the CLI derives its flags and help text from these
+    # clients, so one that moves changes the CLI's surface. Both pins move to
+    # released versions before this ships.
+    "unstract-client @ git+https://github.com/Zipstack/unstract-python-client@a77ef6a",
+    "llmwhisperer-client @ git+https://github.com/Zipstack/llm-whisperer-python-client@7f64caf",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "ruff>=0.6",
+]
+
+[project.scripts]
+unstract = "unstract_cli.__main__:main"
+# `unstract-client` installs an `unstract` script too, so whichever package is
+# installed last wins that name; this one always reaches this CLI.
+unstract-cli = "unstract_cli.__main__:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+# Needed for the git-pinned clients above.
+[tool.hatch.metadata]
+allow-direct-references = true
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/unstract_cli"]
+
+[tool.ruff]
+line-length = 90
+target-version = "py312"
+src = ["src", "tests"]
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "N", "UP", "B", "C4", "SIM"]
+ignore = ["E501"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

src/unstract_cli/__init__.py

@@ -0,0 +1,3 @@
+"""Unstract CLI."""
+
+__version__ = "0.1.0"

src/unstract_cli/__main__.py

@@ -0,0 +1,87 @@
+"""Entry point: turns every failure into an envelope plus a stable exit code.
+
+Click's own error handling is bypassed on purpose. By default it prints prose to
+stderr and exits 1 or 2 with nothing on stdout, which leaves a caller parsing
+stdout with an empty stream and no way to tell a usage error from a server
+failure.
+"""
+
+from __future__ import annotations
+
+import sys
+
+import click
+
+from unstract_cli.app import cli
+from unstract_cli.config import ConfigError
+from unstract_cli.core.errors import CLIError, ExitCode
+from unstract_cli.core.output import AgentMode, OutputFormat, emit_error, resolve_format
+
+
+def _option_from_argv(argv: list[str], *spellings: str) -> str | None:
+    """Best-effort read of one option before Click has parsed anything.
+
+    A failure during parsing still has to be rendered, and the parsed context
+    does not exist yet at that point.
+    """
+    for i, arg in enumerate(argv):
+        for spelling in spellings:
+            if arg.startswith(f"{spelling}="):
+                return arg.split("=", 1)[1]
+            if arg == spelling and i + 1 < len(argv):
+                return argv[i + 1]
+    return None
+
+
+def _format_from_argv(argv: list[str]) -> OutputFormat:
+    """Resolve the format the same way the parsed run would."""
+    try:
+        return resolve_format(
+            _option_from_argv(argv, "--output", "-o"),
+            _option_from_argv(argv, "--agent") or AgentMode.AUTO,
+        )
+    except ValueError:
+        # An unusable value here is Click's error to report, not ours to guess
+        # around; render the failure in the default and let it through.
+        return resolve_format(None)
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = list(sys.argv[1:] if argv is None else argv)
+    fmt = _format_from_argv(args)
+    try:
+        cli.main(args=args, standalone_mode=False)
+    except CLIError as exc:
+        return int(emit_error(exc, fmt))
+    except ConfigError as exc:
+        return int(emit_error(CLIError(str(exc), ExitCode.USAGE), fmt))
+    except click.UsageError as exc:
+        return int(
+            emit_error(
+                CLIError(exc.format_message(), ExitCode.USAGE, hint="Run with --help."),
+                fmt,
+            )
+        )
+    except OSError as exc:
+        # Not a crash worth a traceback: a full disk or an unwritable path is
+        # the caller's to fix, and they still need a parseable envelope.
+        return int(
+            emit_error(
+                CLIError(str(exc), ExitCode.GENERIC, hint="Check the path and disk."),
+                fmt,
+            )
+        )
+    except (click.Abort, KeyboardInterrupt):
+        # Nothing here prompts, so Click's Abort can only mean an interrupt.
+        return int(
+            emit_error(
+                CLIError("Interrupted.", ExitCode.INTERRUPTED, retryable=True), fmt
+            )
+        )
+    except click.exceptions.Exit as exc:  # --help and --version exit through here
+        return int(exc.exit_code)
+    return int(ExitCode.SUCCESS)
+
+
+if __name__ == "__main__":  # pragma: no cover
+    sys.exit(main())