diff --git a/apps/docs/astro.config.mjs b/apps/docs/astro.config.mjs index 659adee..aa0f04f 100644 --- a/apps/docs/astro.config.mjs +++ b/apps/docs/astro.config.mjs @@ -31,8 +31,8 @@ export default defineConfig({ ], }, { - label: 'CLI Reference', - autogenerate: { directory: 'cli' }, + label: 'Tutorials', + autogenerate: { directory: 'tutorials' }, }, { label: 'Configuration', @@ -54,6 +54,10 @@ export default defineConfig({ label: 'Plugins', autogenerate: { directory: 'plugins' }, }, + { + label: 'CLI Reference', + autogenerate: { directory: 'cli' }, + }, ], }), ...(isDev ? [react()] : []), diff --git a/apps/docs/src/components/Footer.astro b/apps/docs/src/components/Footer.astro index 13159fa..afe17a9 100644 --- a/apps/docs/src/components/Footer.astro +++ b/apps/docs/src/components/Footer.astro @@ -25,12 +25,7 @@ const repo = 'https://github.com/obsessiondb/chkit'; - - - - + diff --git a/apps/docs/src/content/docs/cli/init.md b/apps/docs/src/content/docs/cli/init.md index 95f4f5c..b841d13 100644 --- a/apps/docs/src/content/docs/cli/init.md +++ b/apps/docs/src/content/docs/cli/init.md @@ -1,68 +1,91 @@ --- title: "chkit init" -description: "Scaffold a new chkit project with config and example schema files." +description: "Scaffold a chkit project — config, example schema, dependencies, and an optional database connection." sidebar: order: 2 --- -Creates a `clickhouse.config.ts` configuration file and an example schema file in your project. If either file already exists, it is left untouched. +Scaffolds a chkit project in the current directory: writes a `clickhouse.config.ts` and an example schema file, installs dependencies if the project has none, and — on an interactive run — offers to connect a database. ## Synopsis ``` -chkit init +chkit init [flags] ``` ## Flags -No command-specific flags. See [global flags](/cli/overview/#global-flags). +| Flag | Type | Default | Description | +|------|------|---------|-------------| +| `--connect ` | string | — | Preselect a connection path: `claim`, `account`, `clickhouse`, or `later`. Skips the interactive prompt. | +| `--email ` | string | — | Email for the ObsessionDB `claim` / `account` paths (implies an interactive connect step). | +| `--code ` | string | — | One-time email code, to verify a signup without re-prompting. | +| `--org-name ` | string | — | Override the auto-created ObsessionDB organization name. | +| `--yes`, `-y` | boolean | `false` | Skip the connect prompt and just write files — keeps `init` a silent scaffolder for scripts. | + +See [global flags](/cli/overview/#global-flags). ## Behavior -`chkit init` writes two files relative to the current working directory: +`chkit init` runs three steps in order. Each is idempotent, so re-running on an existing project is safe. + +### 1. Scaffold files + +Writes two files relative to the current working directory, leaving any that already exist untouched: + +1. **`clickhouse.config.ts`** — project config with sensible defaults: `schema: './src/db/schema/**/*.ts'`, `outDir: './chkit'`, `migrationsDir: './chkit/migrations'`, `metaDir: './chkit/meta'`, an empty `plugins` array, and a `clickhouse` block reading from `CLICKHOUSE_URL`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, and `CLICKHOUSE_DB`. +2. **`src/db/schema/example.ts`** — a starter `MergeTree` table named `events` with columns `id` (`UInt64`), `source` (`String`), and `ingested_at` (`DateTime64(3)`). -1. **`clickhouse.config.ts`** — project configuration with sensible defaults: - - `schema: './src/db/schema/**/*.ts'` - - `outDir: './chkit'` - - `migrationsDir: './chkit/migrations'` - - `metaDir: './chkit/meta'` - - `plugins: []` - - `clickhouse` block reading from environment variables (`CLICKHOUSE_URL`, `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_DB`) +### 2. Install dependencies -2. **`src/db/schema/example.ts`** — a sample table definition using `MergeTree` engine with columns `id` (UInt64), `source` (String), and `ingested_at` (DateTime64(3)). +If `@chkit/core` does not already resolve from the project, `init` makes the project runnable: it writes a minimal `package.json` when none exists, then installs `chkit`, `@chkit/core`, and `@chkit/plugin-obsessiondb` as dev dependencies using the detected package manager (`npm`, `pnpm`, `yarn`, or `bun`; defaults to `bun`). This is why `init` works in a brand-new empty folder, not just an existing project. A failed install never aborts `init` — it prints the manual install command and continues. -The command is idempotent — running it again on an existing project does nothing. +### 3. Connect a database + +On a TTY — or when `--connect` / `--email` is passed — `init` runs the ObsessionDB onboarding prompt: + +``` +Claim a free ObsessionDB dev instance email code, ready in seconds +I already have an ObsessionDB account log in and pick a service +I already have a ClickHouse instance connect with env vars +Configure later +``` + +Claiming or logging in registers the `@chkit/plugin-obsessiondb` plugin in your config and writes the selected service to `.chkit/obsessiondb.json`. See [Getting Started with ObsessionDB](/obsessiondb/getting-started/) for each path in detail, including the non-interactive and agent-friendly (`--json`) variants. + +Pass `--yes` to skip this step and just write files. In a non-interactive environment (no TTY) without connect flags, `init` skips onboarding entirely — it writes files and prints static next steps (set `CLICKHOUSE_URL`, edit the schema, then `generate` and `migrate`). When an explicit connect path is requested but cannot complete — for example `--connect claim` with no email, or a wrong code — `init` exits non-zero so scripts can detect the failure. ## Examples -**Initialize a new project:** +**Scaffold and connect interactively:** ```sh chkit init ``` -Output: +**Scaffold only, no prompt (CI / scripts):** +```sh +chkit init --yes ``` -Created clickhouse.config.ts -Created src/db/schema/example.ts -Next steps: - 1. Set CLICKHOUSE_URL (and CLICKHOUSE_USER / CLICKHOUSE_PASSWORD / CLICKHOUSE_DB if needed). - 2. Edit src/db/schema/example.ts to match your data. - 3. Run: bunx chkit generate --name init - 4. Run: bunx chkit migrate --apply +**Preselect the claim path (still prompts for the emailed code):** -Docs: https://chkit.obsessiondb.com/getting-started/add-to-existing-project/ +```sh +chkit init --connect claim --email you@example.com ``` -**Run on an existing project (no changes):** +For a fully scripted claim with no prompts, use the two-step OTP flow in [Getting Started with ObsessionDB](/obsessiondb/getting-started/#non-interactive-setup). -```sh -chkit init -# No output — both files already exist -``` +## Exit codes + +| Code | Meaning | +|------|---------| +| 0 | Success | +| non-zero | An explicitly requested connect path could not complete | ## Related commands +- [Getting Started: add to an existing project](/getting-started/add-to-existing-project/) — the guided first-run flow +- [Getting Started with ObsessionDB](/obsessiondb/getting-started/) — every connect path in detail - [`chkit generate`](/cli/generate/) — generate migrations from your schema after init diff --git a/apps/docs/src/content/docs/getting-started/add-to-existing-project.mdx b/apps/docs/src/content/docs/getting-started/add-to-existing-project.mdx index 593286a..530c271 100644 --- a/apps/docs/src/content/docs/getting-started/add-to-existing-project.mdx +++ b/apps/docs/src/content/docs/getting-started/add-to-existing-project.mdx @@ -29,6 +29,10 @@ Add `chkit` and `@chkit/core` as dev dependencies: Edit `src/db/schema/example.ts` to match the table you actually want before moving on. +:::note +On an interactive run, `chkit init` also offers to connect a database — including claiming a **free ObsessionDB dev instance** with a one-time email code, no local ClickHouse required. This guide wires up a connection with environment variables instead (the steps below); pass `--yes` to skip the prompt. See [Getting Started with ObsessionDB](/obsessiondb/getting-started/) for the connect paths, or the [first-schema tutorial](/tutorials/first-schema/) for an end-to-end walkthrough using the free instance. +::: + ## 3. Generate the initial migration `chkit generate` diffs your schema definitions against the previous snapshot and writes migration SQL into `chkit/migrations/`. Pass `--name` to label the migration file. diff --git a/apps/docs/src/content/docs/obsessiondb/overview.md b/apps/docs/src/content/docs/obsessiondb/overview.md index 68ea321..d0f8aba 100644 --- a/apps/docs/src/content/docs/obsessiondb/overview.md +++ b/apps/docs/src/content/docs/obsessiondb/overview.md @@ -13,7 +13,7 @@ chkit ships a dedicated integration with [ObsessionDB](https://obsessiondb.com), - **One schema, two targets** — write `Shared*` engines once and run them against ObsessionDB as-is, or against regular ClickHouse with the `Shared` prefix stripped automatically. - **Service selection** — list services across your organizations, pick a default per project, and override per command without touching config. - **Remote query execution** — once a service is selected, `chkit query` and other SQL-emitting commands route through the ObsessionDB API instead of a local ClickHouse connection. -- **Remote backfills** — `chkit backfill` can submit jobs to ObsessionDB rather than streaming chunks from your machine. +- **Remote backfills** — `chkit plugin backfill` can submit jobs to ObsessionDB rather than streaming chunks from your machine. ## Install diff --git a/apps/docs/src/content/docs/plugins/backfill.md b/apps/docs/src/content/docs/plugins/backfill.md index 0cc6f96..42f0576 100644 --- a/apps/docs/src/content/docs/plugins/backfill.md +++ b/apps/docs/src/content/docs/plugins/backfill.md @@ -1,10 +1,19 @@ --- title: Backfill Plugin description: Plan, execute, and monitor time-windowed backfill operations with async query submission, concurrent execution, and checkpointed progress. +sidebar: + order: 4 + badge: + text: Alpha + variant: caution --- This document covers practical usage of the optional `backfill` plugin. +:::caution[Alpha] +The `backfill` plugin is in alpha. Its API, configuration, and CLI flags may change between releases. +::: + ## What it does - Builds deterministic, immutable backfill plans that divide a time window into chunks. diff --git a/apps/docs/src/content/docs/plugins/codegen.md b/apps/docs/src/content/docs/plugins/codegen.md index 08c38df..556d88e 100644 --- a/apps/docs/src/content/docs/plugins/codegen.md +++ b/apps/docs/src/content/docs/plugins/codegen.md @@ -1,6 +1,8 @@ --- title: Codegen Plugin description: Generate TypeScript row types, optional Zod schemas, ingestion functions, and runtime migration modules from chkit schema definitions. +sidebar: + order: 2 --- This document covers practical usage of the optional `codegen` plugin. diff --git a/apps/docs/src/content/docs/plugins/overview.md b/apps/docs/src/content/docs/plugins/overview.md index b9029e8..f58a205 100644 --- a/apps/docs/src/content/docs/plugins/overview.md +++ b/apps/docs/src/content/docs/plugins/overview.md @@ -1,6 +1,8 @@ --- title: Plugins Overview description: How chkit plugins work and which official plugins are available. +sidebar: + order: 1 --- Plugins extend chkit with capabilities that don't belong in the core CLI — code generation, schema introspection, data backfill, ObsessionDB integration, and anything else you want to bolt on. They're regular npm packages that you register in `clickhouse.config.ts`: diff --git a/apps/docs/src/content/docs/plugins/pull.md b/apps/docs/src/content/docs/plugins/pull.md index 5fac8d9..2b12d13 100644 --- a/apps/docs/src/content/docs/plugins/pull.md +++ b/apps/docs/src/content/docs/plugins/pull.md @@ -1,6 +1,8 @@ --- title: Pull Plugin description: Introspect live ClickHouse tables, views, and materialized views and generate chkit schema files. +sidebar: + order: 3 --- This document covers practical usage of the optional `pull` plugin. diff --git a/apps/docs/src/content/docs/schema/dsl-reference.md b/apps/docs/src/content/docs/schema/dsl-reference.md index 4e53afe..240505a 100644 --- a/apps/docs/src/content/docs/schema/dsl-reference.md +++ b/apps/docs/src/content/docs/schema/dsl-reference.md @@ -45,7 +45,7 @@ const users = table({ { name: 'id', type: 'UInt64' }, { name: 'email', type: 'String' }, ], - engine: 'MergeTree()', + engine: 'MergeTree', primaryKey: ['id'], orderBy: ['id'], }) @@ -67,7 +67,7 @@ const events = table({ { name: 'received_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, { name: 'status', type: 'String', default: 'pending', comment: 'Event processing status' }, ], - engine: 'MergeTree()', + engine: 'MergeTree', primaryKey: ['id'], orderBy: ['org_id', 'received_at', 'id'], partitionBy: 'toYYYYMM(received_at)', @@ -90,7 +90,7 @@ const events = table({ | `database` | `string` | ClickHouse database name | | `name` | `string` | Table name | | `columns` | `ColumnDefinition[]` | Column definitions (see [Columns](#columns)) | -| `engine` | `string` | Engine clause, e.g. `'MergeTree()'`, `'ReplacingMergeTree(ver)'` | +| `engine` | `string` | Engine clause, e.g. `'MergeTree'`, `'ReplacingMergeTree(ver)'` | | `primaryKey` | `string[]` | Primary key columns | | `orderBy` | `string[]` | ORDER BY columns | @@ -109,7 +109,7 @@ const events = table({ | `plugins` | `TablePlugins` | Per-table plugin configuration (see [Plugin configuration](#plugin-configuration)) | :::note -The `engine` field accepts any string. Common engines include `MergeTree()`, `ReplacingMergeTree()`, `SummingMergeTree()`, `AggregatingMergeTree()`, and `CollapsingMergeTree(sign)`. +The `engine` field accepts any string. Common engines include `MergeTree`, `ReplacingMergeTree`, `SummingMergeTree`, `AggregatingMergeTree`, and `CollapsingMergeTree(sign)`. Empty parentheses are optional for parameterless engines (`'MergeTree'` and `'MergeTree()'` are equivalent), and chkit normalizes them when comparing schemas. ::: :::note @@ -130,6 +130,27 @@ Any ClickHouse type string. Parameterized types like `DateTime64(3)`, `Decimal(1 Primitive types recognized by the DSL type system: `String`, `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt128`, `UInt256`, `Int8`, `Int16`, `Int32`, `Int64`, `Int128`, `Int256`, `Float32`, `Float64`, `Bool`, `Boolean`, `Date`, `DateTime`, `DateTime64`. +#### SQL-standard aliases + +chkit passes the `type` string through to ClickHouse verbatim — it does not rewrite it. ClickHouse itself accepts standard SQL type aliases and stores them as its native types, so a table declared with aliases like `BIGINT` or `TEXT` is created successfully: + +| SQL alias | ClickHouse native type | +|-----------|------------------------| +| `TINYINT` | `Int8` | +| `SMALLINT` | `Int16` | +| `INTEGER` / `INT` | `Int32` | +| `BIGINT` | `Int64` | +| `FLOAT` / `REAL` | `Float32` | +| `DOUBLE` | `Float64` | +| `TEXT` / `VARCHAR` / `CHAR` | `String` | +| `TIMESTAMP` | `DateTime` | + +See the ClickHouse [data types reference](https://clickhouse.com/docs/sql-reference/data-types) for the complete alias list. + +:::caution +**Prefer the native type.** chkit compares column types literally. A column declared as `BIGINT` is created as `Int64`, but [`chkit drift`](/cli/drift/) and [`chkit check`](/cli/check/) then compare your declared `BIGINT` against the live `Int64` and report a permanent `changed_column` drift — failing `chkit check --strict` on every run. The [codegen plugin](/plugins/codegen/) likewise recognizes only native names (see [Type system reference](#type-system-reference)) — an alias raises `codegen_unsupported_type`, or emits `unknown` when `failOnUnsupportedType` is `false`. Use the native ClickHouse type (`Int64`, not `BIGINT`) unless you have a specific reason not to. +::: + ### `nullable` (boolean, optional) When `true`, the column type is wrapped in `Nullable(...)` in the generated SQL. diff --git a/apps/docs/src/content/docs/tutorials/first-schema.md b/apps/docs/src/content/docs/tutorials/first-schema.md new file mode 100644 index 0000000..65fc658 --- /dev/null +++ b/apps/docs/src/content/docs/tutorials/first-schema.md @@ -0,0 +1,187 @@ +--- +title: "Tutorial: your first schema" +description: Build a chkit project from an empty folder — migrate the starter table to a live database, insert and query rows, then evolve the schema. +sidebar: + order: 1 +--- + +A hands-on walkthrough from an empty folder to a live, version-controlled table. You'll scaffold a chkit project, deploy the starter `events` table it generates, put data in it, query it back, then add a column and ship the change — the full chkit loop, start to finish. + +Every step is a real command. Run them in order and you'll end with a working project you can keep building on. + +## What you'll need + +- Node.js 20+ or Bun 1.3.5+ +- An email inbox you can reach + +No ClickHouse to install: this tutorial claims a free ObsessionDB dev instance straight from the CLI. The commands use `bun`; `npm`, `pnpm`, and `yarn` work the same way. + +## 1. Create the project + +Start in a new, empty folder and run `chkit init`: + +```sh +mkdir chkit-tutorial +cd chkit-tutorial +bunx chkit@latest init +``` + +In an empty directory, `init` does the full bootstrap: it writes `clickhouse.config.ts` and a starter schema at `src/db/schema/example.ts`, creates a `package.json`, and installs `chkit`, `@chkit/core`, and `@chkit/plugin-obsessiondb` so the project is runnable. + +It then shows the connect prompt: + +``` +Claim a free ObsessionDB dev instance email code, ready in seconds +I already have an ObsessionDB account log in and pick a service +I already have a ClickHouse instance connect with env vars +Configure later +``` + +Choose **Claim a free ObsessionDB dev instance**, enter your email, and paste the 6-digit code from your inbox. chkit creates a personal organization, provisions a free instance, selects it (written to `.chkit/obsessiondb.json`), and registers the ObsessionDB plugin in your config. + +:::note +Already run your own ClickHouse? Pick **I already have a ClickHouse instance** instead and set `CLICKHOUSE_URL`. Every command below works the same against a direct ClickHouse — see [Getting Started with ObsessionDB](/obsessiondb/getting-started/) for the alternatives. +::: + +Confirm the connection works: + +```sh +bunx chkit query "SELECT 1" +``` + +A single row back means you're connected. + +## 2. Look at the starter schema + +`init` scaffolded a table for you at `src/db/schema/example.ts` — an `events` table that's a good shape for ingesting application or analytics events: + +```ts +// src/db/schema/example.ts +import { schema, table } from '@chkit/core' + +const events = table({ + database: 'default', + name: 'events', + engine: 'MergeTree', + columns: [ + { name: 'id', type: 'UInt64' }, + { name: 'source', type: 'String' }, + { name: 'ingested_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, + ], + primaryKey: ['id'], + orderBy: ['id'], + partitionBy: 'toYYYYMM(ingested_at)', +}) + +export default schema(events) +``` + +A `table()` definition maps directly to a ClickHouse `CREATE TABLE`: a `MergeTree` engine, three columns, ordered by `id`, and partitioned by month. `ingested_at` carries `default: 'fn:now64(3)'`, so the database fills it in automatically. The types here (`UInt64`, `String`, `DateTime64(3)`) are ClickHouse-native; see the [Schema DSL reference](/schema/dsl-reference/) for the full type system and table options. + +Use it as-is for now — you'll change it later. + +## 3. Generate the migration + +`chkit generate` diffs your schema against the previous snapshot and writes migration SQL. There's no snapshot yet, so this produces a `CREATE TABLE`: + +```sh +bunx chkit generate --name create_events +``` + +Open the file it wrote under `chkit/migrations/` — chkit shows you the exact SQL before anything is applied: + +```sql +-- operation: create_table key=table:default.events risk=safe +CREATE TABLE default.events +( + id UInt64, + source String, + ingested_at DateTime64(3) DEFAULT now64(3) +) +ENGINE = MergeTree +PARTITION BY toYYYYMM(ingested_at) +ORDER BY id; +``` + +## 4. Apply it + +```sh +bunx chkit migrate --apply +``` + +This runs the pending migration against the instance you claimed and records it in the migration journal. + +## 5. Verify the table exists + +Check migration state, confirm the live schema matches your code, and look at the table directly: + +```sh +bunx chkit status +bunx chkit check +bunx chkit query "DESCRIBE events" +``` + +`status` lists the applied migration, `check` confirms the database matches your TypeScript definitions, and `DESCRIBE` shows the live columns. + +:::caution +ClickHouse and ObsessionDB DDL is eventually consistent — it isn't instant. If `status` or `check` runs immediately after `migrate --apply`, give it a moment and re-run so it doesn't race the cluster. +::: + +## 6. Insert and query rows + +The table is empty. Put a couple of rows in with `chkit query` — `ingested_at` is left out, so the database fills it from its default: + +```sh +bunx chkit query "INSERT INTO events (id, source) VALUES (1, 'web'), (2, 'mobile')" +``` + +Then read them back: + +```sh +bunx chkit query "SELECT count() FROM events" +bunx chkit query "SELECT id, source, ingested_at FROM events ORDER BY id" +``` + +You now have a schema in code and matching data in a live database. + +## 7. Evolve the schema + +Schemas change. Add a `level` column to the `events` table in `src/db/schema/example.ts`: + +```ts + columns: [ + { name: 'id', type: 'UInt64' }, + { name: 'source', type: 'String' }, + { name: 'level', type: 'String' }, + { name: 'ingested_at', type: 'DateTime64(3)', default: 'fn:now64(3)' }, + ], +``` + +Generate a migration for the change and review it — this time it's an `ALTER TABLE`, not a recreate: + +```sh +bunx chkit generate --name add_level_column +``` + +```sql +-- operation: add_column key=table:default.events risk=safe +ALTER TABLE default.events ADD COLUMN level String AFTER source; +``` + +Apply it and confirm the column landed: + +```sh +bunx chkit migrate --apply +bunx chkit check +bunx chkit query "DESCRIBE events" +``` + +`check` passes again, and `DESCRIBE` now lists `level`. That's the whole chkit loop: **edit the schema → `generate` → review the SQL → `migrate` → verify** — repeat it for every change from here on. + +## Where to next + +- [The CLI reference](/cli/overview/) — every command and flag used above +- [Schema DSL reference](/schema/dsl-reference/) — columns, engines, views, and materialized views +- [Configuration](/configuration/overview/) — what `clickhouse.config.ts` controls +- [Getting Started with ObsessionDB](/obsessiondb/getting-started/) — other ways to connect, and non-interactive setup +- [CI/CD integration](/guides/ci-cd/) — run `generate`, `migrate`, and `check` in a pipeline diff --git a/apps/docs/src/styles/custom.css b/apps/docs/src/styles/custom.css index 11e6f4d..97f040c 100644 --- a/apps/docs/src/styles/custom.css +++ b/apps/docs/src/styles/custom.css @@ -1281,6 +1281,14 @@ kbd { .chk-gh-star-icon { color: var(--accent); } +/* In the mobile burger menu the badge renders inside `.sidebar-content`, whose + `.sidebar-content a { display: block }` rule (higher specificity than the bare + `.chk-gh-stars` selector) would override the flex layout — collapsing the pill + to a block and top-aligning its icons against the border. Re-assert the flex + layout with enough specificity to win. */ +.sidebar-content .chk-gh-stars { + display: inline-flex; +} /* ════════════════════════════════════════════════════ Site footer (brand + link columns)