feat: make the SQL engine pluggable and reconstruct types from schema - #21
Open
mpicciolli wants to merge 9 commits into
Open
feat: make the SQL engine pluggable and reconstruct types from schema#21mpicciolli wants to merge 9 commits into
mpicciolli wants to merge 9 commits into
Conversation
Decouples SqlDatabase/SqlEngine from sql.js so any compatible backend can be plugged in (e.g. node:sqlite, better-sqlite3), with samples/ demonstrating both. Also improves column type reconstruction from the normalized schema. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the library’s public surface so cdbToSql/sqlToCdb can operate against a minimal, engine-agnostic SqlEngine/SqlDatabase interface instead of being coupled to sql.js, and adds Node sample adapters demonstrating alternative SQLite backends.
Changes:
- Introduces new public
SqlEngine,SqlDatabase,SqlExecResult, andSqlValuetypes and updates call sites/tests to use them. - Adjusts
sqlToCdbmetadata probing to handle engines that return empty result sets differently. - Adds two Node samples showing
node:sqliteandbetter-sqlite3engine adapters, and updates README documentation accordingly.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| test/roundtrip.test.ts | Updates round-trip test typing to use SqlEngine and removes unnecessary casts. |
| test/mocks/mockSqlDatabase.ts | Simplifies mocks to target the new minimal SqlDatabase interface. |
| test/cdbToSql.test.ts | Updates SQL engine mocks/types to the new engine-agnostic interfaces. |
| src/types.ts | Defines the new public SQL engine/database abstraction types. |
| src/sqlToCdb.ts | Makes DB_STRUCTURE probing more robust against empty/absent result sets. |
| src/index.ts | Re-exports new SQL abstraction types from the package root. |
| src/cdbToSql.ts | Updates cdbToSql signature to accept SqlEngine and relies on new database typing. |
| samples/node-sqlite-engine/README.md | Documents using a Node node:sqlite-backed engine adapter sample. |
| samples/node-sqlite-engine/index.js | Sample script converting a .cdb using the node:sqlite engine adapter. |
| samples/node-sqlite-engine/engine.js | Implements the node:sqlite adapter to SqlEngine/SqlDatabase. |
| samples/node-better-sqlite3-engine/README.md | Documents using a better-sqlite3 engine adapter sample. |
| samples/node-better-sqlite3-engine/package.json | Adds better-sqlite3 dependency for the sample. |
| samples/node-better-sqlite3-engine/index.js | Sample script converting a .cdb using the better-sqlite3 engine adapter. |
| samples/node-better-sqlite3-engine/engine.js | Implements the better-sqlite3 adapter to SqlEngine/SqlDatabase. |
| README.md | Updates API reference and adds guidance/sample links for swapping SQLite engines. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export type ColumnData = Array<string | number | boolean>; | ||
|
|
||
| export interface SqlDatabase extends Database {} | ||
| export type SqlValue = string | number | Uint8Array | null; |
Comment on lines
203
to
207
| - **`SQL`** — a `SqlEngine` (the module returned by `initSqlJs()` satisfies this out of the box). | ||
| - **returns** — a `SqlDatabase` with the CDB tables loaded. | ||
| - **`options.normalize`** — `boolean` (default `false`). Reconstruct PK/FK constraints from PCM naming conventions. See [Normalized schema](#normalized-schema). | ||
| - **`options.indexForeignKeys`** — `boolean` (default `false`). When normalizing, also index every FK column for faster JOINs (roughly doubles the output size). | ||
| - **returns** — a `sql.js` `Database` with the CDB tables loaded. |
|
|
||
| The library's public API is not tied to `sql.js` — it is typed against the minimal, self-contained `SqlEngine`/`SqlDatabase` interfaces exported from the package root, so any object matching that shape works, with no change required to the library: | ||
|
|
||
| - [Node.js — swapping the SQLite engine](./samples/node-sqlite-engine/) — uses the bundled, tested [`node:sqlite`](https://nodejs.org/api/sqlite.html) adapter (`cdb-converter/adapters/node-sqlite`). |
| @@ -0,0 +1,33 @@ | |||
| # Node.js — swapping the SQLite engine | |||
|
|
|||
| This sample shows that `cdb-converter` isn't tied to `sql.js`: it uses the bundled `node:sqlite` adapter (`cdb-converter/adapters/node-sqlite`) as a drop-in replacement for `initSqlJs()`. | |||
|
|
||
| ## Using your own engine | ||
|
|
||
| Any object matching the `SqlEngine`/`SqlDatabase` interfaces exported from `cdb-converter` works in place of `sql.js`. See [`src/adapters/node-sqlite.ts`](../../src/adapters/node-sqlite.ts) for a reference implementation to adapt to other engines (e.g. `better-sqlite3`, `wa-sqlite`). |
| @@ -0,0 +1,35 @@ | |||
| # Node.js — using better-sqlite3 as the SQLite engine | |||
|
|
|||
| This sample shows that `cdb-converter` isn't tied to `sql.js` or the bundled `node:sqlite` adapter: [`engine.js`](./engine.js) is a small hand-written `SqlEngine`/`SqlDatabase` adapter wrapping [`better-sqlite3`](https://github.com/WiseLibs/better-sqlite3), used as a drop-in replacement for `initSqlJs()`. | |||
Comment on lines
+24
to
+26
| function DatabaseEngine(data) { | ||
| return wrap(new Database(data ?? ":memory:")); | ||
| } |
…re logic - Removed the node-better-sqlite3-engine sample and its package.json. - Updated node-cdb-to-sql and node-sql-to-cdb samples to use better-sqlite3 engine instead of sql.js. - Deleted the node-sqlite-engine sample and its associated files. - Introduced better-sqlite3 engine implementation in src/engines/better-sqlite3.ts. - Added sql-js engine creation function in src/engines/sql-js.ts for potential future use. - Updated CLI to utilize better-sqlite3 engine for database operations. - Adjusted tests to validate functionality with better-sqlite3 engine.
…ggable-sql-engine
…ggable-sql-engine
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SqlDatabase/SqlEnginefromsql.js(previously typed viasql.js'sDatabase) so any compatible SQL backend can be plugged insamples/node-sqlite-engineandsamples/node-better-sqlite3-enginedemonstratingnode:sqliteandbetter-sqlite3as drop-in engines