Skip to content

feat: add DuckDB telemetry backend for self-hosted mode (#203)#254

Open
jstojiljkovic wants to merge 20 commits into
mainfrom
feat/duckdb-support
Open

feat: add DuckDB telemetry backend for self-hosted mode (#203)#254
jstojiljkovic wants to merge 20 commits into
mainfrom
feat/duckdb-support

Conversation

@jstojiljkovic

Copy link
Copy Markdown
Collaborator

Adds DuckDB as an opt-in telemetry store for self-hosted deployments (-tags duckdb, CGO required). The main DB stays SQLite (relational/config); only the telemetry DB becomes DuckDB (columnar), written via the Appender API.
Closes #203.

Three mutually-exclusive backends, selected at build time:

Build tag Backend
pgch Postgres + ClickHouse
!pgch && !duckdb dual SQLite
duckdb && !pgch SQLite main + DuckDB telemetry

Benchmarks (CCX13 · €16.49/mo · vs SQLite)

Read: largest fill level the dashboard stays usable (≤ 5 s, no timeout):

Signal SQLite DuckDB
Metrics 1M 10M
Spans 100k 10M
Logs 100k 10M

DuckDB holds 10–100× more queryable rows and flips SQLite's logs cliff.

Write: max sustainable items/sec:

Signal SQLite DuckDB Speedup
Metrics 61,712 85,752 1.4×
Spans 30,508 61,587 2.0×
Logs 4,877 30,480 6.3×

Disk: ~6× smaller than SQLite (auto FSST/Dictionary compression).

@jstojiljkovic jstojiljkovic self-assigned this Jun 30, 2026
@jstojiljkovic jstojiljkovic marked this pull request as ready for review July 1, 2026 14:09
@dusanstanojeviccs

Copy link
Copy Markdown
Collaborator

We have too many separate backends that our folder structure makes no sense anymore.

Instead of doing
repositories/something_duckdb.go
repositories/something_sqlite.go

We should do 3 folders:

repositories/telemetry/duckdb/something.go
repositories/telemetry/sqlite/something.go
repositories/telemetry/clickhouse/something.go

This will allow us to (based on tags) switch between them

We should also update the tags we use when building the projects, right now they are kinda confusing, we should probably do something like
telemetry_ch
telemetry_duckdb
telemetry_sqlite

oltp_sqlite
oltp_pg

This would make tags clear and allow us to implement more backends like victoria metrics/logs/traces easily

@jstojiljkovic

jstojiljkovic commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

We have too many separate backends that our folder structure makes no sense anymore.

Instead of doing repositories/something_duckdb.go repositories/something_sqlite.go

We should do 3 folders:

repositories/telemetry/duckdb/something.go repositories/telemetry/sqlite/something.go repositories/telemetry/clickhouse/something.go

This will allow us to (based on tags) switch between them

We should also update the tags we use when building the projects, right now they are kinda confusing, we should probably do something like telemetry_ch telemetry_duckdb telemetry_sqlite

oltp_sqlite oltp_pg

This would make tags clear and allow us to implement more backends like victoria metrics/logs/traces easily

@dusanstanojeviccs Agreed, separated everything with clean tags, so in the future we can easily add new backends

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adding duckdb for the telemetry data with the appender API for self hosted use

2 participants