Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
{/* prettier-ignore */}
<Price price="0.053" /> per hour for each configured pipeline. <Price price="0.60" /> per Gigabyte of initial sync data. <Price price="3.00" /> per Gigabyte of ongoing replication data.
<Price price="0.053" /> per hour for each configured pipeline. <Price price="0.60" /> per Gigabyte of data processed during initial sync. <Price price="3.00" /> per Gigabyte of data processed during ongoing replication.

| Plan | Configured Pipeline | Initial Sync Data | Ongoing Replication Data |
| ---------- | -------------------------- | ----------------------------- | ----------------------------- |
| Free | - | - | - |
| Pro | <Price price="0.053" />/hr | <Price price="0.60" /> per GB | <Price price="3.00" /> per GB |
| Team | <Price price="0.053" />/hr | <Price price="0.60" /> per GB | <Price price="3.00" /> per GB |
| Enterprise | Custom | Custom | Custom |
| Plan | Configured Pipeline | Initial Sync Data Processed | Ongoing Replication Data Processed |
| ---------- | -------------------------- | ----------------------------- | ---------------------------------- |
| Free | - | - | - |
| Pro | <Price price="0.053" />/hr | <Price price="0.60" /> per GB | <Price price="3.00" /> per GB |
| Team | <Price price="0.053" />/hr | <Price price="0.60" /> per GB | <Price price="3.00" /> per GB |
| Enterprise | Custom | Custom | Custom |

**Data processed** is Postgres row data successfully processed by a pipeline and accepted by its destination. It is measured from the logical row data emitted by Postgres for replication, rather than physical table storage or destination-specific encoding.

For a detailed breakdown of how charges are calculated, refer to [Manage Pipeline usage](/docs/guides/platform/manage-your-usage/pipelines).
Original file line number Diff line number Diff line change
Expand Up @@ -142,13 +142,13 @@ Open [**Database > Replication**](/dashboard/project/_/database/replication), cl

`Lost` means Postgres has already removed WAL files that the pipeline's replication slot needed. The pipeline cannot continue from that slot.

You can recreate the pipeline, or open the pipeline's **Advanced settings**, set **Invalidated slot behavior** to **Recreate**, and start the pipeline again. The pipeline resets its saved table-sync state, creates a new replication slot, and replaces each destination table through a new initial sync. This destructive restart is required for consistency because the old slot can no longer provide every change the pipeline missed, and the new initial sync is billed as initial sync data.
You can recreate the pipeline, or open the pipeline's **Advanced settings**, set **Invalidated slot behavior** to **Recreate**, and start the pipeline again. The pipeline resets its saved table-sync state, creates a new replication slot, and replaces each destination table through a new initial sync. This destructive restart is required for consistency because the old slot can no longer provide every change the pipeline missed, and the data processed during the new initial sync is billed again.

See [Slot statuses](/docs/guides/database/replication/pipelines-monitoring#slot-statuses) for all slot states and what to do next.

## Why is a table in error state?

Table errors occur during the initial sync. To recover, click **View pipeline**, find the affected table, and click its restart action. This restarts that table's initial sync from the beginning, deletes its existing destination data, and bills the copied data as new initial sync data.
Table errors occur during the initial sync. To recover, click **View pipeline**, find the affected table, and click its restart action. This restarts that table's initial sync from the beginning, deletes its existing destination data, and bills the successfully processed row data again.

## How to verify replication is working

Expand Down Expand Up @@ -209,7 +209,9 @@ Removing a table from the publication and restarting the pipeline does not delet

## Can data be processed more than once?

Yes. Pipelines uses at-least-once processing. If a destination accepted a batch but the pipeline did not persist its checkpoint before a retry, restart, or interruption, that batch can be processed again.
Yes. Pipelines uses at-least-once processing. Failed destination write attempts that Pipelines retries are not counted. Data is counted only after the destination acknowledges successful processing.

In rare cases, Pipelines can count an acknowledged batch but crash or be interrupted before its replication checkpoint is persisted. Recovery can then process and count the same data again.

BigQuery uses the replicated source primary key and CDC ordering metadata to converge on the current table state. Pipelines does not provide a history of each delivery attempt that you can query or guarantee exactly-once event processing.

Expand Down
4 changes: 2 additions & 2 deletions apps/docs/content/guides/database/replication/pipelines.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ Follow these steps to configure your destination. Each destination has its own s

Leave these settings at their defaults unless you need to tune initial sync speed, latency, or recovery behavior.

Use **Invalidated slot behavior** carefully. If **Recreate** is selected and the pipeline starts after Postgres has invalidated the main replication slot, the pipeline resets its saved table-sync state, creates a new slot, and replaces each destination table through a new initial sync. This destructive restart is required for consistency because the old slot can no longer provide every change the pipeline missed, and the new initial sync is billed as initial sync data.
Use **Invalidated slot behavior** carefully. If **Recreate** is selected and the pipeline starts after Postgres has invalidated the main replication slot, the pipeline resets its saved table-sync state, creates a new slot, and replaces each destination table through a new initial sync. This destructive restart is required for consistency because the old slot can no longer provide every change the pipeline missed, and the data processed during the new initial sync is billed again.

7. Click **Create and start pipeline** to begin replication

Expand Down Expand Up @@ -329,7 +329,7 @@ Pipelines has the following limitations:
- **Replica identity**: Requirements are destination-specific. Updates and deletes need enough row identity to apply safely. See [BigQuery source table requirements](/docs/guides/database/replication/bigquery#source-table-requirements) for the supported modes.
- **Schema changes**: Currently in beta and limited to BigQuery
- **No user-defined transformations**: Pipelines performs destination-compatible type and name mapping, but doesn't run custom transformations
- **At-least-once processing**: A retry, restart, or interruption can cause a batch to be processed again. BigQuery uses primary-key-based CDC to converge on the current table state. See [Can data be processed more than once?](/docs/guides/database/replication/pipelines-faq#can-data-be-processed-more-than-once) for details.
- **At-least-once processing**: Failed destination write attempts that Pipelines retries are not counted. In rare cases, Pipelines can count an acknowledged batch but crash or be interrupted before its replication checkpoint is persisted. Recovery can then process and count that batch again. BigQuery uses primary-key-based CDC to converge on the current table state. See [Can data be processed more than once?](/docs/guides/database/replication/pipelines-faq#can-data-be-processed-more-than-once) for details.

Destination-specific limitations, such as BigQuery's row size limits, are documented in each destination guide.

Expand Down
35 changes: 31 additions & 4 deletions apps/docs/content/guides/platform/manage-your-usage/pipelines.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,41 @@ title: 'Manage Pipelines usage'

## What you are charged for

You are charged for configured pipelines, initial sync data, and ongoing replication data. Pipelines are charged by the hour for as long as they are configured, including while they are stopped.
You are charged for configured pipelines and pipeline data processed. Data processed is billed at different rates during initial sync and ongoing replication. Pipelines are charged by the hour for as long as they are configured, including while they are stopped.

- **Pipeline hours** measure how long each pipeline remains configured. Delete a pipeline to end this charge.
- **Initial sync data** is the volume of destination-write data processed when a table is first synchronized or synchronized again.
- **Ongoing replication data** is the volume of destination-write data processed for subsequent database changes. It depends on how much your published data changes, not on the source table size, WAL size, or destination's compressed storage size.
- **Initial sync data processed** is the Postgres row data accepted by the destination when a table is first synchronized or synchronized again.
- **Ongoing replication data processed** is the Postgres row data accepted by the destination for subsequent database changes. It depends on how much your published data changes, not on the source table size, WAL size, or destination's compressed storage size.

Destination-provider charges are separate. For example, Google Cloud can charge for BigQuery ingestion, storage, and CDC compute.

## How data processed is measured

Pipeline data processed is the amount of logical row data emitted by Postgres for replication, successfully processed by a pipeline, and accepted by its destination. It is not based on physical table storage or destination-specific encoding, making usage consistent across destinations.

The measurement includes:

- **Initial sync and resynchronization**: Row data emitted by Postgres COPY.
- **Ongoing replication**: Row values Postgres emits for inserts, updates, and deletes. Updates include new row values and any previous identity values Postgres emits. Deletes include the emitted identity values.

Failed destination write attempts that Pipelines retries are not counted. Data is counted only after the destination acknowledges successful processing. In rare cases, Pipelines can count an acknowledged batch but crash or be interrupted before its replication checkpoint is persisted. Recovery can then process and count the same data again.

Data successfully processed again as part of a user-requested resynchronization, table restart, or pipeline reset is also counted again.

### Cost estimates

The Dashboard provides a quick planning estimate of initial sync volume and cost using information already available about your source tables. It is designed to give you a useful indication before initial sync begins without first scanning and encoding all the data that the sync will process.

If an estimate is unavailable, you can still create the pipeline or restart tables.

<Admonition type="note">

Use this estimate as a planning guide rather than an exact quote. The final volume is measured from the data successfully processed during initial sync and can vary based on your published data and filters.

</Admonition>

Actual charges use the logical Postgres row data copied after publication column and row filters and accepted by the destination.

### Usage on your invoice

Usage is shown as "ETL Pipeline Hours", "ETL Copy Backfill Data GB", and "ETL Replicated Data GB" on your invoice.
Expand All @@ -39,7 +66,7 @@ The project has a configured pipeline for the entire month.

### Multiple projects with initial syncs

Multiple projects had a configured pipeline for the entire month and processed initial sync data and ongoing replication data.
Multiple projects had a configured pipeline for the entire month and processed data during initial sync and ongoing replication.

| Line Item | Units | Costs |
| ----------------------------------- | --------- | ----------------------------- |
Expand Down
4 changes: 2 additions & 2 deletions apps/studio/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Project settings are managed outside of the Dashboard. If you use docker compose
- Type: `chore | fix | feature`
- The branch name is arbitrary — just make sure it summarizes the work.
- When you send a PR to `master`, it will automatically tag members of the frontend team for review.
- Review the [contributing checklists](contributing/contributing-checklists.md) to help test your feature before sending a PR.
- Review the main [contributing guide](../../CONTRIBUTING.md) to help test your feature before sending a PR.
- The Dashboard is under active development. You should run `git pull` frequently to make sure you're up to date.

### Developer Quickstart
Expand All @@ -37,7 +37,7 @@ Project settings are managed outside of the Dashboard. If you use docker compose
> **Supabase internal use:** To develop on Studio locally with the backend services, see the instructions in the [internal `infrastructure` repo](https://github.com/supabase/platform/blob/develop/docs/contributing.md).

```bash
# You'll need to be on Node v20
# You'll need to be on Node v22
# in /studio

## For external contributors
Expand Down
2 changes: 2 additions & 0 deletions apps/studio/compat/next/image.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,8 @@ const Image = forwardRef(function Image(
: style

return (
// alt is spread with rest
// eslint-disable-next-line jsx-a11y/alt-text
<img
{...rest}
ref={(node) => {
Expand Down
2 changes: 2 additions & 0 deletions apps/studio/compat/next/legacy/image.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,8 @@ const Image = forwardRef(function Image(
: undefined

return (
// alt is spread with rest
// eslint-disable-next-line jsx-a11y/alt-text
<img
{...rest}
ref={(node) => {
Expand Down
23 changes: 16 additions & 7 deletions apps/studio/components/grid/components/editor/DateTimeEditor.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ import {
Popover,
PopoverContent,
PopoverTrigger,
Tooltip,
TooltipContent,
TooltipTrigger,
} from 'ui'
import { Input } from 'ui-patterns/DataInputs/Input'
import { TimestampInfo, timestampLocalFormatter } from 'ui-patterns/TimestampInfo'
Expand Down Expand Up @@ -149,13 +152,19 @@ function BaseEditor<TRow, TSummaryRow = unknown>({
Set NULL
</Button>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button
variant="default"
icon={<ChevronDown />}
className="px-1 rounded-l-none border-l-0"
/>
</DropdownMenuTrigger>
<Tooltip>
<TooltipTrigger asChild>
<DropdownMenuTrigger asChild>
<Button
aria-label="Date options"
variant="default"
icon={<ChevronDown />}
className="px-1 rounded-l-none border-l-0"
/>
</DropdownMenuTrigger>
</TooltipTrigger>
<TooltipContent side="bottom">Date options</TooltipContent>
</Tooltip>
<DropdownMenuContent className="w-20" align="end">
<DropdownMenuItem onClick={setToNow}>Set to NOW</DropdownMenuItem>
</DropdownMenuContent>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ export function NumberEditor<TRow, TSummaryRow = unknown>({
onChange={onChange}
onBlur={onBlur}
type="number"
aria-label={`${column.name} value`}
/>
)
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { THRESHOLD_COUNT } from '@supabase/pg-meta'
import { keepPreviousData } from '@tanstack/react-query'
import { useParams } from 'common'
import { AlertCircle, ArrowLeft, ArrowRight, HelpCircle, Loader2 } from 'lucide-react'
import { AlertCircle, ArrowLeft, ArrowRight, HelpCircle } from 'lucide-react'
import { useEffect, useRef, useState } from 'react'
import { Button, Tooltip, TooltipContent, TooltipTrigger } from 'ui'
import { Input } from 'ui-patterns/DataInputs/Input'
Expand Down Expand Up @@ -295,11 +295,7 @@ export const Pagination = ({ enableForeignRowsQuery = true }: PaginationProps) =
</div>

{isLoading ? (
<Button
variant="text"
className="w-7"
icon={<Loader2 size={12} className="animate-spin" />}
/>
<Button variant="text" className="w-7" loading aria-label="Get exact row count" />
) : isError ? (
<Tooltip>
<TooltipTrigger asChild>
Expand All @@ -309,6 +305,8 @@ export const Pagination = ({ enableForeignRowsQuery = true }: PaginationProps) =
className="w-7"
loading={isFetching}
icon={<AlertCircle />}
aria-label="Get exact row count"
disabled
/>
</TooltipTrigger>
<TooltipContent side="top">Failed to retrieve count: {error?.message}</TooltipContent>
Expand All @@ -331,6 +329,7 @@ export const Pagination = ({ enableForeignRowsQuery = true }: PaginationProps) =
className="w-7"
loading={isFetching}
icon={<HelpCircle />}
aria-label="Get exact row count"
onClick={() => {
// Show warning if either NOT a table entity, or table rows estimate is beyond threshold
if (rowsCountEstimate === null || count === -1 || count > THRESHOLD_COUNT) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,28 +59,29 @@ export const BatchRestartDialog = ({
[affectedTables, tableSyncCopy]
)

const initialCopyDescription =
const initialSyncDescription =
copiedTables.length === 0 ? (
<li>
<strong>No table will run an initial copy.</strong> New changes will resume streaming
without backfilling existing source rows. There is no additional initial-copy charge.
<strong>No table will run an initial sync.</strong> Replication will resume with new changes
only, without syncing existing source rows. There is no additional initial sync charge.
</li>
) : copiedTables.length === affectedTables.length ? (
<li>
<strong>
{copiedTables.length === 1 ? 'The table' : `All ${copiedTables.length} tables`} will run
an initial copy.
{copiedTables.length === 1
? 'The table will run its initial sync again.'
: `All ${copiedTables.length} tables will run initial sync again.`}
</strong>{' '}
Existing rows will be copied again from the source and billed in addition to previous
initial copies.
Existing source rows will be synced again. Data successfully processed during this initial
sync is billed again.
</li>
) : (
<li>
<strong>
{copiedTables.length} of {affectedTables.length} tables will run an initial copy.
{copiedTables.length} of {affectedTables.length} tables will run initial sync again.
</strong>{' '}
The remaining tables will resume streaming without a backfill. Copied rows are billed in
addition to previous initial copies.
Existing source rows for those tables will be synced again and billed again. The remaining
tables will resume replication with new changes only.
</li>
)

Expand Down Expand Up @@ -127,7 +128,7 @@ export const BatchRestartDialog = ({
{affectedTables.length === 1 ? '' : 's'} in this pipeline from scratch:
</p>
<ul className="list-disc list-inside space-y-1.5 pl-2">
{initialCopyDescription}
{initialSyncDescription}
<li>
<strong>All downstream data will be deleted.</strong> All replicated data will be
removed.
Expand All @@ -150,7 +151,7 @@ export const BatchRestartDialog = ({
<strong>{affectedTables.length} currently failed tables</strong> from scratch:
</p>
<ul className="list-disc list-inside space-y-1.5 pl-2">
{initialCopyDescription}
{initialSyncDescription}
<li>
<strong>Existing downstream data will be deleted.</strong> Replicated data for
these tables will be removed.
Expand Down
Loading
Loading