Skip to content

Add billing event logging for embedding and reranking token usage#2473

Open
Hazel-Datastax wants to merge 24 commits into
mainfrom
hazel/billing
Open

Add billing event logging for embedding and reranking token usage#2473
Hazel-Datastax wants to merge 24 commits into
mainfrom
hazel/billing

Conversation

@Hazel-Datastax
Copy link
Copy Markdown
Contributor

@Hazel-Datastax Hazel-Datastax commented May 18, 2026

What this PR does:
Add billing event logging for embedding and reranking token usage.

Event definition: https://github.com/riptano/cloud-billing/blob/vectorize-usage-design/docs/usage-event-design/vectorize-usage-events.md

Sample logs:

{"id":"f6716d5a-7ca4-4afa-9826-a14561b8c0b7","timestamp":"2026-05-22T00:43:30.549312Z","product":"serverless","event_type":"nvidia_embeddings_tokens","properties":{"usage":7,"region":"CASSANDRA_REGION","resource_type":"serverless_database","resource_id":"SINGLE-TENANT"}}
{"id":"aede7d63-eb64-4c86-9dc3-a8432c07f120","timestamp":"2026-05-22T00:43:30.549335Z","product":"serverless","event_type":"nvidia_egress_bytes","properties":{"usage":84,"region":"CASSANDRA_REGION","resource_type":"serverless_database","resource_id":"SINGLE-TENANT"}}
{"id":"98cb204d-8f8b-4784-aab6-0b7e96b9695d","timestamp":"2026-05-22T00:43:30.549339Z","product":"serverless","event_type":"nvidia_gpu_plane_egress_bytes","properties":{"usage":21832,"region":"CASSANDRA_REGION","resource_type":"serverless_database","resource_id":"SINGLE-TENANT"}}

Which issue(s) this PR fixes:
Fixes #2137

Checklist

  • Changes manually tested
  • Automated Tests added/updated
  • Documentation added/updated
  • CLA Signed: DataStax CLA

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

📈 Unit Test Coverage Delta vs Main Branch

Metric Value
Main Branch 51.67%
This PR 51.79%
Delta 🟢 +0.11%
✅ Coverage improved!

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Unit Test Coverage Report

Overall Project 51.79% -0.14% 🍏
Files changed 67.39% 🍏

File Coverage
TenantFactory.java 100% 🍏
MeteredEmbeddingProviderWrapper.java 97.57% 🍏
Tenant.java 93.62% 🍏
ApiFeature.java 90.43% 🍏
FindAndRerankOperationBuilder.java 82.16% 🍏
Billing.java 81.33% -18.67% 🍏
ModelType.java 74.55% -1.82% 🍏
BillingEvent.java 61.32% -38.68% 🍏
RerankingTask.java 7.64% -3.89%
FindAndRerankCommandResolver.java 0% -27.66%
DataVectorizerService.java 0% -1.86%
SubdomainTenantResolver.java 0% -27.18%

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

📉 Integration Test Coverage Delta vs Main Branch (dse69-it)

Metric Value
Main Branch 72.77%
This PR 72.59%
Delta 🔴 -0.17%
⚠️ Coverage decreased

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Integration Test Coverage Report (dse69-it)

Overall Project 72.59% -0.32% 🍏
Files changed 28.26%

File Coverage
FindAndRerankCommandResolver.java 100% 🍏
MeteredEmbeddingProviderWrapper.java 97.22% 🍏
ApiFeature.java 90.43% 🍏
FindAndRerankOperationBuilder.java 84.86% 🍏
TenantFactory.java 72.34% 🍏
RerankingTask.java 59.17% -2.5%
Tenant.java 45.39% -35.46%
ModelType.java 38.18% -16.36%
DataVectorizerService.java 23.7% 🍏
Billing.java 22.22% -77.78%
BillingEvent.java 0%
SubdomainTenantResolver.java 0% -27.18%

Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEventLogger.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/ModelProvider.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEventLogger.java Outdated
@Hazel-Datastax Hazel-Datastax marked this pull request as ready for review May 18, 2026 21:48
@Hazel-Datastax Hazel-Datastax requested a review from a team as a code owner May 18, 2026 21:48
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

📉 Integration Test Coverage Delta vs Main Branch (hcd-it)

Metric Value
Main Branch 74.19%
This PR 74.01%
Delta 🔴 -0.18%
⚠️ Coverage decreased

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Integration Test Coverage Report (hcd-it)

Overall Project 74.01% -0.32% 🍏
Files changed 28.26%

File Coverage
FindAndRerankCommandResolver.java 100% 🍏
MeteredEmbeddingProviderWrapper.java 97.22% 🍏
FindAndRerankOperationBuilder.java 90.68% 🍏
ApiFeature.java 90.43% 🍏
TenantFactory.java 72.34% 🍏
RerankingTask.java 59.17% -2.5%
Tenant.java 45.39% -35.46%
ModelType.java 38.18% -16.36%
DataVectorizerService.java 23.7% 🍏
Billing.java 22.22% -77.78%
BillingEvent.java 0%
SubdomainTenantResolver.java 0% -27.18%

Copy link
Copy Markdown
Contributor

@tatu-at-datastax tatu-at-datastax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, some thoughts added.

Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEvent.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEventLogger.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEventLogger.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/ModelProvider.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEventLogger.java Outdated
Copy link
Copy Markdown
Contributor

@amorton amorton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some tweaks as discussed, heading in the right direction

Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEvent.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEvent.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEvent.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEventLogger.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/BillingEventLogger.java Outdated
Comment thread src/main/java/io/stargate/sgv2/jsonapi/service/provider/ModelProvider.java Outdated
Comment on lines +130 to +131
'billing.events':
level: INFO
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for info, maybe make a change. Below is an example of setting a logger category to use a different appender / handler

PLAIN_CONSOLE - defines a new appender / handler that will write using a different format. It is a handler of type console , named PLAIN_CONSOLE, with a different formatt than the default handler called "console"

NOTE: we will want to check with billing peeps what format they want.

this %c %m$n from your code would log

billing.event {"the": json"}

take away %c to just log the json on a line by itself.

the category: io.stargate.sgv2.jsonapi.testbench.reporting.TestBenchConsoleWriter says for this named logger, its level is INFO and it would use the PLAIN_CONSOLE appender / handler, AND importantly DO NOT inherit from above - so do not also use the console appender

 log:
    console:
      format: "%-5p [%t] %d{yyyy-MM-dd HH:mm:ss,SSS} %F:%L - %m%n"
    handler:
      console:
        PLAIN_CONSOLE:
          format: "%m%n"
    category:
      # production log level for this category is DEBUG, way too noisy for tests
      'io.stargate':
      # Fine for JVM mode; if building native images, use index-dependency instead
      'io.quarkus.deployment.steps.ReflectiveHierarchyStep':
        level: ERROR
      'io.stargate.sgv2.jsonapi.testbench.reporting.TestBenchConsoleWriter':
        level: INFO
        handlers:
          - PLAIN_CONSOLE
        use-parent-handlers: false

Copy link
Copy Markdown
Contributor

@amorton amorton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hit wrong button, tweaks as discussed

@tatu-at-datastax
Copy link
Copy Markdown
Contributor

I think issue for this is #2137 fwtw.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Emit model usage events

3 participants