Skip to content

feat: add OpenAI-compatible POST /v1/audio/transcriptions endpoint - #234

Open
Kartvya69 wants to merge 1 commit into
iBUHub:mainfrom
Kartvya69:feat/audio-transcriptions-140
Open

feat: add OpenAI-compatible POST /v1/audio/transcriptions endpoint#234
Kartvya69 wants to merge 1 commit into
iBUHub:mainfrom
Kartvya69:feat/audio-transcriptions-140

Conversation

@Kartvya69

Copy link
Copy Markdown

Closes #140

What

Exposes POST /v1/audio/transcriptions by wrapping Gemini's audio input capability into an OpenAI-compatible transcription API, so tools hardcoded to the OpenAI endpoint (e.g. OpenWhispr) work unmodified.

How

  • Minimal hand-rolled multipart/form-data parser (RFC 7578, binary-safe) — no new dependencies
  • Extracts the uploaded audio file and converts it to Gemini inlineData
  • Maps OpenAI request fields where possible:
    • model → Gemini model (suffix parsing reused)
    • prompt → guiding context for the transcription
    • language → language hint
    • response_format → output encoding
    • temperature noted as out of scope for prompted transcription
  • Custom transcription system prompt instructing the model to return only the transcription text
  • Response formats:
    • json (default) → {"text": "..."}
    • text → plain text
    • verbose_json → OpenAI verbose schema (duration, language, segments[] with start/end/text/...) — segment placeholder fields (tokens, avg_logprob, ...) are zeroed since prompted transcription cannot produce them
    • srt / vtt → valid subtitle output
  • Timestamps: for verbose_json/srt/vtt an extra instruction asks the model to emit [MM:SS.mmm -> MM:SS.mmm] text lines, which are parsed into segments. If the model omits timestamps, falls back to a single segment spanning the audio
  • Audio duration estimation (MP3 frame walk incl. VBR + ID3v2 skip, WAV header, Ogg EOS page) for fallback end times — noted in-code as a ceiling; swap for a real probe if exact durations matter
  • Reuses the existing retry / account-switch / request-tracking pipeline

Out of scope (as discussed in #140)

Streaming transcription, diarization. Timestamps come from model prompting, not a decoder — good but not sample-exact.

Testing

Tested end-to-end against a live instance (v1.3.5) with gemini-3.7-flash on a short sound-effect MP3 (~1.8s):

  • json{"text":"Aaaaah!"}
  • textAaaaah!
  • verbose_json{"duration":1.776,"segments":[{"start":0,"end":1.813,"text":"Fah!"}],...}
  • srt / vtt → valid timestamped subtitle output
  • Missing file field → 400 invalid_request_error

Transcribed wording varies between runs on ambiguous sound-effect audio (model interpretation), which matches expected prompted-transcription behavior.

…BUHub#140)

Wrap Gemini audio input into an OpenAI transcription API:

- Hand-rolled multipart/form-data parser (no new dependencies, binary-safe)
- Maps OpenAI fields: file -> inlineData, prompt/language -> prompt text
- Response formats: json, text, verbose_json (OpenAI segment schema),
  srt, vtt with model-prompted [MM:SS.mmm -> MM:SS.mmm] timestamps
- Audio duration estimation (MP3 frame walk incl. VBR, WAV, Ogg) for
  fallback segment end times
- Fallback to single full-length segment when model omits timestamps
- Reuses existing retry / account-switch / request-tracking pipeline

Tested end-to-end against a live instance with gemini-3.7-flash on a
short sound-effect MP3 in all five response formats.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feedback]: Expose POST /v1/audio/transcriptions

1 participant