Conversation
Reference local Whisper (Whisper.net + Whisper.net.Runtime) and NAudio for on-device audio transcription, and declare the microphone device capability needed for live microphone capture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist the user's chosen Whisper model (defaults to balanced multilingual base), trading transcription speed for accuracy and language coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implement on-device transcription backed by Whisper.net (whisper.cpp), running on the CPU packaged or unpackaged on x64/arm64: - File transcription decodes any Media Foundation audio to 16 kHz mono via NAudio and streams Whisper segments as they are recognized, cancellably. - Selectable model (tiny.en / base.en / base / small) via WhisperModelChoice, read from settings; the shared WhisperFactory reloads when the choice changes. - VAD-gated live transcription (LiveAudioTranscriber) from microphone or system-audio loopback: Silero VAD finds speech regions and only sends a region to Whisper once trailing silence marks the utterance complete, so silence is skipped and phrases cut on natural boundaries. - AudioDebugLog writes a timestamped diagnostic log for the transcription path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route audio files (CLI arg, File > Open, drag/drop) to on-device transcription, streaming each Whisper segment into the editor via a non-blocking, cancellable status bar that preserves already-transcribed text. Add a bottom-bar live-transcription toggle with a right-click menu to pick the capture source (microphone or system audio) and the transcription model, both restarting an active session on change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reference local Whisper (Whisper.net + Whisper.net.Runtime) and NAudio for on-device audio transcription, and declare the microphone device capability needed for live microphone capture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist the user's chosen Whisper model (defaults to balanced multilingual base), trading transcription speed for accuracy and language coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implement on-device transcription backed by Whisper.net (whisper.cpp), running on the CPU packaged or unpackaged on x64/arm64: - File transcription decodes any Media Foundation audio to 16 kHz mono via NAudio and streams Whisper segments as they are recognized, cancellably. - Selectable model (tiny.en / base.en / base / small) via WhisperModelChoice, read from settings; the shared WhisperFactory reloads when the choice changes. - VAD-gated live transcription (LiveAudioTranscriber) from microphone or system-audio loopback: Silero VAD finds speech regions and only sends a region to Whisper once trailing silence marks the utterance complete, so silence is skipped and phrases cut on natural boundaries. - AudioDebugLog writes a timestamped diagnostic log for the transcription path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route audio files (CLI arg, File > Open, drag/drop) to on-device transcription, streaming each Whisper segment into the editor via a non-blocking, cancellable status bar that preserves already-transcribed text. Add a bottom-bar live-transcription toggle with a right-click menu to pick the capture source (microphone or system audio) and the transcription model, both restarting an active session on change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Added EtwShowTranscribe setting to enable/disable audio transcription features. Introduced "Show Transcribe" toggle in BottomBarSettings and new menu items in EditTextWindow for transcription actions, shown only if supported and enabled. Synced UI and settings for transcription visibility, source, and model selection. Updated code generation version in Settings.Designer.cs.
Text-Grab.csproj listed Microsoft.WindowsAppSDK.AI/.Foundation/.Runtime/.WinUI twice at differing versions, producing NuGet warning NU1504. The newer set is not internally consistent (Runtime 2.4.0 expects InteractiveExperiences 2.1.6 but 2.1.3 is resolved), so keep the matched 2.2.x components. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The .NET 10 SDK no longer supports running Microsoft.Testing.Platform test projects through the VSTest target, which broke `dotnet test` in CI. Opt into the new runner via global.json and drop the VSTest-only packages (Microsoft.NET.Test.Sdk, xunit.runner.visualstudio, coverlet.collector). OutputType is now Exe as required by the xunit.v3 MTP entry point. xunit.v3 is pinned to 3.2.2 because Xunit.StaFact, used by every WpfFact test, has no build compatible with xunit.v3 4.0.0 yet. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With Microsoft.Testing.Platform, dotnet test no longer accepts a positional project path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enabled capturing and transcribing both microphone and system audio at once via new LiveCaptureSource option and UI updates. Refactored LiveAudioTranscriber for multi-channel input, mixing, and buffer management. Improved VAD/transcription logic for mixed sources. Enhanced concurrency with semaphore locks. Updated UI feedback and error handling. Added quantization-aware Whisper model file handling.
Replaces WindowsAiUtilities/TextRewriter-based translation with new WinAiTranslator, managing on-device Phi Silica model and LAF unlock. Adds batched and streaming translation, improved error handling, and user feedback. Moves language heuristics to LanguageHeuristics.cs. Project now supports LAF tokens via build/env vars. Updates all translation entry points for new API. Cleans up old logic, adds LimitedAccessFeatureUtilities, and improves UI batching and resource management.
Enables secure injection of Windows AI Language Model (LAF) unlock tokens via LAF_TOKEN and LAF_PUBLISHER_ID environment variables, sourced from GitHub Actions secrets in Release.yml and buildDev.yml, and passed to MSBuild publish steps. Updates Text-Grab.csproj to default LafToken and LafPublisherId from environment variables for local and CI builds. Adds Configuring-LAF-Environment-Variables.md to document setup and usage, stressing token secrecy. Builds without secrets still succeed but omit on-device text AI features.
Refactored all Windows AI language model logic into WinAiLanguageModel for shared creation, availability checks, and inference queuing. Standardized error and result handling with WinAiFailure and WinAiGenerationResult. Updated WinAiTranslator, WindowsAiUtilities, and regex extraction to use the shared model and new result types. Improved error reporting in EditTextWindow and clarified documentation throughout.
Added "Summarize as Meeting Notes" to EditTextWindow with UI and handler. Implemented WinAiMeetingNotes utility for splitting, summarizing, and merging long text using Windows AI (Phi Silica). Added unit tests for text splitting. Refactored response cleaning to shared CleanResponse method in WinAiLanguageModel. Updated docs and comments. Improved error handling and user feedback for meeting notes generation.
When the out-of-process Windows AI runtime connection is lost (e.g., "The RPC server is unavailable"), the code now disposes and recreates the language model, retries the request once, and only fails if restart fails. This logic is centralized in `WinAiLanguageModel.GenerateAsync` and `RunWithModelAsync`, using a new `IsConnectionLost` helper. The unlock cache in `LimitedAccessFeatureUtilities` is reset on restart to avoid caching transient failures. All AI features now use the shared model and recovery logic. Public APIs allow manual model restarts. Exception handling distinguishes recoverable from unrecoverable errors, and error messages are clarified.
Wires the existing "New Window with Selected Text" action to Ctrl+N so it doesn't require the menu, and shows the shortcut in the menu item. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously these AI actions overwrote the source text in place. Now the source stays untouched and the generated result opens in a new Edit Text window, matching the existing "open in new window" pattern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds better Local translation
TranscribeAudioFileAsync now accepts an optional hotWords string applied per-call via Whisper's WithPrompt/WithCarryInitialPrompt, plus GetAudioFileInfo (size/duration, no decode) and GetAudioFileFilter for an upcoming file-picker UI. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
IsPickerMode makes a pick always write into DestinationTextBox and close the window, regardless of EditWindowToggleButton, for callers that want QSL purely as a value picker rather than its ETW insert/clipboard flow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New Capture menu item opens OpenMediaWindow: pick an audio file, add hot words for that transcription (typed directly or picked from Quick Simple Lookup via its new picker mode), and see size/duration/model info before starting. Video is a disabled placeholder for a future, unrelated feature. Hot words are transient per-transcription, not persisted. TranscribeAudioFilesAsync is now internal and takes an optional hotWords parameter to support this. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
LookupItem.ToCSVString writes rows as unquoted "ShortValue,LongValue", so a LongValue containing commas (e.g. a saved hot-words batch) split into more than two cells on reload. The parser rejoined those cells with a space, silently losing the commas. Rejoin with the same delimiter that was used to split when reading a CSV row, so the original value round-trips exactly; tab-delimited rows (typed entry, clipboard paste) keep their existing space-joined behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a persisted NotifyOnTranscriptionComplete setting (default on) and a toggle in the Open Audio / Video window, so long-running file transcriptions can notify the user via toast when they finish instead of requiring the editor window to stay in focus.
Adds a persisted IncludeTimecodesInTranscription setting (default off) and a toggle in the Open Audio / Video window. When enabled, each Whisper segment is prefixed with its start time (e.g. [01:23]) and placed on its own line, using timing data Whisper.net already provides per segment.
GetAllTemplates() previously re-read and re-parsed GrabTemplates.json (and could trigger a slow Settings.Save() to keep the legacy setting and file-backed copy in sync) on every call, including every time a menu that lists templates was opened. This was the main cause of a noticeable delay opening EditTextWindow's "Capture" menu. Templates are now resolved once per process (or since the last write) and cached; GetAllTemplates() returns a fresh list copy each time so structural edits by one caller can't leak into another caller's list before being persisted. Test seams (TestFilePath, TestPreferFileBackedMode) invalidate the cache on assignment so the existing test suite's out-of-band file/settings writes still work.
LoadGrabTemplateMenuItems() ran on every "Capture" menu open, rebuilding the template list even when the user was heading for an unrelated item. Moved it to GrabTemplateMenuItem's own SubmenuOpened event so it only runs when that specific "Use Grab Template..." submenu is opened.
GetCaptureLanguagesAsync() had its own copy of the UI-automation / Windows AI / OCR-engine language enumeration logic, duplicating (and never caching) the work LanguageService.GetAllLanguages() already does. The Windows AI readiness checks in particular are real WinRT/WinAppSDK probes that can be slow, especially the first time in a process, so every newly opened EditTextWindow paid that cost again the first time its "Capture" menu was opened. Now builds from the cached LanguageUtilities.GetAllLanguages() and only adds the Tesseract-specific entries on top, spliced in to preserve the original ordering.
Added two new icon files, Select-Black.ico and Select-White.ico, as content. Updated the .csproj to include these icons for build and packaging, ensuring they are copied to the output directory. Removed any previous references to these icons as needed.
Added TrayIconStyle to App.config and Settings.settings as a user-scoped setting with default "Color". Generated property in Settings.Designer.cs with proper attributes. No changes to other settings.
Added RefreshTrayIconStyle call in App.xaml.cs to update the tray icon when applying the system accent color. Introduced TrayIconStyle enum (Color, Monochrome) in Enums.cs to support multiple tray icon appearance modes.
Users can now choose between color or monochrome tray icons via GeneralSettings. Added ApplyTrayIconStyle() in NotifyIconWindow and RefreshTrayIconStyle() utility for immediate updates. UI and logic updated to persist and apply user preference, with thread-safe icon refresh.
Lets users hide the text label on the live transcription bottom bar button, keeping only the mic/speaker icon. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Added real-time progress bars to EditTextWindow and OpenMediaWindow for audio transcription, reflecting actual progress through the audio clip. Updated AudioTranscriptionUtilities to support progress reporting via IProgress<double>. Improved Cancel button logic to handle in-progress transcriptions and updated UI state management for better user feedback. Status text now shows percentage complete during transcription.
Updated Microsoft.WindowsAppSDK packages: AI to 2.4.4, Foundation to 2.3.9, Runtime to 2.4.0, WinUI to 2.3.6. Added InteractiveExperiences 2.1.6. Removed previous versions. Updated PdfPig from 0.1.15 to 0.1.16.
AudioDebugLog is always on, so in Release it grew without bound at %USERPROFILE%\TextGrab-audio-debug.log, recording user file paths in the profile root. It now writes to %LOCALAPPDATA%\Text-Grab\Logs\audio-debug.log and rolls over into audio-debug.prev.log once past 1 MB, tracking the size from a counter rather than a FileInfo per line. Working set now comes from Environment.WorkingSet instead of a Process.GetCurrentProcess() per line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GetFactoryAsync disposed the shared WhisperFactory whenever the selected model changed, which frees the native model every WhisperProcessor built from it decodes against. The transcription status bar is deliberately non-blocking, so switching models or starting live transcription during a long file transcription could take the process down. The factory is now handed out as a refcounted WhisperFactoryLease: a superseded factory is retired rather than disposed, and freed only once the last lease is returned. File transcription holds a lease across the whole decode; LiveAudioTranscriber holds one for the session and releases it in Cleanup, after the processor. Cleanup itself also raced the chunk timer: Timer.Stop leaves already-queued Elapsed callbacks to run, and one could take the processing gate right after StopAsync's flush released it, enumerating _channels as they were cleared and using _processor as it was disposed. StopAsync now sets a _stopping flag that timed passes check, and tears down while holding the same gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three problems in the audio transcription path of EditTextWindow: Streamed segments go into PassedTextControl, but the finally called SyncTextFromActiveEditor(), which pushes the markdown document or the table back over PassedTextControl.Text -- running the whole transcript through the serializer for nothing. A new SyncActiveEditorFromText() syncs in the right direction, reusing the RefreshSpreadsheetFromText / RefreshMarkdownFromText helpers the TextChanged handler already uses. InsertTranscribedText (live transcription) had the same reversed sync. _transcriptionCts was assigned well before the try, with UI setup and EnsureTranscriptionInsertionPoint() in between; anything throwing there left the field non-null and the guard at the top then blocked every later transcription in that window. The setup now runs inside the try that clears it. A non-cancellation failure raised as the window closes showed a MessageBox owned by a torn-down window, so the dialog is now guarded by IsLoaded -- without falling through to the "transcription complete" toast. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SummarizeParagraph returned "ERROR: ..." strings on failure and the handler passed them straight to OpenTextInNewEditTextWindow, so a failure produced a new editor window whose contents read like a real summary and could be saved as one. It now returns a WinAiGenerationResult, the way ExtractRegex and WinAiMeetingNotes.SummarizeAsync already do, and SummarizeMenuItem_Click mirrors MeetingNotesMenuItem_Click: a window only on success, a dialog on failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ReleaseModel disposed _languageModel without taking _modelLock, which GetModelAsync creates the model under. GrabFrame calls it from the translate toggle and from its cleanup, so it could dispose the model while another window was still building one. The real work now lives in ReleaseModelAsync, which takes the lock (RestartModelAsync delegates to it); the sync ReleaseModel is a fire-and-forget wrapper for callers that cannot await. Cleanup no longer disposes the two semaphores. They hold nothing worth reclaiming at exit, and disposing them throws ObjectDisposedException into any pending WaitAsync -- after which InferenceLease.Dispose skips its Release and the inference queue is wedged for good. It disposes the model directly instead, since the process is exiting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every non-success result from the Translate post-grab action showed a modal. NotNeeded is returned whenever the text already looks like the target language, so an English user grabbing English text got a "Nothing to Translate" dialog after every fullscreen grab, and Unavailable produced a "Translation Failed" dialog on every grab on non-Copilot+ hardware. The text is unchanged in both cases, so the dialog told the user nothing. Both are now suppressed on this automatic path; genuine failures still report. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
translationTimer restarts on every draw, resize and OCR refresh, so a second PerformTranslationAsync could start while the first was still awaiting TranslateBatchAsync. Both mutate the shared progress counters, and the first run streamed its results into word borders indexed against the second run's bordersToTranslate list. An isTranslating guard now keeps one pass at a time. It is set immediately before the try -- nothing above it awaits, so no second pass can slip in first, and a throw during the setup cannot leave the flag stuck on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GetAllTemplates returned a new list holding the same GrabTemplate instances as the cache, so a caller that edited a template and then abandoned the edit left the change in the cache -- and in every other caller list -- for the rest of the process. Reads and writes now hand out deep copies through a CloneTemplate helper, which DuplicateTemplate reuses for its own copy. The `??=` on the cache field was also not atomic, so the cache is now guarded by a lock. Storage itself is still unguarded; the class remark says so instead of claiming nothing is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "mm\:ss" format rendered a 1 h 05 m 30 s file as "05:30". Reuse AudioTranscriptionUtilities.FormatTimecode, which already rolls over to h:mm:ss. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Classification
Introduces a new on-device audio transcription feature and related UI, settings, and infrastructure updates.
PR Summary
Adds Whisper-based audio transcription (file and live audio) with user-selectable models and integrates transcription into the main app flows.