feat: unload idle models and show load progress in the notch overlay - #945
feat: unload idle models and show load progress in the notch overlay#945HEREISCB wants to merge 2 commits into
Conversation
Fluid Intelligence and the speech model stay resident from first use until quit. On a 24 GB machine the ~3 GB helper alone pushes everything else into compressed memory and swap during long idle stretches. - Add IdleModelUnloader: restarts a countdown on every model use (ASR stop, AI enhance/rewrite) and releases both runtimes through the existing unloadCachedRuntime and resetTranscriptionProvider paths. Defers while a session is recording or processing. - Add "Unload Models When Idle" to Speech settings (Never/2/5/10/30 min) plus an Unload Now button. - Preload the speech model inside ASRService.start(). The hotkey path only loaded it at stop(), so the first dictation after launch (or any unload) stalled after the key was released. The overlay button path already did this; now every path does. - Resume the streaming preview when the model becomes ready mid-recording. Its first chunk transcribes the buffered prefix, then continues. - Show a "Keep talking" label and progress bar in both notch layouts while the model loads. Load has no real progress, so the bar paces itself on the last measured load time and completes on ready; downloads show the true percentage. Honors Reduce Motion. - Warm Fluid Intelligence only after the speech model is ready so the two loads don't compete. Refs altic-dev#853, altic-dev#806, altic-dev#548, altic-dev#705
github-actions
Bot
added
the
needs screenshots
|
The PR Policy check is blocking this PR because required template information is missing. Please update the PR description with:
Visual files detected:
Screenshots or video are required for UI, UX, settings, onboarding, overlay, menu bar, or visual behavior changes. If this PR has no visual changes, check the no-visual-change box in the template. If this remains incomplete for 48 hours after opening, the PR may be closed. |
Greptile SummaryThis PR adds configurable idle unloading for speech and private-AI models, preloads speech recognition during recording, and displays model-loading progress in the notch overlay.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| var modelIdleUnloadMinutes: Int { | ||
| get { | ||
| guard self.defaults.object(forKey: Keys.modelIdleUnloadMinutes) != nil else { | ||
| return Self.defaultModelIdleUnloadMinutes | ||
| } | ||
| return max(0, self.defaults.integer(forKey: Keys.modelIdleUnloadMinutes)) | ||
| } | ||
| set { | ||
| objectWillChange.send() | ||
| self.defaults.set(max(0, newValue), forKey: Keys.modelIdleUnloadMinutes) | ||
| } | ||
| } |
There was a problem hiding this comment.
When a user selects a non-default idle-unload interval and later restores a settings backup, modelIdleUnloadMinutes is absent from both the backup payload and restore mapping, causing the selection to silently revert to the 10-minute default and potentially unload models despite the user's previous preference.
Knowledge Base Used: Settings and onboarding
Prompt To Fix With AI
This is a comment left during a code review.
Path: Sources/Fluid/Persistence/SettingsStore.swift
Line: 1669-1680
Comment:
**Idle setting lost on restore**
When a user selects a non-default idle-unload interval and later restores a settings backup, `modelIdleUnloadMinutes` is absent from both the backup payload and restore mapping, causing the selection to silently revert to the 10-minute default and potentially unload models despite the user's previous preference.
**Knowledge Base Used:** [Settings and onboarding](https://app.greptile.com/altic/-/custom-context/knowledge-base/altic-dev/fluidvoice/-/docs/settings-and-onboarding.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f550a15a55
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| func unloadForIdle() { | ||
| guard self.isAsrReady, !self.isRunningOrStarting else { return } | ||
| DebugLogger.shared.info("ASRService: unloading speech model after idle", source: "ASRService") | ||
| self.resetTranscriptionProvider() |
There was a problem hiding this comment.
For any Nemotron speech model, this does not release the loaded model: resetTranscriptionProvider() clears the individual provider properties but never removes entries from nemotronProviders, and getNemotronProvider subsequently returns the retained instance. Consequently both the idle timeout and “Unload Now” mark ASR unready without reclaiming the Nemotron model's memory; clear that dictionary as the termination path already does.
Useful? React with 👍 / 👎.
| self.cancel() | ||
| return | ||
| } | ||
| self.schedule(after: .seconds(minutes * 60)) |
There was a problem hiding this comment.
When ASRService.initialize() finds an installed model, it auto-loads it at lines 1454–1460, but that successful path never calls recordActivity(). Since the new timer is otherwise started only after a recording/private-AI request or a settings change, a user who launches the app without dictating never starts the default 10-minute countdown, so the startup-loaded model remains in memory indefinitely.
Useful? React with 👍 / 👎.
github-actions
Bot
removed
the
needs screenshots
|
Hey - Not taking requessts for the load unload part for now as the Parakeet model is not that heavy and would cause UX issues to most of the people and don't want to overload users with options here. I will take care of this mostly for Fluid Intelligence. WIll come to this later overall but 500MB doesn't seem too bad for the model in memory to give ultra low latency similar to cloud models. I Like the notch UI - maybe we can merge that only for model loading if possible. Thanks I'd highly recommend you to make a discussion or smth before you fix sensitive items. Saves a lot of tokens and time for you and I! |
|
Hey, I understand all of the things I would like you to go ahead and test it though, I did work on the UX a lot for this so there won't be a UX issue. But I would want you to test it before you decide anything. This goes for the fluid intelligence as well. Both of the models can be unloaded and loaded with a single shortcut that is used to start the transcription and loading the model doesn't affect the actual experience of the user it doesn't create any latency. I did work on the latency part a lot, but yeah, if you can just test it locally and just give a feedback on this, Maybe this can inspire a new feature you might add in the future. |
2 participants
Description
Once used, Fluid Intelligence (~3 GB helper) and the speech model stay in memory until the app quits. This adds an idle unload, and fixes the user-facing cost of reloading: the hotkey path never preloaded the speech model while recording, so any load happened after the key was released, and the overlay showed nothing while it waited.
What changes:
unloadCachedRuntime/resetTranscriptionProviderpaths. It defers while recording or processing. An "Unload Now" button is included for testing.ASRService.start()now kicks offensureAsrReady()in the background. Previously only the on-screen record button did this; the hotkey path loaded atstop().Measured on this machine: Parakeet TDT reloads in 0.1–0.5 s, Fluid-1 + drafter in ~1.7 s with a warm disk cache. With the preload, any dictation longer than that hides the reload entirely.
Type of Change
Related Issue or Discussion
Refs #853 and #806 (idle unload for Fluid Intelligence), #548 (speech model pinned in memory), #705 (first dictation stalls while the model loads). Closes none directly; #853 and #806 are already closed.
Testing
swiftlint --strict --config .swiftlint.yml Sourcesswiftformat --config .swiftformat SourcesManual: set the timeout to 2 min (or press Unload Now), dictate, confirm the log shows "Idle unload: releasing models"; dictate again and confirm "Model not ready at start; loading in background", "Model became ready mid-recording", and the overlay shows the loading bar before the preview text. Tested in the open-source build; the Fluid Intelligence unload path calls the same
unloadCachedRuntime(reason:)the settings screen already uses on config changes, but I could not observe the closed-source helper's memory drop from this build.Screenshots / Video
Notch overlay (Standard Notch layout) while the speech model reloads mid-recording: "Keep talking" label and progress bar, audio still being captured.
New Speech settings row with the timeout picker and Unload Now button.
Notes