Skip to content

[Blazor] Add multimodal message input components - #68672

Open
kotlarmilos wants to merge 4 commits into
dotnet:javiercn-components-ai-09-predictive-statefrom
kotlarmilos:milosk/components-ai-10-production-app
Open

[Blazor] Add multimodal message input components#68672
kotlarmilos wants to merge 4 commits into
dotnet:javiercn-components-ai-09-predictive-statefrom
kotlarmilos:milosk/components-ai-10-production-app

Conversation

@kotlarmilos

Copy link
Copy Markdown
Member

Overview

This is position 10 in the native Components.AI stack and depends directly on #68335. This three-commit layer adds reusable multimodal message input components for attachments, recorded audio, live speech, submission, and cancellation, then exercises them in a production-oriented vehicle claim sample. The shipping UI runtime remains provider-neutral while the sample uses the real UIAgent -> AGUIChatClient -> HTTP/SSE -> AGUI.Server path and Microsoft Foundry for chat, vision, transcription, and grounded repair research.

Design

MessageInput becomes the composition boundary for text and DataContent. It owns the draft, attachments, validation state, submission state, and cancellation state through MessageInputContext. MessageAttachButton, AudioCaptureButton, LiveSpeechButton, MessageSendButton, and MessageStopButton consume that shared context instead of duplicating composer state.

Recorded voice and live voice have separate behavior. Recorded voice streams interim browser recognition into the textbox and sends the completed audio to the configured transcription callback. Live voice submits finalized browser transcripts directly as conversation turns, keeps listening across model responses and recoverable failures, and stops only when the user explicitly disables it or browser speech recognition reports a fatal error. Assistant responses remain text-only.

Media enters the existing content pipeline through DataContentBlock, DataContentHandler, and MediaContent. This keeps images and audio represented as Microsoft.Extensions.AI.DataContent throughout the agent pipeline instead of introducing provider-specific message types. Microsoft.AspNetCore.Components.Media.Audio provides the corresponding audio rendering surface. Audio capture uses the browser-native MediaRecorder and SpeechRecognition APIs without third-party JavaScript dependencies.

Retries now treat each protocol round as a transaction. A failed user message or continuation is not committed to conversation history, partial response blocks are removed, and retry replays the exact failed round without duplicating the rendered request, text, attachments, tool results, or approval responses.

Implementation

The vehicle claim sample demonstrates the complete composition model:

  • Text, drag-and-drop photos, file selection, recorded voice, and continuous live voice share one MessageInput.
  • Claim state crosses the AG-UI boundary through official state snapshot and RFC 6902 delta events.
  • Damage visualization uses a client-side UI action.
  • Claim submission uses the AG-UI approval and resume flow.
  • Production requests use Microsoft Foundry for conversation, image analysis, transcription, and public web research.
  • E2E tests replace only IClaimAssistantBackend. AGUIChatClient, HTTP, SSE serialization, AGUI.Server, client tools, state mapping, and approval continuations remain production code.
  • Model-provided research links are restricted to normalized absolute HTTP and HTTPS URLs.
  • Evidence limits are enforced consistently in the composer, workflow, and the bounded claim-agent request endpoint.
  • The agent base address supports an explicit ClaimAgent:BaseAddress and otherwise resolves from NavigationManager.BaseUri, preserving reverse-proxy path prefixes.

The sample also covers light, dark, and high-contrast themes, keyboard submission and cancellation, accessible attachment and voice controls, cumulative evidence reassessment, approval rejection, and retry recovery.

Outcome

Validation class Result
Components.AI runtime and component tests 139 passed
Components.Media tests 15 passed
ClaimApp browser scenarios 22 passed, 3 configuration-gated Foundry scenarios skipped
ClaimApp build 0 warnings, 0 errors
Diff validation git diff --check passed

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8305953e-ce14-4a55-b291-77beaac47b0a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8305953e-ce14-4a55-b291-77beaac47b0a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8305953e-ce14-4a55-b291-77beaac47b0a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8305953e-ce14-4a55-b291-77beaac47b0a
@kotlarmilos
kotlarmilos marked this pull request as ready for review August 21, 2026 08:00
@kotlarmilos
kotlarmilos requested a review from a team as a code owner August 21, 2026 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant