diff --git a/README.md b/README.md index 592ca978..001e8687 100644 --- a/README.md +++ b/README.md @@ -102,6 +102,7 @@ cookiecutter templates/examples -o examples - [`@examples/stream`](./examples/stream/README.md) - [`@examples/oauth`](./examples/oauth/README.md) - [`@examples/meetings`](./examples/meetings/README.md) +- [`@examples/ai-file-analysis`](./examples/ai-file-analysis/README.md) ## Links diff --git a/examples/ai-file-analysis/README.md b/examples/ai-file-analysis/README.md new file mode 100644 index 00000000..03c4aac5 --- /dev/null +++ b/examples/ai-file-analysis/README.md @@ -0,0 +1,89 @@ +# AI file analysis + +A Teams bot that reads files attached in personal (1:1) chat and sends the ones it understands to Azure OpenAI. + +One message handler covers both paths: + +- **Basic (no LLM)** replies with an Adaptive Card describing any file the sample cannot analyze, showing the metadata the file API exposes and the bytes that were downloaded. +- **AI** converts supported text files and images into model input and streams the analysis back. + +### Reading the code + +Comments label which of two things a given block is doing: + +- **`FILE RECEIVE`** is the Teams SDK file API. This is the part worth copying into your own app. +- **`SAMPLE GUARDRAIL`** is this sample deciding what it will forward to a model: which formats it accepts, how much text it sends, how many files per message, and whether anything is remembered between turns. These are arbitrary product choices, not SDK or Azure OpenAI requirements. Your app should pick its own. + +The distinction matters because most of the code volume here is guardrails. Receiving a file is only `ctx.files.list()` followed by `download()`. + +## Prerequisites + +- Python +- A Teams bot registration +- A Teams app manifest with `supportsFiles` set to `true` on the bot entry (see [Enable file support in the manifest](#enable-file-support-in-the-manifest)) +- An Azure OpenAI deployment (use a vision-capable model to analyze images). This is optional: without it the example still runs, receives files, and reports each one with an Adaptive Card instead of analyzing it. See [Running without a model](#running-without-a-model). + +## Enable file support in the manifest + +The bot entry in your Teams app manifest must set `supportsFiles` to `true`: + +```json +"bots": [ + { + "botId": "", + "scopes": ["personal"], + "supportsFiles": true + } +] +``` + +Without it, Teams does not enable the attachment UI in the bot's chat, so there is no way to attach a file in the first place and `ctx.files.list()` has nothing to return. + +## Setup + +Add these settings to the example's `.env` alongside your bot credentials: + +```env +AZURE_OPENAI_ENDPOINT=https://.openai.azure.com/ +AZURE_OPENAI_API_KEY= +AZURE_OPENAI_MODEL_DEPLOYMENT_NAME= +AZURE_OPENAI_API_VERSION=2024-10-21 +``` + +Run: + +```bash +uv run src/main.py +``` + +## Running without a model + +The file APIs this example demonstrates do not need a model, so the Azure OpenAI settings above are optional. + +Leave any of them unset and the example starts in metadata-only mode. It still receives, downloads, and reports every +attached file with the Adaptive Card, showing the resolved content type, byte count, scope, and source, so the whole +file round-trip is demonstrable without a model subscription. Only the analysis step is skipped, and the card says so. + +## What happens to an attached file + +1. `ctx.files.list()` returns the files on the incoming activity. +2. Each file is downloaded once, and that in-memory copy is reused instead of refetching through the short-lived Teams download URL. +3. `classify_file` sorts each download into `text`, `image`, or `unsupported`. +4. Unsupported files get the basic Adaptive Card. No model call is made for them. +5. Supported files become OpenAI content parts and are sent in a single request, and the reply is streamed to Teams. + +Image bytes are sent inline as a data URI rather than as a link, so the pre-authorized `tempauth` download URL is never handed to the model. + +## Limits + +The sample accepts up to five files per message. Text input is capped at 100 KB per file and 250 KB per message, and images at 1 MB each. Supported image formats are PNG, JPEG, GIF, and WebP. Anything skipped or truncated produces a message explaining why. + +Because `download()` buffers the whole file first, these caps bound what reaches the model, not network transfer or process memory. + +## Scope + +The AI path is stateless: each message is analyzed on its own, with no conversation memory. That keeps a follow-up question from silently reusing files the user did not attach to it, and keeps images from being resent on every later turn. + +Statelessness here is a **`SAMPLE GUARDRAIL`**, not an SDK or Azure OpenAI constraint. Your app can keep conversation state and reuse previously attached files; this sample opts out so that every analysis is traceable to the files on the message that triggered it. + +There are no tools, citations, feedback, or follow-up suggestions here. See the [`ai-mcp`](https://github.com/microsoft/teams.py/tree/main/examples/ai-mcp) sample for those. diff --git a/examples/ai-file-analysis/pyproject.toml b/examples/ai-file-analysis/pyproject.toml new file mode 100644 index 00000000..5cf7011e --- /dev/null +++ b/examples/ai-file-analysis/pyproject.toml @@ -0,0 +1,18 @@ +[project] +name = "ai-file-analysis" +version = "0.1.0" +description = "Teams bot that analyzes uploaded files with Azure OpenAI" +readme = "README.md" +requires-python = ">=3.11,<4.0" +dependencies = [ + "dotenv>=0.9.9", + "microsoft-teams-apps", + "microsoft-teams-api", + "microsoft-teams-cards", + "openai>=1.60.0", +] + +[tool.uv.sources] +microsoft-teams-apps = { workspace = true } +microsoft-teams-api = { workspace = true } +microsoft-teams-cards = { workspace = true } diff --git a/examples/ai-file-analysis/src/ai/__init__.py b/examples/ai-file-analysis/src/ai/__init__.py new file mode 100644 index 00000000..ae78b9ae --- /dev/null +++ b/examples/ai-file-analysis/src/ai/__init__.py @@ -0,0 +1,17 @@ +""" +Copyright (c) Microsoft Corporation. All rights reserved. +Licensed under the MIT License. +""" + +from .file_context import AnalysisRequest, AnalyzableFile, FileKind, classify_file, prepare_analysis +from .runner import is_ai_configured, run_analysis + +__all__ = [ + "AnalysisRequest", + "AnalyzableFile", + "FileKind", + "classify_file", + "is_ai_configured", + "prepare_analysis", + "run_analysis", +] diff --git a/examples/ai-file-analysis/src/ai/file_context.py b/examples/ai-file-analysis/src/ai/file_context.py new file mode 100644 index 00000000..a1624db6 --- /dev/null +++ b/examples/ai-file-analysis/src/ai/file_context.py @@ -0,0 +1,201 @@ +""" +Copyright (c) Microsoft Corporation. All rights reserved. +Licensed under the MIT License. +""" + +import re +from base64 import b64encode +from dataclasses import dataclass +from typing import Any, List, Literal, Optional + +from microsoft_teams.apps import DownloadedFile + +# SAMPLE GUARDRAIL: every constant below is a product choice made by this sample, not a Teams SDK or Azure OpenAI +# limit. They exist to keep one Teams message from turning into an unbounded model request. Pick your own values. +# +# `download()` buffers the whole file before any of these are checked, so they bound what reaches the model, not +# network transfer or process memory. +MAX_FILES = 5 +MAX_TEXT_BYTES_PER_FILE = 100 * 1024 +MAX_TOTAL_TEXT_BYTES = 250 * 1024 +MAX_IMAGE_BYTES = 1024 * 1024 + +# SAMPLE GUARDRAIL: the formats this sample is willing to forward. The file API itself delivers any attached file type. +IMAGE_CONTENT_TYPES = { + "image/gif", + "image/jpeg", + "image/png", + "image/webp", +} + +TEXT_EXTENSIONS = { + "c", + "cpp", + "cs", + "css", + "csv", + "go", + "h", + "html", + "java", + "js", + "json", + "jsx", + "md", + "py", + "rb", + "rs", + "sh", + "sql", + "toml", + "ts", + "tsx", + "txt", + "xml", + "yaml", + "yml", +} + +_TEXTUAL_CONTENT_TYPE = re.compile(r"\b(json|xml|javascript|yaml|csv|markdown)\b") + +FileKind = Literal["image", "text", "unsupported"] +"""Whether this sample can send a downloaded file to the model, and as what.""" + + +@dataclass +class AnalyzableFile: + """A downloaded file that `classify_file` accepted, paired with its kind.""" + + file: DownloadedFile + kind: Literal["image", "text"] + + +@dataclass +class AnalysisRequest: + """A model request built from the user's message and their analyzable files.""" + + content: List[Any] + + warnings: List[str] + """User-facing explanations for files that were skipped or truncated.""" + + file_count: int + """Number of files whose content reached the model request.""" + + +def classify_file(file: DownloadedFile, extension: Optional[str] = None) -> FileKind: + """ + SAMPLE GUARDRAIL: decides whether a downloaded file can be sent to the model. + + The response MIME type is preferred, but the platform-supplied extension is a necessary fallback, and that part is + a real file-receive detail rather than a sample preference: Teams commonly omits or misclassifies source files, + reporting `.ts` as `video/vnd.dlna.mpeg-tts` for example. + """ + content_type = _base_content_type(file.content_type) + + if content_type in IMAGE_CONTENT_TYPES: + return "image" + + if _is_text_content_type(content_type) or _get_text_extension(extension, file.filename): + return "text" + + return "unsupported" + + +def prepare_analysis(user_text: str, files: List[AnalyzableFile]) -> AnalysisRequest: + """ + Converts already-downloaded files into OpenAI content parts. + + The conversion itself is the AI integration. The caps it enforces along the way are SAMPLE GUARDRAILs, and each one + that drops or shortens a file returns a warning so the user is never left guessing what the model saw. + """ + parts: List[Any] = [ + { + "type": "text", + "text": user_text.strip() or "Please analyze the attached file content.", + } + ] + warnings: List[str] = [] + file_count = 0 + total_text_bytes = 0 + + for entry in files[:MAX_FILES]: + downloaded = entry.file + + if entry.kind == "image": + if len(downloaded.bytes) > MAX_IMAGE_BYTES: + warnings.append(f"{downloaded.filename} was not sent to the model because it is larger than 1 MB.") + continue + + parts.append({"type": "text", "text": f"Attached image: {downloaded.filename}"}) + parts.append( + { + "type": "image_url", + "image_url": { + # FILE RECEIVE: the downloaded bytes are sent inline instead of handing the model the + # pre-authorized `tempauth` download URL, which is a short-lived credential. + "url": _to_data_uri(downloaded.bytes, _base_content_type(downloaded.content_type)), + "detail": "auto", + }, + } + ) + file_count += 1 + continue + + remaining_bytes = MAX_TOTAL_TEXT_BYTES - total_text_bytes + if remaining_bytes <= 0: + warnings.append( + f"{downloaded.filename} was not sent to the model because the combined text-file limit was reached." + ) + continue + + included_bytes = min(len(downloaded.bytes), MAX_TEXT_BYTES_PER_FILE, remaining_bytes) + text = downloaded.bytes[:included_bytes].decode("utf-8", errors="replace") + truncated = included_bytes < len(downloaded.bytes) + total_text_bytes += included_bytes + + lines = [ + f"Attached file: {downloaded.filename}", + "", + "", + text, + ] + if truncated: + lines.append("[File content truncated by the sample.]") + lines.append("") + + parts.append({"type": "text", "text": "\n".join(lines)}) + + if truncated: + warnings.append(f"{downloaded.filename} was truncated before being sent to the model.") + file_count += 1 + + if len(files) > MAX_FILES: + warnings.append( + f"{len(files) - MAX_FILES} supported file(s) were not sent to the model because this sample " + f"analyzes up to {MAX_FILES} files per message. Unsupported files are reported separately." + ) + + return AnalysisRequest(content=parts, warnings=warnings, file_count=file_count) + + +def _base_content_type(content_type: str) -> str: + return content_type.split(";", 1)[0].strip().lower() + + +def _is_text_content_type(content_type: str) -> bool: + return content_type.startswith("text/") or bool(_TEXTUAL_CONTENT_TYPE.search(content_type)) + + +def _get_text_extension(extension: Optional[str], filename: str) -> Optional[str]: + if extension: + normalized = extension.lstrip(".").lower() + elif "." in filename: + normalized = filename.rsplit(".", 1)[-1].lower() + else: + normalized = "" + return normalized if normalized in TEXT_EXTENSIONS else None + + +def _to_data_uri(data: bytes, content_type: str) -> str: + return f"data:{content_type};base64,{b64encode(data).decode('ascii')}" diff --git a/examples/ai-file-analysis/src/ai/runner.py b/examples/ai-file-analysis/src/ai/runner.py new file mode 100644 index 00000000..f0d73153 --- /dev/null +++ b/examples/ai-file-analysis/src/ai/runner.py @@ -0,0 +1,113 @@ +""" +Copyright (c) Microsoft Corporation. All rights reserved. +Licensed under the MIT License. +""" + +import logging +from os import getenv +from typing import Any, Optional, Tuple, cast + +from dotenv import find_dotenv, load_dotenv +from microsoft_teams.api import MessageActivityInput +from microsoft_teams.apps.plugins.streamer import StreamerProtocol +from openai import AsyncAzureOpenAI + +from .file_context import AnalysisRequest + +load_dotenv(find_dotenv(usecwd=True)) + +SYSTEM_PROMPT = """\ +You analyze files supplied by the user. + +Base your answer on the user's message and the attached content. State clearly when the available files do not support +a conclusion. Do not claim to have inspected files that were not included. Keep the response concise and practical.""" + + +REQUIRED_SETTINGS = ( + "AZURE_OPENAI_ENDPOINT", + "AZURE_OPENAI_API_KEY", + "AZURE_OPENAI_MODEL_DEPLOYMENT_NAME", +) + +_client: Optional[AsyncAzureOpenAI] = None + + +def _required(name: str) -> str: + value = getenv(name) + if not value: + raise ValueError(f"{name} is required (set it in .env).") + return value + + +def is_ai_configured() -> bool: + """ + Whether every Azure OpenAI setting this sample needs is present. + + The sample runs without them: run_analysis is skipped and each file is answered with the metadata card + instead, so the Teams file API can still be exercised with no model subscription. Nothing here validates + the values, only that they were supplied. + """ + return all(getenv(name) for name in REQUIRED_SETTINGS) + + +def _get_client() -> Tuple[AsyncAzureOpenAI, str]: + """ + Builds the Azure OpenAI client on first use. + + Deliberately lazy: constructing it at import time would make a missing .env crash the whole bot on + startup, including the metadata-card path that needs no model at all. + """ + global _client + deployment = _required("AZURE_OPENAI_MODEL_DEPLOYMENT_NAME") + if _client is None: + _client = AsyncAzureOpenAI( + azure_endpoint=_required("AZURE_OPENAI_ENDPOINT"), + api_key=_required("AZURE_OPENAI_API_KEY"), + api_version=getenv("AZURE_OPENAI_API_VERSION") or "2024-10-21", + ) + return _client, deployment + + +async def run_analysis(request: AnalysisRequest, stream: StreamerProtocol, log: logging.Logger) -> None: + """ + Sends one stateless request for the current message and streams the reply. + + SAMPLE GUARDRAIL: nothing is carried between turns. A stateful agent would keep history here, but that would let a + later message silently reuse file content the user did not attach to it, and would resend every image on every + following turn. + """ + try: + stream.update("Analyzing files...") + + client, deployment = _get_client() + completion = await client.chat.completions.create( + model=deployment, + messages=[ + {"role": "system", "content": SYSTEM_PROMPT}, + {"role": "user", "content": cast(Any, request.content)}, + ], + stream=True, + ) + + async for chunk in completion: + if not chunk.choices: + continue + text = chunk.choices[0].delta.content + if text: + stream.emit(text) + + stream.emit(MessageActivityInput().add_ai_generated()) + except Exception as err: + message = str(err) + log.error("File analysis failed: %s", message) + stream.clear_text() + rate_limited = getattr(err, "status_code", None) == 429 or message.startswith("429 ") + stream.emit( + MessageActivityInput( + text=( + "The AI service is temporarily rate-limited. Please wait a moment and try again." + if rate_limited + else "I could not analyze those files. Please try again." + ) + ).add_ai_generated() + ) diff --git a/examples/ai-file-analysis/src/file_card.py b/examples/ai-file-analysis/src/file_card.py new file mode 100644 index 00000000..f62cc67a --- /dev/null +++ b/examples/ai-file-analysis/src/file_card.py @@ -0,0 +1,63 @@ +""" +Copyright (c) Microsoft Corporation. All rights reserved. +Licensed under the MIT License. +""" + +from microsoft_teams.apps import DownloadedFile, IncomingFile +from microsoft_teams.cards import AdaptiveCard, Container, Fact, FactSet, TextBlock + +UNSUPPORTED_NOTE = ( + "I downloaded this file but did not analyze it. This sample sends only text files and PNG, JPEG, GIF, " + "or WebP images to the model." +) + + +def unsupported_file_card(file: IncomingFile, downloaded: DownloadedFile, note: str = UNSUPPORTED_NOTE) -> AdaptiveCard: + """ + FILE RECEIVE: the no-LLM response for a file this sample will not send to the model. + + Nothing here touches Azure OpenAI. It reports what the file API exposes (`scope`, `source`, resolved content type) + plus the byte count that was actually downloaded, so the file round-trip is still demonstrated for formats the + model never sees. + + Args: + note: Overrides the closing explanation. Defaults to the unsupported-format wording; the no-model path + passes its own so the card does not imply the file type was the problem. + """ + return AdaptiveCard( + body=[ + Container( + style="emphasis", + items=[ + TextBlock(text="File received", weight="Bolder", size="Large", color="Accent"), + TextBlock(text=downloaded.filename, weight="Bolder", wrap=True), + ], + ), + FactSet( + facts=[ + Fact(title="Type", value=downloaded.content_type), + Fact(title="Size", value=_human_size(len(downloaded.bytes))), + Fact(title="Scope", value=file.scope), + Fact(title="Source", value=file.source), + ] + ), + TextBlock( + text=note, + wrap=True, + is_subtle=True, + spacing="Medium", + ), + ] + ) + + +def _human_size(num_bytes: int) -> str: + if num_bytes < 1024: + return f"{num_bytes} B" + units = ["KB", "MB", "GB"] + value = num_bytes / 1024 + unit = 0 + while value >= 1024 and unit < len(units) - 1: + value /= 1024 + unit += 1 + return f"{value:.1f} {units[unit]}" diff --git a/examples/ai-file-analysis/src/main.py b/examples/ai-file-analysis/src/main.py new file mode 100644 index 00000000..a4a7a835 --- /dev/null +++ b/examples/ai-file-analysis/src/main.py @@ -0,0 +1,102 @@ +""" +Copyright (c) Microsoft Corporation. All rights reserved. +Licensed under the MIT License. + +AI File Analysis Example Bot + +Two kinds of code live in this sample, labeled throughout: + +- `FILE RECEIVE` is the Teams SDK file API itself. This is the part worth copying into your own app. +- `SAMPLE GUARDRAIL` is this sample deciding what it is willing to forward to a model. Those limits are arbitrary + product choices, not SDK requirements, and your app should pick its own. +""" + +import asyncio +import logging +from typing import List + +from ai import AnalyzableFile, classify_file, is_ai_configured, prepare_analysis, run_analysis +from file_card import unsupported_file_card +from microsoft_teams.api import MessageActivity, TypingActivityInput +from microsoft_teams.apps import ActivityContext, App + +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger("ai-file-analysis") + +app = App() + +# SAMPLE GUARDRAIL: the file API needs no model, so the sample stays usable without Azure OpenAI settings. Without +# them it answers every file with the metadata card instead of analyzing it, which keeps download, content type, +# scope, and source demonstrable with no model subscription. +_ai_configured = is_ai_configured() +if not _ai_configured: + logger.warning( + "Azure OpenAI is not configured, so files will be reported but not analyzed. Set AZURE_OPENAI_ENDPOINT, " + "AZURE_OPENAI_API_KEY, and AZURE_OPENAI_MODEL_DEPLOYMENT_NAME in .env to enable analysis." + ) + +NO_MODEL_NOTE = ( + "I downloaded this file, but no model is configured for this sample, so I did not analyze it. " + "Set the Azure OpenAI values in .env to enable analysis." +) + + +@app.on_message +async def handle_message(ctx: ActivityContext[MessageActivity]) -> None: + """Analyze any files attached to the message, and describe the ones this sample cannot read.""" + await ctx.send(TypingActivityInput()) + + # FILE RECEIVE: the files attached to this activity. + attached = await ctx.files.list() + if not attached: + await ctx.send( + "Attach one or more files. I analyze text files and images, and describe anything else I cannot read." + if _ai_configured + else "Attach one or more files. No model is configured, so I will report what I received " + "without analyzing it." + ) + return + + analyzable: List[AnalyzableFile] = [] + + for file in attached: + try: + # FILE RECEIVE: download once. Every read below uses this in-memory copy rather than refetching through the + # short-lived Teams download URL. + downloaded = await file.download() + except Exception as err: + logger.warning("Could not download %s: %s", file.name, err) + await ctx.send(f"I could not download {file.name}.") + continue + + if not _ai_configured: + await ctx.send(unsupported_file_card(file, downloaded, NO_MODEL_NOTE)) + continue + + # SAMPLE GUARDRAIL: the SDK hands over every attached file regardless of type. This sample is what narrows + # that to the formats it will send on. + kind = classify_file(downloaded, file.extension) + + if kind == "unsupported": + await ctx.send(unsupported_file_card(file, downloaded)) + continue + + analyzable.append(AnalyzableFile(file=downloaded, kind=kind)) + + if not analyzable: + return + + # SAMPLE GUARDRAIL: applies this sample's size and count caps and reports anything it dropped or truncated. + analysis = prepare_analysis(ctx.activity.strip_mentions_text().text or "", analyzable) + + for warning in analysis.warnings: + await ctx.send(warning) + + if analysis.file_count == 0: + return + + await run_analysis(analysis, ctx.stream, logger) + + +if __name__ == "__main__": + asyncio.run(app.start()) diff --git a/uv.lock b/uv.lock index a5af9d7c..f29047fa 100644 --- a/uv.lock +++ b/uv.lock @@ -14,6 +14,7 @@ members = [ "a2a", "agent365", "ai-agentframework", + "ai-file-analysis", "botbuilder", "cards", "dialogs", @@ -174,6 +175,27 @@ requires-dist = [ { name = "microsoft-teams-apps", editable = "packages/apps" }, ] +[[package]] +name = "ai-file-analysis" +version = "0.1.0" +source = { virtual = "examples/ai-file-analysis" } +dependencies = [ + { name = "dotenv" }, + { name = "microsoft-teams-api" }, + { name = "microsoft-teams-apps" }, + { name = "microsoft-teams-cards" }, + { name = "openai" }, +] + +[package.metadata] +requires-dist = [ + { name = "dotenv", specifier = ">=0.9.9" }, + { name = "microsoft-teams-api", editable = "packages/api" }, + { name = "microsoft-teams-apps", editable = "packages/apps" }, + { name = "microsoft-teams-cards", editable = "packages/cards" }, + { name = "openai", specifier = ">=1.60.0" }, +] + [[package]] name = "aiohappyeyeballs" version = "2.6.1"