From 29915bbaad2635992b66e192508ca9be21b16361 Mon Sep 17 00:00:00 2001 From: Brandon Werner Date: Fri, 10 Jul 2026 18:35:36 -0700 Subject: [PATCH] docs: refresh repository guidance and Graph provisioning --- .claude/skills/implement-agent-id/SKILL.md | 411 +++----- .github/copilot-instructions.md | 11 +- .github/workflows/docs.yml | 4 +- AGENTS.md | 11 +- CHANGELOG.md | 28 +- CLAUDE.md | 9 +- INSTALL.md | 53 +- README.md | 34 +- TODOS.md | 283 +----- docs/SECURITY-DEBT-PROVISIONER-SECRET.md | 117 +-- .../NEXT-WhatsApp-lightweight-teams-chat.md | 14 +- .../PLAN-agent-identity-by-upn.md | 6 +- .../PLAN-multi-tenant-lightweight-chat.md | 2 +- docs/architecture/PLAN-skills-layer.md | 2 +- docs/architecture/PLAN-windows-port.md | 31 +- .../PLAN-xpia-content-wrapping.md | 6 +- docs/claude-copilot-cli-channel-port.md | 14 +- docs/claude-windows-port.md | 15 +- ...03-certificate-auth-over-client-secrets.md | 14 +- docs/decisions/005-cloud-hosted-memory.md | 2 +- docs/developer/docs-site.md | 62 +- docs/engineering-status.md | 171 +--- docs/getting-started/quickstart.md | 138 +-- docs/guides/customizing-the-body-prompt.md | 10 +- docs/guides/storage-configuration.md | 14 +- docs/index.md | 14 +- docs/openai-windows-agent-identity-port.md | 2 +- .../agent-id-blueprints-and-users.md | 35 +- docs/platform-learnings/entra-agent-users.md | 32 +- .../platform-learnings/microsoft-agent-365.md | 459 ++------- .../msal-entra-agent-ids.md | 51 +- docs/platform-learnings/teams-graph-api.md | 25 +- ...tenant-lightweight-chat-planning-prompt.md | 2 +- .../agent-foundry-create-entra-agent-ids.py | 907 ------------------ .../agent-foundry-entra-provisioning.py | 483 ---------- docs/reference/api/mcp-tools.md | 46 +- docs/reference/mcp-tools.md | 2 +- docs/reference/scripts/operations.md | 11 + docs/reference/scripts/provisioning.md | 6 +- docs/reference/scripts/setup.md | 22 +- docs/reference/setup-script.md | 174 ---- docs/reference/token-flows.md | 112 ++- docs/runbooks/hard-won-learnings.md | 18 +- mkdocs.yml | 178 ++-- scripts/create_entra_agent_ids.py | 35 +- scripts/entra_provisioning.py | 2 +- tests/scripts/test_create_entra_agent_ids.py | 92 ++ tests/scripts/test_entra_provisioning.py | 10 + 48 files changed, 991 insertions(+), 3189 deletions(-) delete mode 100644 docs/reference/agent-foundry-create-entra-agent-ids.py delete mode 100644 docs/reference/agent-foundry-entra-provisioning.py delete mode 100644 docs/reference/setup-script.md diff --git a/.claude/skills/implement-agent-id/SKILL.md b/.claude/skills/implement-agent-id/SKILL.md index eb5b4182..2cfd2bfe 100644 --- a/.claude/skills/implement-agent-id/SKILL.md +++ b/.claude/skills/implement-agent-id/SKILL.md @@ -1,358 +1,217 @@ --- name: implement-agent-id -description: Guide for integrating with Microsoft Entra Agent Identity APIs (Graph beta). Covers authentication, blueprint creation, agent identity provisioning, sponsors, permissions, and known pitfalls. Use when implementing Entra Agent IDs, Agent Identity Blueprints, AgentIdentityBlueprintPrincipal, or working with the Graph beta Agent Identity endpoints. +description: Guide for Microsoft Entra Agent ID and Agent User integration. Covers certificate authentication, stable Graph creation endpoints, sponsors, BlueprintPrincipal creation, permissions, consent, and the three-hop user_fic token flow. --- -# Integrating with Microsoft Entra Agent Identity APIs +# Implementing Microsoft Entra Agent ID -## Overview +Read these repository sources before changing identity or token code: -Microsoft Entra Agent Identity (preview, Nov 2025) provides a new identity primitive for AI agents in Microsoft Entra ID. It creates OAuth2-capable identities (service principals) that represent individual agent instances, organized under an "Agent Identity Blueprint" (application registration). +- `docs/platform-learnings/agent-id-blueprints-and-users.md` +- `docs/platform-learnings/entra-agent-users.md` +- `docs/platform-learnings/msal-entra-agent-ids.md` +- `docs/reference/token-flows.md` +- `docs/runbooks/hard-won-learnings.md` -**Conceptual Model:** -``` -Agent Identity Blueprint (application) ← one per agent "kind" or project - └─ AgentIdentityBlueprintPrincipal (SP) ← must be created explicitly - ├─ Agent Identity (SP): agent-1 ← one per agent instance - ├─ Agent Identity (SP): agent-2 - └─ Agent Identity (SP): agent-3 -``` - -**Graph beta API base:** `https://graph.microsoft.com/beta` - ---- +Microsoft Entra Agent ID and Microsoft Agent 365 reached GA on 2026-05-01, but not every related API is on Microsoft Graph v1.0. Use the endpoint version documented for each object rather than treating the whole surface as beta or stable. -## Critical Pitfalls (Read First) +## Object model -### 1. Azure CLI Tokens Are Rejected +```text +Agent Identity Blueprint (application) + └─ AgentIdentityBlueprintPrincipal (service principal; create explicitly) + ├─ Agent Identity (service principal) + └─ Agent Identity (service principal) + └─ Agent User (user; linked through user_fic) +``` -**Problem:** Azure CLI tokens always include the `Directory.AccessAsUser.All` delegated permission. The Agent Identity APIs **explicitly reject** any token containing this permission, returning a generic 403. +An Agent Identity is a service principal, not a user. Do not create a password-backed fake user to represent an agent. -**Solution:** You MUST use a dedicated app registration with `client_credentials` flow: +## Non-negotiable constraints -```python -from azure.identity import ClientSecretCredential +### Use a dedicated provisioner identity -credential = ClientSecretCredential( - tenant_id="", - client_id="", - client_secret="", -) -token = credential.get_token("https://graph.microsoft.com/.default") -``` +Azure CLI user tokens contain `Directory.AccessAsUser.All`; Agent Identity APIs reject those tokens with a hard 403. Use Azure CLI only to bootstrap the dedicated provisioner app and identify the signed-in sponsor. -**DO NOT** use `DefaultAzureCredential` or `AzureCliCredential` — they will produce tokens with `Directory.AccessAsUser.All` and every Agent Identity API call will fail with 403. +The provisioner authenticates with a certificate credential. Keep the private key in the OS credential store and purge legacy password credentials. Entrabot implements this in: -Auto-provisioning the app registration via `az ad app create` is the recommended approach. See the reference implementation in `scripts/create-entra-agent-ids.py`. +- `scripts/entra_provisioning.py` +- `scripts/create_entra_agent_ids.py` -### 2. Sponsors Are Required +Do not add a client secret fallback. -**Problem:** Both Blueprint and Agent Identity creation require a `sponsors@odata.bind` field. Without it, you get: `400: No sponsor specified. Please provide at least one sponsor.` +### Parse Azure CLI output as JSON -**Rules:** -- Sponsors must be **User** references — ServicePrincipals are NOT valid -- Use the `/users/{objectId}` URL format (not `/directoryObjects/` or `/servicePrincipals/`) -- Since you're using `client_credentials` (no user context), you CANNOT use `GET /me` to get the user ID. Use `az ad signed-in-user show --query id -o tsv` instead. +CLI warnings can corrupt TSV output. Request JSON and parse the `id` field: ```python -# Get sponsor user ID (az CLI has the user's auth context) result = subprocess.run( - ["az", "ad", "signed-in-user", "show", "--query", "id", "-o", "tsv"], - capture_output=True, text=True, + ["az", "ad", "signed-in-user", "show", "-o", "json"], + check=True, + capture_output=True, + text=True, ) -user_id = result.stdout.strip() - -# Add to any Blueprint or Agent Identity creation body -body["sponsors@odata.bind"] = [ - f"https://graph.microsoft.com/beta/users/{user_id}" -] +user_id = json.loads(result.stdout)["id"] ``` -### 3. BlueprintPrincipal Must Be Created Separately +### Sponsors are user references -**Problem:** Creating a Blueprint (`POST /applications`) does NOT auto-create its BlueprintPrincipal (SP). Without the BlueprintPrincipal, all Agent Identity creation fails with: `400: The Agent Blueprint Principal for the Agent Blueprint does not exist.` - -**Solution:** Always create the BlueprintPrincipal immediately after the Blueprint: +Blueprint and Agent Identity creation require at least one sponsor. Bind a Microsoft Graph v1.0 user reference: ```python -# Step 1: Create Blueprint -blueprint_body = { - "@odata.type": "Microsoft.Graph.AgentIdentityBlueprint", - "displayName": "My Agent Blueprint", - "sponsors@odata.bind": [f"https://graph.microsoft.com/beta/users/{user_id}"], -} -resp = requests.post(f"{GRAPH_BASE}/applications", headers=headers, json=blueprint_body) -app_id = resp.json()["appId"] - -# Step 2: Create BlueprintPrincipal (REQUIRED — not auto-created) -sp_body = { - "@odata.type": "Microsoft.Graph.AgentIdentityBlueprintPrincipal", - "appId": app_id, -} -requests.post(f"{GRAPH_BASE}/servicePrincipals", headers=headers, json=sp_body) -``` - -**Also important:** If you're implementing idempotent scripts that skip Blueprint creation when it already exists, you MUST check for and create the BlueprintPrincipal on the skip path too. A previous run may have created the Blueprint but crashed before creating the SP. - -### 4. Permission Propagation Takes 30-120+ Seconds - -After `az ad app permission admin-consent`, newly-granted Agent Identity permissions don't appear in tokens immediately. The token endpoint serves cached claims. - -**Solution:** Retry with fresh tokens: - -```python -for attempt in range(5): - token = credential.get_token("https://graph.microsoft.com/.default") - # Try the actual operation - resp = requests.post(url, headers=auth_header(token), json=body) - if resp.status_code == 403: - wait = 20 * (attempt + 1) - time.sleep(wait) - continue - break -``` - -Key insight: `credential.get_token()` returns cached tokens. For `ClientSecretCredential`, the cache is based on token lifetime (usually 1hr). But Entra's token endpoint itself may serve tokens with stale claims for 30-120s after a permission change. The retry loop handles this. - ---- - -## Required Permissions - -### Minimum for Blueprint + Agent Identity Creation - -There are **18 Agent Identity-specific** Graph application permissions. They can be discovered dynamically: - -```bash -az ad sp show --id 00000003-0000-0000-c000-000000000000 \ - --query "appRoles[?contains(value, 'AgentIdentity')].{id:id, value:value}" -o json -``` - -**Core permissions needed:** -| Permission | Purpose | -|-----------|---------| -| `Application.ReadWrite.All` | Read/write applications (for Blueprint CRUD) | -| `AgentIdentityBlueprint.Create` | Create new Blueprints | -| `AgentIdentityBlueprint.ReadWrite.All` | Read/update Blueprints | -| `AgentIdentityBlueprintPrincipal.Create` | Create BlueprintPrincipals | -| `AgentIdentity.Create.All` | Create Agent Identities | -| `AgentIdentity.ReadWrite.All` | Read/update Agent Identities | - -**Microsoft Graph API ID** (constant across all tenants): `00000003-0000-0000-c000-000000000000` - -**Application.ReadWrite.All role ID**: `1bfefb4e-e0b5-418b-a88f-73c46d2cc8e9` - -In practice, granting all 18 Agent Identity permissions plus `Application.ReadWrite.All` is the safest approach — the granular permission set is underdocumented and it's unclear which exact subset is needed for which operations. - -### Admin Consent - -All these are **Application permissions** (not delegated), so they require tenant admin consent: - -```bash -az ad app permission admin-consent --id +sponsors = [f"https://graph.microsoft.com/v1.0/users/{user_id}"] ``` -Admin consent may fail with 404 if the service principal hasn't replicated yet. Retry with 10-40s backoff: +Do not bind a service principal, group, or `/directoryObjects/` reference. -```python -for attempt in range(4): - wait = 10 * (attempt + 1) - time.sleep(wait) - rc, _, err = run_az(["ad", "app", "permission", "admin-consent", "--id", client_id]) - if rc == 0: - break -``` +### Create BlueprintPrincipal explicitly ---- +Creating a Blueprint does not create its BlueprintPrincipal. Always create or verify the principal immediately after creating or discovering the Blueprint, including idempotent resume paths. -## API Reference +## Current creation endpoints -### Create Agent Identity Blueprint +### Blueprint -``` -POST https://graph.microsoft.com/beta/applications -``` +```http +POST https://graph.microsoft.com/v1.0/applications/microsoft.graph.agentIdentityBlueprint +Content-Type: application/json -```json { - "@odata.type": "Microsoft.Graph.AgentIdentityBlueprint", - "displayName": "My Agent Blueprint", - "description": "Optional description", - "sponsors@odata.bind": [ - "https://graph.microsoft.com/beta/users/{user-object-id}" - ] + "displayName": "My Agent Blueprint", + "description": "Optional description", + "sponsors@odata.bind": [ + "https://graph.microsoft.com/v1.0/users/{sponsor-object-id}" + ] } ``` -Returns: Application object with `appId` (GUID) and `id` (object ID). +Persist both returned identifiers: -### Create BlueprintPrincipal +- `appId`: Blueprint client/application ID used by token requests. +- `id`: Blueprint directory-object ID used by object-specific Graph paths. -``` -POST https://graph.microsoft.com/beta/servicePrincipals -``` +### BlueprintPrincipal + +```http +POST https://graph.microsoft.com/v1.0/servicePrincipals/microsoft.graph.agentIdentityBlueprintPrincipal +Content-Type: application/json -```json { - "@odata.type": "Microsoft.Graph.AgentIdentityBlueprintPrincipal", - "appId": "{blueprint-appId-from-step-1}" + "appId": "{blueprint-app-id}" } ``` -### Create Agent Identity +### Agent Identity -``` -POST https://graph.microsoft.com/beta/servicePrincipals -``` +```http +POST https://graph.microsoft.com/v1.0/servicePrincipals/microsoft.graph.agentIdentity +Content-Type: application/json -```json { - "@odata.type": "Microsoft.Graph.AgentIdentity", - "displayName": "my-agent-instance", - "agentIdentityBlueprintId": "{blueprint-appId}", - "sponsors@odata.bind": [ - "https://graph.microsoft.com/beta/users/{user-object-id}" - ] + "displayName": "my-agent-instance", + "agentIdentityBlueprintId": "{blueprint-app-id}", + "sponsors@odata.bind": [ + "https://graph.microsoft.com/v1.0/users/{sponsor-object-id}" + ] } ``` -Returns: ServicePrincipal object. The `appId` or `id` field is the Entra Agent ID (UUID). +The Agent Identity has no backing application object. Do not call application password APIs for it. -### Find Existing Blueprint +### Agent User -``` -GET https://graph.microsoft.com/beta/applications?$filter=displayName eq 'My Agent Blueprint' -``` - -### Find Existing Agent Identity - -``` -GET https://graph.microsoft.com/beta/servicePrincipals?$filter=displayName eq 'my-agent-instance' -``` - -### Check BlueprintPrincipal Exists +Agent User creation remains on Microsoft Graph beta: +```http +POST https://graph.microsoft.com/beta/users ``` -GET https://graph.microsoft.com/beta/servicePrincipals?$filter=appId eq '{blueprint-appId}' -``` - ---- -## Complete Integration Sequence +Use the current request shape in `scripts/create_entra_agent_ids.py` and `docs/platform-learnings/entra-agent-users.md`. License the resulting user before relying on Teams or Outlook. -The correct order of operations: +## Permissions and consent -1. **Create dedicated app registration** (`az ad app create`) -2. **Create its service principal** (`az ad sp create --id `) -3. **Add permissions** (`az ad app permission add --id --api 00000003-... --api-permissions =Role =Role ...`) - - Note: Each `=Role` must be a **separate argument** to `--api-permissions`, not a joined string -4. **Grant admin consent** (`az ad app permission admin-consent --id `) — retry with backoff -5. **Wait 30s** for permission propagation to token endpoint -6. **Acquire token** via `ClientSecretCredential` with `client_credentials` flow -7. **Verify permissions** by attempting a test blueprint creation, retry with fresh tokens if 403 -8. **Get sponsor user ID** via `az ad signed-in-user show --query id -o tsv` -9. **Create Blueprint** (`POST /applications` with `AgentIdentityBlueprint` type + sponsors) -10. **Create BlueprintPrincipal** (`POST /servicePrincipals` with `AgentIdentityBlueprintPrincipal` type) -11. **Create Agent Identities** (`POST /servicePrincipals` with `AgentIdentity` type + sponsors + `agentIdentityBlueprintId`) +Discover Agent Identity application roles from the Microsoft Graph service principal rather than copying an old fixed list: -### Idempotency - -All steps should be idempotent — check for existing resources before creating: -- Blueprint: filter `/applications` by `displayName` -- BlueprintPrincipal: filter `/servicePrincipals` by `appId` -- Agent Identity: filter `/servicePrincipals` by `displayName` - -Agent Identities are **durable** — they should survive environment teardowns (infra destroy/recreate). Only delete them when decommissioning the project entirely. - ---- +```bash +az ad sp show \ + --id 00000003-0000-0000-c000-000000000000 \ + --query "appRoles[?contains(value, 'AgentIdentity')].{id:id,value:value}" \ + -o json +``` -## Reference Implementation +Grant the dedicated provisioner only the roles required by the provisioning operations, then grant admin consent. Expect service-principal and permission propagation delays; retry with bounded backoff and acquire a fresh token for each attempt. -See `scripts/create-entra-agent-ids.py` in this repo for a battle-tested implementation that handles all of the above, including: -- Auto-provisioning the dedicated app registration via `az ad` CLI -- Dynamic discovery of all 18 Agent Identity permissions -- Admin consent with retry and SP propagation handling -- Token verification with actual blueprint creation probe -- Idempotent blueprint, BlueprintPrincipal, and agent identity creation -- Sponsor assignment from `az ad signed-in-user show` -- Workload Identity Federation for OAuth2 token acquisition -- azd env integration for credential and ID storage +Entrabot writes Agent User delegated consent through: ---- +```http +POST https://graph.microsoft.com/v1.0/oauth2PermissionGrants +``` -## OAuth2 Token Flow (Workload Identity Federation) +Some tenants require `startTime` even when a newer Microsoft example omits it. Preserve the compatibility behavior in the repository helper. -Agent Identities authenticate using **Managed Identity + Workload Identity Federation** — NOT password credentials (which are explicitly blocked). +## Autonomous three-hop token flow -### Architecture +All requests use: +```text +https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token ``` -Container App (user-assigned MI) - → ManagedIdentityCredential.get_token("api://{blueprint-app-id}/.default") - → Azure AD token exchange (MI token → Agent ID token) - → JWT with oid = MI principal, aud = api://{blueprint-app-id} - → Backend validates JWT signature + claims -``` - -### Setup Steps -1. **Create federated identity credential** on the Blueprint: +### Hop 1: Blueprint certificate to T1 -``` -POST /applications/{blueprint-obj-id}/microsoft.graph.agentIdentityBlueprint/federatedIdentityCredentials +```text +client_id={blueprint_app_id} +scope=api://AzureADTokenExchange/.default +fmi_path={agent_identity_app_id} +grant_type=client_credentials +client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer +client_assertion={certificate_signed_jwt} ``` -```json -{ - "name": "aim-fic-budget-report", - "issuer": "https://login.microsoftonline.com/{tenant-id}/v2.0", - "subject": "{mi-principal-id}", - "audiences": ["api://AzureADTokenExchange"] -} +### Hop 2: Agent Identity FIC exchange to T2 + +```text +client_id={agent_identity_app_id} +scope=api://AzureADTokenExchange/.default +grant_type=client_credentials +client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer +client_assertion={T1} ``` -2. **Caller acquires token** using its MI: +### Hop 3: Agent User resource token -```python -from azure.identity import ManagedIdentityCredential -cred = ManagedIdentityCredential(client_id=mi_client_id) -token = cred.get_token(f"api://{blueprint_app_id}/.default") -# Include in request: Authorization: Bearer {token.token} +```text +client_id={agent_identity_app_id} +scope=https://graph.microsoft.com/.default +grant_type=user_fic +client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer +client_assertion={T1} +user_id={agent_user_object_id} +user_federated_identity_credential={T2} +requested_token_use=on_behalf_of ``` -3. **Backend validates token** using PyJWT + Microsoft JWKS: +`user_id` is Entrabot's canonical selector; Microsoft also documents `username={agent_user_upn}` as an alternative. For Azure Blob Storage, keep Hops 1 and 2 and request `https://storage.azure.com/.default` at Hop 3. -```python -import jwt -from jwt import PyJWKClient -jwks_client = PyJWKClient(jwks_uri) -signing_key = jwks_client.get_signing_key_from_jwt(token) -claims = jwt.decode(token, signing_key.key, algorithms=["RS256"], - audience=f"api://{blueprint_app_id}", - issuer=f"https://sts.windows.net/{tenant_id}/") -``` +Check every token response for an `error` key before reading `access_token`. Never log tokens, assertions, certificate private keys, or full token responses. -### Key Learnings +## Delegated mode is separate -- **Agent Identities do NOT support `passwordCredentials`** — you get `PropertyNotCompatibleWithAgentIdentity`. Use federated credentials (MI or certificate) instead. -- **Agent Identities are SPs without backing application objects** — you cannot use `/applications/{appId}/addPassword`. The SP has no corresponding application. -- **Federated credentials go on the Blueprint**, not the Agent Identity SP. Use the `.../microsoft.graph.agentIdentityBlueprint/federatedIdentityCredentials` path. -- **The `subject` is the MI's `principalId`** (object ID), not the client ID. -- **The `audiences` must be `["api://AzureADTokenExchange"]`** — this is the token exchange audience, not your API audience. -- **The Blueprint must have an Application ID URI set** — callers request tokens for `api://{blueprint-app-id}/.default`, but this scope won't resolve unless `identifierUris` includes `api://{app-id}`. Set it via `PATCH /applications/{id}` with `{"identifierUris": ["api://{app-id}"]}`. -- **Token issuer varies by endpoint version** — v1.0 tokens use `https://sts.windows.net/{tenant}/`, v2.0 tokens use `https://login.microsoftonline.com/{tenant}/v2.0`. Accept both in validation. -- **Token audience may or may not have `api://` prefix** — accept both `api://{app-id}` and `{app-id}` bare when validating the `aud` claim. +`ENTRABOT_MODE=delegated` uses MSAL browser authentication with device-code fallback and represents the signed-in human. It does not provide Agent User attribution. Agent Blueprints are confidential clients and cannot be turned into OAuth public clients for PKCE or device-code flows; use a separate app registration when both patterns are required. ---- +## Implementation checklist + +1. Create or recover the certificate-backed provisioner app. +2. Discover and grant required Graph application permissions. +3. Grant admin consent and wait for propagation with bounded retries. +4. Resolve the sponsor from JSON Azure CLI output. +5. Create or discover the Blueprint with the v1.0 subtype endpoint. +6. Create or verify BlueprintPrincipal with the v1.0 subtype endpoint. +7. Create or discover the Agent Identity with the v1.0 subtype endpoint. +8. Create or discover the Agent User on Graph beta. +9. Assign required Microsoft 365 licenses. +10. Create delegated consent records for Graph and optional Storage. +11. Store private key material only in the platform credential store. +12. Verify a three-hop resource token has `idtyp=user` and the Agent User object ID. -## Known Limitations (as of March 2026) - -1. **Preview API only** — all endpoints are under `/beta`, not `/v1.0` -2. **Sponsors must be Users** — ServicePrincipals and Groups are not accepted -3. **`/me` endpoint unavailable** in `client_credentials` flow — must use CLI for user context -4. **No "quick start" permission bundle** — must discover and grant 18+ individual permissions -5. **BlueprintPrincipal not auto-created** — requires explicit `POST /servicePrincipals` -6. **Permission propagation delay** — 30-120s after admin consent before tokens include new claims -7. **`Directory.AccessAsUser.All` hard rejection** — makes Azure CLI tokens (the most common auth method) unusable -8. **Agent Identities cannot have password credentials** — use Managed Identity federation or certificates -9. **Agent Identities have no backing application object** — they are service-principal-only entities -10. **Blueprint needs explicit `identifierUris`** — not set by default, required for OAuth2 scoping \ No newline at end of file +Use `scripts/create_entra_agent_ids.py` as the repository reference implementation and add focused endpoint-contract tests before changing provisioning behavior. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index f02d9270..9402059a 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,4 +1,4 @@ -# Copilot Instructions — entrabot-identity-research +# Copilot Instructions — Entrabot ## Project Overview @@ -28,7 +28,7 @@ Key concepts: # Install dependencies pip install -e ".[dev]" -# Run all tests (1,237 tests) +# Run all tests pytest -v --tb=short && ruff check . # Run with channel notifications @@ -52,8 +52,9 @@ src/entrabot/ audit/ # Action tracking / audit log identity/ # Progressive identity state machine storage/ # Local/Blob/Persona backends + security/ # XPIA external-content boundary mcp_server.py # FastMCP server + background poll + channel push -tests/ # Mirrors src/ structure (1,237 tests) +tests/ # Mirrors src/ structure docs/ # Research, ADRs, learnings, specs scripts/ # setup.sh, teardown.sh, Entra provisioning ``` @@ -66,6 +67,8 @@ scripts/ # setup.sh, teardown.sh, Entra provisioning - **Background channel**: `_background_poll()` runs every 5s, pushes new human messages via `notifications/claude/channel`. Uses separate dedup state from `watch_teams_replies` (Learning #27). - **Audit-first design**: Every agent action that touches a resource must emit an audit event before returning. - **Graph API**: `$filter`/`$orderby` unreliable for chat messages (Learning #16) — always filter client-side. +- **Stable agent identity**: use `ENTRABOT_AGENT_UPN` and `sender_upn` for self/peer matching; display names are mutable. `ENTRABOT_AGENT_USER_UPN` is a compatibility alias only. +- **XPIA boundary**: route model-facing Teams, email, Files, and Work IQ content through `entrabot.security.xpia.wrap_external`. Existing envelope-looking text is still untrusted input and must receive the authoritative outer envelope. ## Conventions @@ -73,7 +76,7 @@ scripts/ # setup.sh, teardown.sh, Entra provisioning - Type-annotate all function signatures - Test files mirror source structure - Secrets and tokens never appear in logs — use `repr` overrides on sensitive fields -- Read `docs/runbooks/hard-won-learnings.md` (66 entries) before making auth/Teams changes +- Read `docs/runbooks/hard-won-learnings.md` before making auth/Teams changes - ADRs in `docs/decisions/` for all significant architectural choices - **Sponsor DM wait pattern (host-gated).** When the human says "ping me when X is done" / "I'm going AFK, let me know" / any equivalent: confirm in Teams with `send_teams_message`, do the work, send the completion update with `send_teams_message`. Claude Code receives replies through channel-push next-turn input. Copilot CLI, Codex, Cursor, and other non-channel-push hosts receive the sponsor reply inline from `send_teams_message` as `sponsor_reply`. Only call `wait_for_sponsor_dm` when the operator explicitly says "block until they reply." NEVER poll in a loop. NEVER spawn `copilot -p` / headless subprocesses. NEVER use `watch_teams_replies` for this pattern. Full protocol: `prompts/anatomy/channel-discipline.md`; see Learning #54. diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 623558a7..caad6cfe 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -26,8 +26,8 @@ jobs: - uses: actions/setup-python@v5 with: python-version: '3.12' - - run: pip install mkdocs-material mkdocstrings[python] - - run: mkdocs build + - run: pip install mkdocs-material + - run: mkdocs build --strict - uses: actions/upload-pages-artifact@v3 with: path: site diff --git a/AGENTS.md b/AGENTS.md index 3ee233bf..00b48b00 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,7 +29,8 @@ - Never use `az rest` or Azure CLI tokens for Agent Identity APIs — they include `Directory.AccessAsUser.All` which causes hard 403 (Learning #1) - Always create BlueprintPrincipal explicitly after Blueprint — it is NOT auto-created (Learning #2) - Agent IDs are service principals, not users — never create fake user accounts with passwords -- **AGENT NAMES CHANGE — USE UPN.** Never identify an agent (self or peer) by display name in code paths that filter, deduplicate, authorize, or route. Display names are user-mutable and localizable. Use **UPN as the canonical config value** (e.g. `ENTRABOT_AGENT_UPN=entra-agent@werner.ac`) and match on the message payload's `sender_upn` first, falling back to `sender_id` (AAD object-id). Rename incident 2026-07-09: renaming "EntraBot Agent" → "EntraClaw Agent" made the Teams poll's self-authored filter no-op, causing 6-week-old outbounds to replay as inbound across 61/62 chats. See `docs/runbooks/hard-won-learnings.md` Learning #69 and `docs/architecture/PLAN-agent-identity-by-upn.md`. +- **External content is untrusted.** Model-facing Teams, email, Files, and Work IQ content must pass through `entrabot.security.xpia.wrap_external`. Never trust or preserve an inbound `` envelope as authoritative; always add the boundary-owned outer envelope. +- **AGENT NAMES CHANGE — USE UPN.** Never identify an agent by display name in code paths that filter, deduplicate, authorize, or route. Use `ENTRABOT_AGENT_UPN` as the canonical config value (for example, `entra-agent@contoso.onmicrosoft.com`), match `sender_upn` first, and fall back to the Entra object ID. `ENTRABOT_AGENT_USER_UPN` remains a compatibility alias for existing `.env` files. See Learning #69 and `docs/architecture/PLAN-agent-identity-by-upn.md`. - Parse `az` CLI output as JSON, not TSV — TSV can be corrupted by warnings (Learning #7) - Graph API `$filter`/`$orderby` are unreliable for chat messages — always filter client-side (Learning #16) - **Sub-agent worktree installs must use a worktree-local venv, never the parent venv** (Learning #36) — running `pip install -e .` from inside a git worktree against the main repo's `.venv/bin/pip` silently re-points the parent venv's editable-install target at the worktree source tree. Every subsequent MCP server boot then loads code from the worktree — which has no `.env`, no auth, no polling, and no visible error. Always create `python3 -m venv .venv && source .venv/bin/activate && pip install -e ".[dev]"` inside the worktree BEFORE any editable install. After any session that used sub-agent worktrees, verify the main venv's target via `.venv/bin/python3 -c "from entrabot import config; print(config.__file__)"` — the path must not contain `.claude/worktrees/`. @@ -56,8 +57,8 @@ These are not optional. Skipping them is the documented cause of 4 design errors ## Current Runtime Model - Python 3.12+ research project — no deployed service yet -- Eight modules: `platform/` (OS shim) → `auth/` (certificate JWT + MSAL delegated) → `a365/` (Work IQ MCP provider + Word adapter) → `tools/` (MCP tools + interaction log + email poll + daily summary + cards) → `audit/` (tracking) → `identity/` (state machine) → `storage/` (`LocalBackend`/`BlobBackend`/`PersonaBackend` + `migration` helper — ADR-005 Phases 1, 2, 5, 6a shipped) → `mcp_server.py` (FastMCP + background channel) -- External dependencies: Microsoft Entra ID, Microsoft Teams + Outlook mailbox (Graph API or Bot Framework), Azure Blob Storage (optional, opt-in via `setup.sh --use-cloud-memory`) +- Core runtime components: `platform/` (OS shim) → `auth/` (certificate JWT + MSAL delegated) → `a365/` (Work IQ MCP provider + Word adapter) → `tools/` (MCP tools + interaction log + email poll + daily summary + cards) → `audit/` (tracking) → `identity/` (state machine) → `storage/` (`LocalBackend`/`BlobBackend`/`PersonaBackend` + `migration` helper — ADR-005 Phases 1, 2, 5, 6a shipped) → `mcp_server.py` (FastMCP + background channel) +- External dependencies: Microsoft Entra ID, Microsoft Teams + Outlook mailbox (Microsoft Graph), Azure Blob Storage (optional, opt-in via `setup.sh --use-cloud-memory`) - **No default group chat.** Every Teams tool requires an explicit `chat_id`. Chats come from `create_chat`, the persisted `watched_chats` file, or the auto-discovery sweep over `/me/chats`. - **Body-first prompt.** `prompts/agent_system.md` loads at boot with `@include` expansion of `prompts/anatomy/*.md`. Persona-sati output (if configured) is appended AFTER the body and cannot override body rules. - Two auth modes via `ENTRABOT_MODE`: `agent_user` (three-hop), `delegated` (MSAL). Agent memory has a **parallel third hop** against `https://storage.azure.com/.default` (`acquire_agent_user_storage_token`). @@ -122,12 +123,12 @@ Note: efferent-copy may mechanically cover body-tool observe but not bootstrap/r ## Read These First -- `docs/engineering-status.md` — current state, test count (1,237), next steps +- `docs/engineering-status.md` — current state and next steps - `prompts/agent_system.md` + `prompts/anatomy/*.md` — the body prompt that governs your behaviour (security, channel discipline, identity/tools) - `docs/architecture/DESIGN-persona-sati-integration.md` — mind-body split design - `docs/decisions/005-cloud-hosted-memory.md` — cloud memory spec - `prompts/agent_system.md.archive` — original monolithic prompt, kept for reference -- `docs/runbooks/hard-won-learnings.md` — 66 learnings, read before making changes +- `docs/runbooks/hard-won-learnings.md` — read before making changes ## Commands diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ab5a8a1..d838820b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,31 @@ # Changelog +## Unreleased + +### Added + +- Rename-safe agent matching through canonical `ENTRABOT_AGENT_UPN` with object-ID fallback. +- Boundary-owned XPIA envelopes for model-facing Teams, email, Files, and Work IQ content. +- `read_email`, `read_interactions`, and `bootstrap_body_state`, bringing the MCP surface to 37 tools. +- Windows status tooling and expanded Windows setup/teardown guidance. + +### Changed + +- Blueprint, BlueprintPrincipal, and Agent Identity creation now use the dedicated Microsoft Graph v1.0 subtype endpoints; Agent User creation remains on beta. +- Documentation, quickstarts, API references, script references, and GitHub Pages navigation were refreshed against current code and platform APIs. +- GitHub Pages builds run in strict mode. + +### Fixed + +- Self-authored Teams messages remain filtered after an Agent User display-name change. +- Forged `` text can no longer bypass the authoritative XPIA boundary. +- A UTC-midnight-dependent test no longer fails intermittently on Windows CI. + +### Removed + +- Bot Gateway mode and its M365 Agents SDK dependency (ADR-006). +- The unused Claude pull-request review workflow. + ## v0.1 — 2026-05-21 First public release. Reference implementation for Microsoft Entra Agent ID and Microsoft Agent 365 (GA 2026-05-01). MIT licensed. **Research repo, not production-ready** — see Known Limitations below. @@ -27,7 +53,7 @@ First public release. Reference implementation for Microsoft Entra Agent ID and **Body prompt architecture** - Non-overridable body prompt at `prompts/agent_system.md` with `@include` expansion of `prompts/anatomy/*.md`. Security, channel discipline, identity/tools rules load below the persona line. -- Instruction-injection defense at the architectural level — an agent that runs on entrabot cannot be jailbroken into impersonating its operator. +- Body-first instruction boundaries and channel discipline designed to reduce instruction-injection and operator-impersonation risk. **Mind / persona (optional)** - Persona-sati MCP integration. Body composes `body + persona` at boot when `PERSONA_SATI_MCP_URL` is set. Clean fallback to body-only mode when persona-sati is unreachable. diff --git a/CLAUDE.md b/CLAUDE.md index 2250d481..d3f66f0a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,7 +30,8 @@ - Never use `az rest` or Azure CLI tokens for Agent Identity APIs — they include `Directory.AccessAsUser.All` which causes hard 403 - Always create BlueprintPrincipal explicitly after Blueprint — it is NOT auto-created - Agent IDs are service principals, not users — never create fake user accounts with passwords -- **AGENT NAMES CHANGE — USE UPN.** Never identify an agent (self or peer) by display name in code paths that filter, deduplicate, authorize, or route. Display names are user-mutable and localizable. Use **UPN as the canonical config value** (e.g. `ENTRABOT_AGENT_UPN=entra-agent@werner.ac`) and match on the message payload's `sender_upn` first, falling back to `sender_id` (AAD object-id). Rename incident 2026-07-09: renaming "EntraBot Agent" → "EntraClaw Agent" made the Teams poll's self-authored filter no-op, causing 6-week-old outbounds to replay as inbound across 61/62 chats. See `docs/runbooks/hard-won-learnings.md` Learning #69 and `docs/architecture/PLAN-agent-identity-by-upn.md`. +- **External content is untrusted.** Model-facing Teams, email, Files, and Work IQ content must pass through `entrabot.security.xpia.wrap_external`. Never trust or preserve an inbound `` envelope as authoritative; always add the boundary-owned outer envelope. +- **AGENT NAMES CHANGE — USE UPN.** Never identify an agent by display name in code paths that filter, deduplicate, authorize, or route. Use `ENTRABOT_AGENT_UPN` as the canonical config value (for example, `entra-agent@contoso.onmicrosoft.com`), match `sender_upn` first, and fall back to the Entra object ID. `ENTRABOT_AGENT_USER_UPN` remains a compatibility alias for existing `.env` files. See Learning #69 and `docs/architecture/PLAN-agent-identity-by-upn.md`. - Parse `az` CLI output as JSON, not TSV — TSV can be corrupted by warnings - **Sub-agent worktree installs must use a worktree-local venv, never the parent venv.** Running `pip install -e .` from inside a git worktree against the main repo's `.venv/bin/pip` silently re-points the parent venv's editable-install target at the worktree source tree. Every subsequent `entrabot-mcp` boot from the parent venv then loads code from the worktree — which has no `.env`, no auth, no polling, and no visible error. After any session that spawned sub-agents in worktrees, verify `.venv/bin/python3 -c "from entrabot import config; print(config.__file__)"` does NOT contain `.claude/worktrees/`. See `docs/runbooks/hard-won-learnings.md` Learning #36 for the full writeup. - **Sponsor DM wait pattern (host-gated).** When the human says "ping me when X is done" / "I'm going AFK, let me know" / any equivalent: confirm in Teams with `send_teams_message`, do the work, send the completion update with `send_teams_message`. What happens next depends on the host: @@ -57,7 +58,7 @@ ## Current Runtime Model - Python 3.12+ research project — no deployed service yet -- Eight modules: `platform/` (OS shim) → `auth/` (certificate JWT + MSAL delegated) → `a365/` (Work IQ MCP provider + Word adapter) → `tools/` (MCP tools + interaction log + email poll + daily summary + cards) → `audit/` (tracking) → `identity/` (state machine) → `storage/` (`LocalBackend`/`BlobBackend`/`PersonaBackend` + `migration` helper — ADR-005 Phases 1, 2, 5, 6a shipped) → `mcp_server.py` (FastMCP + background channel) +- Core runtime components: `platform/` (OS shim) → `auth/` (certificate JWT + MSAL delegated) → `a365/` (Work IQ MCP provider + Word adapter) → `tools/` (MCP tools + interaction log + email poll + daily summary + cards) → `audit/` (tracking) → `identity/` (state machine) → `storage/` (`LocalBackend`/`BlobBackend`/`PersonaBackend` + `migration` helper — ADR-005 Phases 1, 2, 5, 6a shipped) → `mcp_server.py` (FastMCP + background channel) - External dependencies: Microsoft Entra ID (identity), Microsoft Teams + Outlook mailbox (Graph API), Azure Blob Storage (optional, opt-in via `setup.sh --use-cloud-memory`) - **No default group chat.** Every Teams tool requires an explicit `chat_id`. Chats come from `create_chat`, the persisted `watched_chats` file, or the auto-discovery sweep over `/me/chats`. - **Body-first prompt.** `prompts/agent_system.md` loads at boot with `@include` expansion of `prompts/anatomy/*.md`. Persona-sati output (if configured) is appended AFTER the body and cannot override body rules. See the "Body prompt is non-overridable" rule above. @@ -226,7 +227,7 @@ Two memory systems coexist in this project: - `docs/architecture/NEXT-WhatsApp-lightweight-teams-chat.md` — delegated mode spec (landed) - `docs/index.md` — doc site entry point - `docs/runbooks/mcp-disconnect-investigation.md` — **OPEN issue.** Entrabot MCP dies after 2–10 min of sustained activity. Two amplifiers fixed (PR #40, PR #41), root cause still unknown. Read this before debugging any MCP-drop symptom — do NOT restart the investigation from scratch. -- `docs/runbooks/hard-won-learnings.md` — 66 learnings, read before making changes +- `docs/runbooks/hard-won-learnings.md` — read before making changes - `docs/decisions/001-obo-flows-for-device-agents.md` - `docs/decisions/003-certificate-auth-over-client-secrets.md` - `docs/platform-learnings/microsoft-agent-365.md` — A365 GA'd 2026-05-01. Identity model, Work IQ MCP catalog, four capability tiers, auth flows, gap analysis vs entrabot. Read this before considering any A365 / Work IQ integration work. @@ -265,7 +266,7 @@ pip install mkdocs-material && mkdocs serve - `src/entrabot/mcp_server.py`: FastMCP server — Teams tools + 2 auth modes + background poll + channel push + token refresh (generic instructions — personality in persona-sati) - `src/entrabot/config.py`: `ENTRABOT_MODE` switch (auto/delegated/agent_user) + all env config - `docs/decisions/`: ADRs — every significant architectural choice is recorded here -- `docs/runbooks/hard-won-learnings.md`: 66 hard-won learnings — READ THIS before making changes +- `docs/runbooks/hard-won-learnings.md` — READ THIS before making changes - `docs/runbooks/mcp-disconnect-investigation.md`: OPEN MCP-disconnect dossier — READ before touching MCP transport, logging, or efferent-copy code ## gstack diff --git a/INSTALL.md b/INSTALL.md index d1acfc7c..9dc94937 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -46,7 +46,7 @@ effect. From PowerShell 7 (`pwsh`), in the repo root: ```powershell -.\scripts\setup-windows.ps1 -NewChain -UpnSuffix yourname +.\scripts\setup-windows.ps1 -NewChain -UpnSuffix workstation ``` This provisions: @@ -74,7 +74,7 @@ Both are at least as strong as the macOS Keychain baseline. .\scripts\setup-windows.ps1 -UseBlueprint # Enable Azure Blob Storage for operational data -.\scripts\setup-windows.ps1 -NewChain -UpnSuffix yourname -UseCloudMemory +.\scripts\setup-windows.ps1 -NewChain -UpnSuffix workstation -UseCloudMemory ``` ### Teardown @@ -83,8 +83,10 @@ Both are at least as strong as the macOS Keychain baseline. .\scripts\teardown-windows.ps1 ``` -Removes MCP registrations from `claude.json` and `copilot mcp-config.json`. -Leaves the certificate and `.env` intact. +Removes Blueprint certificates from the Windows Certificate Store, deletes +`%LOCALAPPDATA%\entrabot`, removes certificate settings from `.env`, and +unregisters the MCP servers. It does **not** delete the tenant-side Agent User, +Agent Identity, or Blueprint. --- @@ -92,36 +94,25 @@ Leaves the certificate and `.env` intact. ### Prerequisites -Install manually or via Homebrew: +Run the repository prerequisite helper: ```bash -# Python 3.12+ -brew install python@3.12 - -# Azure CLI -brew install azure-cli - -# Git (usually pre-installed on macOS) -brew install git +./scripts/prereqs-macos.sh ``` -No build tools needed — macOS ships with the required C compiler via Xcode -Command Line Tools: - -```bash -xcode-select --install -``` +It installs or verifies Python 3.12+, Azure CLI, GitHub CLI, Node.js, Claude Code, +`jq`, and the Xcode Command Line Tools through Homebrew. ### Setup -```bash -./scripts/setup.sh -``` - -Or with a fresh identity chain: +Choose exactly one identity mode: ```bash -./scripts/setup.sh --new --with-upn-suffix=yourname +# Create a fresh identity chain +./scripts/setup.sh --new --with-upn-suffix=workstation + +# Or attach to an existing Blueprint +./scripts/setup.sh --use-blueprint= ``` #### Certificate storage @@ -132,7 +123,8 @@ the `keyring` Python package. No PEM files on disk. #### With Azure Blob Storage ```bash -./scripts/setup.sh --use-cloud-memory +./scripts/setup.sh --new --with-upn-suffix=workstation --use-cloud-memory +# Or add --use-cloud-memory to an existing --use-blueprint invocation. ``` ### Teardown @@ -166,7 +158,7 @@ sudo dnf install azure-cli ### Setup ```bash -./scripts/setup.sh --new --with-upn-suffix=yourname +./scripts/setup.sh --new --with-upn-suffix=workstation ``` #### Certificate storage @@ -197,7 +189,12 @@ After setup completes on any platform, verify the three-hop flow works: # Mac/Linux: source .venv/bin/activate # Test three-hop token acquisition -python -c "from entrabot.tools.teams import acquire_agent_user_token; print(acquire_agent_user_token()[:40])" +python - <<'PY' +from entrabot.config import get_config +from entrabot.tools.teams import acquire_agent_user_token + +print(acquire_agent_user_token(get_config())[:40]) +PY ``` You should see a 40-character token prefix. If you get an AADSTS error, check: diff --git a/README.md b/README.md index 6df8f079..68163527 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,11 @@ # Entrabot: Identity Research for Microsoft 365 Agents -Entrabot is a Python MCP server that gives a device-local agent its own Entra **Agent ID** and an **Agent User** that has all the capabilities of a human user in a Microsoft tenant. It can have a Teams presence and be invited to meetings to chat with your colleagues 1:1, a mailbox it can monitor and respond to, create and edit Word documents, make PowerPoint presentations, and allows you to access your CLI. The agent signs in autonomously, sends Teams messages from its own account, and writes audit events against its own object ID. It runs on macOS, Linux, and Windows, and works with Claude Code, Copilot CLI, or any MCP-speaking client. +Entrabot is a Python MCP server that gives a device-local agent its own Entra **Agent ID** and **Agent User**. The agent can sign in autonomously, send and receive Teams messages from its own account, use its mailbox, work with Microsoft 365 files, and write audit events against its own object ID. It runs on macOS, Linux, and Windows and works with Claude Code, Copilot CLI, or any MCP-speaking client. **All you need to get started is:** -* A Free Microsoft 365 Developer tenant (sign up at ) +* A Microsoft 365 development tenant where you can create app registrations and grant admin consent * A license that includes Teams and Outlook (E3 or E5 dev tenant licenses work) * Python 3.12 installed locally @@ -25,7 +25,7 @@ A device-local MCP server that turns an LLM agent into a first-class principal i It is for developers building agents on Microsoft 365 who want the security posture to match the architecture. The agent's smarts are up to you. entrabot gives it a secure seat at the table and the keys to the kingdom; what it does with that power is your call. -The body prompt (`prompts/agent_system.md` plus `prompts/anatomy/*.md`) is non-overridable and loads before any user turn. Security rules, channel discipline, and instruction-injection defense are baked in below the persona line. An agent that runs on entrabot cannot be jailbroken into impersonating its operator. +The body prompt (`prompts/agent_system.md` plus `prompts/anatomy/*.md`) loads before any optional persona. Security rules, channel discipline, and external-content boundaries are enforced by the body and reinforced in code. These controls reduce instruction-injection risk; they are not an absolute guarantee against every jailbreak technique. --- @@ -38,7 +38,7 @@ entrabot is the device-side glue for a set of platform primitives Microsoft ship - **Conditional Access for agents** — GA. Apply CA policies to Agent Identity sign-ins the same way you apply them to users. - **ID Protection for agents** — GA. Risk scoring and remediation against the agent's own object. - **FastMCP** — the Python MCP server framework. entrabot registers every Teams, Outlook, Files, Word, audit, and identity tool through it. -- **Three-hop certificate chain** — Blueprint token (cert JWT) → Agent Identity token (federated identity credential) → Agent User token (`user_fic` grant). No client secret in flight. Private key in macOS Keychain, Windows TPM via CNG, or Linux Secret Service. +- **Three-hop certificate chain** — Blueprint token (cert JWT) → Agent Identity token (federated identity credential) → Agent User token (`user_fic` grant). No client secret in flight. Private key in macOS Keychain, Windows CNG (TPM-backed when available), or Linux Secret Service. entrabot connects these. The Blueprint is provisioned via Graph. The Agent User is licensed and visible in Teams. The MCP server runs locally, mints tokens against Entra without a human, and exposes the resulting capability surface to the agent. @@ -89,10 +89,10 @@ Full walkthrough in [`docs/architecture/system-overview.md`](docs/architecture/s Mac or Linux: ```bash -git clone https://github.com/brandwe/entrabot-identity-research.git -cd entrabot-identity-research -./scripts/setup.sh --new --with-upn-suffix=yourname -source .venv/bin/activate +git clone https://github.com/microsoft/entrabot.git +cd entrabot +./scripts/prereqs-macos.sh # macOS; see INSTALL.md for Linux/Windows +./scripts/setup.sh --new --with-upn-suffix=workstation claude --dangerously-load-development-channels server:entrabot ``` @@ -127,7 +127,7 @@ While the agent is blocked waiting on a Teams reply (any host that calls `wait_f (•ᴗ•) zZz... listening for Teams DM [42s] (Ctrl+C to break) ``` -Frames cycle (`ʕ•ᴥ•ʔ waiting on sponsor`, `(´･ω･`) sponsor hasn't replied yet`, `(◕‿◕) still here, still waiting`, …) every ~30s with elapsed time. Ctrl+C breaks out cleanly. Full host-by-host protocol: [`docs/claude-copilot-cli-channel-port.md`](docs/claude-copilot-cli-channel-port.md) and [`prompts/anatomy/channel-discipline.md`](prompts/anatomy/channel-discipline.md). +Frames cycle (`ʕ•ᴥ•ʔ waiting on sponsor`, `(´･ω･`) sponsor hasn't replied yet`, `(◕‿◕) still here, still waiting`, …) every ~30s with elapsed time. Ctrl+C breaks out cleanly. Full host-by-host protocol: [`docs/claude-copilot-cli-channel-port.md`](docs/claude-copilot-cli-channel-port.md) and [`prompts/anatomy/channel-discipline.md`](prompts/anatomy/channel-discipline.md). On Windows, use `pwsh -File status-windows.ps1` for the equivalent status surface. After setup, use `./status.sh` as the canonical health and identity check: @@ -146,14 +146,14 @@ The full doc site: **** Direct pointers: - [Quickstart](docs/getting-started/quickstart.md) — five minutes from clone to first Teams message -- [MCP tool reference](docs/reference/mcp-tools.md) — every tool, every parameter -- [Setup script reference](docs/reference/setup-script.md) — every `setup.sh` flag +- [MCP tool reference](docs/reference/api/mcp-tools.md) — every tool, every parameter +- [Setup script reference](docs/reference/scripts/setup.md) — every `setup.sh` and `setup-windows.ps1` flag - [Script reference](docs/reference/scripts/operations.md) — status, health, DM, email, setup, teardown, and diagnostic scripts - [Token flows](docs/reference/token-flows.md) — the three hops, annotated - [System overview](docs/architecture/system-overview.md) — how the modules fit together -- [Architecture decisions](docs/decisions/README.md) — ADRs 001–005 +- [Architecture decisions](docs/decisions/README.md) — ADRs 001–006 - [Platform learnings](docs/platform-learnings/) — Entra Agent ID constraints, Agent 365, MSAL, OS-specific notes -- [Hard-won learnings](docs/runbooks/hard-won-learnings.md) — 66 non-obvious gotchas; read before changing auth or Teams code +- [Hard-won learnings](docs/runbooks/hard-won-learnings.md) — non-obvious gotchas; read before changing auth or Teams code - [Engineering status](docs/engineering-status.md) — what's shipped, what's open, what's next --- @@ -172,7 +172,7 @@ This is a research repo, not a production service. It runs reliably on a develop - Storage: `LocalBackend` (default) and `BlobBackend` (Azure Blob Storage, opt-in via `setup.sh --use-cloud-memory`) - Body-first prompt architecture with optional persona layer from a separate MCP (`persona-sati`) - Audit fails closed: if the audit write fails, the action does not proceed -- 1,237 tests; `pytest -v && ruff check .` gate every commit +- `pytest -v --tb=short && ruff check .` gates every commit **Persona-sati host bootstrap:** Hosts that attach persona-sati must call `bootstrap_session` before the first substantive answer because FastMCP instructions do not reliably reach the LLM prompt. If `mind_contract_available` is false, operate in body-only mode. When the mind contract is available, follow the per-turn cognition tools: `observe` around external tools, `reflect` for durable observations, and `recall` when a returned memory excerpt is insufficient. @@ -182,12 +182,12 @@ This is a research repo, not a production service. It runs reliably on a develop |---|---| | macOS | Shipped — Keychain-backed cert storage, full three-hop flow | | Linux | Works — Secret Service (libsecret) backend | -| Windows | Shipped, acceptance-tested on ARM64 Windows 11 — TPM-backed CNG cert storage | +| Windows | Shipped, acceptance-tested on ARM64 Windows 11 — CNG cert storage with TPM-first/software fallback | **Open:** -- AppContainer sandbox spike on Windows for stronger process isolation -- A few platform-edge bugs tracked in [`docs/engineering-status.md`](docs/engineering-status.md) (Agent Identity missing `Application.Read.All`; `add_file_comment` Word 404; persona-sati 12h MCP refresh bug paused at the Blueprint public-client constraint) +- MXC/AppContainer sandbox integration is under review and is not part of the current `main` runtime. +- Long-session MCP disconnect investigation and several scheduler/cursor precision fixes remain tracked in [`docs/engineering-status.md`](docs/engineering-status.md). --- diff --git a/TODOS.md b/TODOS.md index 1c3591ef..fa4744bb 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,266 +1,41 @@ -# TODOS +# Engineering TODOs -## Shipped (pending merge) +**Last reviewed:** 2026-07-10 +**Current status:** [`docs/engineering-status.md`](docs/engineering-status.md) -### Cursor-replay bug: identify self-authored by UPN, not display name -2026-07-09 incident — after the agent was renamed ("EntraBot Agent" → "EntraClaw Agent"), the Teams poll's self-authored filter no-op'd on display-name comparison and 6-week-old outbounds replayed as fresh inbound across 61/62 watched chats. +Keep this file limited to actionable work. Completed design history belongs in ADRs, changelog entries, or architecture plans marked **Shipped**. -Fix landed on this branch: +## In progress / merge queue -- `filter_human_messages` now matches on canonical UPN + AAD object-id fallback; display name is no longer an identity predicate anywhere in the code path. -- `read` (Teams Graph adapter) emits `sender_upn` alongside `sender_id`. -- All four callers in `mcp_server.py` (background poll, `send_teams_message` auto-wait, `watch_teams_replies`, `wait_for_sponsor_dm`) source UPN + object-id from config instead of the hard-coded `"EntraBot Agent"` string. -- `whoami` surfaces `agent_upn` + `agent_object_id` for observability. -- New Non-Negotiable "AGENT NAMES CHANGE — USE UPN" in `CLAUDE.md` and `AGENTS.md`; Learning #69 in `docs/runbooks/hard-won-learnings.md`. -- One-shot migration script `scripts/migrate_cursors_to_upn.py` with `--dry-run` and `--verify` flags. Idempotent; bumps `last_ts` past now + seeds `seen_ids_tail` with recent self-authored message IDs. -- Test count: 1562 → 1576 (+14 new tests across `tests/tools/test_watch.py::TestFilterHumanMessages` and `tests/test_cursor_migration.py`). +- [ ] **MXC sandbox integration (PR #86).** Rebase against current `main`, resolve conflicts, rerun the full test/lint gate in a worktree-local virtual environment, and update runtime docs only after merge. +- [ ] **Windows command-injection hardening (PR #76).** Re-evaluate the diff against current Windows process-launch helpers and either update and merge it or close it with a superseding reference. +- [ ] **Long-session MCP disconnect.** Continue from `docs/runbooks/mcp-disconnect-investigation.md`; do not restart the investigation without incorporating the existing evidence. -Remaining before close: Brandon reviews the diff, runs the migration script live against the 62 cursor blobs, restarts entrabot, and confirms the next poll pass is clean. +## Reliability backlog -## P0 — DO NEXT +- [ ] **Script-toolkit documentation closeout.** Keep Unix and Windows setup/status/teardown references synchronized with script help and actual cleanup behavior. +- [ ] **Blob environment test isolation.** Prevent host `ENTRABOT_BLOB_*` values from leaking into tests while preserving fail-closed half-configuration checks. +- [ ] **MCP server orphan cleanup.** Ensure abnormal host termination does not leave duplicate background pollers or stale child processes. +- [ ] **Daily-summary scheduler fixes.** Make timezone/day-boundary behavior deterministic and keep retries idempotent. +- [ ] **Email cursor precision.** Advance cursors without skipping same-timestamp messages or replaying already-delivered mail. +- [ ] **Long-running persona-sati authentication.** Replace restart-based recovery after bearer expiry without trying to make an Agent Blueprint act as an OAuth public client. -### persona-sati: implement /authorize + /token PKCE flow (blocks Claude Code SSE) -Claude Code v2.1.152 now does MCP OAuth 2.1 discovery and ignores `.mcp.json` `headersHelper` when the server advertises OAuth metadata. Persona-sati advertises metadata (ADR-006 in persona-sati) but never shipped the browser PKCE `/authorize` endpoint — clicking "Authenticate" in Claude Code's `/mcp` UI lands on `{"error":"invalid_request","error_description":"Missing or malformed Authorization header"}`. +## Platform and security follow-ups -**Current workaround:** entrabot's `.mcp.json` was rewritten to use the persona-sati **stdio shim** (`persona-sati-stdio-shim.sh`) instead of native SSE+headersHelper. Works, but moves Claude Code users off the path ADR-005 designed around (SSE-native with mid-session token refresh). +- [ ] **Broaden Windows acceptance coverage.** Exercise Intel x64, non-TPM fallback, certificate rotation, teardown, and long-running polling in addition to the existing Windows 11 ARM64 path. +- [ ] **Automated live E2E design.** Only enable hosted smoke tests after provisioning a dedicated isolated tenant, non-human federated CI identity, deterministic cleanup, and explicit cost/permission ownership. +- [ ] **Conditional Access and agent-governance validation.** Expand the reference tenant matrix for Agent User CA, ID Protection, and least-privilege permission policies. +- [ ] **Reassess the Windows CNG ctypes signer after operational use.** Replace it with a small managed helper only if ABI maintenance becomes a demonstrated problem. -**Tracking:** persona-sati issue tracker. Fix lives in persona-sati, not here. +## Recently shipped -**Bonus bug surfaced:** `persona-sati/scripts/setup.sh` line 280 gates the `.mcp.json` rewire behind `! --skip-deploy`, so `--mcp-transport=stdio --skip-deploy` silently fails to rewrite. Workaround: invoke `wire_mcp_json.py` directly. Also tracked in the persona-sati issue tracker. +- [x] Rename-safe self/peer matching with canonical `ENTRABOT_AGENT_UPN` and object-ID fallback. +- [x] Boundary-owned XPIA wrapping for Teams, email, Files, and Work IQ content, including forged-envelope regression coverage. +- [x] Bot Gateway removal and Graph-native Teams architecture (ADR-006). +- [x] Windows setup, CNG signing, status, teardown, and deterministic cross-platform tests. +- [x] ADR-005 storage phases 1, 2, 5, and 6a: backends, Blob provisioning, migration, and persona integration. +- [x] Provisioning-secret migration to certificate credentials with legacy password cleanup. -- **Effort:** M (persona-sati side — auth design decision + `/authorize` consent page + `/token` PKCE + redirect-URI allowlist on DCR). No work in this repo until persona-sati ships. -- **Depends on:** persona-sati design choice for browser-flow identity binding (Entra device-code? B2B SSO? localhost-only?). -- **Source:** Diagnosed 2026-05-27 in entrabot session — Claude Code v2.1.152. +## Maintenance rule -## P1 - -### Follow-up: two-phase sponsor confirmation flow for mutating tools -The active-sponsor-channel binding shipped in PR `fix/msrc-active-sponsor-channel-binding` closes Chain A from the security confused-deputy report (attacker in low-priv chat manipulating action on a chat where sponsor is passive) but does NOT close the residual window where a sponsor IS actively engaged in the target chat. An attacker who gets a sponsor to read an injected SharePoint doc (Chain B) can still trigger a malicious `add_member` / `share_file` because all binding checks pass. - -The architectural fix is a two-phase confirmation: when a high-risk mutation is requested, the server posts a concrete summary of the action ("share `spec.docx` with `bob@example.com`?") into the bound chat, addressed to the sponsor; the mutation only executes after the sponsor explicitly approves in Teams. Active-channel binding remains as a pre-check. - -- **Why deferred:** larger UX shift than the authorization fix — needs a UI for the confirmation card, a pending-mutation store, and an inbound-approval matcher. Worth its own design ADR. -- **Effort:** L -- **Source:** Internal security report 2026-06-04; rubber-duck review on the authorization fix plan flagged this as the only path to fully close Chain B without LLM-level taint tracking. - -### Follow-up: `read_file` content sanitization / spotlighting -Document content currently enters LLM context unguarded. Sponsor-readable but attacker-authored content can carry instructions that survive into tool-call reasoning. Pair with a content-spotlighting pass on `read_file` output so injected instructions are marked as data, not commands. - -- **Status update (2026-07-09):** Primary defense LANDED on `feat/xpia-content-wrapping` — every `read_file` / `read_email` / `read_teams_messages` / `read_word_document` / `read_a365_text_file` return now wraps the model-facing body in `…` at the tool boundary, and the body prompt was updated to teach the model that envelope-wrapped content is data. Boundary-enforced, not model-enforced. See `docs/architecture/PLAN-xpia-content-wrapping.md` (landing in PR #99) and Learning #70. Residual scope in this TODO: LLM-level taint tracking, spotlighting on adversarial patterns inside the envelope, and any follow-up hardening as adversarial evidence lands. Do NOT close this TODO — the wrap is the first mechanical layer, not the full defense. -- **Why deferred:** distinct from the security authorization fix; covers a wider class of prompt-injection vectors than just the add/share confused-deputy. -- **Effort:** M -- **Source:** Internal security report 2026-06-04, Chain B. - -### Script toolkit final phase: README + GitHub Pages script reference -After the remaining script-toolkit phases land, do a dedicated documentation -closeout pass so the new operational surface is discoverable and stable. - -Required deliverables: -- Update `README.md` to make `./status.sh` the canonical one-command status - entry point, and mention that `./scripts/setup.sh --status` delegates to the - same consolidated status implementation. -- Add a GitHub Pages/MkDocs reference page under `docs/reference/` that lists - every supported root and `scripts/` entry point, what it does, whether it is - read-only or mutating, required prerequisites/permissions, common flags, and - safe example invocations. -- Add that page to `mkdocs.yml` under the Reference section and link it from - `docs/index.md` and the README's project-layout / docs sections. -- Include deprecated compatibility wrappers such as `scripts/health_check.py`, - explaining their canonical replacement rather than leaving users to infer it. -- Verify the docs site builds (`mkdocs build`) and that every documented script - has at least one non-destructive smoke command (`--help`, `--dry-run`, - `--json`, `--health-only`, or equivalent) validated during the final pass. - -- **Effort:** S-M (README + one reference page + nav + smoke-check table) -- **Depends on:** All script-toolkit behavior settling, especially status, - health, setup/status delegation, cert inventory, sponsor/license reporting, - and non-mutating token acquisition paths. -- **Source:** Script-toolkit follow-up request, 2026-05-19. - -### ADR-005 Phase 2: MemoryBackend protocol + Local/Blob impls -Land the next phase of cloud-hosted memory. Spec: `docs/decisions/005-cloud-hosted-memory.md` §"Implementation phases" (Phase 2 row). Define `MemoryBackend` protocol in `src/entrabot/storage/backend.py` with `LocalBackend` (current behavior) and `BlobBackend` (uses Phase 1 `BlobStore`). Route `interaction_log.py`, `daily_summary.py`, and memory-file access through it. Driven by `ENTRABOT_KEEP_MEMORY_LOCAL` env var. -- **Effort:** S (~150 LOC + tests) -- **Depends on:** Phase 1 (`f900ba1`, shipped) -- **Source:** ADR-005 - -### Test isolation: interaction_log tests leak into production blob when ENTRABOT_BLOB_ENDPOINT is set -The `tmp_data_dir` fixture in `tests/tools/test_interaction_log.py` sets `ENTRABOT_DATA_DIR` to a pytest tmp path but does NOT clear `ENTRABOT_BLOB_ENDPOINT` / `ENTRABOT_BLOB_CONTAINER`. Since Phase 2/5 routed `log_interaction` and `read_day` through `get_backend()`, the factory reads those env vars and returns `BlobBackend` — which hits the real production container and ignores the tmp_data_dir. Result: 10 tests in `test_interaction_log.py` fail on any machine with the blob env configured (passed on the Phase 6a author's machine because they hadn't exported those vars). Observed 2026-04-17 during Phase 6a review: test run produced 443 passed / 10 failed; failing tests were reading 75 real chat entries from blob when they expected 2 from the tmp dir. -Fix: make the `tmp_data_dir` fixture (and any sibling fixture that patches config) also `monkeypatch.delenv("ENTRABOT_BLOB_ENDPOINT", raising=False)` + same for `ENTRABOT_BLOB_CONTAINER`. Consider a session-scoped autouse fixture that unsets blob env for *all* tests unless a test opts in. Also audit other test files that might have the same latent bug (`test_daily_summary.py`, `test_email_poll.py`, anywhere using `get_backend()`). -- **Effort:** S (~30 LOC — fixture edit + audit) -- **Source:** Phase 6a review 2026-04-17; failure is pre-existing on main, not introduced by Phase 6a - -### PersonaBackend.pull_all() missing mtime-newer-local check (Phase 6d scope) -`src/entrabot/storage/persona.py` `pull_all()` currently overwrites local files unconditionally — cloud is authoritative on pull. The persona-persistence plan §4.2 specified: "If local is newer (happens if session was offline), leave it (to be pushed next)." Phase 6a shipped without that check for the safe-starting-point framing, but it's a race-loss risk: if a session writes a memory file offline, the next online session's SessionStart pull will clobber it before the PostToolUse-Write push fires. The mitigation of this was planned for Phase 6d (ETag-based conflict resolution) but the simple mtime check should land sooner. -Fix: compare local file mtime vs blob's last-modified on pull, skip overwrite if local is newer, add to `PersonaReport` a new `skipped_local_newer` counter. Test: pytest fixture with a local file newer than the (fake) blob's content → pull_all must leave it. -- **Effort:** XS (~20 LOC + 2 tests) -- **Depends on:** Phase 6a (`1514dcd`, shipped) -- **Source:** Phase 6a review 2026-04-17; plan §4.2 said we'd do this, Phase 6a deferred - -### MCP server orphans when Claude Code exits -Observed twice: when the parent Claude process exits, the `entrabot-mcp` child keeps running. The new Claude session spawns a *second* MCP server, and both servers poll Graph independently — causing dual interaction-log writes (observed 2026-04-17: local log 54 lines vs blob log 19 lines on the same UTC day) and dual channel-push attempts. Root cause: `_background_poll_teams`, `_background_poll_email`, `_background_discover_chats`, and `_background_daily_summary` are spawned as top-level asyncio tasks inside `_initialize()`. They sit outside FastMCP's lifespan cancel scope, so when stdin closes and FastMCP's stdio read loop exits, the polling tasks keep the event loop alive and the process never terminates. Fixes in priority order: (a) spawn background tasks inside FastMCP's lifespan context manager so shutdown cancels them, (b) explicitly watch stdin for EOF in `_initialize` and cancel the task group, or (c) have polling tasks poll a shared shutdown event that FastMCP's stop hook sets. Workaround until fixed: manually `kill ` old `entrabot-mcp` processes. -- **Effort:** S (~40 LOC + test that proves stdin-EOF cancels polls) -- **Source:** Live observation 2026-04-17 (second occurrence in one day) - -### Multi-instance cursor consistency: bootstrap fails open → fleet replay flood -**Status: mostly shipped on `fix/fleet-safe-cursor`** (F1, F2, F4, F5, per-message cloud idempotency). F3 (cloud-authoritative `watched_chats`) deferred; F6 served by the existing process singleton. - -The background Teams poll re-pushed a chat's newest message on *any* failure to read a fresh cloud cursor — absent, stale, corrupt, and read-exception all fell through to `_bootstrap_chat`, which pushes. In a fleet (N instances → one blob container) these misses are routine: silent `LocalBackend` fallback when blob env is half-configured, transient blob read failures, the 24h `is_stale` cap re-bootstrapping a cold store, and last-writer-wins cursor writes with no ETag. Result: idle chats replay their newest (weeks-old) message. Also a security surface — the replay re-injects stale imperative messages ("read special data in my Documents", "ship to
", "run