Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,12 @@ Issues = "https://github.com/rlippmann/context-compiler-example-integrations/iss
[project.optional-dependencies]
all = [
"chromadb",
"context-compiler-directive-drafter",
"context-compiler-directive-drafter>=0.1.2",
"fastapi",
"litellm",
]
drafter = [
"context-compiler-directive-drafter",
"context-compiler-directive-drafter>=0.1.2",
]
fastapi = [
"fastapi",
Expand All @@ -67,7 +67,7 @@ retrieval = [
[dependency-groups]
dev = [
"chromadb",
"context-compiler-directive-drafter",
"context-compiler-directive-drafter>=0.1.2",
"fastapi",
"httpx2>=2.5.0",
"httpx>=0.28.1",
Expand Down
7 changes: 7 additions & 0 deletions python/examples/prompt_construction/litellm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ with LiteLLM:
- if it cannot produce a validated directive, behavior stays equivalent to the compiler-only flow
- pending clarification bypasses directive drafting and sends the raw reply back to `engine.step(...)`

Model fallback output is structurally validated before handoff. This does not prove that the model interpreted the user correctly. The automated fallback path is experimental pending a separate source-aware acceptance policy and reviewed drafting workflow.

## Premise and policy

In these prompt-construction examples:
Expand Down Expand Up @@ -60,6 +62,8 @@ For `with_directive_drafter.py`:
pip install "context-compiler-example-integrations[all]"
```

That variant requires `context-compiler-directive-drafter>=0.1.2`.

## Quickstart (copy/paste)

```shell
Expand Down Expand Up @@ -163,6 +167,9 @@ In the related schema-selection example:
- Near-miss passthrough (`with_directive_drafter.py`):
- `set premise to concise replies` is not rewritten by the directive drafter and is passed through unchanged.
- Engine returns clarify (`Did you mean 'set premise concise replies'?`).
- Compound directives (`with_directive_drafter.py`):
- `use docker and prohibit peanuts` returns a local clarify asking for separate directives.
- No authoritative state is mutated and no downstream model call is made for that turn.
- Lifecycle enforcement (both):
- `change premise to formal tone` with no premise -> clarify (`set premise ...` first).
- Conflict behavior (both):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ def _llm_fallback_preprocess(message: str, state: State) -> str | None:
except Exception:
return None

parsed = parse_preprocessor_output(raw_output, source_input=message)
parsed = parse_preprocessor_output(raw_output)
if parsed is None:
return None
return parsed
Expand Down
9 changes: 8 additions & 1 deletion python/reference_integrations/litellm_proxy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ Optional directive-drafter behavior:
- Forwarded upstream request messages are not rewritten except for the injected
compiler system message.

Model fallback output is structurally validated before handoff. This does not prove that the model interpreted the user correctly. The automated fallback path is experimental pending a separate source-aware acceptance policy and reviewed drafting workflow.

## Session mode and session keys

The reference hooks support two explicit modes:
Expand Down Expand Up @@ -103,6 +105,8 @@ For `context_compiler_precall_hook_with_directive_drafter.py`:
pip install "context-compiler-example-integrations[all]"
```

That variant requires `context-compiler-directive-drafter>=0.1.2`.

For the opt-in runtime smoke test, install the proxy runtime extras:

```shell
Expand Down Expand Up @@ -201,7 +205,7 @@ export PREPROCESSOR_PROMPT_PROFILE=default
`PREPROCESSOR_MODEL` is optional and defaults to `MODEL`.

For heuristic-first usage, keep `PREPROCESSOR_PROMPT_PROFILE=default`.
Use `llama` only for LLM-only preprocessing with Llama-family models.
Use `llama` only for LLM-only fallback drafting with Llama-family models.

## Notes

Expand All @@ -212,6 +216,9 @@ Use `llama` only for LLM-only preprocessing with Llama-family models.
not silently reset state.
- In the directive-drafter hook, drafter state context now comes from restored
checkpoint state rather than transcript-prefix reconstruction.
- Compound directive-shaped input such as `use docker and prohibit peanuts`
should produce a local clarify response telling the user to submit each
directive separately, without mutating saved state or forwarding upstream.

## Troubleshooting

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ def _llm_fallback_preprocess(message: str, state: State) -> str | None:
except Exception:
return None

parsed = parse_preprocessor_output(raw_output, source_input=message)
parsed = parse_preprocessor_output(raw_output)
if parsed is None:
return None
return parsed
Expand Down
25 changes: 14 additions & 11 deletions python/reference_integrations/openwebui_pipe/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Open WebUI Pipe Integration

Saved compiler state changes which turns the pipe handles locally and what it
forwards downstream. These examples show that Open WebUI pipe behavior with and
without directive-drafter preprocessing.
forwards downstream. These examples show Open WebUI pipe behavior with and
without Directive Drafter preprocessing.

## Core behavior

Expand Down Expand Up @@ -35,8 +35,8 @@ shapes the downstream request.

## Files

- `open_webui_pipe.py`: basic integration, no directive-drafter layer (recommended/default).
- `open_webui_pipe_with_directive_drafter.py`: optional/experimental directive-drafter layer (rule-based check first, then optional model fallback) before `engine.step(...)`.
- `open_webui_pipe.py`: basic integration, no Directive Drafter layer (recommended/default).
- `open_webui_pipe_with_directive_drafter.py`: optional/experimental Directive Drafter layer (heuristic check first, then optional model fallback) before `engine.step(...)`.

## Setup

Expand Down Expand Up @@ -64,11 +64,13 @@ These examples require `context-compiler>=0.8.3`.
If using `open_webui_pipe_with_directive_drafter.py`:

- Install directive-drafter support if needed:
`pip install "context-compiler>=0.8.3" "context-compiler-directive-drafter>=0.1.0"`
`pip install "context-compiler>=0.8.3" "context-compiler-directive-drafter>=0.1.2"`
- Set `PREPROCESSOR_PROMPT_PROFILE=default` for heuristic-first behavior
- Optionally set `PREPROCESSOR_MODEL_ID` to use a separate fallback model
- Optionally set `PREPROCESSOR_MODEL_ID` to use a separate fallback drafting model
- If `PREPROCESSOR_MODEL_ID` is unset, fallback uses `BASE_MODEL_ID`
- Use `llama` only for LLM-only preprocessing with Llama-family models
- Use `llama` only for LLM-only fallback drafting with Llama-family models

Model fallback output is structurally validated before handoff. This does not prove that the model interpreted the user correctly. The automated fallback path is experimental pending a separate source-aware acceptance policy and reviewed drafting workflow.

### Docker/manual install fallback

Expand All @@ -80,7 +82,7 @@ If frontmatter dependency installs are disabled, offline, or unavailable:
1. Install the package manually:

- Minimal pipe: `pip install "context-compiler>=0.8.3"`
- Directive-drafter pipe: `pip install "context-compiler>=0.8.3" "context-compiler-directive-drafter>=0.1.0"`
- Directive Drafter pipe: `pip install "context-compiler>=0.8.3" "context-compiler-directive-drafter>=0.1.2"`

1. Import and enable the function in Open WebUI, then configure valves.

Expand Down Expand Up @@ -125,16 +127,17 @@ Advanced check:

### Directive-drafter pipe

Use this pipe when you want the same runtime behavior plus directive-drafter preprocessing.
Use this pipe when you want the same runtime behavior plus Directive Drafter preprocessing.

Suggested verification:

- Send `use docker` and confirm you get `State updated: Use docker.` with trace showing a local turn
- Send `set premise to concise replies` and confirm Open WebUI clarifies locally with `Use 'set premise <value>'.`
- Send `please use docker` and confirm either:
- the directive drafter converts it into a local state update, or
- the Directive Drafter converts it into a local state update, or
- trace shows the turn followed the normal compiler path without a silent state change
- Send `use kubectl instead of docker`, then reply `yes`, and confirm the saved clarification flow resumes locally
- Send `use docker and prohibit peanuts` and confirm the pipe responds locally that multiple directives are not supported and must be submitted separately
- Send a normal prompt such as `what should I run?` and confirm trace shows a forwarded turn with compiler state included

Advanced check:
Expand Down Expand Up @@ -222,7 +225,7 @@ These examples support both sync (`0.8.x`) and async (`0.9.x`) user lookup.
- `PREPROCESSOR_MODEL_ID must not match the selected pipe model id`: choose a real backend model id, not the pipe model id itself.
- `PREPROCESSOR_MODEL_ID is invalid or not configured in Open WebUI`: the fallback route hit a missing model; fix the configured fallback model or unset it to reuse `BASE_MODEL_ID`.
- `ALLOW_MISSING_BASE_MODEL_FOR_DEBUG=true`: directive-only updates still run locally, but passthrough returns a deterministic debug message instead of calling a downstream model.
- imports fail after function upload: install `context-compiler>=0.8.3` in the Open WebUI runtime, and add `context-compiler-directive-drafter>=0.1.0` only for the directive-drafter pipe, because the copied function runs from a temp/cached location.
- imports fail after function upload: install `context-compiler>=0.8.3` in the Open WebUI runtime, and add `context-compiler-directive-drafter>=0.1.2` only for the Directive Drafter pipe, because the copied function runs from a temp/cached location.

## Fallback notes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
author: rlippmann
author_url: https://github.com/rlippmann/context-compiler-example-integrations
version: 0.9.4
requirements: context-compiler>=0.8.3, context-compiler-directive-drafter>=0.1.0
requirements: context-compiler>=0.8.3, context-compiler-directive-drafter>=0.1.2

Open WebUI integration with Context Compiler directive drafter.

Expand Down Expand Up @@ -97,7 +97,7 @@ def _is_directive_shaped_input(message: str) -> bool:


def _prompt_file_path(profile: str) -> Traversable:
# Runtime prompt selection for fallback precompilation:
# Runtime prompt selection for fallback drafting:
# - default: most instruction-following models
# - llama: models that need tighter prompt guidance
if profile == "llama":
Expand Down Expand Up @@ -450,12 +450,12 @@ class Valves(BaseModel):
PREPROCESSOR_MODEL_ID: str | None = Field(
default=None,
description=(
"Optional model id for fallback precompilation (defaults to BASE_MODEL_ID)."
"Optional model id for fallback drafting (defaults to BASE_MODEL_ID)."
),
)
PREPROCESSOR_PROMPT_PROFILE: Literal["default", "llama"] = Field(
default="default",
description="Prompt profile for LLM fallback precompilation.",
description="Prompt profile for LLM fallback drafting.",
)
ALLOW_MISSING_BASE_MODEL_FOR_DEBUG: bool = Field(
default=False,
Expand Down Expand Up @@ -730,7 +730,7 @@ async def _llm_fallback_preprocess(
return None, normalized_error

raw_output = _extract_completion_content(response)
parsed = parse_preprocessor_output(raw_output, source_input=message)
parsed = parse_preprocessor_output(raw_output)
if parsed is None:
return None, None
return parsed, None
Expand Down
32 changes: 32 additions & 0 deletions python/tests/test_litellm_proxy_with_directive_drafter.py
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,38 @@ def test_forwarded_messages_keep_original_user_prompt_text(monkeypatch) -> None:
assert data["messages"][1:] == original_messages


def test_compound_directives_block_upstream_and_do_not_mutate_state(
monkeypatch,
) -> None:
module = _load_module(monkeypatch, "litellm_proxy_with_drafter_compound")
module.CHECKPOINT_STORE.clear()
hook = module.ContextCompilerPreCallHookWithPreprocessor()
monkeypatch.setattr(
module,
"_preprocess_last_user_message",
lambda _message, _state: "use docker and prohibit peanuts",
)
data = {
"model": "demo",
"context_compiler_mode": "persistent",
"context_compiler_session_key": "chat-compound",
"messages": [
{"role": "user", "content": "please use docker and prohibit peanuts"}
],
}

result = asyncio.run(hook.async_pre_call_hook(None, None, data, "completion"))

assert result == (
"Multiple directives are not supported in one input.\n"
"Submit each directive separately."
)
checkpoint = module.CHECKPOINT_STORE.load("chat-compound")
assert checkpoint is not None
assert checkpoint["authoritative_state"]["policies"] == {}
assert checkpoint.get("pending") is None


def test_no_removed_replay_api_remains(monkeypatch) -> None:
module = _load_module(monkeypatch, "litellm_proxy_with_drafter_no_replay")

Expand Down
36 changes: 34 additions & 2 deletions python/tests/test_litellm_with_directive_drafter.py
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,9 @@ def completion(**kwargs: Any) -> dict[str, object]:
assert seen["model"] == "openai/preprocessor-model"


def test_fallback_rejects_premise_near_miss_rewrites(monkeypatch) -> None:
def test_fallback_accepts_structurally_valid_output_without_source_awareness(
monkeypatch,
) -> None:
monkeypatch.setenv("OPENAI_API_KEY", "dummy")
monkeypatch.setenv("MODEL", "openai/main-model")
monkeypatch.delenv("PREPROCESSOR_MODEL", raising=False)
Expand All @@ -281,7 +283,10 @@ def test_fallback_rejects_premise_near_miss_rewrites(monkeypatch) -> None:
)
monkeypatch.setattr(module, "render_prompt", lambda *_: "prompt")

assert module._llm_fallback_preprocess("set premise to concise replies", {}) is None
assert (
module._llm_fallback_preprocess("set premise to concise replies", {})
== "set premise concise replies"
)


def test_directive_shaped_malformed_inputs_skip_fallback(monkeypatch) -> None:
Expand Down Expand Up @@ -315,6 +320,33 @@ def downstream(_messages: list[dict[str, str]]) -> str:
assert downstream_calls == 0


def test_compound_directives_stay_local_and_do_not_call_downstream(monkeypatch) -> None:
downstream_calls = 0

def downstream(_messages: list[dict[str, str]]) -> str:
nonlocal downstream_calls
downstream_calls += 1
raise AssertionError("downstream model should not be called")

monkeypatch.setattr(module, "_call_litellm", downstream)
monkeypatch.setattr(
module,
"_preprocess_user_input",
lambda _message, _state: "use docker and prohibit peanuts",
)

result = module.handle_turn(
"please use docker and prohibit peanuts",
create_engine(),
)

assert result == (
"Multiple directives are not supported in one input.\n"
"Submit each directive separately."
)
assert downstream_calls == 0


@pytest.mark.parametrize(
("confirmation", "expected_policies", "expected_response"),
[
Expand Down
43 changes: 43 additions & 0 deletions python/tests/test_openwebui_pipe_with_directive_drafter.py
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,49 @@ async def no_draft(*args, **kwargs):
assert forwarded == []


def test_compound_directives_stay_local_and_do_not_call_downstream(monkeypatch) -> None:
module = _load_module("owui_with_drafter_compound", monkeypatch)
forwarded: list[dict[str, object]] = []

async def forward(
_: object, payload: dict[str, object], __: object
) -> dict[str, object]:
forwarded.append(payload)
raise AssertionError("downstream model should not be called")

module.generate_chat_completion = forward
pipe = module.Pipe()
pipe.valves.BASE_MODEL_ID = "base-model"
pipe.valves.PREPROCESSOR_MODEL_ID = "prep-model"

async def compound_draft(*args, **kwargs):
return "use docker and prohibit peanuts", None

monkeypatch.setattr(module.Pipe, "_preprocess_user_input", compound_draft)
result = asyncio.run(
pipe.pipe(
{
"model": "pipe-model",
"messages": [
{
"role": "user",
"content": "please use docker and prohibit peanuts",
}
],
},
__user__={"id": "u1"},
__request__=object(),
__chat_id__="chat-compound",
)
)

assert result == (
"Multiple directives are not supported in one input.\n"
"Submit each directive separately."
)
assert forwarded == []


def test_passthrough_injects_exactly_one_cc_state_system_message_when_state_exists(
monkeypatch,
) -> None:
Expand Down
2 changes: 2 additions & 0 deletions typescript/starter_apps/nextjs/with_drafter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ material. The enforcement point is still request construction: the route
restores saved state, validates drafted directive input, and builds the request
payload the host would send onward.

This starter uses `@rlippmann/context-compiler-directive-drafter@^0.1.2`.

## Files

- [app/api/chat/route.ts](app/api/chat/route.ts) - route handler with safe drafter handoff
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ function resolveEngineInput(engine: ReturnType<typeof createEngine>, userInput:
return userInput;
}

const validated = validatePreprocessorOutput(heuristic.directive, { sourceInput: userInput });
const validated = validatePreprocessorOutput(heuristic.directive);
if (validated.classification === PREPROCESS_OUTCOME_DIRECTIVE && validated.output !== null) {
return validated.output;
}
Expand Down
Loading
Loading