Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 20 additions & 19 deletions .bazelrc.ai_checker
Original file line number Diff line number Diff line change
Expand Up @@ -12,27 +12,28 @@
# *******************************************************************************

###############################################################################
## GitHub Copilot SDK - Environment (config:copilot)
## GitHub Copilot SDK - Environment
###############################################################################
# The Copilot CLI needs HOME (for stored OAuth credentials) and proxy vars
# (to reach api.github.com behind a corporate proxy).
# No environment configuration is required here. The AI analysis runs at TEST
# time and the AI test rules bake the required environment-variable inheritance
# (HOME, the GitHub tokens, and proxy variables) into each target via
# RunEnvironmentInfo. Running `bazel test //path/to:my_ai_check` inherits those
# variables from your shell automatically — no --config or --test_env needed.
#
# These are scoped to --config=copilot so they don't affect other builds.
# The AI checker BUILD target applies this config automatically via a
# test --config=copilot line below.
# Provide credentials by exporting one of COPILOT_GITHUB_TOKEN / GH_TOKEN /
# GITHUB_TOKEN, or by logging in via the Copilot CLI (writes ~/.copilot/config.json).
#
# Auth docs: https://github.com/github/copilot-sdk/blob/main/docs/auth/index.md

# Auth
build:copilot --action_env=HOME
build:copilot --action_env=COPILOT_GITHUB_TOKEN
build:copilot --action_env=GH_TOKEN
build:copilot --action_env=GITHUB_TOKEN

# Proxy (Node.js checks both upper and lowercase)
build:copilot --action_env=HTTP_PROXY
build:copilot --action_env=HTTPS_PROXY
build:copilot --action_env=NO_PROXY
build:copilot --action_env=http_proxy
build:copilot --action_env=https_proxy
build:copilot --action_env=no_proxy
###############################################################################
## Project-specific guidelines (optional)
###############################################################################
# General + element-type guidelines are built in. Project-specific details
# (e.g. requirement levels, architecture levels) are injected as graded rules
# via label flags, set once here instead of on every AI test target.
#
# Point them at your own filegroup of .md files, or reuse the bundled SCORE
# examples shown below.
#
# build --//validation/ai_checker:project_guidelines=//validation/ai_checker:score_project_guidelines
# build --//validation/ai_checker:project_architecture_guidelines=//validation/ai_checker:score_project_architecture_guidelines
6 changes: 4 additions & 2 deletions bazel/rules/rules_score/docs/user_guide/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ The `tags = ["manual"]` attribute is strongly recommended to prevent the rule fr
Run the check explicitly with:

```bash
bazel test //my/package:feature_requirements_ai_check --config=copilot
bazel test //my/package:feature_requirements_ai_check
```

| Attribute | Type | Required | Description |
Expand All @@ -210,7 +210,9 @@ bazel test //my/package:feature_requirements_ai_check --config=copilot
| `score_threshold` | string | no | Minimum average quality score from 0 to 10 to pass the test (default: `"0.0"`) |
| `guidelines` | label | no | Filegroup of guideline Markdown files to override the built-in guidelines |

**Output files** (written to `bazel-bin/`):
**Output files** (the AI analysis runs at test time; reports are written to the
test's undeclared-outputs archive at
`bazel-testlogs/<package>/<name>/test.outputs/outputs.zip`):

| File | Content |
|---|---|
Expand Down
14 changes: 14 additions & 0 deletions bazel/rules/rules_score/examples/seooc/design/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ load(
"//bazel/rules/rules_score:rules_score.bzl",
"architectural_design",
)
load("//validation/ai_checker:ai_checker.bzl", "architecture_ai_test")

architectural_design(
name = "sample_seooc_design",
Expand All @@ -29,3 +30,16 @@ architectural_design(
],
visibility = ["//visibility:public"],
)

# Background context (read-only) for the AI architecture review.
filegroup(
name = "design_context",
srcs = ["design_context.md"],
)

architecture_ai_test(
name = "sample_seooc_design_ai_test",
context = ":design_context",
designs = [":sample_seooc_design"],
tags = ["manual"],
)
40 changes: 40 additions & 0 deletions bazel/rules/rules_score/examples/seooc/design/design_context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
<!-- ----------------------------------------------------------------------------
Copyright (c) 2025 Contributors to the Eclipse Foundation

See the NOTICE file(s) distributed with this work for additional
information regarding copyright ownership.

This program and the accompanying materials are made available under the
terms of the Apache License Version 2.0 which is available at
https://www.apache.org/licenses/LICENSE-2.0

SPDX-License-Identifier: Apache-2.0
----------------------------------------------------------------------------- -->

# Design Context: Sample SEooC

Background material provided to the AI reviewer as read-only reference. It is
**not** graded; it only helps the reviewer interpret the architecture under
review.

## Scope

This is a Safety Element out of Context (SEooC). The component is developed
without a concrete vehicle-level item, so assumptions of use (AoU) stand in for
the missing system context.

## Components

- The static design (`static_design.puml`) describes the component structure and
its public interfaces.
- The dynamic design (`dynamic_design.puml`) describes the runtime interaction
between the component and its environment.
- The public API (`public_api.puml`) defines the interfaces exposed to
integrators.

## Assumptions of Use

- Integrators are responsible for satisfying the documented assumptions of use
before relying on the component's safety claims.
- The execution environment provides the resources declared in the static
design.
111 changes: 95 additions & 16 deletions validation/ai_checker/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -28,47 +28,127 @@ filegroup(
# Default requirements engineering guidelines
filegroup(
name = "default_guidelines",
srcs = glob(["guidelines/*.md"]),
srcs = ["guidelines/general.md"] + glob(["guidelines/requirements/*.md"]),
visibility = ["//visibility:public"],
)

# Core AI checker library (analysis framework)
# Default architecture design guidelines
filegroup(
name = "default_architecture_guidelines",
srcs = ["guidelines/general.md"] + glob(["guidelines/architecture/*.md"]),
visibility = ["//visibility:public"],
)

# Empty default for the project-guideline flags below. Consumer repos override
# the flags in their .bazelrc to inject project-specific guidelines once,
# instead of setting an attribute on every AI test target.
filegroup(
name = "empty_guidelines",
srcs = [],
visibility = ["//visibility:public"],
)

# Project-specific requirement guidelines (graded). Override once via:
# build --//validation/ai_checker:project_guidelines=//path/to:my_guidelines
label_flag(
name = "project_guidelines",
build_setting_default = ":empty_guidelines",
visibility = ["//visibility:public"],
)

# Project-specific architecture guidelines (graded). Override once via:
# build --//validation/ai_checker:project_architecture_guidelines=//path/to:my_guidelines
label_flag(
name = "project_architecture_guidelines",
build_setting_default = ":empty_guidelines",
visibility = ["//visibility:public"],
)

# Example SCORE project guidelines, referenced by the flags above when a repo
# opts into the SCORE process levels.
filegroup(
name = "score_project_guidelines",
srcs = ["guidelines/project/score_requirement_levels.md"],
visibility = ["//visibility:public"],
)

filegroup(
name = "score_project_architecture_guidelines",
srcs = ["guidelines/project/score_architecture_levels.md"],
visibility = ["//visibility:public"],
)

# Core AI checker library (analysis framework).
# Deliberately free of any AI-SDK / LangChain dependency: it depends only on
# the AnalysisAgent interface, which concrete agents implement.
# Core analysis framework + extractors + the AnalysisAgent interface.
# No AI-SDK / LangChain dependency. The agents/ subpackage (concrete backends)
# is intentionally excluded — those are separate libraries below.
py_library(
name = "ai_checker_core",
srcs = glob(["src/ai_checker/*.py"]),
srcs = glob([
"src/ai_checker/*.py",
"src/ai_checker/extractors/*.py",
"src/ai_checker/reports/*.py",
]),
# Jinja2 report templates are loaded at runtime relative to the package,
# so they must travel in the library's runfiles.
data = glob(["src/ai_checker/reports/*.j2"]),
imports = ["src"],
visibility = ["//visibility:public"],
deps = [
"@trlc//trlc",
requirement("bigtree"),
requirement("jinja2"),
requirement("markupsafe"),
requirement("pydantic"),
requirement("pydot"),
requirement("pyyaml"),
],
)

# LangChain adapter for GitHub Copilot SDK
# Default AI backend: GitHub Copilot SDK agent (no LangChain).
py_library(
name = "copilot_langchain",
name = "copilot_agent",
srcs = [
"src/copilot_adapter/__init__.py",
"src/copilot_adapter/_client_manager.py",
"src/copilot_adapter/_errors.py",
"src/copilot_adapter/_message_converter.py",
"src/copilot_adapter/_preflight.py",
"src/copilot_adapter/_tool_converter.py",
"src/copilot_adapter/copilot_langchain.py",
"src/ai_checker/agents/__init__.py",
"src/ai_checker/agents/_client_manager.py",
"src/ai_checker/agents/_errors.py",
"src/ai_checker/agents/_preflight.py",
"src/ai_checker/agents/copilot_agent.py",
],
imports = ["src"],
visibility = ["//visibility:public"],
deps = [
requirement("langchain-core"),
":ai_checker_core",
requirement("github-copilot-sdk"),
requirement("pydantic"),
],
)

# Default orchestrator (uses GitHub Copilot SDK as default AI backend)
# Optional LangChain adapter: wraps any LangChain BaseChatModel (e.g. ChatOpenAI)
# as an AnalysisAgent. Used only when a custom langchain model is supplied via a
# custom ai_model target's create_agent().
py_library(
name = "langchain_agent",
srcs = [
"src/ai_checker/agents/__init__.py",
"src/ai_checker/agents/_errors.py",
"src/ai_checker/agents/langchain_agent.py",
],
imports = ["src"],
visibility = ["//visibility:public"],
deps = [
":ai_checker_core",
requirement("langchain-core"),
],
)

# Default orchestrator (uses the Copilot SDK agent as default AI backend).
# Intentionally does NOT depend on :langchain_agent — the default path is
# LangChain-free. A consumer wanting the LangChain path supplies a custom
# ai_model target whose create_agent() returns a LangChainAgent and which
# depends on :langchain_agent itself.
py_binary(
name = "orchestrator",
srcs = ["src/ai_checker/orchestrator.py"],
Expand All @@ -77,8 +157,7 @@ py_binary(
visibility = ["//visibility:public"],
deps = [
":ai_checker_core",
":copilot_langchain",
requirement("langchain-core"),
":copilot_agent",
],
)

Expand Down
Loading
Loading