Skip to content

Improve batch scan visibility: add README usage guide and rename contrib/multilingual → contrib/batch_scan #258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
WhereIs38 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Improve batch scan visibility: add README usage guide and rename contrib/multilingual → contrib/batch_scan #258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,33 @@ skillspector scan ./my-skill/ --format markdown --output report.md
skillspector scan ./my-skill/ --format sarif --output report.sarif
```

### Batch Scanning

Scan entire directories of skills in parallel from `contrib/batch_scan/`:

```bash
python -m contrib.batch_scan.batch_scan ./my-skills/ --no-llm
python -m contrib.batch_scan.batch_scan ./my-skills/ --workers 20 -f json -o report.json
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 20
```

Supports multilingual detection (zh/ja/ko) and terminal/JSON/Markdown output.

For LLM scans with higher concurrency, configure multiple API keys following
[`.env.example`](contrib/batch_scan/.env.example) — the pool improves throughput
and resilience, provided the keys don't share an account-level rate limit.

See the [contrib guide](contrib/batch_scan/docs/) for details.

> **Note on LLM support:** The default configuration targets DeepSeek as the
> cheapest public option. DeepSeek-Chat is
> [expected to sunset](https://api-docs.deepseek.com/), and the contributor
> does not have hardware to test against local models. The batch scanner was
> originally tested with OpenAI-compatible endpoints — DeepSeek's lack of
> structured-output support required manual JSON-parsing patches. If you can
> contribute a more universal backend (Ollama, vLLM, or a different provider),
> PRs are very welcome.
### Suppressing False Positives (baseline)

Suppress known/accepted findings so the risk score reflects only un-triaged
Expand Down
24 changes: 24 additions & 0 deletions contrib/batch_scan/.env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# SkillSpector Batch Scanner — DO NOT COMMIT
#
# Copy to the repository root as .env:
# cp contrib/batch_scan/.env.example .env
#
# =============================================================================
# Multi-key pool (recommended for batch scans)
# =============================================================================
#
# Format: key|base_url|model, separated by semicolons.
# Add as many keys as you want — the pool distributes requests across them.
# ⚠️ Only helps if keys don't share an account-level rate limit.
#
SKILLSPECTOR_API_KEYS="sk-or-xxx1|https://api.deepseek.com|deepseek-chat;sk-or-xxx2|https://api.deepseek.com|deepseek-chat;sk-or-xxx3|https://api.openai.com/v1|gpt-5.4"

# Force OpenAI-compatible provider mode
SKILLSPECTOR_PROVIDER=openai

# Single-key fallback (ignored when SKILLSPECTOR_API_KEYS is set)
OPENAI_API_KEY=sk-or-xxxxxxxxxxxxxxxxxxxxxxxx
OPENAI_BASE_URL=https://api.deepseek.com

SKILLSPECTOR_MODEL=deepseek-chat
SKILLSPECTOR_LOG_LEVEL=WARNING
32 changes: 16 additions & 16 deletions contrib/multilingual/CONTRIBUTING.md → contrib/batch_scan/CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,20 +10,20 @@
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
cp contrib/multilingual/.env.example .env # edit with your API keys
cp contrib/batch_scan/.env.example .env # edit with your API keys
```

Verify everything works:
```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 8
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8
```

---

## Project Map

```
contrib/multilingual/
contrib/batch_scan/
├── batch_scan.py # CLI entry + ThreadPoolExecutor (start here)
├── runner.py # graph.invoke() wrapper + 7 patches + pool wiring (core)
├── gap_fill.py # GapFillAnalyzer — LLM pass for 8 uncovered rules
Expand Down Expand Up @@ -63,30 +63,30 @@ contrib/multilingual/

```bash
# All 164 tests
python contrib/multilingual/tests/tests-pro/random_numbered.py # 120 unit (seed=42)
python contrib/multilingual/tests/test_pool_wiring.py # 4 smoke checks
python contrib/multilingual/tests/test_monkeypatch_invasiveness.py # 14 thematic
python contrib/multilingual/tests/test_monkeypatch_fragility.py # 26 thematic
python contrib/batch_scan/tests/tests-pro/random_numbered.py # 120 unit (seed=42)
python contrib/batch_scan/tests/test_pool_wiring.py # 4 smoke checks
python contrib/batch_scan/tests/test_monkeypatch_invasiveness.py # 14 thematic
python contrib/batch_scan/tests/test_monkeypatch_fragility.py # 26 thematic

# Review-themed only
python -m unittest \
contrib.multilingual.tests.test_monkeypatch_invasiveness \
contrib.multilingual.tests.test_monkeypatch_fragility -v
python contrib/multilingual/tests/test_pool_wiring.py
contrib.batch_scan.tests.test_monkeypatch_invasiveness \
contrib.batch_scan.tests.test_monkeypatch_fragility -v
python contrib/batch_scan/tests/test_pool_wiring.py

# Mutation test
python contrib/multilingual/tests/tests-pro/mutation_max.py
python contrib/batch_scan/tests/tests-pro/mutation_max.py

# End-to-end (fixture suite)
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 8
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 8 --no-llm
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8 --no-llm
```

**Three commands catch most regressions:**
```bash
python contrib/multilingual/tests/tests-pro/random_numbered.py
python contrib/multilingual/tests/test_pool_wiring.py
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 8
python contrib/batch_scan/tests/tests-pro/random_numbered.py
python contrib/batch_scan/tests/test_pool_wiring.py
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8
```

---
Expand Down
0 contrib/multilingual/__init__.py → contrib/batch_scan/__init__.py
View file
Open in desktop
File renamed without changes.
0 contrib/multilingual/annotation.py → contrib/batch_scan/annotation.py
View file
Open in desktop
File renamed without changes.
0 contrib/multilingual/api_pool.py → contrib/batch_scan/api_pool.py
View file
Open in desktop
File renamed without changes.
6 changes: 3 additions & 3 deletions contrib/multilingual/batch_scan.py → contrib/batch_scan/batch_scan.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,9 @@
Usage::
python -m contrib.multilingual.batch_scan ./skills/ --no-llm
python -m contrib.multilingual.batch_scan ./skills/ -f json -o report.json
python -m contrib.multilingual.batch_scan ./skills/ --lang zh --workers 8
python -m contrib.batch_scan.batch_scan ./skills/ --no-llm
python -m contrib.batch_scan.batch_scan ./skills/ -f json -o report.json
python -m contrib.batch_scan.batch_scan ./skills/ --lang zh --workers 8
"""

from __future__ import annotations
Expand Down
0 contrib/multilingual/detection.py → contrib/batch_scan/detection.py
View file
Open in desktop
File renamed without changes.
0 contrib/multilingual/discovery.py → contrib/batch_scan/discovery.py
View file
Open in desktop
File renamed without changes.
4 changes: 2 additions & 2 deletions contrib/multilingual/docs/DESIGN.md → contrib/batch_scan/docs/DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

```
CLI
│ python -m contrib.multilingual.batch_scan ./tests/fixtures/ --workers 7
│ python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --workers 7
│
▼
batch_scan.py :: main()
Expand Down Expand Up @@ -187,7 +187,7 @@ HTTP-level timeouts (Patch 6) prevent most hangs from reaching the 90s ceiling.
## File layout

```
contrib/multilingual/
contrib/batch_scan/
├── __init__.py # package init + dotenv preload
├── batch_scan.py # CLI + ThreadPoolExecutor
├── runner.py # graph wrapper + setup_deepseek_compat()
Expand Down
76 changes: 38 additions & 38 deletions contrib/multilingual/docs/README.md → contrib/batch_scan/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Zero changes to upstream `src/skillspector/`.
## What it does

```
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 7
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 7
```

1. Finds all `SKILL.md`-containing directories under the input root
Expand All @@ -38,7 +38,7 @@ source .venv/bin/activate
pip install -e .

# Copy and edit the environment template
cp contrib/multilingual/.env.example .env
cp contrib/batch_scan/.env.example .env
```

The `.env` file needs these keys (see `.env.example` for the full template):
Expand All @@ -60,19 +60,19 @@ The `.env` file needs these keys (see `.env.example` for the full template):
### Static-only (fast, no API keys needed)

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --no-llm
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --no-llm
```

### Full LLM scan

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 7
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 7
```

### Test with built-in fixtures

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 8
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8
```

23 skills designed to exercise every detection rule.
Expand Down Expand Up @@ -197,7 +197,7 @@ static rules, LLM finds 2–8 additional issues per skill.
skillspector scan ./tests/fixtures/malicious_skill/ -f json -o upstream.json

# Batch — scan all skills
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f json -o batch.json
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o batch.json
```

Key differences in batch output:
Expand All @@ -211,63 +211,63 @@ Key differences in batch output:
### Scan (LLM mode)

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 7 # default
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 1 # sequential, easy to read
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 20 # high throughput
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 7 # default
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 1 # sequential, easy to read
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 20 # high throughput
```

### Scan (static-only, no API keys)

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --no-llm
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --no-require-llm --no-llm # skip LLM even for non-English
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --no-llm
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --no-require-llm --no-llm # skip LLM even for non-English
```

### Output formats

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal # default (Rich)
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f json -o report.json
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f markdown -o report.md
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal # default (Rich)
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o report.json
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f markdown -o report.md
```

### Fixture test (built-in 23 skills)

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 8
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f terminal --workers 8 --no-llm
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f json -o report.json --workers 8
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f terminal --workers 8 --no-llm
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o report.json --workers 8
```

### Language override

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --lang auto --workers 4 # detect (default)
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --lang zh -f terminal --workers 4
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --lang ja -f terminal --workers 4
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --lang ko -f terminal --workers 4
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --lang en -f terminal --workers 4 # skip gap-fill
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang auto --workers 4 # detect (default)
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang zh -f terminal --workers 4
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang ja -f terminal --workers 4
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang ko -f terminal --workers 4
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --lang en -f terminal --workers 4 # skip gap-fill
```

### Debugging

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --workers 1 -V # single worker + verbose
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --workers 4 -V
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --workers 1 -V # single worker + verbose
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --workers 4 -V
skillspector scan ./tests/fixtures/malicious_skill/ --no-llm # verify upstream works
```

### Compare upstream vs batch

```bash
skillspector scan ./tests/fixtures/malicious_skill/ -f json -o upstream.json
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f json -o batch.json --workers 4
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o batch.json --workers 4
```

### CI

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f json -o report.json --workers 8
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o report.json --workers 8
if [ $? -eq 0 ]; then echo "All clean"; fi
```

Expand All @@ -294,7 +294,7 @@ if [ $? -eq 0 ]; then echo "All clean"; fi

```bash
# Single worker + verbose output — easiest to read
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --workers 1 -V
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --workers 1 -V

# Verify upstream still works
skillspector scan ./tests/fixtures/malicious_skill/ --no-llm
Expand All @@ -304,7 +304,7 @@ skillspector scan ./tests/fixtures/malicious_skill/ --no-llm

```bash
# Static-only + skip LLM requirement even for non-English skills
python -m contrib.multilingual.batch_scan ./tests/fixtures/ --no-require-llm --no-llm
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ --no-require-llm --no-llm
```

## Exit codes
Expand All @@ -318,7 +318,7 @@ python -m contrib.multilingual.batch_scan ./tests/fixtures/ --no-require-llm --n
CI usage:

```bash
python -m contrib.multilingual.batch_scan ./tests/fixtures/ -f json -o report.json
python -m contrib.batch_scan.batch_scan ./tests/fixtures/ -f json -o report.json
if [ $? -eq 0 ]; then
echo "All clean"
fi
Expand Down Expand Up @@ -354,30 +354,30 @@ See `DESIGN.md` for architecture details and `docs/archive/FUTURE_WORK.md` for s
# === All 164 tests ===

# Unit tests — random order (seed=42, 120 tests)
python contrib/multilingual/tests/tests-pro/random_numbered.py
python contrib/batch_scan/tests/tests-pro/random_numbered.py

# Pool wiring smoke test (4 checks)
python contrib/multilingual/tests/test_pool_wiring.py
python contrib/batch_scan/tests/test_pool_wiring.py

# Monkey-patch invasiveness (14 tests)
python contrib/multilingual/tests/test_monkeypatch_invasiveness.py
python contrib/batch_scan/tests/test_monkeypatch_invasiveness.py

# Monkey-patch fragility (26 tests)
python contrib/multilingual/tests/test_monkeypatch_fragility.py
python contrib/batch_scan/tests/test_monkeypatch_fragility.py

# === Convenience ===

# All review-themed tests in one command
python -m unittest \
contrib.multilingual.tests.test_monkeypatch_invasiveness \
contrib.multilingual.tests.test_monkeypatch_fragility -v
python contrib/multilingual/tests/test_pool_wiring.py
contrib.batch_scan.tests.test_monkeypatch_invasiveness \
contrib.batch_scan.tests.test_monkeypatch_fragility -v
python contrib/batch_scan/tests/test_pool_wiring.py

# Mutation test — 30 injected bugs across 4 risk areas
python contrib/multilingual/tests/tests-pro/mutation_max.py
python contrib/batch_scan/tests/tests-pro/mutation_max.py

# Sequential pytest (if pytest installed)
pytest contrib/multilingual/tests/tests-pro/ -v
pytest contrib/batch_scan/tests/tests-pro/ -v
```

## For PR Reviewers
Expand Down
0 contrib/multilingual/docs/REVIEW_RESPONSE.md → contrib/batch_scan/docs/REVIEW_RESPONSE.md
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion ...al/docs/archive/ARCHITECTURE_DEEP_DIVE.md → ...an/docs/archive/ARCHITECTURE_DEEP_DIVE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ SKILLSPECTOR_PROVIDER env var
The contrib layer sits entirely outside upstream. It imports upstream classes as parents and wraps upstream functions:

```
contrib/multilingual/
contrib/batch_scan/
├── batch_scan.py ← CLI + ThreadPoolExecutor
├── runner.py ← graph.invoke() wrapper + 7 safety patches
├── gap_fill.py ← GapFillAnalyzer(LLMAnalyzerBase)
Expand Down
6 changes: 3 additions & 3 deletions ...ltilingual/docs/archive/DESIGN_HISTORY.md → ...batch_scan/docs/archive/DESIGN_HISTORY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
1. Zero changes to `src/skillspector/`
2. Subclass and wrap, don't rewrite
3. Output comparable with standard single-skill scan
4. All extensions in `contrib/multilingual/`
4. All extensions in `contrib/batch_scan/`

---

Expand All @@ -23,7 +23,7 @@
### Four-layer model

```
CLI layer python -m contrib.multilingual.batch_scan
CLI layer python -m contrib.batch_scan.batch_scan
Scheduling layer ThreadPoolExecutor(max_workers=N)
API Pool layer ApiKeyPool (multi-key scheduler)
Graph layer graph.invoke() per skill (upstream, untouched)
Expand Down Expand Up @@ -96,7 +96,7 @@ Chose stdlib `unicodedata` over ML-based detectors (e.g., `langdetect`, `fasttex
### Files created (9 source + tests + docs)

```
contrib/multilingual/
contrib/batch_scan/
├── __init__.py # Package init + dotenv pre-loading
├── discovery.py # Recursive SKILL.md finder
├── detection.py # Unicode script-ratio detection
Expand Down
Loading