Raw evaluation data supporting the paper:
Tokai: Cognitive-State-Aware Agentic Productivity System for Neurodivergent Users
Seth Austin Harding, Hao-Yuan Chen, Shih-Wei Liao, Li-Wei Ko, Tony Siu, Ming-Fong Sie
| Folder | Description |
|---|---|
tokagent-evaluation/ |
Results from a structured 93-trial technical evaluation of TokAgent's tool-calling accuracy |
user-feedback/ |
Anonymized responses to a 31-question open-ended feedback survey completed by 20 testers (to be added) |
aggregate-analytics/ |
Aggregate usage statistics exported from Supabase (to be added after June 30, 2026) |
31 prompts × 3 runs = 93 total trials
| Metric | Value |
|---|---|
| Tool Selection Accuracy (TSA) | 87.0% (60/69) |
| Parameter Extraction Accuracy (PEA) | 95.0% (57/60) |
| Multi-Tool Accuracy (MTA) | 25.0% (3/12) |
| False Positive Rate (FPR) | 0.0% (0/12) |
| False Negative Rate (FNR) | 21.0% (17/81) |
| Mean Rounds to Completion | 1.0 |
| Mean Latency (all 93 runs) | 4.70s |
- Neural metrics (held constant): Focus 35.6 · Bio Energy 84 · Mental Fatigue 18.0 · Working Memory 42.0 · Sleep Quality 57
- Data source mode: Self-Report
- Language: English only
- Latency measurement: Manual stopwatch, ±0.2s margin
- State reset between runs: Chat history cleared; workspace state (tasks, timers, medication log) restored to baseline before each run
- Account: Dedicated test account created for evaluation; tested in isolation from personal account
See tokagent-evaluation/methodology.md for full details.
User feedback was collected anonymously. No personally identifiable information is included in this repository. Tester IDs (T01–T20) are arbitrary and carry no connection to real identities.
Citation will be added upon publication.
CC BY 4.0 — free to use with attribution.