AI Fact-Checker

A fact-checking system with multi-factor confidence scoring, citation analysis, and source credibility evaluation.

Features

• Literature search via PMC Open Access (polars-dovmed) with agent-browser fallback (Bing)
• Multi-factor confidence scoring with calibrated intervals
• Citation analysis via CrossRef and Semantic Scholar
• Linguistic confidence analysis and hedging detection
• Biomedical validation via PubTator3
• Console, JSON, Markdown, and HTML outputs
• Cached citation lookups
• Source credibility scoring based on domain heuristics

Architecture

flowchart LR
  A[Claim] --> B[Search via PMC OA + agent-browser fallback]
  B --> C[Source aggregation]
  C --> D[Citation analysis]
  D --> E[Model analysis via OpenRouter]
  E --> F[Confidence scoring]
  F --> G[Report and JSON output]

Loading

Quick Start

Installation

git clone https://github.com/fmschulz/scifactchecker
cd scifactchecker
npm install
npm run setup:agent-browser
cp .env.example .env

Edit .env and add your OpenRouter API key. Set the PMC OA API key in ~/.config/polars-dovmed/.env or POLARS_DOVMED_API_KEY.

Basic Usage

npm start
npm start "Climate change is caused by human activity"
npm start "Vaccines are safe" --output json
npm run dev "5G causes COVID-19"

CLI Options

factchecker [options] [claim]

Options:
  -V, --version          output the version number
  -o, --output <format>  Output format: console, json, markdown, html (default: "console")
  -s, --save             Save results to file (default: true)
  -v, --verbose          Verbose output with detailed analysis
  --no-citations         Disable citation analysis
  --model <model>        OpenRouter model to use (default: "openai/gpt-4o-mini")
  --confidence-only      Only output confidence score
  -h, --help             display help for command

Output Example

AI Fact-Checker

ENHANCED FACT-CHECKING AGENT STARTING
Claim: "Do giant viruses infect humans"
Timestamp: 1750608471861
------------------------------------------------------------
Step 1: Search (PMC OA + agent-browser fallback)...
Step 2: Combining and deduplicating sources...
Total unique sources: 20
Step 3: Enhanced citation analysis...
Citation Analysis: Processing 20 sources
Citation Analysis Complete: 6/20 sources with citation data
Citation coverage: 30.0%
Average citations: 127
Step 4: AI analysis with enhanced context...
Step 5: Multi-factor confidence analysis...
Enhanced Confidence Analysis complete (312ms)

ENHANCED FACT-CHECK COMPLETE
------------------------------------------------------------
VERDICT: Likely False
CONFIDENCE: 71% [65-77%]
EVIDENCE QUALITY: Moderate
SOURCES ANALYZED: 20
HIGH-CREDIBILITY SOURCES: 12
CITATION COVERAGE: 30.0%
UNCERTAINTY ESTIMATE: +/-6%
UNCERTAINTY FLAGS:
- limited confidence analysis
- limited citation coverage
------------------------------------------------------------
EXPLANATION:
The claim that giant viruses infect humans is not strongly supported by the available evidence...
------------------------------------------------------------
Step 6: Saving enhanced results to JSON...
Complete enhanced results saved to: results_factchecked/factcheck_enhanced_1750608471861.json

Configuration

OPENROUTER_API_KEY=sk-or-v1-xxxxx
OPENROUTER_MODEL=openai/gpt-4o-mini
OPENROUTER_LOGPROBS_MODEL=openai/gpt-4o-mini
YOUR_SITE_URL=http://localhost:3000
YOUR_SITE_NAME=AI Fact Checker
SEMANTIC_SCHOLAR_API_KEY=
POLARS_DOVMED_API_KEY=
FACTCHECKER_SEARCH_URL=https://www.bing.com/search?q=
AGENT_BROWSER_BIN=

Project Structure

scifactchecker/
|-- src/
|   |-- index.js
|   |-- core/
|   |   |-- FactChecker.js
|   |   |-- agentBrowserSearch.js
|   |   `-- sourceCredibility.js
|   |-- apis/
|   |   |-- crossref.js
|   |   |-- semanticScholar.js
|   |   `-- pubtator3.js
|   |-- confidence/
|   |   |-- citationAnalyzer.js
|   |   |-- tokenAnalyzer.js
|   |   `-- enhancedScoring.js
|   `-- utils/
|-- tests/
`-- results_factchecked/

Testing

npm test
npm run lint
npm run format

Documentation

• API reference: docs/API.md
• Architecture: docs/ARCHITECTURE.md
• Development: docs/DEVELOPMENT.md
• Changelog: CHANGELOG.md

Limitations

• Search uses PMC OA when available and falls back to agent-browser (Bing) when needed.
• Citation APIs have rate limits without API keys.
• English only.

License

MIT. See LICENSE.