🔍 Image Forgery Detection via Dual-Stream Residual Network

A deep learning approach to detecting image tampering using parallel RGB and Error Level Analysis (ELA) feature streams.

---

📑 Table of Contents

• Overview
• Key Features
• Quick Start / Usage
• Approach
• Model Architecture
• ELA (Error Level Analysis)
• Dataset & Preprocessing Pipeline
• Training Pipeline
• Evaluation & Results
• Project Structure
• Future Work

---

📖 Overview

This project implements a Dual-Stream Residual Network for detecting forged (tampered) images. The model simultaneously processes two representations of every input image:

1. RGB Stream — Learns visual features from the original image.
2. ELA Stream — Learns compression artifact inconsistencies from the Error Level Analysis representation.

By fusing features from both streams, the model captures both high-level visual semantics and low-level forensic signals, achieving ~89% accuracy on the validation set with an AUC of 0.9853.

✨ Key Features

• Dual-Stream Architecture: Combines standard RGB visual data with ELA compression artifacts for high-confidence detection.
• In-Memory ELA Generation: Generates Error Level Analysis representations on the fly without heavy disk I/O, saving storage and time.
• Auto-Recovery Training: Built-in checkpointing on Google Drive ensures Colab session timeouts don't wipe out your progress.
• Custom Residual Blocks: Efficient ResNet-style blocks with roughly 2.5 million parameters designed specifically to fit into memory-constrained environments.
• Mixed-Precision Training: Leverages torch.cuda.amp to maximize batch sizes and speed up training on standard GPUs like the T4.

---

🚀 Quick Start / Usage

1. Environment Setup

The notebook is optimized for Google Colab with GPU (T4) acceleration.

# Mount Google Drive (for dataset & checkpoint persistence)
from google.colab import drive
drive.mount('/content/drive')

2. Dataset Organization

Organize your dataset as follows:

dataset/
├── Authentic/
│   ├── img001.jpg
│   ├── img002.jpg
│   └── ...
└── Forged/
    ├── img001.jpg
    ├── img002.jpg
    └── ...

3. Training

Open Dual_Stream_Model.ipynb in Google Colab and run all cells sequentially. The auto-recovery system will automatically resume from the last checkpoint if training is interrupted.

4. Inference

# Load the best model
model = DualStreamResNet(num_classes=2)
checkpoint = torch.load('dual_stream_best_94.77%.pth', map_location=DEVICE)
model.load_state_dict(checkpoint)
model.eval()

# Prepare input
rgb_tensor = val_transforms(image).unsqueeze(0).to(DEVICE)
ela_image = convert_to_ela_ram(image, quality=90)
ela_tensor = val_transforms(ela_image).unsqueeze(0).to(DEVICE)

# Predict
with torch.no_grad():
    output = model(rgb_tensor, ela_tensor)
    _, predicted = torch.max(output, 1)
    result = "Forged" if predicted.item() == 1 else "Authentic"

---

🧠 Approach

The core insight is that forged regions in JPEG images exhibit different compression characteristics than authentic regions. By re-saving the image at a known quality and computing the pixel-wise difference (ELA), tampered areas light up — but this signal is often too subtle for human eyes. Our dual-stream network is designed to combine both visual and forensic cues for robust forgery detection.

flowchart LR

A["Input Image"]
B["RGB Representation"]
C["ELA Representation"]
D["RGB Feature Stream"]
E["ELA Feature Stream"]
F["Feature Fusion (Concatenation)"]
G["Classifier Head"]
H{"Prediction"}
I["Authentic"]
J["Forged"]

A --> B
A --> C
B --> D
C --> E
D --> F
E --> F
F --> G
G --> H
H -->|Class 0| I
H -->|Class 1| J

Loading

---

🏗️ Model Architecture

The DualStreamResNet consists of ~2.55 million trainable parameters and processes two parallel 224×224 input images.

High-Level Architecture

graph TB

subgraph InputLayer["Input Layer"]
RGB["RGB Image\nShape: 1×3×224×224"]
ELA["ELA Image\nShape: 1×3×224×224"]
end

subgraph RGBStream["RGB Stream (1.2M Params)"]
R1["Conv2d\n3→32\n3×3 + BN + ReLU"]
R2["ResidualBlock\n32→64"]
R3["MaxPool\n2×2"]
R4["ResidualBlock\n64→128"]
R5["MaxPool\n2×2"]
R6["ResidualBlock\n128→256"]
R7["MaxPool\n2×2"]
R8["AdaptiveAvgPool\n1×1"]
R9["Flatten\n256"]

R1 --> R2 --> R3 --> R4 --> R5 --> R6 --> R7 --> R8 --> R9
end

subgraph ELAStream["ELA Stream (1.2M Params)"]
E1["Conv2d\n3→32\n3×3 + BN + ReLU"]
E2["ResidualBlock\n32→64"]
E3["MaxPool\n2×2"]
E4["ResidualBlock\n64→128"]
E5["MaxPool\n2×2"]
E6["ResidualBlock\n128→256"]
E7["MaxPool\n2×2"]
E8["AdaptiveAvgPool\n1×1"]
E9["Flatten\n256"]

E1 --> E2 --> E3 --> E4 --> E5 --> E6 --> E7 --> E8 --> E9
end

subgraph Classifier["Classifier Head (132K Params)"]
C1["Concat\n512"]
C2["Linear\n512→256\nBN + ReLU"]
C3["Dropout\np=0.6"]
C4["Linear\n256→2"]
C5["Authentic / Forged"]

C1 --> C2 --> C3 --> C4 --> C5
end

RGB --> R1
ELA --> E1
R9 --> C1
E9 --> C1

Loading

Residual Block Design

Each stream uses custom Residual Blocks with skip connections, enabling stable gradient flow during deep network training.

graph TB

subgraph ResidualBlock["Residual Block"]

Input["Input\nC_in"]

Conv1["Conv2d\nC_in → C_out\n3×3\npad=1\nno bias"]
BN1["BatchNorm2d\nC_out"]
ReLU1["ReLU"]

Conv2["Conv2d\nC_out → C_out\n3×3\npad=1\nno bias"]
BN2["BatchNorm2d\nC_out"]

Skip["Skip Connection\nConv2d 1×1 + BN\n(if C_in ≠ C_out)"]

Add["Element-wise Add"]
ReLU2["ReLU\nOutput C_out"]

Input --> Conv1
Conv1 --> BN1
BN1 --> ReLU1
ReLU1 --> Conv2
Conv2 --> BN2
BN2 --> Add

Input --> Skip
Skip --> Add

Add --> ReLU2

end

Loading

Key Design Choices:

• No bias in Conv2d layers — since BatchNorm absorbs the bias term
• 1×1 skip projections — used when channel dimensions change between input and output
• ReLU after residual addition — standard post-addition activation

Feature Stream Pipeline

Each stream (RGB and ELA) follows identical topology but learns independent weights:

Stage	Layer	Output Shape
0	Conv2d(3→32, 3×3, pad=1) + BatchNorm2d + ReLU	[32, 224, 224]
1	ResidualBlock(32→64)	[64, 224, 224]
2	MaxPool2d(kernel=2, stride=2)	[64, 112, 112]
3	ResidualBlock(64→128)	[128, 112, 112]
4	MaxPool2d(kernel=2, stride=2)	[128, 56, 56]
5	ResidualBlock(128→256)	[256, 56, 56]
6	MaxPool2d(kernel=2, stride=2)	[256, 28, 28]
7	AdaptiveAvgPool2d(output_size=1)	[256, 1, 1]
8	Flatten	[256]

Classifier Head

After both streams produce their 256-dimensional feature vectors, they are concatenated into a 512-dimensional vector and passed through the classifier:

Layer	Details	Output Shape
Linear	512 → 256	[256]
BatchNorm1d	Normalizes the 256-dim features	[256]
ReLU	Activation	[256]
Dropout	p=0.6 (aggressive regularization)	[256]
Linear	256 → 2 (binary classification)	[2]

---

🔍 ELA (Error Level Analysis)

ELA is a forensic technique that reveals compression inconsistencies in JPEG images. The process is performed in-memory (RAM) to save disk space:

flowchart TB
    A["📷 Original Image"] --> B["Re-save as JPEG\nat quality=90"]
    B --> C["Re-open\nRe-saved Image"]
    A --> D["Compute Pixel-wise\nAbsolute Difference"]
    C --> D
    D --> E["Scale × 20\n(Amplify Differences)"]
    E --> F["🖼️ ELA Image"]
    
    style A fill:#4CAF50,color:#fff
    style F fill:#FF9800,color:#fff

Loading

Implementation Details:

def convert_to_ela_ram(img, quality=90):
    buffer = BytesIO()
    img.save(buffer, 'JPEG', quality=quality)
    buffer.seek(0)
    resaved = Image.open(buffer)
    ela = ImageChops.difference(img, resaved)
    extrema = ela.getextrema()
    max_diff = max([ex[1] for ex in extrema])
    if max_diff == 0:
        max_diff = 1
    scale = 255.0 * 20.0 / max_diff
    ela = ImageEnhance.Brightness(ela).enhance(scale)
    return ela

• Quality 90 is used for re-saving — a high-enough quality that authentic areas remain virtually unchanged.
• 20× amplification magnifies subtle differences, making tampered regions visually (and computationally) distinguishable.
• The entire process runs via BytesIO (in-RAM), avoiding disk I/O overhead.

---

🗂️ Dataset & Preprocessing Pipeline

The DualStreamDataset is a custom PyTorch dataset that dynamically generates ELA images alongside the original RGB images.

graph TB
    A["Dataset Folder"] --> B["ImageFolder<br/>auto class mapping"]
    B --> C["Load Image Path + Label"]

    subgraph DatasetPipeline [DualStreamDataset __getitem__]
        C --> D["Open Image - PIL"]
        D --> E["Convert to RGB"]
        E --> F["Generate ELA<br/>in-memory"]
        E --> G["Apply RGB Transforms"]
        F --> H["Apply ELA Transforms"]
    end

    G --> I["rgb_tensor, ela_tensor, label"]
    H --> I

    subgraph Transforms [Transforms - Train]
        T1["Resize 256"] --> T2["RandomResizedCrop 224"]
        T2 --> T3["RandomHorizontalFlip"]
        T3 --> T4["RandomRotation 15 deg"]
        T4 --> T5["ColorJitter 0.2, 0.2, 0.2, 0.1"]
        T5 --> T6["ToTensor"]
        T6 --> T7["Normalize<br/>mean: 0.485, 0.456, 0.406<br/>std: 0.229, 0.224, 0.225"]
    end

    subgraph ValTransforms [Transforms - Validation]
        V1["Resize 256"] --> V2["CenterCrop 224"]
        V2 --> V3["ToTensor"]
        V3 --> V4["Normalize<br/>mean: 0.485, 0.456, 0.406<br/>std: 0.229, 0.224, 0.225"]
    end

Loading

Key details:

• 80/20 split — Dataset is split into training and validation subsets using torch.utils.data.random_split().
• Batch size: 32
• Workers: 4 (multi-process data loading).
• Pin memory: Enabled for faster GPU transfer.
• Class Mapping: Alphabetical (Authentic = 0, Forged = 1).

---

📈 Training Pipeline

Training Workflow

graph TB
    A["Start Training"] --> B["Mount Google Drive"]
    B --> C["Load Dataset and Create DataLoaders"]
    C --> D["Initialize Model - DualStreamResNet"]
    D --> E{Checkpoint Exists?}
    E -->|Yes| F["Load Checkpoint"]
    E -->|No| G["Fresh Start"]
    F --> H1["Set model.train"]
    G --> H1
    H1 --> H2["Iterate over batches"]
    H2 --> H3["Forward pass with autocast"]
    H3 --> H4["Compute CrossEntropyLoss"]
    H4 --> H5["Backward pass with GradScaler"]
    H5 --> H6["Optimizer step - AdamW"]
    H6 --> H7["Accumulate metrics"]
    H7 --> I1["Set model.eval"]
    I1 --> I2["torch.no_grad"]
    I2 --> I3["Compute val loss and accuracy"]
    I3 --> J{Val Acc Improved?}
    J -->|Yes| K["Save Best Model"]
    J -->|No| L["Continue"]
    K --> M["Save Latest Checkpoint"]
    L --> M
    M --> N["ReduceLROnPlateau Scheduler Step"]
    N --> O{More Epochs?}
    O -->|Yes| H1
    O -->|No| P["Plot Results"]

Loading

Auto-Recovery & Checkpointing

The system implements robust auto-recovery via Google Drive integration:

graph LR
    S1["model_state_dict"] --> CP["Checkpoint .pth"]
    S2["optimizer_state_dict"] --> CP
    S3["epoch"] --> CP
    S4["best_val_accuracy"] --> CP
    S5["train/val losses"] --> CP
    S6["train/val accuracies"] --> CP
    S7["scaler_state_dict"] --> CP
    S8["scheduler_state_dict"] --> CP
    CP -->|Saved to| GDrive["Google Drive"]
    GDrive -->|Restored from| Resume["Resume Training"]

Loading

Checkpoint Strategy:

• latest_checkpoint.pth — Saved every epoch for auto-recovery.
• best_model.pth — Saved only when validation accuracy improves.
• Google Drive mount — Ensures persistence across Colab sessions.

Hyperparameters

Parameter	Value	Notes
Optimizer	AdamW	Weight decay regularization built-in
Learning Rate	1e-4	Initial learning rate
Weight Decay	1e-3	L2 regularization
LR Scheduler	ReduceLROnPlateau	Reduces LR by factor 0.5 after 3 epochs of no improvement
Loss Function	CrossEntropyLoss	Standard for multi-class classification
Mixed Precision	✅ GradScaler + autocast	Faster training, lower memory
Dropout	0.6	Aggressive regularization in classifier head
Batch Size	32	Perfect size for standard GPU memory (T4)
Image Size	224 × 224	Standard for ResNet architectures
Epochs	26	Sufficient convergence time

---

📊 Evaluation & Results

Training Curves

The training curves show:

• Training accuracy steadily climbs to ~96% by epoch 26.
• Validation accuracy is more volatile (typical of forgery detection tasks) but stabilizes in the ~89–95% range.
• Training loss converges smoothly, while validation loss shows spikes (indicating the model encounters challenging batches).

ROC Curve & Confusion Matrix

• AUC = 0.9853 — Excellent discrimination between authentic and forged images.
• Confusion Matrix reveals:
  • 1,196 True Negatives (Authentic correctly identified)
  • 1,042 True Positives (Forged correctly caught)
  • 284 False Positives (Authentic misclassified as Forged)
  • 1 False Negative (Forged missed — extremely rare!)

Classification Report

Class	Precision	Recall	F1-Score	Support
Authentic	1.00	0.81	0.89	1,480
Forged	0.79	1.00	0.88	1,043
Accuracy			0.89	2,523
Macro Avg	0.89	0.90	0.89	2,523
Weighted Avg	0.91	0.89	0.89	2,523

Key Observations:

• Near-perfect recall for Forged images (1.00) — The model almost never misses a forgery.
• High precision for Authentic images (1.00) — When the model says authentic, it's virtually always correct.
• The trade-off is that some authentic images are flagged as forged (284 false positives), which can be tuned via threshold adjustment.

Model Summary

Metric	Value
Total Parameters	2,548,098
Trainable Parameters	2,548,098
Model Size	~10.19 MB
Forward/Backward Pass Size	~590.88 MB
Total Mult-Adds	17.35 GFLOPS
Best Validation Accuracy	94.77%

---

📁 Project Structure

google colab/
├── Dual_Stream_Model.ipynb      # Complete training & evaluation notebook
├── dual_stream_best_94.77%.pth  # Best saved model weights
├── Results_Images/
│   ├── Graphs.png               # Training/validation loss & accuracy curves
│   ├── output.png               # ROC curve & confusion matrix
│   ├── Model_architecture.png   # Visual architecture diagram
│   └── Model_architecture.svg   # Vector format architecture diagram
└── README.md                    # This file

---

🔮 Future Work

• 📦 Model Export — Convert the best .pth model to ONNX format for production deployment.
• 📊 Extended Evaluation — Precision-Recall curves and per-class failure mode analysis.
• ⚙️ Hyperparameter Tuning — Fine-tune learning rate, dropout rate, and augmentation strategies to reduce validation variance.
• 🖥️ Inference UI — Build a simple web interface for single-image forgery prediction.
• 🔬 Attention Mechanisms — Integrate channel/spatial attention to improve feature fusion.
• 📚 Cross-Dataset Validation — Test generalization on CASIA, Columbia, and other forgery benchmarks.

---

Built with PyTorch 🔥 | Trained on Google Colab ☁️