---

A Benchmark Suite for Federated Learning Across Multiple Modalities

This repository provides a comprehensive benchmark suite for simulating federated learning (FL) across four distinct data modalities: image, audio, text, and video. It is designed as a research framework to explore, evaluate, and compare various federated learning algorithms on state-of-the-art models and standard datasets.

The entire suite is built using Flower as the core federated learning framework. Each sub-project is self-contained and leverages modern machine learning libraries, including PyTorch, Hugging Face (transformers, datasets, peft), and Hydra for flexible configuration.

Key Features

• Multi-Modality Benchmarking: Conduct federated learning experiments on a diverse set of tasks:
  • Image Classification on CIFAR-10.
  • Audio Classification using the Audio Spectrogram Transformer (AST).
  • Video Classification on UCF101 with the VideoMAE model.
  • Text Fine-Tuning of Large Language Models (Llama) with LoRA.
• Advanced Federated Algorithms: Easily switch between and compare multiple FL strategies, including FedAvg, FedProx, FedAdam, and FedYogi.
• State-of-the-Art Models: Implements powerful, modern architectures tailored for each data type.
• Realistic Data Simulation: Features robust simulation of non-IID (Not Independent and Identically Distributed) data across clients using Dirichlet partitioning, modeling real-world data heterogeneity.
• Modular and Reproducible: Each modality is a self-contained project with its own environment, documentation, and configuration, ensuring clarity and reproducibility. The text, audio, and video projects use Hydra for powerful and flexible experiment management.

Repository Structure

The repository is organized into distinct directories for each modality, plus a shared script directory for analysis.

.
├── audio/            # Federated Audio Classification experiments
│
├── image/            # Federated Image Classification experiments
│
├── text/             # Federated LLM Fine-Tuning experiments
│
├── video/            # Federated Video Classification experiments
│
├── script/           # Scripts for post-experiment analysis and plotting
│
└── README.md         # This file (Project Overview)

Getting Started

To get started with an experiment for a specific modality, please follow these general steps:

1. Navigate to a Sub-project: Choose the modality you are interested in and change into its directory.

   # For example, to work with the video project:
   cd video/

2. Follow the Sub-project's README: Each modality has its own detailed README.md file with specific instructions for:

   • Downloading required datasets and pre-trained models.
   • Setting up the Python environment and installing dependencies from its requirements.txt.
   • Configuring and running experiments.

3. Run Experiments: Use the provided scripts and instructions in the sub-directory's README.md to launch centralized or federated training runs.

4. Analyze Results: After running experiments, you can use the scripts in the /script directory to aggregate results and generate figures and heatmaps for comparison.

Sub-Project Overview

This table provides a high-level summary of each experimental suite. For detailed setup and usage instructions, please refer to the linked README.md in each sub-directory.

Modality	Task	Dataset(s)	Model(s)	Configuration	Details
Image	Image Classification	CIFAR-10	ResNet-18, PyramidNet	Python script with argparse	image/README.md
Audio	Audio Classification	AudioSet, ESC-50	Audio Spectrogram Transformer (AST)	Hydra (.yaml)	audio/README.md
Video	Video Classification	UCF101	VideoMAE	Hydra (.yaml)	video/README.md
Text	LLM Fine-Tuning	Shakespeare, Dolly	Llama + LoRA	Hydra (.yaml)	text/README.md

Analysis Scripts (/script)

The /script directory contains tools for post-hoc analysis of experiment results. These scripts are designed to parse output files (e.g., .csv logs) from the various experiment directories to generate comparative plots, tables, and figures for research papers or presentations.