Diffusion Sampling Efficiency – Retrieval-based Improvements

This project explores sampling efficiency in Latent Diffusion Models using retrieval-based techniques. It builds on top of ReDi method and is part of the Master's thesis "Enhancing the Efficiency of Diffusion Models".

🔍 Project Goals

• Implement and evaluate retrieval-based diffusion with Retrieval Key Compression (PCA and PQ) to optimize memory usage and lookup speed for stored latent vectors.
• Implement an adaptive skipping technique for retrieval-based diffusion that adjusts the number of skipped denoising steps based on retrieval confidence.
• Evaluate performance using Domain-Specific Knowledge Bases, including:
  • ID-2K: a synthetic interior design dataset
  • COCO-10K: a subset of MS-COCO filtered for furniture-related prompts
• Benchmark improvements over standard DDIM/Stable Diffusion baselines using:
  • CLIPScore (text-image alignment)
  • PickScore (human preference modeling)
  • FID (distribution similarity)
  • Inception Score (IS) (image diversity and confidence)
• [WIP] Integrate a Value Refinement Module: an auxiliary lightweight U-Net that refines retrieved latents to improve quality in the later denoising steps.

📂 Structure

• Prompts preparation for ID-2K and COCO-10K
• Main pipeline: Trajectories generation, experiments with key compression and demos
• Adaptive skipping: Retrieval logic and compression tools
• Retrieval Diffusion module: All functions to work with Retrieval Diffusion (Knowledge Base, Trajectories, Generation, Neighbours search). Partially based on ReDi approach.
• Results: All the artifacts of experimentation
• Data: Lightweight files with prompts etc. Main datsets with trajectories and all prompts are kept in HuggingFace dataset
• Metrics: Functions for computing FID, CLIPScore, PickScore and IS

✅ Requirements

Dependencies installation and virtual environment creation is managed by PDM

Install dependencies:

pdm install