VIRTUAL TRY-ON

This project presents an end-to-end AI-driven platform for immersive virtual try-on experiences. Users can preview garments on their own images, discovering how clothing items fit and look in a realistic manner. The system leverages cutting-edge computer vision and deep learning technologies to deliver a seamless try-on functionality.

HOW IT WORKS

The Virtual Try-On pipeline consists of a pre-processing stage followed by a two-stage generative model based on Thin-Plate Spline (TPS) transformation. The two key generative components are the Geometric Matching Module (GMM) and the Context-Aware Generator (CAG).

1. Pre-Processing Pipeline

Before generating the final try-on image, the input images undergo a robust pre-processing pipeline:

• Human Parsing: Uses the Self-Correction-Human-Parsing model to segment the person into different body parts and clothing regions.
• Cloth Masking: Employs the Segment Anything Model (SAM) to generate a precise segmentation mask for the target garment.
• Pose Detection: Uses PoseNet to extract 2D pose keypoints from the person's image.
• Skeleton Generation: The extracted keypoints are used to draw an OpenPose-style skeleton image, providing structural guidance.

2. Geometric Matching Module (GMM)

The purpose of the GMM is to transform the target garment into a warped version that aligns accurately with the person’s pose and body shape.

Architecture:

• Feature Extractor: A custom VGG-like convolutional network with 6 Conv-BatchNorm-ReLU layers that extracts deep features from both the person's segmentation map and the target garment.
• Feature Correlation: Computes the correlation between the extracted features of the person and the garment, capturing the spatial relationships.
• Feature Regression: A convolutional and fully connected network that processes the correlation tensor to predict the spatial transformation parameters (theta) for a Thin-Plate Spline (TPS) grid.
• TPS Transformation: Applies the predicted transformation to a spatial grid, warping the target garment smoothly to fit the person.

3. Context-Aware Generator (CAG)

The CAG synthesizes the final virtual try-on image by combining the warped garment from the GMM with various contextual elements (like the person's body shape, hair, and background).

Architecture:

• The generator employs a 6-block architectural design.
• Each block consists of sequential Convolutional and ReLU layers, followed by Context-Aware Normalization (CAN).
• Context-Aware Normalization (CAN): The contextual inputs (segmentation, original cloth, warped cloth, and cloth-agnostic person image) are processed through Convolutional/Deconvolutional pathways to compute scaling (alpha) and shifting (beta) parameters. These parameters modulate the normalized feature maps at each resolution block.
• Upsampling: Each block concludes with a ConvTranspose2d layer for progressive upsampling, refining visual details hierarchically to ensure semantic consistency and generate a highly realistic final try-on image.