Skip to content
Snippets Groups Projects
Select Git revision
  • 482654f6bc434305346606ff1751d8c0908a20b5
  • master default protected
2 results

sensorapp

Name Last commit Last update
images
Deep_Optimal_Transport_Presentation.pdf
README.md
model_fashion_mnist.pt
ot-image-restoration.ipynb
requirements.txt
README.md

README

Team : Fatma Zahra Zaghouani, Islem Chaari, Léo Rouppert

Description

The paper "Deep Optimal Transport: A Practical Algorithm for Photo-realistic Image Restoration" presents an image restoration algorithm that allows control over the perceptual quality and/or mean square error (MSE) of any pre-trained model, enabling a trade-off between these aspects during testing. This few-shot algorithm requires only about a dozen images restored by the model to significantly enhance the perceptual quality or MSE for new images without additional training.

The approach is based on theoretical findings that connect the minimum MSE predictor with the predictor that minimizes MSE under a perfect perceptual quality constraint, achieved by optimally transporting the output of the former to match the source data distribution. This optimal transport is approximated by a linear transformation in the latent space of a variational autoencoder, computed in closed-form using empirical means and covariances.

Installation

Make sure Python is installed on your machine. Then, install the required dependencies with:

pip install -r requirements.txt

Usage

Run the notebook with the parameters you want to test. A pretrained VAE model on the Fashion-MNIST dataset is provided.

Difference Between the Original and Simplified Transport Operator

The original optimal transport operator based on the paper has been modified for computational simplification while retaining its fundamental properties.

In the original version, the transport operator is defined as:

T_{p_x \longrightarrow p_y}(x) = \Sigma_x^{-\frac{1}{2}} \left(\Sigma_x^{\frac{1}{2}} \Sigma_{y} \Sigma_x^{\frac{1}{2}}\right)^{\frac{1}{2}} \Sigma_x^{-\frac{1}{2}} (x - \mu_{x}) + \mu_y

where

x \sim \mathcal{N}(\mu_{x}, \Sigma_x)
and
y \sim \mathcal{N}(\mu_{y}, \Sigma_y)
.

In the simplified version, the transport operator is rewritten by replacing the successive multiplications with simpler transform :

T_{p_{x} \longrightarrow p_{y}}(x) = \Sigma_y^{\frac{1}{2}} \Sigma_x^{-\frac{1}{2}} (x - \mu_x) + \mu_y

This reformulation reduces the number of matrix multiplications and facilitates the inversion of matrices, improving the numerical stability of the model while maintaining a transformation close to the original. This approach led to some successful results whereas the original transport operator failed to improve perceptual quality.

Example Results

Results Curve

References

Author

This code was developed by Léo Rouppert.