An Extensive Review of Generative Adversarial

Network Based Image Denoising Methods

Ch. prasad, Dr. Sreejyothsna Ankam (Senior Assistant Professor)
ABSTRACT:
One of the main challenges in image processing is image denoising, which aims to eliminate
noise while keeping crucial details. This topic has benefited greatly from recent developments in
Generative Adversarial Networks (GANs), which provide fresh approaches to age-old denoising
issues. This paper looks at several approaches and their efficacy in providing a thorough
overview of GAN-based picture denoising techniques. I examine the development of GAN
architectures used for denoising tasks, starting with simple models and moving up to more
complex versions like self-supervised Noise2Noise methods and Wasserstein GANs (WGANs).
Important topics covered include single picture denoising and the modification of GANs to
handle blurriness. In addition to offering suggestions for future research paths in the field, our
assessment aims to shed light on the advantages and disadvantages of the GAN-based denoising
techniques that are already in use.

Keywords: Generative Adversarial Networks (GANs), Image Denoising, Noise2Noise,

Wasserstein GAN (WGAN), Image Quality Enhancement,

1.INTRODUCTION
Image denoising is a fundamental task in image processing, crucial for enhancing image quality
by removing noise that often degrades visual clarity. Noise can be introduced during image
capture due to factors like environmental conditions, low-light settings, or limitations of imaging
sensors. Traditional image denoising techniques include filtering-based methods and deep
learning-based approaches. Filtering techniques work by adjusting pixel values based on defined
parameters and have been widely used to improve image sharpness and color accuracy.
However, these methods often struggle with complex noise patterns, especially in high-noise
scenarios.
Deep learning methods have advanced denoising by learning to differentiate noise from the true
image signal using large datasets of paired noisy and clean images. In particular, the
Noise2Clean (N2C) approach trains a network to convert a noisy image into its clean
counterpart. This, however, requires a considerable amount of paired data, which can be difficult
to obtain, especially in blind noise scenarios where only a single noisy image is available.
To address this limitation, Noise2Noise (N2N) methods were developed, demonstrating that
neural networks can be trained on pairs of independently noisy images to effectively reduce
noise without needing clean data. The N2N approach adapts to noise by balancing the average
noise patterns within the noisy inputs and outputs, improving denoising performance.
Nonetheless, Noise2Noise still requires multiple noisy images for training, making it less
practical for cases where only a single noisy image is available or in scenarios with unpredictable
noise patterns.
This paper introduces a self-augmented approach to enhance the N2N framework, aiming to
address single-image denoising in blind noise settings. The proposed method leverages self-
supervised learning by generating augmented noisy images that mimic the distribution of the
original noisy image. This self-augmentation allows for training using only the noisy image
itself, eliminating the dependency on paired noisy data. The self-augmented model is validated
using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM)
metrics, with results indicating that the approach achieves competitive performance in single-
image denoising tasks. This method opens new possibilities for image denoising, particularly in
fields where obtaining clean or multiple noisy images is impractical, such as in medical imaging
or satellite data analysis.

2. METHODOLOGY

The methodology involves implementing a self-augmented noisy image generation for single-
image denoising within a Noise2Noise (N2N) framework. Initially, a self-augmented network is
trained by feeding a noisy image set as both input and validation data, producing a similar noisy
output that differs slightly from the input due to inherent learning imperfections in deep learning.
This process creates a new set of noisy images which, alongside the original noisy images, form
input-validation pairs for the N2N denoising network. Three models—ResNet, U-Net, and
Vision Transformer (ViT)—were tested for self-augmentation, with U-Net and ViT yielding the
best results. The self-augmented noisy image generation approach emphasizes variance
normalization in the loss function to prevent overfitting and improve the distribution of noise
across RGB channels. The methodology was evaluated using standard datasets and real-world
noisy images with performance metrics like PSNR and SSIM, demonstrating effectiveness in
denoising without requiring noise-free images.

2.1 Self-Augmented Noisy Image Generation:

This methodology begins with generating self-augmented noisy images to create a secondary
noisy image set from a single noisy image, addressing the challenge of requiring two noisy
images with different noise for traditional Noise2Noise setups. A self-augmented network
produces slightly altered versions of a noisy image, using the original as both input and
validation data. Through training imperfections, the network generates outputs mimicking the
original noise distribution with subtle variations, suitable for Noise2Noise denoising. Three
models—ResNet, U-Net, and Vision Transformer (ViT)—were tested for this approach. ResNet
prevents data loss with skip connections, U-Net retains spatial context via an encoder-decoder
structure, and ViT uses attention mechanisms to enhance feature retention. The models were
optimized with mean squared error (MSE) loss and variance normalization, which adapts to RGB
noise distribution and ensures realistic noisy images for effective denoising.Application in the
Paper: The network generates a set of noisy images that resemble the original noisy image but
are different enough to be used for training the Noise2Noise denoising network. This strategy is
essential in scenarios where only one noisy image is available for training.

2.2 Deep Learning Architectures for Self-Augmentation:

Fig 2.2.1: learning structure

For the self-augmentation process, this paper explored three prominent deep learning
architectures: ResNet, U-Net, and Vision Transformer (ViT), each selected for its unique
capacity to handle image feature extraction and reconstruction, which are essential for generating
realistic noisy images.
 ResNet employs a convolutional neural network (CNN) structure with residual
connections, which link earlier layers directly to later layers, helping to retain important
features from initial layers throughout the network. This skip-connected design reduces
data loss during training and is especially beneficial for tasks requiring precise feature
preservation. However, due to these residual connections, ResNet occasionally produces
slight pixel errors in the generated noisy images as it learns to reconstruct fine details.
 Vision Transformer (ViT) introduces an attention-based mechanism that divides the input
image into patches and processes them independently, learning relationships between
patches using attention layers rather than standard convolutions. This patch-based
approach, which uses key-query-value matching, makes ViT well-suited for capturing
distributed noise patterns. ViT demonstrated a high degree of accuracy in simulating
noisy images closely aligned with the original noise characteristics, as it effectively
captures both local and global noise distributions across image patches.
 U-Net operates as an encoder-decoder network designed to compress and reconstruct
image features, making it well-suited for tasks requiring feature localization. The
architecture includes skip connections between corresponding encoder and decoder
layers, which help retain spatial details and enhance noise distribution uniformity in the
output. This structure is commonly used in medical imaging but also proves effective for
self-augmented noisy image generation, as it balances noise across image regions while
preserving feature continuity.

Fig2.2.2: U-Net architecture

Each model was trained with mean squared error (MSE) loss as the primary optimization target,
alongside variance normalization to control the noise distribution across RGB channels,
particularly in uniform regions where standard MSE can lead to overfitting. The addition of
variance normalization ensures that each architecture adapts more effectively to the noise
characteristics, thereby generating noisy outputs that serve as reliable training pairs for the
Noise2Noise denoising network. Through this methodology, I was able to evaluate each
architecture's effectiveness in self-augmented noisy image generation, with ViT and U-Net
showing particular strengths in maintaining consistent noise distributions.
3. GANs (Generative Adversarial Networks):
In addition to ResNet, U-Net, and Vision Transformer (ViT), experimented with Generative
Adversarial Networks (GANs) for self-augmentation. GANs consist of a generator, which
creates new images, and a discriminator, which evaluates their authenticity against the original
noisy input. Using a U-Net generator for its feature-preserving properties and a CNN-based
discriminator enhanced with VGG-19 for subtle noise differentiation, the GAN trained on noisy
images with a dual-loss approach: MSE for content loss and binary cross-entropy for adversarial
loss. This setup aimed to produce realistic noisy images while minimizing pixel-level
differences. Despite challenges like overfitting and minor artifacts in regions of high feature
variability, careful tuning of loss functions mitigated these issues, enabling the GAN to generate
noisy images closely resembling the input. However, GANs proved less consistent than ViT and
U-Net due to the complexities of adversarial training, though they provided valuable insights into
capturing detailed noise patterns in self-augmentation.
4. Self-Augmented Noise2Noise algorithm
In scenarios where only a single noisy image is available, the self-augmented N2N approach is
particularly valuable. The methodology includes an additional step to simulate a second noisy
version from a single noisy image, enabling the Noise2Noise process even in single-image cases.
 Fig 2.2.3 flow diagram for noise 2 noise algorithm
 Self-Augmentation of Noisy Images:
o The model first creates a second noisy version of each image by learning to
approximate the noise distribution in the original noisy image.
o This process involves training a deep learning model (e.g., U-Net, ResNet, or
Vision Transformer) to create a noisy output that closely resembles the input
noisy image without directly replicating it.
o To prevent overfitting, the training uses a modified loss function that combines
MSE with variance normalization. This encourages the model to focus on the
underlying structure of the image rather than memorizing the noise pattern.
 Self-Supervised Noise2Noise Training:
o The original noisy image serves as the input, and the self-augmented noisy
version serves as the target during training.
o By minimizing the loss between the original noisy image and the self-augmented
noisy image, the network learns to reduce noise.
 Network Architecture:
o Various architectures can be used for this self-augmentation step, such as U-Net
(an encoder-decoder structure) or Vision Transformer (ViT), which have shown
better results in handling noise.
 o In particular, U-Net and ViT are effective because they capture relationships
within image patches and avoid overfitting, which helps maintain image quality
while denoising.
 Evaluation Metrics:
o The model's performance is evaluated using metrics like Peak Signal-to-Noise
Ratio (PSNR) and Structural Similarity Index Measure (SSIM), which quantify
how well the noise is removed while preserving image details.

LIMITATIONS & GAPS:

The study acknowledges several limitations. First, the self-augmented method tends to overfit at
higher noise levels, particularly with models like ResNet and GANs, resulting in pixel errors in
the denoised images. Another limitation is its reduced performance on real-world noisy images,
such as those from the NIND and SIDD datasets, which often have noise distributions not
aligned with the Gaussian noise model assumed in this approach. Additionally, the dependence
on the MSE loss function alone was found to cause inaccuracies in noise generation, and though
variance normalization improved results, it did not completely eliminate the problem. Lastly, the
proposed method struggles with low-level noise, where residual noise artifacts remain, indicating
the need for further improvement in handling diverse noise types beyond Gaussian noise.

3.LITERATURE SURVEY
3.1 RELATED WORK
Qu, Z., Zhang, Y., Sun, Y., & Lin, X[1] has proposed a paper A New Generative Adversarial
Network for Texture Preserving Image Denoising. The Noise2Noise framework builds on
classical and deep learning-based denoising methods. Traditional techniques used filtering and
optimization, while modern approaches, like CNNs (e.g., DnCNN) and autoencoders, learn
mappings from noisy to clean images. GANs further enhance denoising by preserving textures
through adversarial training, making them ideal for detail-sensitive applications.

Samik Banerjee, Sukhendu Das[2], SD-GAN: Structural and Denoising GAN reveals facial parts
under occlusion. Classical methods like Non-Local Means and BM3D use filtering and statistical
models for noise removal. Deep learning models like DnCNN leverage CNNs trained on clean-
noisy pairs for improved denoising, while GANs enhance perceptual quality through adversarial
training. Noise2Noise innovates by using noisy image pairs instead of clean references, enabling
denoising without clean data. Self-supervised extensions adapt this approach for single-image
denoising by simulating multiple noisy versions.

Xin Cheng , Jingmei Zhou , Jiachun Song, and Xiangmo Zhao[3], A Highway Traffic Image
Enhancement Algorithm Based on Improved GAN in Complex Weather Conditions. Traditional
methods like histogram equalization and Retinex struggle with complex textures and
overexposure in challenging weather. Deep learning models, such as GCANet and object
detection networks, improve clarity and adapt to varying conditions, while GANs enhance
underwater image clarity but face synthetic-to-real distribution challenges. However, many
methods still struggle with fine details and color fidelity, highlighting the need for better
solutions.

SONGKUI CHEN 1 , DAMING SHI[4], Image Denoising With Generative Adversarial

Networks and Its Application to Cell Image Enhancement. The paper critiques traditional MSE-
based denoising for producing blurry results and limited texture recovery. Feature extraction
methods are also constrained by task-specific sensitivity. The proposed framework addresses
these issues using Wasserstein GANs (WGANs), improving detail recovery by focusing on the
distribution of clean images, particularly for cell image denoising.

HAIBO ZHANG 1 AND KOUICHI SAKURA[5], Conditional Generative Adversarial Network-

Based Image Denoising for Defending Against Adversarial Attack. Key strategies for improving
adversarial robustness include Adversarial Logit Pairing (ALP), gradient traps, and high-level
representation guided denoisers. Free adversarial training reduces training time while
maintaining accuracy, and GAN-based defenses operate without prior attack knowledge.
Purification techniques by Li et al. and Hwang et al. address adversarial perturbations,
highlighting diverse methods to enhance model security against threats.

Shaobo Zhao, Sheng Lin , Xi Cheng[6], Dual-GAN Complementary Learning for Real-World
Image Denoising. Notable denoising methods like DnCNNs, FFDNet, CBDNet, RIDNet, and
SADNet address image noise using deep learning. However, many rely on L1 or L2 losses,
leading to blurring and limited real-world effectiveness. Handling complex, spatially varying
noise has driven the development of strategies combining denoised image and noise learning.
Single GAN-based methods face challenges with complexity and training, prompting the
proposed DGCL strategy to enhance performance and overcome these limitations.

HAIJUN HU 1 , BO GAO3 , ZHIYUAN SHEN[7], Image Smear Removal via Improved

Conditional GAN and Semantic Network. Image smear removal techniques often face challenges
with paired training data and reconstruction loss in traditional methods like autoencoders. GANs
offer a solution by generating realistic images, but issues like error oscillations and gradient
instability persist, leading to the adoption of stable models like Wasserstein GANs (WGAN).
Recent methods use architectures like VGG-16 for improved feature extraction, combining low-
and high-dimensional features. However, visual consistency in complex backgrounds remains a
challenge. The proposed method addresses this by leveraging GANs and multi-scale feature
fusion to enhance restoration accuracy and realism.

Xin Jin1 , Ying Hu1 , Chu-Yue Zhang1[8], This image restoration method combines GANs with
multi-scale feature fusion to address limitations in traditional algorithms requiring paired data.
While GAN-based approaches often complicate training and produce unnatural results, this
method uses a VGG-16 encoder-decoder to fuse low- and high-dimensional features, enhancing
restoration quality. Incorporating WGAN principles and L1 loss improves similarity to target
images. Experiments show improved accuracy and visual consistency, though challenges remain
in complex backgrounds, which future work aims to address.

LUAN THANH TRINH AND TOMOKI HAMAGAMI[9], Latent Denoising Diffusion GAN:
Faster Sampling, Higher Image Quality. The Latent Denoising Diffusion GAN (LDDGAN)
improves image generation by addressing the slow inference of diffusion models while
enhancing quality and diversity. Using pre-trained autoencoders, it compresses images into a
low-dimensional latent space for faster sampling and efficiency. A Weighted Learning strategy
balances adversarial and reconstruction losses, boosting image quality and diversity. LDDGAN
achieves state-of-the-art speeds and competitive quality compared to models like DiffusionGAN,
showing promise for real-time applications and advancing high-fidelity diffusion models by
moving beyond Gaussian latent spaces.

IBRAHIM H. EL-SHAL 1 , OMAR M. FAHM[10], License Plate Image Analysis Empowered

by Generative Adversarial Neural Networks. License Plate Recognition (ALPR) employs
various methods to improve detection and recognition under challenging conditions. Traditional
techniques like edge-based, color-oriented, and texture-based methods face limitations with
complex backgrounds, lighting variations, and computational demands. Recent advancements
focus on Deep Learning, particularly DCNNs and GANs, which excel in feature learning and
generalization. Notable innovations include SRGAN for enhanced image quality and GAN
adaptations for license plate enhancement and character recognition. These advancements
underscore the need for robust, real-world ALPR solutions.

Huihui Li , Cang Gu , Dongqing Wu , Gong Cheng[11], Multiscale Generative Adversarial

Network Based on Wavelet Feature Learning for SAR-to-Optical Image Translation. The
document reviews advancements in image-to-image translation (I2IT), focusing on the
challenges of translating synthetic aperture radar (SAR) images to optical images (S2OIT). It
highlights the limitations of general I2IT methods due to discrepancies between the image types.
Approaches like cGANs and CycleGAN have been used for image matching but often struggle
with visual quality and noise interference. The review calls for tailored strategies to address these
challenges, emphasizing the need for deep learning techniques, including CNNs and GANs, to
improve visual realism and detail retention in S2OIT tasks.

WENDA LI 1,2,3 AND JIAN WANG[12], Residual Learning of Cycle-GAN for Seismic Data
Denoising. The Residual Cycle-GAN (RCGAN) improves seismic data denoising by integrating
residual learning into a Cycle-GAN framework, addressing the limitations of traditional methods
that require manual parameter selection. RCGAN uses data augmentation to enhance training
efficiency and adaptability to real seismic data, generating labeled datasets from noisy unlabeled
data. It outperforms methods like FXDM and DnCNN by better suppressing random noise while
preserving important data details in both synthetic and real seismic datasets.

ASAVARON LIMSUEBCHUEA 1 , RAKKRIT DUANGSOITHONG[14], Self-Augmented

Noisy Image for Noise2Noise Image Denoising. The related work discusses image degradation
in processing, categorized into convolution and additive signal degradation. Convolution
degradation distorts images due to processing algorithms or environmental factors, while
additive degradation arises from interference signals causing abnormal pixel behavior. The study
reviews conventional filtering methods that require parameter tuning and highlights the
advantages of deep learning techniques, particularly the Noise2Noise strategy, which effectively
removes noise without needing clean reference images, addressing challenges in single-image
and blind noise scenarios

Joonyoung Song , Jae-Heon Jeong , Dae-Soon Park [15], Unsupervised Denoising for Satellite
Imagery Using Wavelet Directional Cycle GAN. Satellite image denoising includes traditional
methods like Total Variation (TV), which preserves high-frequency information, and Low-Rank
Matrix Recovery (LRMR), which uses the low-rank properties of clean images. However, these
methods require paired noisy and clean images, which are hard to obtain. Recent deep learning
approaches, like deep image prior and Noise2Void, face issues with computational complexity
and noise assumptions. This highlights the need for innovative methods like WavCycleGAN,
which denoises satellite imagery without paired datasets, addressing challenges in remote
sensing.

4.RESULTS & DISSCUSSION

Algorithm Training Data Blind Noise Single Image Complexity

Capability Capability
Noise2Clean Clean & noisy No No High
image pairs
Noise2Noise Noisy image Limited Limited Moderate
pairs
Self2Self Single noisy Limited Yes Moderate
image
Proposed Single noisy Yes Yes Moderate
image (self-
aug.)
Table 4.1: Comparison of Learning Characteristics
 Table 4.2: PSNR (dB) and SSIM Results on BSD300 Dataset

CONCLUSION
This research presents a novel approach to enhancing the Noise2Noise (N2N) algorithm for
image denoising, specifically targeting single-image and blind noise scenarios by generating
noisy images from limited datasets. By leveraging the imperfections inherent in the deep learning
training process, the proposed self-augmented noisy image network enables the generation of
new noisy images that can be used for training and validation, effectively facilitating noise
removal. Experimental results demonstrate that this self-augmentation strategy achieves
performance comparable to other unsupervised denoising methods at lower noise levels,
although it struggles with higher noise levels and real-world images due to a lack of
understanding of image features. The findings underscore the importance of accurately
estimating noise characteristics in real-world applications to enhance denoising performance and
minimize pixel errors.

references
1. Qu, Z., Zhang, Y., Sun, Y., & Lin, X(2018), A New Generative Adversarial Network
for Texture Preserving Image Denoising ,IEEE,5356-5361
2. Samik Banerjee, Sukhendu Das(2020), SD-GAN: Structural and Denoising GAN
reveals facial parts under occlusion,arxiv,1-24
3. Xin Cheng , Jingmei Zhou , Jiachun Song, and Xiangmo Zhao,(2023), A Highway
Traffic Image Enhancement Algorithm Based on Improved GAN in Complex
Weather Conditions,IEEE,8716-8726
4. SONGKUI CHEN 1 , DAMING SHI(2020), Image Denoising With Generative
Adversarial Networks and Its Application to Cell Image Enhancement ,IEEE, 82819-
82831
5. HAIBO ZHANG 1 AND KOUICHI SAKURA(2022), Conditional Generative
Adversarial Network-Based Image Denoising for Defending Against Adversarial
Attack,IEEE Access, 169031-169043
6. Shaobo Zhao, Sheng Lin , Xi Cheng(2024), Dual-GAN Complementary Learning
for Real-World Image Denoising,IEEE, 355-366
7. HAIJUN HU 1 , BO GAO3 , ZHIYUAN SHEN(2020), Image Smear Removal via
Improved Conditional GAN and Semantic Network,IEEE Access, 113104-113111
8. Xin Jin1 , Ying Hu1 , Chu-Yue Zhang1(2020), Image restoration method based on
GAN and multi-scale feature fusion ,IEEE Xplore, 2305-2310
9. LUAN THANH TRINH AND TOMOKI HAMAGAMI(2024), Latent Denoising
Diffusion GAN: Faster Sampling, Higher Image Quality,IEEE Access, 78161-78172
10. IBRAHIM H. EL-SHAL 1 , OMAR M. FAHM(2022), License Plate Image Analysis
Empowered by Generative Adversarial Neural Networks (GANs) ,IEEE Access,
30846-30857
11. Huihui Li , Cang Gu , Dongqing Wu , Gong Cheng/(2022), Multiscale Generative
Adversarial Network Based on Wavelet Feature Learning for SAR-to-Optical Image
Translation,IEEEE, 5236115-5236125
12. WENDA LI 1,2,3 AND JIAN WANG(2021), Residual Learning of Cycle-GAN for
Seismic Data Denoising,IEEE Access, 11585-11597
13. Qi-Feng Sun1 · Jia-Yue Xu1 · Han-Xiao Zhang1(2021), Random noise suppression
and super-resolution reconstruction algorithm of seismic profle based
on GAN,Springer, 2107–2119
14. ASAVARON LIMSUEBCHUEA 1 , RAKKRIT DUANGSOITHONG(2024), Self-
Augmented Noisy Image for Noise2Noise Image Denoising,IEEE Access, 71076-
71087
15. Joon young Song , Jae-Heon Jeong , Dae-Soon Park (2021), Unsupervised
Denoising for Satellite Imagery Using Wavelet Directional CycleGAN,IEEE Xplore,
6823-6839