Scribd
Upload
62 views21 pages

Generative Adversarial Networks: Akrit Mohapatra Ece Department, Virginia Tech

This document discusses Generative Adversarial Networks (GANs) which use two neural networks competing against each other to generate new data instances that match the training data distribution. GANs have been successfully used to generate natural images and are useful for unsupervised learning tasks. However, GANs are difficult to train and often fail to converge on large problems.

Uploaded by

fatema.tuj.jahura
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
62 views21 pages

Generative Adversarial Networks: Akrit Mohapatra Ece Department, Virginia Tech

This document discusses Generative Adversarial Networks (GANs) which use two neural networks competing against each other to generate new data instances that match the training data distribution. GANs have been successfully used to generate natural images and are useful for unsupervised learning tasks. However, GANs are difficult to train and often fail to converge on large problems.

Uploaded by

fatema.tuj.jahura
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 21

Generative Adversarial

Networks
AKRIT MOHAPATRA
ECE DEPARTMENT, VIRGINIA TECH
What are GANs?

 System of two neural networks competing against each other


in a zero-sum game framework.
 They were first introduced by Ian Goodfellow et al. in 2014.
 Can learn to draw samples from a model that is similar to
data that we give them.
Components of GANs

Noise

Slide credit – Victor Garcia


Overview of GANs
Source: https://ishmaelbelghazi.github.io/ALI
Discriminative Models

 A discriminative model learns a function that maps the input data (x)
to some desired output class label (y).

 In probabilistic terms, they directly learn the conditional


distribution P(y|x).
Generative Models

 A generative model tries to learn the joint probability of the input data
and labels simultaneously i.e. P(x,y).

 Potential to understand and explain the underlying structure of the


input data even when there are no labels.
How GANs are being used?

 Applied for modelling natural images.

 Performanceis fairly good in comparison to other


generative models.

 Useful for unsupervised learning tasks.


Why GANs?

 Use a latent code.


 Asymptotically consistent (unlike variational methods) .
 No Markov chains needed.
 Often regarded as producing the best samples.
How to train GANs?

 Objective of generative network - increase the error rate of the


discriminative network.
 Objective of discriminative network – decrease binary classification
loss.
 Discriminator training - backprop from a binary classification loss.
 Generator training - backprop the negation of the binary classification
loss of the discriminator.
Loss Functions

Generator Discriminator
Generated bedrooms. Source: “Unsupervised Representation Learning with Deep Convolutional Generative
Adversarial Networks” https://arxiv.org/abs/1511.06434v2
“Improved Techniques for Training GANs” by
Salimans et. al

 One-sided Label smoothing - replaces the 0 and 1 targets for a classifier


with smoothed values, like .9 or .1 to reduce the vulnerability of neural
networks to adversarial examples.

 Virtual batch Normalization - each example x is normalized based on


the statistics collected on a reference batch of examples that are chosen
once and fixed at the start of training, and on x itself.
Original CIFAR-10 vs. Generated CIFAR-10 samples
Source: “Improved Techniques for Training GANs” https://arxiv.org/abs/1606.03498
Variations to GANs

 Several new concepts built on top of GANs have been introduced –


 InfoGAN – Approximate the data distribution and learn
interpretable, useful vector representations of data.
 Conditional GANs - Able to generate samples taking into account
external information (class label, text, another image). Force G to
generate a particular type of output.
Major Difficulties

 Networks are difficult to converge.

 Ideal goal – Generator and discriminator to reach some


desired equilibrium but this is rare.

 GANs are yet to converge on large problems (E.g. Imagenet).


Common Failure Cases

 Thediscriminator becomes too strong too quickly and the generator


ends up not learning anything.

 The generator only learns very specific weaknesses of the


discriminator.

 The generator learns only a very small subset of the true data
distribution.
So what can we do?

 Normalize the inputs


 A modified loss function
 Use a spherical Z
 BatchNorm
 Avoid Sparse Gradients: ReLU, MaxPool
 Use Soft and Noisy Labels
 DCGAN / Hybrid Models
 Track failures early (D loss goes to 0: failure mode)
 If you have labels, use them
 Add noise to inputs, decay over time
Conclusions

 Train GAN – Use discriminator as base model for transfer learning and
the fine-tuning of a production model.

 A well-trainedgenerator has learned the true data distribution well -


Use generator as a source of data that is used to train a production
model.
Dive Deeper?

Ian Goodfellow’s NIPS 2016 Tutorial

Available online.
References

 https://tryolabs.com/blog/2016/12/06/major-advancements-deep-learning-2016/
 https://blog.waya.ai/introduction-to-gans-a-boxing-match-b-w-neural-nets-b4e5319cc935
#.
6l7zh8u50
 https://en.wikipedia.org/wiki/Generative_adversarial_networks
 http://blog.aylien.com/introduction-generative-adversarial-networks-code-tensorflow/
 https://github.com/soumith/ganhacks

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy