IA generativa: Aplicacions

Valero Laparra
www.uv.es/lapeva
www.uv.es/lapeva/gandia.pdf

2
 Modelos Generativos
Definiciones formales

DEFINICIÓN 1

- Generativos p(X,Y)

- Condicionales p(Y|X)
(a veces a estos se les llama modelos discriminativos)

- Discriminativos Y = f(X)

3
 Modelos Generativos
Definiciones formales

DEFINICIÓN 2

- Generativos p(X|Y)

- Discriminativos p(Y|X)

4
 Modelos Generativos
Definiciones
DEFINICIÓN 3 formales
Es un modelo que se usa para generar datos

5
 Modelos Generativos
Modelo Básico: histograma (+CDF)

6
 Modelos Generativos
Modelo Básico: modelo de distribución

7
Modelos Generativos
Curse of dimensionality

8
En ML en los últimos años, Deep Generative Models:
- Generative Adversarial Networks (GANs)
- Variational Autoencoders (VAEs)
- Normalized Flows
- Diffusion
- Autoregresive
- Style transfer

9
Pero antes…

- Gaussian mixture model

- Hidden Markov model
- Probabilistic context-free grammar
- Bayesian network
- Averaged one-dependence estimators
- Latent Dirichlet allocation
- RBIG: https://www.uv.es/lapeva/papers/Laparra11.pdf
- Portilla & Simoncelli: http://www.cns.nyu.edu/~lcv/texture/
…

10
Tienen relación con Autoencoders

11
Tienen relación con entrenamiento No supervisado

12
Tienen relación con entrenamiento No supervisado

13
A jugar!
14
 Generar datos
This person does not exist: https://thispersondoesnotexist.com/

https://thisxdoesnotexist.com/
15
 Síntesis: Imágenes

GLOW

Paper:

https://arxiv.org/abs/1807.03039

Jugar:

https://openai.com/blog/glow/

16
Síntesis condicional: Imágenes

Pix2pix (imágenes)

https://affinelayer.com/pixsrv/

17
Síntesis condicional: Texto

Genéricos: BERT, GPT3, PALM, Bloom (Open Source) GPT3, ChatGPT, GPT4...

2019 Megatron-11b (GPT2)

https://app.inferkit.com/demo

(2022) ChatGPT
https://openai.com/blog/chatgpt/

Específicos:

Matemáticas→ Minerva:
https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html

Código→ CoPilot:
https://github.com/features/copilot
18
19
20
21
22
Síntesis condicional: Imagen | Texto
CLIP: Evaluate concepts in Images

Blog/jugar:
https://openai.com/blog/clip/

Code:
https://github.com/openai/CLIP

Jugar:
http://Wombo.art

23
Síntesis condicional: Imagen | Texto

DALL·E Blog/jugar:
Creating Images from Text https://openai.com/blog/dall-e/

Code (VAE):
https://github.com/openai/dall-e

Ru DALL·E:
https://rudalle.ru/en/

24
Síntesis condicional: Imagen | Texto
GLIDE
Images from Text by Diffusion

Paper:

https://arxiv.org/pdf/2112.10741.pdf

Code and Colab examples (Jugar):

https://github.com/openai/glide-text2im
25
Síntesis condicional: Imagen | Texto

Re-entrenando

+ Investigador + Investigador gammer + Investigador

trabajando estilo pintado estilo clásico

GLIDE, DALL·E 2, Midjourney, Flamingo...

Stable Difussion (Open Source):

https://stability.ai/ - https://dreamstudio.ai/generate 26
Edición de imagen con texto:

Imagen editor (Google)

https://imagen.research.google/editor/

27
Síntesis condicional: Video | Texto
Multimodal NLP
Text2Video
https://huggingface.co/spaces/PAIR/Text2Video-Zero

Gen-2
https://research.runwayml.com/gen2

28
Síntesis condicional: Video | Texto

Phenaki (Google)
https://phenaki.video/

Imagen-Video (Google)
https://imagen.research.google/video/

Make-a-Video (Meta)
https://makeavideo.studio/

También Video → Video

29
Síntesis condicional: Texto | Audio

Whisper OpenAI
https://openai.com/research/whisper
https://huggingface.co/openai/whisper-large-v2

LALAL.ai
https://www.lalal.ai/guides/how-to-create-an-acapella-from-a-song/

30
Otras condiciones
ControlNet

+ Smurf (pitufo) + Indiana Jones + Heavy Metal + Disney

31
https://huggingface.co/spaces/hysts/ControlNet-v1-1
 Multimodal: Imagen | Texto, imagen, dibujo...

Gaugan2

Blog/jugar:
http://gaugan.org/gaugan2

33
 Imagen|Imagen : Superresolución

JUGAR
SR3:
https://iterative-refinement.github.io/ https://deepai.org/machine-learning-model/torch-srgan

Palette: https://bigjpg.com/
https://iterative-refinement.github.io/palette/ 34
http://waifu2x.udp.jp/
Imagen|Imagen : Colorización

· “Colorful Image Colorization”

https://arxiv.org/abs/1603.08511

· Colorization example:
http://richzhang.github.io/colorization/

· JUGAR Colorization:
http://demos.algorithmia.com/colorize-photos/

35
Imagen|Imagen : Inpainting

Palette:
https://iterative-refinement.github.io/palette/

NVIDIA Demos:
https://www.nvidia.com/en-us/research/ai-demos/

36
 Imagen | Contenido + Estilo

Estilo

Contenido

JUGAR: https://goart.fotor.com/
https://reiinakano.com/arbitrary-image-stylization-tfjs/
37
Deep Generative Models

38
 GENERATIVE MODELS

- Generative Adversarial Networks (GANs)

- Variational Autoencoders (VAEs)
- Normalized Flows
- Diffusion
- Auto-regresive
- Style transfer

39
 GENERATIVE MODELS

- Generative Adversarial Networks (GANs)

- Variational Autoencoders (VAEs)
- Normalized Flows
- Diffusion
- Auto-regresive
- Style transfer

40
 Generative Adversarial Networks

Generative Adversarial Networks

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio
https://arxiv.org/abs/1406.2661
41
 Generative Adversarial Networks

Generative Adversarial Networks

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio
https://arxiv.org/abs/1406.2661
42
Generative Adversarial Networks

43
Generative Adversarial Networks

GANs Conditional GANs Coupled GANs

44
 GENERATIVE MODELS

- Generative Adversarial Networks (GANs)

- Variational Autoencoders (VAEs)
- Normalized Flows
- Diffusion
- Auto-regresive
- Style transfer

45
Mas Info: GANs

Tutorial:
https://towardsdatascience.com/generative-adversarial-network-gan-for-dummies-a-step-by-step-tutorial-fdefff170391

Papers:
GANs https://arxiv.org/abs/1406.2661
Conditional GANs https://arxiv.org/abs/1611.07004
CycleGANs https://arxiv.org/abs/1703.10593

Jugar:
https://affinelayer.com/pixsrv/

Code:
https://keras.io/examples/generative/conditional_gan/

https://www.tensorflow.org/tutorials/generative/dcgan
46
 Variational Autoencoders

Kingma, Diederik P.; Welling, Max (2013-12-20). "Auto-Encoding Variational Bayes".

47
arXiv:1312.6114 [stat.ML].
Variational Autoencoders

https://transcranial.github.io/keras-js/#/mnist-vae

48
Mas Info: VAEs

Tutorial:
https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73

Paper:
https://arxiv.org/abs/1312.6114

Jugar:
https://transcranial.github.io/keras-js/#/mnist-vae

Code:
https://keras.io/examples/generative/vae/

49
 GENERATIVE MODELS

- Generative Adversarial Networks (GANs)

- Variational Autoencoders (VAEs)
- Normalized Flows
- Diffusion
- Auto-regresive
- Style transfer

50
 Normalizing Flows

Data Known
Distribution Distribution

Deep Density Destructors - Inouye & Ravikumar - ICML 2018

Normalizing Flows: An Introduction and Review of Current Methods - Kobyzev et al. (2019) - arxiv
Normalizing Flows for Probabilistic Modeling and Inference - Papamakarios et al. (2019) - arxiv
51
Normalizing Flows

52
Normalizing Flows

55
 Normalizing Flows

Gaussianization (RBIG)
➢Feature-wise Gaussianization (e.g. Histogram)
➢Rotation (e.g. PCA)

57
 Diffusion Models
Mas Info: Flows

· Awesome Normalizing Flows

https://github.com/janosh/awesome-normalizing-flows

· Libreria NFLOWS

https://github.com/bayesiains/nflows

https://github.com/bayesiains/nflows/blob/master/examples/moons.ipynb

· Gaussianization Flows

https://github.com/IPL-UV/gaussflow

https://colab.research.google.com/drive/1loLFPNK6hCMNpUIqB3ixMcC7iwoen0-g?usp=sharing

58
 GENERATIVE MODELS

- Generative Adversarial Networks (GANs)

- Variational Autoencoders (VAEs)
- Normalized Flows
- Diffusion
- Auto-regresive
- Style transfer

59
 Diffusion Models

https://yang-song.github.io/assets/img/score/celeba_large.gif
60
https://yang-song.github.io/assets/img/score/cifar10_large.gif
Diffusion Models

61
Mas Info: Diffusion Models
Muy buena explicación teórica

https://lilianweng.github.io/lil-log/2021/07/11/diffusion-models.html

Mas desde el punto de vista práctico (incluye ejemplos en Colab)

https://yang-song.github.io/blog/2021/score/

MODELOS

Palette:
https://iterative-refinement.github.io/

Stable Diffusion:
https://stablediffusionweb.com/

62
 GENERATIVE MODELS

- Generative Adversarial Networks (GANs)

- Variational Autoencoders (VAEs)
- Normalized Flows
- Diffusion
- Auto-regresive
- Style transfer

63
Autoregresive Generative Models

Explicit model
64
Autoregresive Generative Models

Explicit model
65
Autoregresive Generative Models

66
Mas Info: Autorregresivos
Curso Toronto
http://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/readings/L15%20Autoregressive%20a
nd%20Reversible%20Models.pdf

Explicación PixelCNN (y familia)

https://towardsdatascience.com/autoregressive-models-pixelcnn-e30734ede0c1

PixelRNN, PixelCNN++
https://towardsdatascience.com/auto-regressive-generative-models-pixelrnn-pixelcnn-32d192911173

CÓDIGO

Pixel CNN en Keras

https://keras.io/examples/generative/pixelcnn/

Deep Autoregressive Models (PixelCNN)

https://github.com/Mind-the-Pineapple/Autoregressive-models

68
 GENERATIVE MODELS

- Generative Adversarial Networks (GANs)

- Variational Autoencoders (VAEs)
- Normalized Flows
- Diffusion
- Auto-regresive
- Style transfer

69
Style Transfer

70
Style Transfer

71
Style Transfer

72
Style Transfer

73
 Style Transfer

74
https://fierce-bayou-68989.herokuapp.com/
 GENERATIVE MODELS

Mas Info: Style Transfer

Medium
https://medium.com/tensorflow/neural-style-transfer-creating-art-with-deep-learning-using-tf-keras-and-eager-execution-7d541ac31398

Wikipedia
https://en.wikipedia.org/wiki/Neural_style_transfer

Paper
https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf

CÓDIGO

En Keras
https://keras.io/examples/generative/neural_style_transfer/

En TF
https://www.tensorflow.org/tutorials/generative/style_transfer

JUGAR

https://deepart.io/

https://reiinakano.com/arbitrary-image-stylization-tfjs/

https://style-transfer.makeoptim.com/ 75
www.uv.es/lapeva/gandia.pdf

Gràcies! 76