Group Name-IT2201

Mentor – Mr. Khustar Ansari

Assistant Professor
BIT SINDRI , DHANBAD JHARKHAND Department of CSE
DEPARTMENT OF HIGHER, TECHNICAL
EDUCATION AND SKILL DEVELOPMENT Name-Abhijit Saurabh (19030480001)
Name-Abhinav Kumar (19030480002)
Name-Aditya Gaurav Bhagat (19030480004)

Image Colorization Using Deep Learning

Mentor : Mr. Khustar Ansari

Group Members:
Abhijit Saurabh
Abhinav Kumar
Aditya Gaurav Bhagat

BIT SINDRI, JHARKHAND UNIVERSITY OF TECHNOLOGY

Presentations:
SYNOPSIS : 19th October , 2022
REVIEW 1 : 14th December, 2022
REVIEW 2 : 4th January, 2023
Pre Submission: 7th January,2023
CONTENTS:

1. Introduction
2. Literature Review
3. Problem Statement
4. Project Objective
5. Methodology
6. Expertimental Layout
7. Results and Conclusions
8. References
 Introduction
• The project is colorization of grey scale image using Deep Learning.

• Previous methods leverage the deep neural network to map input grayscale
images to plausible color outputs directly. Although these learning-based methods
have shown impressive performance, they usually fail on the input images that
contain multiple objects.

• Our project makes use of an off the shelf colorization model by Richard Zhang and
an off-the-shelf object detection model to crop out the images.
 Introduction
• Instead of colouring the entire image at once, our model extracts objects and
colors them individually .

• For this we require instance level features which can be done easily only when
the image is in LAB format instead of RGB.

• In RGB all the three channels include values of lightness as well as colors. But in
Lab only a single dedicated channel is used to store information about lightness.
 Introduction
Similar to the RGB color space,there is another color space named
as CIELAB color space. The CIELAB color space has three channels.
But unlike the RGB color space, CIELAB encodes color information
differently:
• The L channel encodes lightness intensity only
• The a channel encodes green-red.
• And the b channel encodes blue-yellow
 Literature Review
S.No Authors Techniques used Dataset Result
Colorization with a deep CNN and a well-
chosen objective function can come closer to
Richard Zhang, Phillip
Convolutional Neural ImageNet training producing results indistinguishable from real
1 Isola, and Alexei A
Network (CNN) set color photos. This method not only provides a
Efros (2016)
useful graphics output, but can also be viewed
as a pretext task for representation learning
Satoshi Iizuka, Edgar
Model exploits the semantic context of each
Simo-Serra, and Convolutional Neural
2 Places dataset image with the global features, allowing it to
Hiroshi Ishikawa Network (CNN)
properly colorize without any intervention
(2016)

This fully automatic colorizer produces strong

Gustav Larsson,
results, improving upon previously leading
Michael Maire, and Convolutional Neural ImageNet/ctest10k
3 methods by large margins on all datasets tested
Gregory Network (CNN) dataset
and also propose a new large-scale benchmark
Shakhnarovich (2017)
for automatic image colorization
 Literature Review

S.No Authors Techniques used Dataset Result

Richard Zhang, Jun-Yan

  Benefit of this system is that the
Zhu, Phillip Isola,
Covolutional Neural network predicts user-intended
4 Xinyang Geng, Angela S.
Network (CNN) actions based on learned semantic
Lin, Tianhe Yu, and ImageNet dataset similarities
Alexei A. Efros (2017)

Experiments
When compared with Zhang et al. ,
mainly
this approach is preferred in 80.0% of
Chenyang Lei and K nearest neighbor conducted on the
5 the comparisons on the DAVIS
Qifeng Chen (2019) (KNN) DAVIS dataset
dataset and 88.8% of the comparisons
and the Videvo
on the Videvo dataset
dataset
Problem Statement
• Coloring of Grey-Scale image.

• Address the problem of contrast and brightness in the output image

colored using deep learning.

• Address the problem of false colorization i.e certain colorization

networks fails to predict the colors accurately when the image
contains a lot of objects . The model fails to identify the objects from
the background.
Fig 1.1 Output image of colorization using Zhnag’s
model presented at ECCV
Project Objective
• To observe if the proposed colorization method gains any
improvement over existing colorization methods.

• To observe the metrics on which the image quality can be improved.

 Methodology
• Object detection using pre trained model
(using Detectron2)

• Training the colourization model based on Zhang’s model

• Training instance colourization network

• Create a Fusion Network that takes instance features and full image
features as input .
Methodology

Fig. Process Flow Diagram

Object Detection
We need to detect all the bounding boxes for the images.
These bounding boxes are used for object detection. We
employ an off the-shelf pre-trained network, Mask R-CNN as
our object detector. This network is present in detectron2
which is a PyTorch-based modular object detection library
provided by Facebook AI Research
 Instance-aware Image Colorization model

1.Full Image Colorization Network: We would start to train our full image
colorization branch based on the siggraph retrained’s pretrained weight

2.Instance Colorization Network: We would use the full image colorization

branch's weight as our instance colorization branch's pretrained weight.

3.The third network , ie , the fusion network takes input from each layer of
both the full image colorization network and instance colorization network.
EXPERIMENTAL SETUP:

• DATA SET :
i. Stanford Vision Lab 2020,”Stanford University,Priceton University”,”
https://www.image-net.org/download.php”
ImageNet Dataset is used for training the Full-image colorization network
and Instance colorization network:
ii. Microsoft,Facebook,MightyAI 2021,”Coco Stuff Dataset”,
https://cocodataset.org/download
COCO-Stuff dataset contains a wide variety of natural scenes with multiple
objects present in the image. This dataset is used for performance
evaluation and validation.
Experimental Setup

Tools Used:
• CUDA 10.1
• Pytorch >= 1.5
• Python3
• Detectron2
• OpenCV-Python
• Pillow/scikit-image
 RESULTS AND CONCLUSIONS

Fig. 5.1 Mark Twain Image Colorization Using Our Approach

Colorization Results
Colorization Results
References
[1]. Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image
colorization." European conference on computer vision. Springer, Cham,
2016.

[2]. Yuxin Wu and Alexander Kirillov and Francisco Massa andWan-Yen

Lo and Ross Girshick, “Detectron2”, 2018.
https://github.com/facebookresearch/detectron2.
THANK YOU