KNOWLEDGE BASED

SCENE GRAPH GENERATION

IN MEDICAL FIELD
SRS Literature Survey

PROJECT GUIDE :

DEPARTMENT OF COMPUTER SCIENCE, SJEC

INTRODUCTION

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 1.1 Problem Statement
Knowledge-Based Scene Graph Generation in the Medical
Field.
Scene understanding in the medical field based on visual context using object
detection and representing it in the form of knowledge graphs in order to derive
conclusions or understand contexts.

This problem statement highlights understanding the hospital environment and

deciding the relationship between the objects detected in the image.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 1.2 Scope

We hope to use this project to help the visually impaired and the elderly to
better understand what’s in front of them. This is also a huge leap for research
areas and robot training.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 1.3 Definitions
Knowledge Graph : is a set of data points connected by relations that describe a
domain, for instance, a business, an organization, or a field of study.
● It is a powerful way of representing data.
● The concept of Knowledge Graphs borrows from the Graph Theory.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

LITERATURE SURVEY

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 1. Object Detection Meets Knowledge Graphs- Supporting Datasets”

Used two different approaches to quantify background knowledge as semantic consistency -

knowledge based and frequency based method

Semantic consistency are attribute values for the same logical concepts expressed and defined the
same way throughout the system/context.

● Analyzed the connections between simple object detection and a knowledge graph network.
● Claimed that semantic consistency can be applied to ‘any’ object detection model and increase
performance using transfer learning.
● The claim can not be reproduced and confirmed for the knowledge graph approaches.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

1 . “[Re] Object Detection Meets Knowledge Graphs - Supporting Datasets”
Lemmens, Jarl, Jancura, Pavol, Dubbelman, Gijs, & Elforai, Hala.

Dataset/ tools used Approaches Limitations

“[Re] Object Detection Mee COCO, VGG -16 Multiple Object detection with Failed to showcase or put out
ts Knowledge Graphs - Sup bounding boxes and labels any code implementation of the
porting Datasets” same.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 2. Few-Shot Object Detection Method Based on Knowledge
Reasoning

Enunciates generating knowledge graphs for a scene or a scene graph.

● Scene graph arranges the logical and spatial representation of a graphical scene as a
collection of nodes in a graph or tree structure.
● KG based reasoning or relation reasoning - combines visual information and
knowledge reasoning methods.
● Few-shot learning is when a machine is taught how to use data to learn from a
specific point of view.
● Used this semantic relationship to integrate knowledge reasoning and visual
information into a unified framework to achieve optimal compatibility.
● Recognized object instances and their attributes.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 2 . “Few-Shot Object Detection Method Based on Knowledge Reasoning “
Jianwei Wang and Deyun Chen

Dataset/ tools used Approaches Limitations

“Few-Shot Object R-CNN, Few shot learning - Generation of scene graphs. -Failed to showcase or put out
Detection Method Based on -Combined visual any code implementation of the
same.
Knowledge Reasoning “ information and knowledge
reasoning methods. - Limited to general semantic
structures of knowledge graphs.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 3. Object Detection Meets Knowledge Graphs.
● Developed a revolutionary framework for knowledge-aware object detection that
makes it possible to incorporate outside knowledge, such as knowledge graphs, into
any object detection algorithm.
● Embedding knowledge into object detection, a young paradigm that is yet only
effective for visual tasks.
● Knowledge graphs are used to derive and measure semantic consistency that
generalizes to new images with unrecognized contexts.
● Intend to investigate or build knowledge graphs that are specifically specialized to
visual tasks in future work.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

3. “Object detection meets knowledge graph “
Yuan Fang, Kingsley Kuan, Jie Lin, Cheston Tan & Vijay Chandrasekhar

Dataset/ tools used Approaches Limitations

“Object detection meets MSCOCO 15 - Quantify semantic -Only with the re-optimization
knowledge graphs “ PASCAL 07 consistency from knowledge using KG-CNet it identifies all
objects correctly otherwise it
graphs that can generalize to
identifies the objects partially.
new images with unobserved
contexts.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

4. Centroid-Tracking-Aided Robust Object Detection for Hospital Object

● Measuring and deriving semantic consistency from knowledge graphs,

which generalizes to new images with unfamiliar contexts.
● The objective of the current research effort is to identify hospital beds
and detect them so that robots can recognize them.
● It is advantageous to make use of the object detector’s ability to classify
objects, which causes it to automatically select every area that contains
hospital beds as the target object while retaining the ability to tell one
object from another, even if they are members of the same class.
● The new score is obtained by taking into account movement distance
score,which determines displacement
DEPARTMENT inSCIENCE,
OF COMPUTER relationSJEC to the size of the object.
4. “Centroid-Tracking-Aided Robust Object Detection For Hospital Object “
Fabiola Maria Teresa Retno Kinasih, Carmadi Machbub, Arief Syaichu Rohman & Lenni Yulianti

Dataset/ tools used Approaches Limitations

“Centroid-Tracking-Aided YOLO - Object -Not suitable for tracking fast-

Robust Object Detection For Detection Using novel moving objects as the location
will significantly differ in a
Hospital Object “ Knowledge-aware
short period.
framework.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 5. The More You Know: Using Knowledge Graphs for Image Classification

● Graph Search Neural Network (GSNN) is used which focuses on the features from
the image and selects relevant subsets of the input graph and predicts the output on
the various nodes.
● To propagate the beliefs from the initial nodes to the adjacent nodes, a method called
a per-node scoring function that tells how important the node is.
● The nodes of the graph could have represented anything from human relationships
to a computer program.
● Visual Genome is used in which each image is labeled with objects, attributes, and
relationships between objects entered by human annotators as well as the COCO
dataset.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

5. The More You Know: Using Knowledge Graphs for Image Classification
Kenneth Marino, Ruslan Salakhutdinov, Abhinav Gupta Carnegie Mellon University

Dataset/ tools used Approaches Limitations

The More You Know: Faster R-CNN, -Graph Search Neural -The GSNN and the framework
COCO, Visual Network (GSNN) is used that is used for vision problems
Using Knowledge Graphs Genome based on input and only is completely general
for Image Classification
chooses to expand nodes
that are useful for the
final output.
-All models are trained
with Stochastic Gradient
Descent, except GSNN
which is trained using
ADAM.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

6. Knowledge-Based Scene Graph Generation with Visual Contextual Dependency

● KVCD,this model combines external knowledge and visual features to determine the
global contextual information in a scene and thus generates a more balanced scene
graph.
● Feature extraction module uses object detection to determine the visual features of an
image.
● Relational reasoning module uses the visual features extracted by the previous
module and applies a novel approach to combine external knowledge with these visual
features.
● Visual dependency constraint module applies two losses to balance the model’s
reliance on the two types of knowledge applied in the relational reasoning module.
● The objectives are to detect objects in an image, to identify the relationships between
object pairs, and to use graph structures to visualize these objects and their
relationships. DEPARTMENT OF COMPUTER SCIENCE, SJEC
 6. Knowledge-Based Scene Graph Generation with Visual Contextual
Dependency
Dataset/ tools used Approaches Limitations

Knowledge-Based Scene Visual Genome, -given an image, detect -the computational complexity
R-CNN object of method is high.
Graph Generation with the object categories and
detector network
Visual Contextual with a their relationships. -due to significant categories
Dependency ResNeXt-101-FPN -The object features and imbalance in the dataset, the
backbone, bounding boxes in the model over-fits large sample
VSPNET. relationships & may ignore
input images are extracted
by the feature extraction small-sample relationships.
module.
-The visual relationship
is introduced to learn and
adjust the visual
dependencies in the
model

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 7. Artificial intelligence for Medical Imaging
● Integration of Knowledge Representation techniques in Machine Learning
applications.
● Motivated by the complementary strengths and weaknesses that could lead to a new
generation of hybrid intelligent systems.
● They hypothesize that knowledge graphs, which naturally provide domain
background knowledge in a machine-readable format, could be integrated into
Explainable Machine Learning Approaches to help them provide more meaningful,
insightful, and trustworthy explanations.
● Drawback: inability to explain their decisions in a way that humans can easily
understand them.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

7. Artificial intelligence for Medical Imaging

Dataset/ tools used Approaches Limitations

Artificial intelligence for inter-object statistical - Related to the research areas of - Failed to reduce the
Medical Imaging information to initialize object detection architectures computational complexity
the edge weights in the - Especially their classification
graph structure. losses and the usage of -Failed to obtain unbiased
commonsense knowledge in features
computer vision.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

8. Contrastive Object Detection Using Knowledge Graph Embeddings

● Related to the research areas of object detection architectures, especially their

classification losses and the usage of commonsense knowledge in computer vision.
● Two stages: 1. first filter image regions containing objects from a set of search
anchors.
2. Extracting visual features from which it regresses enclosing bounding boxes as
well as predicts a class label from a multi-set of classes for each object proposal in
isolation
● COCO dataset:data depicts complex everyday scenes containing common objects in
their natural context.
● Objects are labeled using per-instance segmentation to aid in precise object
localization.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 Contrastive Object Detection Using Knowledge Graph Embeddings
8.
Dataset/ tools used Approaches Limitations

Contrastive Object Using a systematic -Reasoning over knowledge - The KBX-systems that
Detection Using literature review graphs can be performed by analyzed exploit knowledge
Knowledge Graph Methodology, COCO means of standard graphs as silos of facts.
Embeddings dataset was used knowledge representation. - From which relevant triples
- formalisms allowing to are aggregated to support or
describe and label entities and explain a given observation,
without following any
the relationships between them.
particular semantic structure

DEPARTMENT OF COMPUTER SCIENCE, SJEC

PROPOSED SYSTEM

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 3.1 MOTIVATION

As humans we are able to understand the scene because of the general common sense
and the reasoning ability that is present within us.
It is difficult for a machine to do so.
We need a system that can derive conclusions about the scene by depending on visual
context and knowledge graphs.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 3.2 CONTRIBUTION
In other words, we define our project to understand what's happening in the
scene through context and relationships between objects.
Rather than identifying the objects around us, we provide an artificial mind to
1. Develop connections and relations between the objects detected from the
scene
2. Derive conclusions from them.
In the future, it can be used for training robots that can be used in the hospital
environment.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 3.3 COMPARISON TO EXISTING WORK

Existing work Proposed work

Dataset used COCO, YOLO, R-CNN, YOLO with transfer learning

Knowledge graphs Visual Genome, KVCD, comQA Neo4j, custom, GNN

Goals Knowledge-based models using visual Constraint environment by focusing

context. No given constraints. application towards the medical field.

Limitations -No code implementation. -Limited to constraints defined by the

-Limited to theoretical concepts and environment.
general knowledge graphs were given. -Bounded accuracy
-Dataset inadequacy

DEPARTMENT OF COMPUTER SCIENCE, SJEC

STATE OF ART :
Our project is an advancement to the current object detection technologies as we go one
step further and provide the detection with some understanding behind the entities it
detects.
We make use of knowledge graphs to maintain and create a relationship between the
objects detected.
This problem statement highlights understanding the hospital environment and deciding
the relationship between the objects detected in the image.

DEPARTMENT OF COMPUTER SCIENCE, SJEC

THE COMPLETE APPROACH

Fig: the complete approach

DEPARTMENT OF COMPUTER SCIENCE, SJEC

SUMMARY TABLE :

Tools used for object detection R-CNN, VGG-16 , COCO , YOLO datasets

Knowledge graph implementation Neo4j

Other tools/ concepts Usage of custom dataset, automatic Graph

generation, applied NLP, Transfer learning,
OpenCV

DEPARTMENT OF COMPUTER SCIENCE, SJEC

 REFERENCES
1.Fang, Yuan, Kingsley Kuan, Jie Lin, Cheston Tan, and Vijay Chandrasekhar. "Object detection meets knowledge graphs." International Joint Conferences on
Artificial Intelligence, 2017.

2.Wang, Jianwei, and Deyun Chen. "Few-Shot Object Detection Method Based on Knowledge Reasoning." Electronics 11, no. 9 (2022): 1327.
3.Fang, Yuan, Kingsley Kuan, Jie Lin, Cheston Tan, and Vijay Chandrasekhar. "Object detection meets knowledge graphs." International Joint Conferences on
Artificial Intelligence, 2017.

4.Kinasih, Fabiola Maria Teresa Retno, Carmadi Machbub, Lenni Yulianti, and Arief Syaichu Rohman. "Centroid-Tracking-Aided Robust Object Detection for
Hospital Objects." In 2020 6th International Conference on Interactive Digital Media (ICIDM), pp. 1-5. IEEE, 2020.

5.Marino, Kenneth, Ruslan Salakhutdinov, and Abhinav Gupta. "The more you know: Using knowledge graphs for image classification." arXiv preprint
arXiv:1612.04844 (2016).

6.Zhang, Lizong, Haojun Yin, Bei Hui, Sijuan Liu, and Wei Zhang. "Knowledge-Based Scene Graph Generation with Visual Contextual Dependency."
Mathematics 10, no. 14 (2022): 2525.

7.Tran, Khanhvi, Johan Peter Bøtker, Arash Aframian, and Kaveh Memarzadeh. "Artificial intelligence for medical imaging." In Artificial Intelligence in
Healthcare, pp. 143-162. Academic Press, 2020.

8.Lang, Christopher, Alexander Braun, and Abhinav Valada. "Contrastive object detection using knowledge graph embeddings." arXiv preprint arXiv:2112.11366
(2021).

DEPARTMENT OF COMPUTER SCIENCE, SJEC

THANK YOU

DEPARTMENT OF COMPUTER SCIENCE, SJEC