A Secure Forensic Evidence Sharing Platform Using

Blockchain Technology

MAJOR PROJECT REPORT

Submitted in partial fulfillment of the requirements for the award of the degree

of

BACHELOR OF TECHNOLOGY

in

COMPUTER SCIENCE & ENGINEERING

by

AVDHESH CHAUDHARY AMAR SINHA CHIRAG AGGARWAL

EN. No: 02315002719 EN. No: 04415002719 EN. No: 04615002719

Guided by
Dr. Vikrant Shokeen
Assistant Professor,CSE

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

MAHARAJA SURAJMAL INSTITUTE OF TECHNOLOGY
(AFFILIATED TO GURU GOBIND SINGH INDRAPRASTHA UNIVERSITY, DELHI)
DELHI – 110058
2019-23

I
 CANDIDATE’S DECLARATION

It is hereby certified that the work which is being presented in the B. Tech Minor
Project Report entitled "A Secure Forensic Sharing Platform Using Blockchain
Technology" in partial fulfillment of the requirements for the award of the degree of
Bachelor of Technology and submitted in the Department of Computer Science &
Engineering of MAHARAJA SURAJMAL INSTITUTE OF TECHNOLOGY,
New Delhi (Affiliated to Guru Gobind Singh Indraprastha University, Delhi) is
an authentic record of our own work carried out during a period from March 2023 to
June 2023 under the guidance of Dr. Vikrant Shokeen, Assistant Professor ,CSE.

The matter presented in the B. Tech Minor Project Report has not been submitted by
me for the award of any other degree of this or any other Institute.

AVDHESH CHAUDHARY AMAR SINHA CHIRAG AGGARWAL

EN. No: 02315002719 EN. No: 04415002719 EN. No: 04615002719

II
 CERTIFICATE

This is to certify that the above statement made by the candidate is correct to the best
of my knowledge. He/She/They are permitted to appear in the External Minor Project
Examination

(Dr. Vikrant Shokeen) (Dr. Rinky

Dwivedi)
Assistant Professor,CSE HOD,CSE

III
 ACKNOWLEDGEMENT

We express our deep gratitude to Dr. Vikrant Shokeen, Assistant Professor,

Department of Computer Science & Engineering for his valuable guidance and
suggestion throughout my project work. We are thankful to Dr. Koyel Datta Gupta,
Associate Professor for their valuable guidance.

We would like to extend my sincere thanks to Head of the Department, Dr. Rinky
Dwivedi for his time to time suggestions to complete my project work. I am also
thankful to Col(Dr.)Ranjit Singh, Principal for providing me the facilities to carry
out my project work.

AVDHESH CHAUDHARY AMAR SINHA CHIRAG AGGARWAL

EN. No: 02315002719 EN. No: 04415002719 EN. No: 04615002719

IV
 TABLE OF CONTENTS

TITLE PAGE
i
CANDIDATE DECLARATION
ii
CERTIFICATE iii
ACKNOWLEDGEMENT iv
TABLE OF CONTENTS v
LIST OF FIGURES vii
LIST OF TABLES viii
ABSTRACT ix

Chapter 1: Introduction 1 – 10
1.0 Introduction 1
1.1 Architecture 3
1.1.1 Dataset 3
1.1.2 Preprocessing Testing Test 4
1.1.3 Inception V3 6
1.1.3 Convolutional Neural Network 6
1.1.3 Grouping of Features maps
in to a single Pattern 7
1.1.3 Recurrent Neural Network 7
2.0 Motivation 8
3.0 Objective 9
4.0 Summary of the Report 9
Chapter 2: Project Design & Implementation 11 – 19
2.1 Data Sources 11
2.2 Feed the Videos to Network 13
2.3 Transfer Learning &
Feature Extraction 14
2.4 Label Encoding 16

V
 2.5 CNN Model Processing 16
2.6 The Sequence Model 17
2.7 Prediction 19

Chapter 3: Result & Discussion 20 – 23

Chapter 4: Conclusion 24 – 25

VI
 LIST OF FIGURES

Figure 1. This figure is from Show, attend and tell visualization

Figure 2. Workflow of Anomaly Recognition System
Figure 3. Videos of 14 Malicious Activities
Figure 4. Object detection using inception V3
Figure 5. A Working Of Deep CNN
Figure 6. A LSTM Cell Structure
Figure 7. (a) Accuracy of Model 1
Figure 7. (b) Loss of Model 1
Figure 8. (a) Accuracy of Model 2
Figure 8. (b) Loss of Model 2
Figure 9. (a) Accuracy of Model 3
Figure 9. (b) Loss of Model 3
Figure 10. (a) Accuracy of Model 3
Figure 10. (b) Loss of Model 3
Figure 11. (a) Accuracy of Model 3
Figure 11. (b) Loss of Model 3
Figure 12. (a) Accuracy of Model 3
Figure 12. (b) Loss of Model 3

VII
 LIST OF TABLES

Table 1. Performance Comparison of all models

Table 2. Details about the Optimized Model.

VIII
 ABSTRACT

The growing need for secure and tamper-proof forensic evidence sharing has led to
the exploration of innovative technologies such as blockchain. This research paper
presents a novel approach to building a secure forensic evidence sharing platform
using blockchain technology, with a specific focus on utilizing file storage and file
sharing features on the blockchain. The proposed platform leverages a local
blockchain network established using Hardhat, Ether.js, Truffle, and Solidity,
combined with a frontend developed using React JS. In addition, the InterPlanetary
File System (IPFS) network, facilitated by Pinata, is utilized for efficient and
decentralized storage of files.
The platform incorporates blockchain's inherent characteristics of immutability,
decentralization, and transparency to enhance the security and integrity of forensic
evidence. By storing evidence files on the IPFS network and linking their metadata to
the blockchain, the platform ensures that evidence remains tamper-proof and
accessible only by authorized parties. Smart contracts are employed to enforce secure
and automated sharing of evidence among relevant stakeholders, enabling efficient
collaboration while maintaining data privacy and confidentiality.
The research paper evaluates the platform's security, privacy, and performance
aspects. Security measures include cryptographic techniques, access control
mechanisms, and data encryption, ensuring the confidentiality and integrity of shared
evidence. Privacy considerations encompass pseudonymity and anonymity to protect
the identities of involved parties. Performance and scalability of the platform are
assessed through various experiments, including file upload and download times,
transaction throughput, and network latency.
The results demonstrate the viability and effectiveness of the proposed platform in
providing a secure forensic evidence sharing solution. The utilization of blockchain
technology, alongside IPFS for file storage, enhances data integrity, prevents
unauthorized access, and enables efficient collaboration among forensic investigators

IX
and other relevant stakeholders. The research findings contribute to the field of digital
forensics and provide a practical framework for developing similar secure evidence
sharing platforms.

X
 CHAPTER 1: INTRODUCTION

1.0 INTRODUCTION
An essential part of the criminal justice system is played by forensic science. In order
to establish facts that can aid in the investigation and conviction of criminals or the
exoneration of innocent individuals, forensic scientists assess, analyse, and scrutinise
evidence from crime scenes and other places. The most often used forensic science
research lab specialisations are forensic genetic biology (DNA), trace proof
examination, forensic biochemistry, firearms, latent fingerprint investigation and
handwriting analyses, tool marks study, fire and explosives research, forensic
toxicology, and digital evidence.

In addition to forensic laboratories, additional intelligence disciplines include forensic

pathology, forensic nursing, forensic psychology, forensic entomology, and forensic
engineering. Practitioners of these professions could be found at medical examiners
and coroners offices, institutes, or private clinics. Intelligence scientists collect,
maintain, and analyse scientific evidence during the investigation. While some
intelligence scientists were visiting the crime scene to gather evidence on their own,
others were taking on a laboratory role, analysing the material presented to them by
other people. However, some have participated in the analysis of financial, banking,
or other numbers that will be used to investigate financial crime, and maybe employed
as consultants for private companies, academics, or government employees. With the
fast rise in cybercrime in today's digital environment, digital evidence is becoming
increasingly important for the provenance of person-toperson cybercrime links. The

1
chain of custody for digital evidence has its own set of issues. Block chain technology
is a distributed system that uses ledger based transactions, which could store linked
records within the range of a decentralized database within the P2P network. The
advantage of using block chain in digital forensics is to ensure secure and reliable
sharing of data between the sender and the receiver [1]. When a crime is committed,
the corpse is sent to the forensic lab where they analyse the body and generate the
report of the evidence. Evidence gathered at a crime scene is crucial in solving the
case and bringing the persons involved to justice. It is extremely difficult to guide a
case in the proper path without proof. As a result, safeguarding this evidence from any
type of tampering is critical. Maintaining the integrity of evidence requires proper
handling and packing. Enigma is a secure forensic data sharing application in which
the pieces of evidence are securely transferred from the forensic lab to the police
station. In [15] the paper focuses on implementing deep learning technology based on
different methodologies to classify different malware families. In [16] the paper
highlights the role and importance of image processing in the service industry and
how location-based image querier would be used for extracting important values to
improve services. In [17]the paper discusses the role of video in painting in detecting
missing frames of unvarying images in the field of video analytics.
1.1 Problem Statement
It is observed that forensic reports are changed most of the times during the
investigation process. Unrecognized sources may change data of forensic reports. In
most of the cases, due to this the judgment may not be fair and justice may not be
provided to the needy. This is due to the lack of security in the existing systems. The
existing forensic data sharing system includes sharing data via emails, third party
websites, physical forms, etc which leads to issues like of data security, allowing
unknown websites to access confidential data, modification of data content, and
viruses of infecting shared files, etc. Hence, this problem faced by the people brings
to rise the need of implementing a system that ensures the complete security of
forensic report/data, all the way starting from their generation until the time they used
finally for the purpose of serving justice to the needful. Our aim is to develop a data
sharing system that is used to send the forensic report from the pathology lab to the
police station using Blockchain Technology.
1.1 System Overview

2
The system is conceptualized, designed and proposed by us in a manner which mainly
aims towards increasing the overall security of confidential or forensic data sharing
while maintaining the authenticity as well as integrity of the system. The system
designed by us will take the forensic report as an input initially from the pathology lab
which will be further processed to the doctor. After a brief and thorough examination
of the document by the doctor, the status of the report will be marked as verified by
the doctor if the report is found to be authentic. Further, the tamperproof verified
report can be viewed by the police station for further examination.
1.2 Blockchain in Forensics
One of the primary benefits of storing data in a digital form is its versatility.
Authorized individuals can quickly have access to it. Without inflicting any harm to
the actual document, several copies can be made and saved. It is readily available
from every location on the planet. It is possible to transfer many documents at once.

It enhances efficiency since searching for information preserved in digital medium

takes only a few seconds compared to searching for traditional papers, which takes
much longer. Evidence documentation can be damaged by natural or manmade
calamities. As a result, including blockchain technology into the field of Forensic
Science will reduce the chances of these papers being corrupted. This technology will
also eradicate human error. It is vital to incorporate such technologies into forensic
evidence management systems as the world moves toward digitalization.
1.2.1 Private Blockchain:

3
A private blockchain is a permissioned blockchain since it is run by a network
administrator and only approved users can connect to the network. The network is
controlled by one or more organizations, which makes it necessary to conduct
transactions via third parties. Only the parties involved in the transaction will know
about it in this sort of blockchain; others won't be able to access it, making the
transaction private.
1.2.2 Public Blockchain:
Public blockchains, sometimes referred to as permissionless blockchains, are totally
open and strictly adhere to the decentralisation principle. Public blockchains include
ones like Bitcoin and Ethereum. Anybody with access to the network can contribute
blocks to the chain. In contrast to private blockchains, where the identities of the
parties to a transaction are kept secret, public blockchains are likewise mostly
anonymous.
1.3 Smart Contracts in Blockchain
Smart contract is the agreement involving two parties that are maintained in computer
code. It does not necessitate the involvement of a third party. Smart contracts function
similarly to traditional contracts in which specific code can be inserted immediately
and the parties concerned can check the respective code before the deadline. Smart
contracts include certain terms and circumstances that must be adhered to.

Adhering to agreements, defining regulations and executing the business logic,

relying on blockchain technology for encryption to help protect and authenticate all

4
messages it contains, processing, and lastly updating the blockchain network are all
part of a smart contract's anatomy. It is stored in a database because it works with
blockchain and is irreversible. A smart contract's transactions must be handled
primarily by blockchain technology, which eliminates the need for a third party and,
as a result, saves time.
1.4 Security and Privacy:
Security Benefits of Blockchain for File Storage and Sharing: Blockchain technology
offers several security benefits that can enhance the security of file storage and
sharing in the context of a forensic evidence sharing platform.
Immutable Records: The immutability of blockchain ensures that once a file is stored
on the blockchain, it cannot be modified or tampered with without detection. Each file
is associated with a unique cryptographic hash, making it possible to verify the
integrity and authenticity of the shared files. This feature provides strong evidentiary
value, ensuring that the forensic evidence remains tamper-proof and reliable
throughout its life cycle.
Distributed Storage: Blockchain-based file storage leverages decentralized networks,
such as IPFS, to distribute files across multiple nodes. This distributed storage
architecture enhances resilience against data loss or unauthorized access. Even if
some nodes in the network fail or are compromised, the files remain accessible and
retrievable from other nodes, ensuring high availability and data redundancy.
Encryption: Blockchain platforms can incorporate encryption techniques to secure the
stored files. Encryption algorithms can be applied to the files before they are uploaded
to the IPFS network or stored on the blockchain. This ensures that the files are
protected and can only be accessed by authorized parties with the appropriate
decryption keys.
Access Control: Smart contracts in blockchain-based systems enable fine-grained
access control mechanisms. Access to shared files can be governed by predefined
rules and conditions, allowing only authorized individuals or entities to view or
modify the files. This reduces the risk of unauthorized access and ensures that
sensitive forensic evidence is accessible only to authorized stakeholders.

b. Privacy Considerations in a Blockchain-based System: Privacy is a critical aspect

when dealing with forensic evidence sharing, as it involves sensitive and confidential

5
information. Blockchain-based systems can address privacy concerns through various
mechanisms:
Encryption: As mentioned earlier, encryption techniques can be employed to encrypt
the shared files. This ensures that the content of the files remains confidential, even if
they are stored on a distributed network. Encryption helps protect the privacy of the
forensic evidence from unauthorized access or disclosure.
Anonymity and Pseudonymity: Blockchain platforms can allow users to participate
under pseudonyms, protecting their identities while still maintaining the transparency
and accountability of transactions. This pseudonymity feature can be beneficial for
forensic evidence sharing, as it provides privacy for individuals involved in sensitive
investigations.
Off-Chain Storage: While blockchain provides a secure and tamper-proof ledger, it
may not be suitable for storing large files directly. In such cases, off-chain storage
solutions like IPFS can be utilized. IPFS allows files to be stored in a decentralized
manner while their hashes and metadata are recorded on the blockchain. This
approach balances the need for data privacy with the efficiency of file storage and
retrieval.
Permissioned Access: In private or consortium blockchains, access to the network can
be restricted to authorized participants. This ensures that only trusted entities have
access to the shared files, enhancing privacy and reducing the risk of data breaches.
By employing encryption, anonymity, pseudonymity, and permissioned access, a
blockchain-based forensic evidence sharing platform can address privacy concerns
while maintaining the security and integrity of the shared files.
1.5 Performance and Scalability:
By employing encryption, anonymity, pseudonymity, and permissioned access, a
Performance and scalability are crucial factors to consider when implementing file
storage and sharing on a blockchain-based platform. The decentralized and distributed
nature of blockchain introduces unique challenges that need to be addressed to ensure
efficient and scalable operations.
Transaction Throughput: Blockchain platforms, especially public ones, often face
limitations in terms of transaction throughput. The consensus mechanisms, such as
Proof of Work (PoW) or Proof of Stake (PoS), require time-consuming computations,

6
leading to slower transaction processing speeds. This can impact the speed at which
files are stored and shared on the blockchain.
File Size and Storage Limitations: Storing large files directly on the blockchain can be
inefficient and impractical due to the limited storage capacity of each block.
Additionally, replicating large files across multiple nodes in a decentralized network
can consume significant resources and hinder scalability.
Network Latency: The decentralized nature of blockchain involves communication
and synchronization among multiple nodes. This can introduce network latency,
especially when performing file storage and retrieval operations that require
coordination among different network participants. Latency can impact the overall
performance and responsiveness of the file storage and sharing system.

b. Solutions and Optimizations for Performance and Scalability: Off-Chain Storage:

To address the limitations of storing large files directly on the blockchain, an
approach called off-chain storage can be employed. Off-chain storage solutions, such
as the InterPlanetary File System (IPFS), allow files to be stored in a distributed
manner while their hashes and metadata are recorded on the blockchain. This
approach improves scalability by reducing the data stored on the blockchain itself
while maintaining the benefits of decentralized file storage and retrieval.
Layer-2 Solutions: Layer-2 solutions, such as state channels or sidechains, can be
utilized to improve the scalability of file storage and sharing on the blockchain. These
solutions enable faster and more efficient transactions by conducting certain
operations off the main blockchain. By leveraging these mechanisms, the overall
throughput and performance of the system can be significantly improved.
Caching and Data Compression: Implementing caching mechanisms can help enhance
the performance of file storage and retrieval operations. By caching frequently
accessed files or metadata, the system can reduce the need for repeated blockchain
interactions, resulting in faster access times. Additionally, data compression
techniques can be applied to reduce the storage footprint of files and optimize
network bandwidth usage.
Sharding: Sharding is a technique that involves partitioning the blockchain network
into smaller subsets called shards, where each shard processes a portion of the
transactions. This approach improves scalability by parallelizing transaction
processing across multiple shards. Sharding can be particularly beneficial for file

7
storage and sharing, as it allows for concurrent processing of multiple file
transactions, reducing overall latency.
Optimized Consensus Mechanisms: Exploring alternative consensus mechanisms that
offer faster transaction processing, such as Proof of Authority (PoA) or Delegated
Proof of Stake (DPoS), can help improve the performance and scalability of the file
storage and sharing system. These consensus mechanisms prioritize efficiency and
throughput, making them suitable for applications that require high transaction
speeds. By implementing off-chain storage, utilizing layer-2 solutions, employing
caching and compression techniques, exploring sharding, and optimizing consensus
mechanisms, the performance and scalability challenges associated with file storage
and sharing on the blockchain can

2.0 MOTIVATION
A blockchain-based data sharing framework is proposed by authors in [25]. A
framework contributes to the version control of the documents so that the changes can
be tracked easily. Multiple users can collaborate on a blockchain platform to negotiate
on document changes. Therefore, multiple versions of documents can be stored in a
network without the involvement of third party. Moreover, a decentralized storage,
IPFS, is used to store the data. Ethereum smart contracts are used to develop an
interaction among multiple entities such as owners and users of the document.
Data privacy and confidentiality is ensured by authors . ABE along with key
policy are the important techniques to secure digital content. In this paper, inner
product encryption and re-encryption techniques are applied where decryption is only
possible if private key has an inner product with a value of zero. A specific set of
attributes are specified by the owner of data to grant access. Hence, it achieves fine-
grained access control.
Authors in proposed a decentralized storage system with digital content
protection. A secret key is distributed to the user by the owner of the data. While
releasing the secret key, the owner defines the access policy to have a control over the
usage of sensitive information. Attribute encryption scheme is adapted by authors to
accomplish their goal.
From the existing literature discussed above, we have gained motivation to work
on the digital data sharing using blockchain. Most of the researchers have worked on

8
similar domain, yet there is much scope to enhance and modify the existing work to
contribute for the benefit of research community.
.
3.0 OBJECTIVE
1. Utilize file storage and file sharing features on a blockchain network to enhance the
security and integrity of forensic evidence.
2. Establish a local blockchain network using Hardhat, Ether.js, Truffle, and Solidity
to enable secure and transparent transactions related to evidence sharing.
3. Integrate a front end developed with React JS to provide a user-friendly interface
for evidence upload, download, and sharing.
4. Leverage the Inter Planetary File System (IPFS) network, facilitated by Pinata, for
efficient and decentralized storage of evidence files.
5. Evaluate the security, privacy, and performance aspects of the proposed platform,
comparing it with traditional centralized approaches.
6. Contribute to the field of digital forensics by providing a practical framework and
insights for developing secure and efficient evidence sharing platforms using
blockchain technology

4.0 SUMMARY OF THE REPORT

CCTV surveillance systems are commonly used to ensure the safety and security of
public and private spaces. However, manual monitoring of surveillance footage can
be tedious and time-consuming, making it difficult to promptly identify and respond
to potential threats. In this paper, we present a real-time threat detection system for
CCTV surveillance that utilizes deep learning models to detect and classify levels of
high movement in video frames. By treating videos as segments and defining
anomalous (threatening) and normal (safe) segments, our system is able to
continuously monitor surveillance footage in real-time and identify potential threats,
such as abuse, burglaries, explosions, shootings, fighting, shoplifting, road accidents,
arson, robbery, stealing, assault, and vandalism. To evaluate the performance of our
system, we conducted extensive experiments on a large dataset of CCTV footage and
achieved promising results. Our system has the potential to significantly improve the
efficiency and effectiveness of CCTV surveillance, enabling faster response times and
enhanced security for individuals.

9
In our approach, we consider normal and anomalous videos as bags and video
segments as instances in multiple instance learning (MIL), and automatically learn a
deep anomaly ranking model that predicts high anomaly scores for anomalous video
segments. Furthermore, we introduce sparsity and temporal smoothness constraints in
the ranking loss function to better localize anomaly during training. We also introduce
a new large-scale first of its kind dataset of 128 hours of videos. It consists of 1900
long and untrimmed real-world surveillance videos, with 13 realistic anomalies such
as fighting, road accident, burglary, robbery, etc. as well as normal activities. This
dataset can be used for two tasks. First, general anomaly detection considering all
anomalies in one group and all normal activities in another group. Second, for
recognizing each of 13 anomalous activities. Our experimental results show that our
MIL method for anomaly detection achieves significant improvement on anomaly
detection performance as compared to the state-of-the-art approaches. We provide the
results of several recent deep learning baselines on anomalous activity recognition.
The low recognition performance of these baselines reveals that our dataset is very
challenging and opens more opportunities for future work.

10
CHAPTER 2: PROJECT DESIGN & IMPLEMENTATION

2.1 Data Sources

The Dataset Below is Divided in to Previous and Current Dataset
Previous datasets
We briefly review the existing video anomaly detection datasets in this section. The
UMN dataset [2] consists of five different staged videos, where people walk around
and after some time start running in different directions. The anomaly is characterized
by only running action. UCSD Ped1 and Ped2 datasets [27] contain 70 and 28
surveillance videos, respectively. Those videos are captured at only one location. The
anomalies in the videos are simple and do not reflect realistic anomalies in video
surveillance, e.g. people walking across a walkway, non pedestrian entities (skater,
biker and wheelchair) in the walkways. Avenue dataset [28] consists of 37 videos.
Although it contains more anomalies, they are staged and captured at one location.
Similar to [27], videos in this dataset are short and some of the anomalies are
unrealistic (e.g. throwing paper). Subway Exit and Subway Entrance datasets [3]
contain one long surveillance video each. The two videos capture simple anomalies
such as walking in the wrong direction and skipping payment. BOSS [1] dataset is
collected from a surveillance camera mounted in a train. It contains anomalies such as
harassment, person with a disease, panic situation, as well as 6482 normal videos. All
anomalies are performed by actors. Abnormal Crowd [31] introduced a crowd
anomaly dataset which contains 31 videos with crowded scenes only. Overall, the
previous datasets for video anomaly detection are small in terms of the number of
videos or the length of the video. Variations in abnormalities are also limited. In
addition, some anomalies are not realistic.
Our datasets
Due to the limitations of previous datasets, we construct a new large-scale dataset to
evaluate our method. It consists of long untrimmed surveillance videos which cover
13 realworld anomalies, including Abuse, Arrest, Arson, Assault, Accident, Burglary,
Explosion, Fighting, Robbery, Shooting, Stealing, Shoplifting, and Vandalism. These
anomalies are selected because they have a significant impact on public safety. We
compare our dataset with previous anomaly detection datasets in Table 1.

11
 Video collection. To ensure the quality of our dataset, we train ten annotators
(having different levels of computer vision expertise) to collect the dataset. We search
videos on YouTube and LiveLeak 1 using text search queries (with slight variations
e.g. “car crash”, “road accident”) of each anomaly. In order to retrieve as many videos
as possible, we also use text queries in different languages (e.g. French, Russian,
Chinese, etc.) for each anomaly, thanks to Google translator. We remove videos
which fall into any of the following conditions: manually edited, prank videos, not
captured by CCTV cameras, taking from news, captured using a hand-held camera,
and containing compilation. We also discard videos in which the anomaly is not clear.
With the above video pruning constraints, 950 unedited real-world surveillance videos
with clear anomalies are collected. Using the same constraints, 950 normal videos are
gathered, leading to a total of 1900 videos in our dataset. In Figure 2, we show four
frames of an example video from each anomaly.
Annotation. For our anomaly detection method, only video-level labels are required
for training. However, in order to evaluate its performance on testing videos, we need
to know the temporal annotations, i.e. the start and ending frames of the anomalous
event in each testing anomalous video. To this end, we assign the same videos to
multiple annotators to label the temporal extent of each anomaly. The final temporal
annotations are obtained by averaging annotations of different annotators. The
complete dataset is finalized after intense efforts of several months.
Training and testing sets. We divide our dataset into two parts: the training set
consisting of 800 normal and 810 anomalous videos (details shown in Table 2) and
the testing set including the remaining 150 normal and 140 anomalousvideos. Both
training and testing sets contain all 13 anomalies at various temporal locations in the
videos. Furthermore, some of the videos have multiple anomalies. The distribution of
the training videos in terms of length (in minute) is shown in Figures 3. The number
of frames and percentage of anomaly in each testing video are presented in Figures 4
and 5, respectively.

12
2.2 Feed the videos to a Network

Extracting frames from the captured CCTV recordings is the first step of this
approach. The work extracts the frame after a fixed and small interval of time (say 1
sec). This extracted frame is then resized to the dimension 299x299 pixels which are
the standard input dimensions for InceptionV3. The preprocess_input function is
meant to adequate the resized image to the format that the model requires.

13
2.3 Transfer Learning & Feature Extraction :

Images are nothing but input (X) to our model. As you may already know that any
input to a model must be given in the form of a vector. We need to convert every
image into a fixed sized vector which can then be fed as input to the neural network.
For this purpose, we opt for transfer learning by using the Inception V3 model
(Convolutional Neural Network).Inception V3 is not the first model coming from the
ResNet family. The original model was called the Inception V3 and was another
milestone in the CV domain back in 2015. The main motivation behind this model
was to avoid poor accuracy as the model went on to become deeper. Additionally, if
you are familiar with Gradient Descent, you would have come across the Vanishing
Gradient issue – the Inception V3 model aimed to tackle this issue as well. Inception
V3 also follows a similar technique with just more layers) The code for this is as
follows

14
15
2.4 Label Encoding

StringLookup layer encode the class labels as integers.

2.5 CNN Model Processing

First, we run every frame from every video through Inception, saving the output from
the final pool layer of the network. So we effectively chop off the top classification
part of the network so that we end up with a 2,048-d vector of features that we can pass
to our RNN.

16
2.6 The sequence model

we convert those extracted features into sequences of extracted features. If you recall
from our constraints, we want to turn each video into a 40-frame sequence. So we
stitch the sampled 40 frames together, save that to disk, and now we’re ready to train
different RNN models without needing to continuously pass our images through the
CNN every time we read the same sample or train a new network architecture.
Now, we can feed this data to a sequence model consisting of recurrent layers like
GRU.

17
For the RNN, we use a single, 4096-wide LSTM layer, followed by a 1024 Dense
layer, with some dropout in between. This relatively shallow network outperformed all
variants where we tried multiple stacked LSTMs.

2.7 Prediction

18
CHAPTER 3 : RESULT AND DISCUSSION

19
In this work, we have trained 6 variations of our approach by altering different
parameters and refining the dataset. The output layer of the RNN in model 1 has two
neurons which are used to classify the entire dataset into 2 categories i.e. threat and
safe. The anomalies considered for this model are Abuse, Arrest, Assault, and Arson
along with a set of normal videos. The videos used are untrimmed and contain several
unwanted footages. There are 940 chunks of un-shuffled frames with each chuck of
30 frames extracted at an interval of 1 second. The optimiser and loss function used
for training this model are Adam and mean_squared_error resp. Fig. 7. (a) and (b)
shows that the model trained is overfitted and producing fluctuating output with poor
testing and training accuracy

Fig. 7. (a) Accuracy of Model 1; (b) Loss of Model 1

To overcome the imperfections caused due to overfitting in model 1, the size of each
chunk is reduced and the regularization [21] parameter is set to 0.01. The added term
is considered to control the excessively fluctuating function. This prevents the
coefficients to take extreme values. This forms the second model, Model 2. Fig. 8. (b)
reflects that the loss function of model 2 is better tuned as compared to that of model
1. Moreover, overfitting is reduced to some extent as shown in Fig. 8. (a) producing a
comparatively well-performing model.

Fig. 8. (a) Accuracy of Model 2; (b) Loss of Model 2

Later, model 3 is designed to reduce overfitting to a considerable level. Along with
regularization, the dataset is cleaned by manually trimming each video. Trimming is

20
done to exclude the unwanted and useless footage from the videos that are causing
improper training of the model. The dataset was shuffled this time which was not
done in the case of model 1 and model 2. Moreover, the optimizer for model 3 is
changed from Adam with the learning rate of 0.001 to SGD with the learning rate of
0.01.

Fig. 9. (a) Accuracy of Model 3; (b) Loss of Model 3

Fig. 9. (a) shows that cleaning the dataset and changing the optimizer played an
important role in reducing the overfitting in the trained model. But, the accuracy of
the model is not as per expectations. Fig. 9. (b) presents a smooth curve for loss
function with no fluctuation.
To enhance the accuracy of the model, eight more classes of anomalies; Road
Accidents, Burglar, Explosion, Shooting, Fighting, Shoplifting, Robbery, Stealing,
and Vandalism are added to the dataset. Also, the size of the chunks is decreased to 8.
Model 4 is created to classify the segments of video into 13 different categories rather
than 2 to give a better description of the anomaly detected by the algorithm.

Fig. 10. (a) Accuracy of Model 4; (b) Loss of Model 4

Fig. 10. (a) and (b) shows that increasing the dataset alone, is not sufficient to
improve the testing accuracy of the model.
In model 5, the loss function is changed from mean_square_error to
categorical_crossentropy. The mean squared error (MSE) or mean squared deviation

21
of an estimator measures the average of the squares of the errors while
categorical_crossentropy (CCE) is a softmax activation plus a Cross-Entropy loss. It
sets up a classification problem between classes more than 2 for every class in C.

Where,yi to desired value and yi to the actual value obtained

where, M: number of classes (Arson, Burglary, Shooting, etc), log: the natural log, s:
is the CNN score for each positive class, 1/M: scaling factor to make the loss invariant
and p: predicted probability observation o is of class c

Fig. 11. (a) Accuracy of Model 5; (b) Loss of Model 5

Changing the error function has boosted the testing accuracy to a considerable amount
and loss function is also converging faster as shown in Fig. 11. (a) and (b) resp.

Fig. 12. (a) Accuracy of Model 6; (b) Loss of Model 6

Thus, all the changes integrated so far have a positive impact on the overall
performance of the model.

22
Finally, the entire dataset is augmented by vertically flipping the dataset. Model 6 is
trained on a dataset, double in size of the dataset used in model 5. This is done to
optimise the validation accuracy of the model. Fig. 12. (a) shows that the training
accuracy of the model increases logarithmically in accordance with the training curve.
Fig. 12. (b) plots the loss function of the trained model.
Table 1. Performance Comparison of all models

Model 6 has comparatively the best overall performance amongst all the six models
that have been implemented during the course of this work as shown in Table 1. Table
1 present the results obtained by six models that have been explained so far.
Furthermore, we have tested model 6 on the self-collected dataset to examine its
application in realtime scenarios.

Chapter 4: Conclusion

23
This work suggests an approach to spot variation from the norm in real-world CCTV
recordings. The normal data alone may not be effective to distinguish abnormalities in
these recordings. Therefore, to handle the complexity of these realistic anomalies,
both normal and anomalous videos have been considered and hence, maximised the
accuracy of the model. Furthermore, to prevent the efforts-requiring temporal
annotations of abnormal sections in training recordings, a general model of anomaly
detection has been learned utilizing two distinct neural networks with a poorly
labelled dataset. A rarely processed large-scale anomaly dataset consisting of 12 real-
world anomalies has been utilized for learning with the aim of validating the
suggested approach. The experimental results obtained during the work conclude that
our suggested anomaly detection approach performs significantly better than the
previously used methods.
Table 2. Details about the Optimized Model.

As specified in Table 2., this Threat Recognition Model classifies the anomalies into
thirteen categories: Abuse, Burglar, Explosion, Shooting, Fighting, Shoplifting, Road
Accidents, Arson, Robbery, Stealing, Assault, Vandalism, and Normal. The model
concatenates 8 frames to form a chunk. The optimizer used in this model is Adam and
the error function is categorical_crossentropy. The model uses three different types of
activation function; Relu, Sigmoid, and Softmax. Moreover, to further increase the
testing accuracy of the model the dataset has been doubled by flipping the videos
horizontally. Hence, the overall accuracy of the model is 97.23% with reduced
overfitting. Eventually, to implement this model in real-time, it is necessary to
consider all the hardware constraints explained in this work. Hence, a proper
implementation plan will reduce the computation power, optimise the use of resources
and eventually reduce the overall cost of the system.

24
 REFRENCES

25
[1] J. Kooij, M. Liem, J. Krijnders, T. Andringa, and D. Gavrila. Multi-modal human
aggression detection. Computer Vision and Image Understanding, 2016.
[2] S. Mohammadi, A. Perina, H. Kiani, and M. Vittorio. Angry crowds: Detecting
violent events in videos. In ECCV, 2016.
[3] Convolutional Neural Network (CNN) in Keras,
https://towardsdatascience.com/building-a-convolutional-neural-network-cnn-in-
keras329fbbadc5f5
[4] Recurrent Neural Networks (RNN) in Keras,
https://towardsdatascience.com/understanding-lstm-and-its-quick-implementation-in-
keras-forsentiment-analysis-af410fd85b47
[5] W. Li, V. Mahadevan, and N. Vasconcelos. Anomaly detection and localization in
crowded scenes. TPAMI, 2014.
[6] X. Cui, Q. Liu, M. Gao, and D. N. Metaxas. Abnormal detection using interaction
energy potentials. In CVPR, 2011.
[7] T. Hospedales, S. Gong, and T. Xiang. A Markov clustering topic model for
mining behaviour in the video. In ICCV, 2009.
[8] Y. Zhu, I. M. Nayak, and A. K. Roy-Chowdhury. Context-aware activity
recognition and anomaly detection in video. In IEEE Journal of Selected Topics in
Signal Processing, 2013.
[9] L. Kratz and K. Nishino. Anomaly detection in extremely crowded scenes using
Spatio-temporal motion pattern models. In CVPR, 2009.
[10] How to Automate Surveillance Easily with Deep Learning,
https://medium.com/nanonets/how-to-automate-surveillance-easily-with-
deeplearning-4eb4fa0cd68d, 2018.
[11] C. Lu, J. Shi, and J. Jia. Abnormal event detection at 150 fps in Matlab. In ICCV,
2013.
[12] IR. Mehran, A. Oyama, and M. Shah. Abnormal crowd behaviour detection using
the social force model. In CVPR, 2009.
[13] I. Saleemi, K. Shafique, and M. Shah. Probabilistic modelling of scene dynamics
for applications in visual surveillance. TPAMI, 31(8):1472– 1485, 2009.
[14] B. Zhao, L. Fei-Fei, and E. P. Xing. Online detection of unusual events in videos
via dynamic sparse coding. In CVPR, 2011.
[15] Unusual crowd activity dataset of the University of Minnesota. In
http://mha.cs.umn.edu/movies/crowdactivity-all.avi.

26
[16] A. Adam, E. Rivlin, I. Shimshoni, and D. Reinitz. Robust real-time unusual event
detection using multiple fixed-location monitors. TPAMI, 2008.
[17] Boss dataset, http://www.multitel.be/image/researchdevelopment/research-
projects/boss.php.
[18] Waqas Sultani, Chen Chen, Mubarak Shah. Real-world anomaly detection in
surveillance videos. In IEEE/CVF Conference, 2018.
[19] Data Augmentation, https://medium.com/nanonets/how-to-use-deep-learning-
when-you-have-limited-data-part-2-data-augmentationc26971dc8ced
[20] Transfer learning from pre-trained models,
https://towardsdatascience.com/transfer-learning-from-pre-trained-models-
f2393f124751, 2018.

27