Handwritten Text Recognition Using Deep Learning

Techniques
L.Navya1 MD.Farhan Ali2 K.Pavan Sai3
Computer Science and Engineering Computer Science and Engineering Computer Science and Engineering
B V Raju Institute of Technology B V Raju Institute of Technology B V Raju Institute of Technology
Narsapur,Medak(D),Telangana(S)-502313 Narsapur,Medak(D),Telangana(S)-502313 Narsapur,Medak(D),Telangana(S)-502313
21211a05e5@bvrit.ac.in 21211a05g9@bvrit.ac.in 21211a05e0@bvrit.ac.in

K.Shyam4 Alabazar Ramesh5

Computer Science and Engineering Computer Science and Engineering
B V Raju Institute of Technology B V Raju Institute of Technology
Narsapur,Medak(D),Telangana(S)-502313 Narsapur,Medak(D),Telangana(S)-502313
21211a05e1@bvrit.ac.in ramesh.a@bvrit.ac.in

Abstract—”HANDWRITING RECOGNITION” has consuming and costly. An excellent example is the automated
generated a lot of attention in the realms of pattern recognition processing of bank checks, which eliminates the need for a
and machine learning due to its applicability in a variety of large number of employees that may not be as efficient as
disciplines. Both optical character recognition (OCR) and hand-
written character recognition (HCR) have a defined operation computerized systems. OCR technology plays a crucial role
sphere. In the near future, character recognition technology may in digitizing handwritten medieval calligraphy and converting
be essential for exercising and digitising paper documents in typewritten documents into digital form. This simplifies the
order to create a paperless environment. This action categorises retrieval of information, as one no longer needs to search
each handwritten word in order to convert handwritten notes through piles of documents. Organizations are meeting the cri-
into digital form. Input text or photos are directly identified
by handwriting; many automated process systems also look at teria for digitally preserving important data, legal documents,
these materials. Numerous sorts of handwritten characters, such and ensuring educational continuity, among other things.
as integers, figures, cursive notation, symbols, and scripts in
various languages have emerged as a result of the development The development and categorization of features
of handwriting. In multitudinous situations where it’s necessary based on patterns are essential components of an OCR system.
to exercise large amounts of handwritten data, analogous as Handwritten OCR has gained popularity as a branch of OCR.
the identification of addresses and postcodes on envelopes, the
interpretation of amounts on bank checks, document analysis, It can further be divided into offline and online systems,
and hand verification, automatic recognition of handwritten text depending on the input data. Online systems involve dynamic
can be truly helpful. therefore, a computer is demanded in order input based on the movement of a pen tip, including speed,
for document processing to be as simple as possible. angle, position, and locus point. On the other hand, offline
systems use static input in the form of scanned photographs.
Index Terms—–Optical Character Recognition (OCR),
Handwritten Character Recognition (HCR),Convolutional Neu- Online systems are considered more sophisticated and
ral Networks(CNN),Recurrent Neural Networks(RNN), Feauture complex as they overcome the issue of overlapping data in
Extraction. offline systems.

I. I NTRODUCTION
The development of the first handwritten recognition II. L ITERATURE S URVEY
system took place in the 1950s with the aim of emulating hu- In 1959, Grimsdale made a significant early attempt
man reading abilities, enabling computers to edit and read doc- in the field of character recognition research. In the early
uments similar to humans. It has greatly improved the human- 1960s, a lot of research activity had an approach as its
machine interface and accelerated the process of automation in foundation. Described as the analysis-by-synthesis approach,
various industries. However, in today’s digital environment, the it was Eden’s idea in 1968. Later, all syntactic (structural)
importance of character recognition and handwriting analysis approaches to character recognition adopted this idea.
has diminished. This is evident from the increasing number
of recognition systems being created and implemented in A machine learning-based method for word extrac-
industries that require high efficiency. Handwriting recognition tion and detection from photos using CNNs was proposed by
systems that identify letters, characters, and numbers allow the authors of [1]. Text region localization and text recognition
users to simplify complex tasks that would otherwise be time- are the two key steps of the method. The usefulness of
the suggested approach is shown by experimental results to categorise and identify various Handwriting letters and
on common benchmark datasets for precisely extracting and digits using the training data that was saved in the system’s
recognising text from images. The suggested method has the database. The process of handwriting recognition involved
ability to automate the process of text extraction and detection several stages.The final system successfully met the prescribed
from images in a number of applications, including document standards in terms of recognition and accuracy. Additionally,
analysis, image understanding, and information retrieval. It the current research findings can be extended to include
is possible to do more study to increase the precision and character recognition in different languages. Leveraging the
effectiveness of our suggested approach and see whether it machine learning models developed in this research, various
can be used in real-world situations. types of handwritten materials such as books, newspapers, and
notes can be converted into digital text format.
The study’s authors [2] suggested an OCR system
that can read handwritten notes on handouts and determine The primary topic of the study [5] is the identifica-
their contents. They reasoned that utilising a single type of tion of Devanagari scripts. It presents a challenge because of
OCR technology wouldn’t produce accurate results because the language’s complexity and the variety of modifiers utilised.
the targets’ handouts contain both Japanese and mathematical Deep learning techniques have been used to categorise basic
calculations. They will therefore include two separate OCR Devanagari characters. As opposed to the conventional method
technologies in their suggested solution as a result. The best of splitting a word into many modifiers, the proposed system
OCRs to combine are Tesseract and Mathpix, which are good segments words character-by-character. For the objective of
at understanding mathematical formulas and Japanese writing, obtaining high-level features for recognition, various CNN
respectively. There is no guarantee that the real recognition approaches were employed.
results will follow the same pattern as the OCR score.
CNN and its architecture for OCR are explored in
The authors of [3] suggest a technique for auto- length in the paper of [6]. CNN is a deep learning method
matically cropping handwritten letters from the image. Six that is employed for character recognition. The architectures
testing images are picked from 24 scanned documents in of the various CNN algorithms that are utilised for categorising
the handwritten dataset in the Punjabi language. 29 letters handwritten character recognition are surveyed. The suggested
are used in the test and 267 letters are used in the training. technique uses Telugu character data to discuss and train a
The effectiveness of the method demonstrates an increase in VGG-16 architecture for handwriting recognition. The out-
detection accuracy as well as the facilitation of the detection comes show how accurately datasets used for training and
process, which results in a decrease in time and expenses. testing performed.
The proposed strategy achieves 98.77% accuracy, which is
superior to state-of-the-art techniques. The process of cropping In comparison to the prior approach, the suggested
a photograph involves choosing a portion of the image and system from the paper [7] achieves relatively good identifi-
further cutting it apart from the rest of the image. Typically, cation rates for both the evaluation of text line elements and
this modification of handwritten documents is accomplished that of a collection of horizontal, vertical, and skewed lines.
by successfully completing a three-step process Step 1: Using The suggested system includes a handwritten text recognition
segmentation algorithms and annotated XML files, the scanned neural network with 5 CNN, 2 RNN, and character probability
input page is divided into the handwritten word images. matrix outputs. Finally, this detection and implementation
Step 2:photos are trained using the YOLOv3 object detection via Tensor flow offers suggestions for improving accuracy.
method. Step 3: After that, the bounding boxes and cropping Currently, FASTAG deployment in India and digital book
of the handwritten letters are produced. translation from one language to another are the key appli-
cation areas for this. The subsequent work entails enhancing
This study’s primary goal [4] was to create a system accuracy by lowering various aspects of handwriting, such as
for classifying and identifying handwriting characters and normalising contrast, normalising skew, normalising slant, and
numerals. In today’s digital age, it is crucial to be able to normalising size of the text or character in handwriting, as
recognise characters and numbers, particularly in organisations well as additional techniques for enhancing neural network
that work with handwritten papers that require computer accuracy, such as data augmentation.
systems to analyse. Organisations and individuals can perform
difficult tasks with the aid of handwriting categorization and In the study referenced as [8], a solution was pre-
recognition systems. The existing technology processed and sented for the problem of processing online handwritten docu-
read handwritten characters and digits using neural networks. ments in a free-form manner. The objective was to categorize
Convolution neural networks (CNN) were advantageous for multi-class handwritten documents into different types such as
the system thanks to training data that made it simple to texts, tables, formulas, and drawings. The primary focus of the
recognise characters and numbers. CNN made it possible study was stroke classification, as it plays a crucial role in the
for the OCR system to be more sensitive to various object automatic layout analysis of handwritten document recognition
aspects, just like the human visual system. This made it simple systems. The research introduces a novel hierarchical neural
network-based architecture specifically designed for stroke V. E XISTING W ORK
classification.
OCR (Optical Character Recognition) technology
has several disadvantages and limitations that can affect its
For online hand written diagram recognition, the au-
performance and accuracy. Some of the major disadvantages
thors of Paper [9] have suggested a novel and general multitask
of OCR include: Limited accuracy: OCR technology may
learning architecture based on GNNs. The proposed frame-
not be able to accurately recognize all types of fonts and
work may efficiently encode complicated spatial and temporal
characters, especially if they are distorted or have poor image
relationships for symbol segmentation and recognition, as
quality. Language limitations: OCR technology is limited
shown by experiments on two flowchart benchmark datasets
by the languages it is trained to recognize. It may not be
and one finite automata dataset. Our approach performs sig-
able to recognize characters from languages that it has not
nificantly better than several recently proposed approaches.
been trained on. Document layout: OCR may have difficulty
A large-scale, publicly accessible dataset of handwritten ten-
recognizing text if the document has complex layouts, such
diagrams called CASIA-OHFC has also been given, and pre-
as tables or multiple columns. Image quality: OCR accuracy
liminary tests have been run on it to serve as a benchmark. The
can be affected by poor image quality, such as low resolution,
suggested framework is suitable for general online handwritten
blurriness, or shadows.
diagram and freehand sketch recognition tasks and can be
extended and modified accordingly. The subsequent works to
VI. P ROPOSED W ORK
further enhance recognition performance.
To address these limitations, proposed work in OCR
The SVM classifier is used in this research [10] to technology includes:
develop a handwritten Kannada character recognition system.
Numerous OCR-based algorithms exist for identifying certain 1) Improving accuracy through deep learning: OCR can
individuals. The process of extracting the text involves several benefit from deep learning techniques such as CNNs
different processes and methods. As a result, the algorithm’s and RNNs to improve accuracy and recognize a wider
efficiency is degraded, and the number of features and compu- range of fonts and characters.
tation time grow. The SVM multiclass One-vs-One technique, 2) Multilingual OCR: OCR technology can be trained to
which is more suitable for Kannada character recognition, is recognize characters from multiple languages, making it
used by the algorithm and is based on two-fold characterisa- more versatile and useful for a wider range of applica-
tion, which produces accurate results. SVM classifiers are used tions.
in this work to measure character recognition by using pixel 3) Layout analysis: OCR can be combined with layout
values. Due to its advantage in nonlinear and high dimensional analysis algorithms to improve the recognition of text
pattern recognition as well as its superior approximation and in documents with complex layouts.
generalisation abilities, SVM performs well while working 4) Image enhancement: OCR accuracy can be improved by
with a small sample size. The following effort will encompass using image enhancement techniques such as deblurring,
the alphabets’ final 49 characters. denoising, and contrast adjustment to improve the qual-
ity of the input image.
III. P ROBLEM S TATEMENT
VII. M ETHODS
• Making Computer capable to read or recognize the human
written text and then convert its observation to digital A. Data Set Description
form of text. The data used to train the model is obtained from the
• The factors like low-resolution input, or when there are IAM dataset. A portion of the larger IAM dataset utilised only
variations in lightning conditions or poor paper quality for handwriting recognition tasks is the IAM off-line HTR
affect the robustness and reliability of the existing system. dataset. Over 9,000 text lines totalling over 9,000 words were
• Variability in handwriting across different individuals can written by 657 distinct authors. A transcription of the text is
also pose challenges for recognition systems, leading to included with each line of text and is kept in an ASCII file.
errors in recognition results. This effects the Accuracy of It is usual practise to train and test handwriting recognition
the given system. systems using this dataset. The following are some of the main
features and specifics of the IAM off-line HTR dataset:
IV. O BJECTIVES
1) The dataset’s around 1,539 pages of handwritten text
1) A model with Artificial Intelligence to capture the hand- samples make up its size.
written text. 2) There are texts in both German and English.
2) Filtering the text to its best form. 3) Annotations: To ensure the right text is used for training
3) Segmentation and applying Artificial Intelligence and evaluation purposes, each text sample in the dataset
(CNN). is accompanied by ground truth transcriptions.
 4) Data Format: Line images or word images of the dataset specified vocabulary or lexicon. Another strategy
are frequently included, along with plain text transcrip- is to predict potential word sequences based on
tions of the corresponding ground truth data. context by using language models, such as n-grams
5) Variability: The dataset includes a variety of writing or recurrent neural networks.
styles, variations in the quality of handwriting, and 4) Post-processing: To increase the precision and coherence
various document layouts. of the recognised text, the post-processing module takes
extra actions. It might entail activities like spelling
B. Modules and grammar checks, as well as language-specific post-
The proper transcription and interpretation of hand- processing regulations.
written text is a challenging process that requires the em- 5) Evaluation and Feedback: This module evaluates
ployment of many modules and approaches. Here are a few the reliability and standard of the recognised text
essential modules that handwritten note recognition software and offers suggestions for system enhancements. To
frequently uses: assess the recognition accuracy and pinpoint areas
for improvement, it might entail human validation or
1) Preprocessing: The handwritten notes are first prepared comparison with real-world data.
for subsequent analysis by the preprocessing module.
Image improvement, noise reduction, binarization (turn-
ing an image black and white), de-skewing (adjusting
the notes’ orientation), and segmentation (detection and
separation of individual lines or phrases) are frequently
involved.
2) Feature Extraction: The feature extraction module seeks
to extract pertinent data from the handwritten notes
that have been previously analysed. It takes out several Fig. 2. Flow Diagram
aspects, such as stroke shape, direction, curvature, or
texture, that can indicate the visual qualities of the hand- Together, these components enable the recognition
writing. The HOG, SIFT, and CNNs for deep learning- of handwritten notes, transforming them into text that can
based feature extraction are popular feature extraction be read by computers and searched. The effectiveness of
methods. each module has a significant impact on the system’s overall
accuracy and dependability in recognising handwritten notes.
Within each module, researchers and practitioners are always
investigating and developing new methods and strategies to
boost the efficiency of handwritten text recognition.
VIII. A RCHITECTURE D IAGRAM

Fig. 1. Enhanced image

3) Handwriting Recognition: The module for handwriting

recognition is in charge of turning the features that were
extracted into text. Character and word recognition are
the two sub-modules that make up this module.
a) Character Recognition: Using the extracted fea-
tures, this sub-module focuses on identifying spe- Fig. 3. Architecture Diagram
cific characters. It often requires the use of machine
learning techniques that are trained on a labelled IX. R ESULT AND D ISCUSSIONS
dataset of handwritten characters, such as SVMs, The input images,and the outputs that are obtained
HMMs, or RNNs. when the model is trained in IAM dataset are follows:
b) Word Recognition: The word recognition sub-
module assembles the recognised characters into
full words after they have been identified as char- A Handwritten Text Recognition (HTR) system based on Ten-
acters. This can be accomplished using methods sorFlow (TF) has been created and trained using the IAM off-
like lexicon-based recognition, in which recognised line HTR dataset. This system is capable of processing images
character sequences are matched to words using a containing single words or text lines (comprising multiple
 R EFERENCES
[1] Shivani Surana, Komal Pathak, Mehul Gagnani , Vidhan Shrivastava,
Mahesh T R, Sindhu Madhuri G, ”Extraction and Detection from
Images using Machine Learning Techniques”, International Conference
on Electronics and Renewable Systems(ICEARS)-IEEE,2022.
Fig. 4. Input Image
[2] Yasuhiro Kobayashi, Shunya Mimuro, Shin-nosuke Suzuki, Yousuke
Iijima, Akira Okada, “Basic research on a handwritten note image
recognition system that combines two OCRs,25th International Confer-
ence on Knowledge -Based and Intelligent Information &Engineering
Systems,2021-ELSEVIER.

Fig. 5. Ouput Text [3] Aanchal, Nidhi, Preeti, Gurupratap, ”Automatic Cropping of Handwrit-
ten Scanned Documents with Object Detection Algorithm”, Interna-
tional Conference on Machine Learning and Data Engineering,2023-
TABLE I ELSEVIER.
P ERFORMANCE A NALYSIS OF HTR S YSTEM ON VARIOUS T RAINING
M ETHODS AND DATASETS [4] Sara Aqab, Muhammad Usman Tariq, ”Handwriting Recognition using
Artificial Intelligence Neural Network and Image Processing,(IJACSA)
Dataset Method Accuracy International Journal of Advanced Computer Science and Applica-
tions,vol.11,No.7,2020
MMO and UCSD CNN, BLSTM, LSTM 95%
IAM, RIMES CNN, LSTM 86.37-90.5% [5] Yash Gurav, Priyanka Bhagat, Rajeshri Jadhav, Swathi Sinha, “De-
Chars74k FLM 92-95% vanagiri Handwritten Character Recognition using Convolution Neural
Networks”, Proc.of the 2nd International Conference on Electrical,
Self-built Shui character CNN 93.3% Communication, and Computer Engineering(ICECCE),12-13 June 2020.
IAM CNN, RNN, CTC 96%
[6] Naragudem Sarika, Nageswararao Sirisala, Muni Sekhar Velpuru,”CNN
based Optical Character Recognition and Applications , Proceedings of
the Sixth International Conference on Inventive Computation Technolo-
words) and providing recognized text as output. However, it’s gies [ICICT 2021] IEEE Xplore Part Number: CFP21F70-ART; ISBN:
worth noting that the model experiences character errors at 978-1-7281-8501-9
a rate of around 10%. Despite this, it achieves a reasonable [7] Nithin Gupta, Neha Goyal, “Machine Learning Tensor Flow Based Plat-
recognition rate, successfully recognizing approximately 75% form for Recognition of Handwritten Written Text”, 2021 International
of the words in the validation set. Conference on Computer Communication and Informatics (ICCCI -
2021), Jan. 27-29, 2021.

[8] Illya Degtyarenko, Ivan Deriuga, Andrii Grygoriev, Serhii Polot-

X. C ONCLUSIONS skyi, Volodymyr Melnyk, Dmytro Zakharchuk and Olga Radyvo-
nenko,”Hierarchical Recuurent Neural Network For Handwritten Strokes
Classification”, 2021,IEEE.
Using this system’s pre-processing and sequential
CNN and RNN training, an adaptive technique for offline [9] Xiao-Long Yun, Yan-Ming Zhang, Fei Yin, and Cheng-Lin Liu, ”In-
paragraph recognition is provided. the photos in the first stance GNN: A Learning Framework for Joint Symbol Segmentation
and Recognition in Online Handwritten Diagrams”,2021,IEEE.
paragraph are separated into line pictures using OpenCV
contour preprocessing techniques, and then line images are [10] Kusumika Krori Dutta, Sunny Arokia .Swamy, Anushua Banerjee, Divya
further processed into word images that are sent into the NN Rashi B, Chandan R, Deepak Vaprani,”Kannada Character Recognition
model layers during recognition. The RNN layers further Using Multi-Class SVM Method”,2021,IEEE.
process the output of the CNN layers. The CTC receives [11] W. Hussain, S. Anwar, and M. Majid, ”Handwritten Character Recogni-
the output of the RNN layers and decodes the output text. tion with Convolutional Neural Networks: A Deep Learning Approach,”
The outcomes show the potential of using CNN and RNN in in IEEE Access, 2019.
tandem to continuously improve accuracy. [12] N. Kumar, M. R. Rajagopalan, and R. J. Ramteke, ”Handwritten Char-
acter Recognition using Convolutional Neural Networks and Transfer
Learning,” in Procedia Computer Science, 2018.

[13] K. Simonyan, A. Zisserman Very Deep Convolu- tional Networks for

Large-Scale Image Recognition arXiv technical report, 2014

[14] Alex Graves, Santiago Fernndez, Faustino Gomez, Jrgen Schmidhuber,

Fig. 6. Example Connectionist temporal classification: labelling unsegmented sequence
data with recurrent neural networks, Proceedings of the 23rd interna-
tional conference on Machine learning. 2006

[15] P. Bhattacharya, S. K. Parui, and S. K. Bandyopadhyay, ”Deep Learning

Techniques for Handwritten Character Recognition: A Comprehensive
Review,” in International Journal of Image, Graphics and Signal Pro-
cessing, 2018.
Fig. 7. Output Text