10 IX September 2022

https://doi.org/10.22214/ijraset.2022.46803
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue IX Sep 2022- Available at www.ijraset.com

Handwritten Mathematical Expression Solver

using CNN
Abhiruchi Patil1, Teena Varma2
1 2
Undergraduate Student, Assistant Professor, Department of Computer Engineering, Xavier Institute of Engineering, Mumbai,
India

Abstract: Mathematics is the universal language. It is the core of all the scientific discoveries and is used extensively in all
possible fields. Hence, there arises the need for fast computation and instant results. This has led to the development of various
technologies and machines which would ease the strain on humans and also reduce the error and delays caused by them. Our
project caters to these needs by reducing the effort on user side. We want to make the experience of experimenting with
equations as simple as possible. The application is capable of performing fast calculations on human handwritten mathematical
expressions. Basic calculations like arithmetic and trigonometric functions can be done along with simultaneous equations. The
expression is extracted from the image and each character, symbol and number is recognised and then the entire expression is
further being solved displaying all possible solutions. The application developed is simple and user-friendly tool that can
leverage the power of existing powerful math packages.
Keywords: Handwritten Mathematical Expression, Character recognition, Convolutional Neural Network (CNN), Symbol
recognition, Image processing.
I. INTRODUCTION
Discovery, that is the word we associate with finding something new. The reason for this is that this universe already has everything
it requires, it's just in a form that mankind has yet to comprehend or discover. Mathematics is the simplest way to understand the
universe's transformations and intricacies. The language of mathematics which can describe the nature of the universe to the shapes
and structure and pattern of almost all objects that we perceive through our senses. Take any aspect and you will find a
mathematical description for it. For years mankind has tried to decipher this mathematical code by numerous methods and formulae
which were worked upon for years and years. And today we have reduced the time gap to get answers for tedious methods and
procedures with the help of new technologies and gadgets. The most general example would be a calculator which is used instead of
manually working out mathematical methods thus reducing the percentage of human error associated with it. But using the
calculator to solve the humongous equations and find answers for them is a tedious task since it requires the equations to be entered
accurately. Therefore, another need for innovation came to life and thus technology which could scan documents and interpret what
is written and further find the answer for it to ease the daily humdrum. This is where machine learning and neural networks come
into the picture, where a model is built in such a way that it can think and solve the problems in the same way a human brain would
do but swiftly and efficiently. Mathematical equations form an integral part of most research work, so researchers use such
mathematical tools to save time and boost efficiency keeping in mind the complexities and syntax of the chosen tool. If the syntax
rules are not followed properly, one might not get the desired output. Another concern is the task of entering all the equations and
expressions properly leaving no gap for human error since the system will be performing the calculations based on the input given.
This requires a lot of time and human intervention and also good knowledge of the tool being used. A convenient solution to this
can be to build a user-friendly tool that would capture the image of a mathematical equation, recognise the equation embedded in it
and present the user with the required solution. This is precisely the tool we have developed and discussed throughout this paper.
Computers can be made to think and act like a human would, by forming a neural network and also retaining their computational
superiority. In our project, we have attempted to simplify the interaction between humans and computers pertaining to the
processing and solving mathematical equations. We have focused on building a model which is simple to understand, by not
restricting the user with various syntax rules and complexities. In today’s world there are many tools, simulators which are
available for mathematical purposes. However, more the features, more the awareness about the particular tool is required from the
user. To shorten this gap of excessive awareness on the user’s side, the machine could be trained to understand the user’s needs
more accurately with very less effort from the concerned user. This being the ideology we have tried to build a machine learning
model which will take the image input of the mathematical expression written by the user and identify the input expression and
provide the answer with accuracy achieved while building the model.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 1205
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue IX Sep 2022- Available at www.ijraset.com

II. LITERATURE SURVEY

The Internet is the key to connecting with the world and helping us learn. It is one of the greatest pillars to educate many in some
way or the other. But, in the field of mathematics, considering the use of Greek letters and special symbols in mathematical
equations, it is difficult to give these as an input to the internet. This issue has been addressed as we have Latex to represent these
equations, however using Latex requires some training and intensive practice. On the other-hand, some development is done in
recognition of these expressions using machine learning. Mathematical expression recognition typically consists of two major stages:
symbol recognition and structural analysis. Both symbol recognition and structure analysis of two-dimensional patterns have been
extensively studied for decades. The recognition of characters requires segmentation. Segmentation has been studied by many and
includes various approaches like use of modular systems [3], X-Y cutting [4], recursive X-Y cut[5]. Symbol recognition also uses
some approaches like template matching [4][6][7], structural approaches by Haton[8] and Chan and Yeung[9] and statistical
approaches by Chen and Yin [10], Fateman and Tokuyasu [11], Ha [12] using neural networks. Offline and online are the two ways
by which mathematical expression recognition can be done. Here we have used the offline way in which the user input is a complete
expression of the problem to be solved. Offline recognition has been attempted by building relation trees [2][4], expression trees
built through top-down and bottom-up approaches [5], use of nearest neighbours [15][16], application of convex hull [17].
Recent works done in the field includes the works of Zanibbi [18] which recognizes the math expressions online, sketched on the
computer. Then an Offline Handwritten Mathematical Recognition [19] which also uses CNN and gives detail information on
segmenting the expressions effectively to detect the horizontal link, upper link, superscript link, sub-expression, subscript link and
lower link. The recognition accuracy of neural network-based handwritten recognition systems can be improved by changing the
features considered in the classification and segmentation stages. Edge detection was used in the segmentation stage, followed by
morphological operations, and special features such as skew, standard deviation, mean, variance, and so on were used instead of
common features such as thickness, thinness, and area [20].

III. PROPOSED MODEL

The proposed model consists of two phases mainly, first the handwritten expression recognition and second is expression evaluation
phase. The input image provided by the user is pre-processed and segmented to recognize and predict each individual character,
number and symbols [1]. After recognition the mathematical expression is formed which is further processed using python libraries
to provide the calculations. The Block Diagram (Fig. 1) represents the procedure in which entire processing is carried out. The Use
Case Diagram (Fig. 2) shows all the use case scenarios that a user encounters while using the application.

Image Number &

Input Image Symbols Expression Expression
Pre- Output
Image Segmenta- Classification Formation Evaluation
Processing
tion

Fig. 1 Block Diagram of Proposed System

Fig. 2 Use Case Diagram of Proposed System

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 1206
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue IX Sep 2022- Available at www.ijraset.com

IV. IMPLEMENTATION
The model created is a sequential model. The project implementation is divided into two phases, Handwritten Expression
Recognition Phase and Expression Evaluation Phase.

A. Dataset
The dataset was used from Kaggle and it consisted images of handwritten digits from 0 to 9 along with various math symbols and
alphabets. It consists of symbols from the Greek alphabet, English alphanumeric characters, math and set operators, basic predefined
math functions (such as sin, cos etc.) and math symbols (such as sum, sqrt, delta etc.). Every specific element has more than 100
images for the model to train on and predict. Dataset is fairly diverse since it contains handwritten samples of different individuals
which helps the CNN model to train on various parameters. Samples without any variation won’t be helpful since they usually lead
to overfitting. In this project we have used limited number of symbols and alphabets rather than considering all the 26 alphabets for
computation purposes. The dataset could be expanded even further to perform even more complex calculations.

B. Pre-processing
1) Binarization: The conversion of an input image into a bi-level target image is binarization. The term bi indicates two which
means that image pixels are divided into two groups, black and white pixels. The primary goal of image binarization is to sort
an image into foreground and background text. The given input image may consist of noise or other unwanted information
which can affect the processing of the expression. Hence, the conversion of the input image into binarized image is necessary.

Fig. 3 Original Input Image

Fig. 4 Binarized Image

2) Contour based Segmentation: Contours are defined as the line connecting all the points along an image's borders that have the
same intensity. Contours are useful for shape analysis, determining the size of an object of interest, and object detection. The
findContours() function in OpenCV aids in the extraction of contours from images. It is most effective with binary images.
These contours segregate the handwritten characters, symbols and numbers based on their intensities in binarized image which
are than further used for recognition.

Fig. 5 Contours Detected from Input Image

C. Building Model
The model is built using Keras. Necessary modules and layers from keras are imported which would form the model. The to-
categorial function is imported for forming the different classes for the classification of symbols and numbers. As many varieties of
symbols, characters and numbers are involved, many classes are formed. The model created is a Sequential model. The Convolution
and MaxPooling layer are applied one after another. The input layer is encompassed in the Convolution layer. This layer applies the
convolution operation to the input layer and then passes the result to the next layer. The work of this layer is to change all the pixels
in its responsive field into a single value. By this a vector is obtained which is given to the MaxPooling layer.

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 1207
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue IX Sep 2022- Available at www.ijraset.com

The MaxPooling layer is used to reduce the dimensionality of the vector by replacing each cell in the array with the max value of
the sub matrix part of the required size around the cell in consideration. Dropout layer is included to avoid overfitting of the model.
It randomly sets the outgoing edges of neurons that make up hidden layers to 0 at each update. While the data is passing and
switching from one format to another, the weight ordering may change. This may affect the output and hence the Flatten layer is
used. It adds an extra channel to the batch size. It doesn’t much affect the output but is used as a form of prevention. The Dense
layer is the main part of the model where the neural networks are formed and which result in the correct prediction. Three layers of
Dense are applied to the model for better accuracy with activation=’ReLU’. The model works like a brain to analyse and classify
from the data given in training and based on that, predict into which class the given component can be classified.

The architecture used for our CNN model is described as follows:

Layer 1: Convolutional Layer
• Input Shape = (28, 28, 1)
• Filter size = (5, 5)
• ReLU activation function

Layer 2: Pooling Layer

• MaxPooling
• Pool size = (2, 2)

Layer 3: Convolutional Layer

• Filter size = (3, 3)
• ReLU activation function

Layer 4: Pooling Layer

• MaxPooling
• Pool size = (2, 2)

Layer 5: Dropout Layer

• Drop probability = 0.2

Layer 6: Flatten Layer

Layer 7: Fully Connected Layer

• 206 output neurons
• ReLU activation function

Layer 8: Fully Connected Layer

• 128 output neurons
• ReLU activation function

Layer 9: Fully Connected Layer

• 50 output neurons
• ReLU activation function

Layer 10: Fully Connected Layer

• 22 output neurons
• Softmax Activation function

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 1208
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue IX Sep 2022- Available at www.ijraset.com

D. Expression Evaluation
After the recognition of the characters and symbols, the expression is passed to the function to solve and display the solution. For all
the mathematical processing python library called SymPy is used [21]. SymPy depends on mpath which is a python library for real
and complex floating-point arithmetic with arbitrary precision. The final answer after calculation is displayed on the GUI based
page.

V. RESULTS

Fig. 6 Home Page of the Application

Fig. 7 Simultaneous Equations Solution Page

Fig. 8 Trigonometric Functions Solution Page

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 1209
 International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 10 Issue IX Sep 2022- Available at www.ijraset.com

VI. CONCLUSIONS AND FUTURE SCOPE

This project focuses on developing a machine learning approach to solve math problems along with character and symbol
recognition. The motivation was to create a user friendly application which will be simple and easy to use, without feeling the need
to study the usage of the tool or complex syntaxes. It reduces the task on user side as direct images can be given to the application
and then get the equivalent solution for it. The discussed web application can solve all the basic arithmetic calculations,
trigonometric calculations and simultaneous equations in two variables.
With more rigorous training and extension of dataset the capability and accuracy of the model in general can be increased. It can
further be extended to solve more complex calculations like differential equations and integrations. Also, the product tool can be
made more efficient where it would recognize and solve equations written in groups and reduce the task from the user’s side to input
individual equation images.

VII. ACKNOWLEDGMENT
The authors would like to express their gratitude to Kaggle for making the handwritten math symbols’ dataset available for public
use. We also sincerely thank the skilful developers of Keras, Tensorflow and Python for their ceaseless efforts in maintaining the
libraries.
REFERENCES
[1] Xue-Dong Tian, Hai-Yan Li, Xin-Fu Li, & Li-Ping Zhang. (n.d.). Research on Symbol Recognition for Mathematical Expressions. First International
Conference on Innovative Computing, Information and Control - Volume I (ICICIC’06). doi:10.1109/icicic.2006.506
[2] Okamoto M. Recognition of mathematical expressions by using the layout structure of symbols. InProc. 1st Int. Conf. Document Analysis and Recognition,
1991 1991 (pp. 242-250).
[3] Faure and Z. Wang, “Automatic perception of the structure of handwritten mathematical expressions,” Computer Processing of Handwriting, pp. 337 –361,
1990Faure C, Wang ZX. Automatic perception of the structure of handwritten mathematical expressions. In- Computer processing of handwriting 1990 (pp.
337-361).
[4] M. Okamoto and A. Miyazawa. An experimental implementation of a document recognition system for papers containing mathematical expressions. In H. S.
Baird, H. Bunke, and K. Yamamoto, editors, Structured Document Image Analysis, pages 3653. Springer- Verlag,Berlin, 1992.
[5] Ha J, Haralick RM, Phillips IT. Understanding mathematical expressions from document images. InProceedings of 3rd International Conference on Document
Analysis and Recognition 1995 Aug 14 (Vol. 2, pp. 956-959). IEEE.
[6] Chou PA. Recognition of equations using a two-dimensional stochastic context-free gram- mar. InVisual Communications and Image Processing IV 1989
Nov 1 (Vol. 1199, pp. 852- 865). International Society for Optics and Photonics.
[7] Yasutomo Nakayama. 1989. Mathematical formula editor for CAI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
(CHI ’89). Association for Computing Machinery, New York, NY, USA, 387–392. DOI:https://doi.org/10.1145/67449.67523
[8] A. Belaid and J. -P. Haton, ”A Syntactic Approach for Handwritten Mathematical Formula Recognition,” in IEEE Transactions on Pattern Analysis and
Machine Intelligence, vol. PAMI-6, no. 1, pp. 105-111, Jan. 1984, doi: 10.1109/TPAMI.1984.4767483.
[9] Chan KF, Yeung DY. Recognizing on-line handwritten alphanumeric characters through flexible structural matching. Pattern recognition. 1999 Jul
1;32(7):1099-114.
[10] L. H. Chen and P. Y. Yin. A system for on-line recognition of handwritten mathematical expressions. Computer Processing of Chinese and Oriental
Languages, 6(1):1939, June 1992.
[11] Fateman RJ, Tokuyasu T, Berman BP, Mitchell N. Optical character recognition and parsing of typeset mathematics1. Journal of Visual Communication and
Image Representation. 1996 Mar 1;7(1):2-15.
[12] Ha J, Haralick RM, Phillips IT. Understanding mathematical expressions from document images. InProceedings of 3rd International Conference on Document
Analysis and Recognition 1995 Aug 14 (Vol. 2, pp. 956-959). IEEE.
[13] Faure C, Wang ZX. Automatic perception of the structure of handwritten mathematical expressions. In Computer processing of handwriting 1990 (pp. 337-
361).
[14] Pfeiffer JJ. Parsing graphs representing two dimensional figures. InProceedings IEEE Workshop on Visual Languages 1992 Sep 15 (pp. 200-206). IEEE.
[15] Lee HJ, Wang JS. Design of a mathematical expression recognition system. InProceedings of 3rd International Conference on Document analysis and
Recognition 1995 Aug 14 (Vol. 2, pp. 1084-1087). IEEE
[16] Chan KF, Yeung DY. Mathematical expression recognition: a survey. International Journal on Document Analysis and Recognition. 2000 Aug 1;3(1):3-15.
[17] Miller EG, Viola PA. Ambiguity and constraint in mathematical expression recognition. InAAAI/IAAI 1998 Jul 1 (pp. 784-791).
[18] Zanibbi R, Blostein D. Recognition and retrieval of mathematical expressions. International Journal on Document Analysis and Recognition (IJDAR). 2012
Dec 1;15(4):331-57.
[19] L. D’souza and M. Mascarenhas, ”Offline Handwritten Mathematical Expression Recognition using Convolutional Neural Network,” 2018 International
Conference on Information , Communication, Engineering and Technology (ICICET), 2018, pp. 1-3, doi: 10.1109/ICI- CET.2018.8533789.
[20] Sagar Shinde, Dr. R. B. Waghulade, Dr. D. S. Bormane,“A new neural network based algorithm for identifying handwritten mathematical equations ”,
International Conference on Trends in Electronics and Informatics, 2017.
[21] SymPy Python Library https://www.sympy.org/en/index.html

©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 1210