Handwritten math expression parser
Seshat is an open-source system for recognizing handwritten
mathematical expressions. Given a sample represented as a sequence of
strokes, the parser is able to convert it to LaTeX or other formats
like InkML or MathML. You can see an example of application of this
parser in
where seshat is the underlying engine.
This parser has been developed by Francisco Álvaro as part of his PhD
thesis while he was a member of the [PRHLT research center] 1 at
[Universitat Politècnica de València] 2.
Seshat represents a state-of-the-art system that has participated in
several [international competitions] 3, and it was awarded the best
system trained on the competition dataset in:
The math expression recognition model that seshat implements is described in:
and it is the main part of my PhD thesis.
Seshat is released under the [GNU General Public License version 3.0 (GPLv3)] 5
Seshat is written in C++ and should work on any platform, although
it has only been tested in Linux.
This software integrates the open-source [RNNLIB library] 4
for symbol classification. The code of RNNLIB has been slightly
modified and directly integrated in seshat, thus, it is not
necessary to download it. However, it requires the [Boost C++
Libraries] 6 (headers only).
Finally, the parser accepts input files in two formats: InkML and
SCGINK. There is a example of each format in folder
“SampleMathExps”. Seshat uses the [Xerces-c library] 7 for parsing
InkML in C++.
Seshat is written in C++ and it only requires Makefile and g++ to
compile it. Once the required tools and libraries are available, you
can proceed with the installation of seshat as follows:
Obtain the package using git:
$ git clone https://github.com/falvaro/seshat.git
Or [download it as a zip file] 8
Go to the directory containing the source code.
If the include files of boost libraries are not in the path, add
it to the FLAGS variable in the file Makefile (“-I/path/to/boost/”).
Compile seshat
$ make
As a result, you will have the executable file “seshat” ready to
recognize handwritten math expressions.
Run seshat without arguments and it will display the command-line interface:
$ Usage: ./seshat -c config -i input [-o output] [-r render.pgm]
-c config: set the configuration file
-i input: set the input math expression file
-o output: save recognized expression to 'output' file (InkML format)
-r render: save in 'render' the image representing the input expression (PGM format)
-d graph: save in 'graph' the description of the recognized tree (DOT format)
There are two example math expressions in folder “SampleMathExps”. The
following command will recognize the expression (x+y)^2 encoded in
“exp.scgink”
$ ./seshat -c Config/CONFIG -i SampleMathExps/exp.scgink -o out.inkml -r render.pgm -d out.dot
This command outputs several information through the standard output, where the last line will
provide the LaTeX string of the recognized math expression. Furthermore:
An image representation of the input strokes will be rendered in “render.pgm”.
The InkML file of the recognized math expression will be saved in “out.inkml”.
The derivation tree of the expression provided as a graph in DOT
format will be saved in “out.dot”. The representation of the graph
in, for example, postscript format can be obtained as follows
$ dot -o out.ps out.dot -Tps
It should be noted that only options “-c” and “-i” are mandatory.
If you use seshat for your research, please cite the following reference:
@article{AlvaroPR16,
title = "An integrated grammar-based approach for mathematical expression recognition",
author = "Francisco \'Alvaro and Joan-Andreu S\'anchez and Jos\'e-Miguel Bened\'{\i}",
journal = "Pattern Recognition",
volume = "51",
pages = "135 - 147",
year = "2016",
issn = "0031-3203"
}
Seshat was the [Goddess of writing] 9 according to Egyptian
mythology, so I liked this name for a handwritten math expression
parser. I found out about seshat because my colleague of the PRHLT
[Daniel Ortiz-Martínez] 10 developed [Thot] 11, a great
open-source toolkit for statistical machine translation; and Thot is
the [God of Knowledge] 12 according to Egyptian mythology.