neoml

Machine learning framework for both deep learning and traditional algorithms

668

104

C++

NeoML

Desktop Build Status
Python Build
iOS Build
Android Build

NeoML is an end-to-end machine learning framework that allows you to build, train, and deploy ML models. This framework is used by ABBYY engineers for computer vision and natural language processing tasks, including image preprocessing, classification, document layout analysis, OCR, and data extraction from structured and unstructured documents.

Key features:

Neural networks with support for over 100 layer types
Traditional machine learning: 20+ algorithms (classification, regression, clustering, etc.)
CPU and GPU support, fast inference
ONNX support
Languages: Python, C++, Java, Objective-C
Cross-platform: the same code can be run on Windows, Linux, macOS, iOS, and Android

Build and install
Getting started
API description
License

Build and install

Supported platforms

The full С++ library version has been tested on the platforms:

Target OS	Compiler	Architecture
Windows 7+ (CPU and GPU)	MSVC 2019+	x86, x86_64
Ubuntu 14+ (CPU)	gcc 5.4+	x86_64
MacOS 10.11+ (CPU)	Apple clang 12+	arm64, x86_64
iOS 11+ (CPU, GPU)	Apple clang 12+	arm64-v8a, x86_64
Android 5.0+ (CPU), Android 7.0+ (GPU)	clang 7+	armeabi-v7a, arm64-v8a, x86, x86_64

The inference Java and Objective-C library versions have been tested on the platforms:

Target OS	Compiler	Architecture
iOS 11+ (CPU, GPU)	Apple clang 12+	arm64-v8a, x86_64
Android 5.0+ (CPU), Android 7.0+ (GPU)	clang 7+	armeabi-v7a, arm64-v8a, x86, x86_64

Third party

The library is built with CMake (recommended versions 3.18 and later).

For best CPU performance on Windows, Linux and macOS we use Intel MKL.

When processing on a GPU, you can optionally use CUDA (version 11.2 upd.1) on Windows or Linux and Vulkan (version 1.1.130 and later) on Windows, Linux or Android.

We also use Google Test for testing and Google Protocol Buffers for working with ONNX model format.

We use very convinous generator of JIT code xbyak for speeding up some convolutions on x86_64 processors.

Build fully functional C++ version

See here for instructions on building the C++ library version for different platforms.

Build inference versions for Java and Objective-C

See here for instructions on building the Java and Objective-C versions that would only run the trained neural networks.

Getting started

Several tutorials with sample code will help you start working with the library:

API description

Basic principles

The library was developed with these principles in mind:

Platform independence

The user interface is completely separated from the low-level calculations implemented by a math engine.

The only thing you have to do is to specify at the start the type of the math engine that will be used for calculations. You can also choose to select the math engine automatically, based on the device configuration detected.

The rest of your machine learning code will be the same regardless of the math engine you choose.

Math engines independence

Each network works with one math engine instance, and all its layers should have been created with the same math engine. If you have chosen a GPU math engine, it will perform all calculations. This means you may not choose to use a CPU for “light” calculations like adding vectors and a GPU for “heavy” calculations like multiplying matrices. We have introduced this restriction to avoid unnecessary synchronizations and data exchange between devices.

Multi-threading support

The math engine interface is thread-safe; the same instance may be used in different networks and different threads.

Note that this may entail some synchronization overhead.

However, the neural network implementation is not thread-safe; the network may run only in one thread.

ONNX support

NeoML library also works with the models created by other frameworks, as long as they support the ONNX format. See the description of import API. However, you cannot export a NeoML-trained model into ONNX format.

Serialization format

The library uses its own binary format (implemented by CArchive, CArchiveFile) to save and load the trained models.

GPU support

Processing on GPU often helps significantly improve performance of mathematical operations. The NeoML library uses GPU both for training and running the models. This is an optional setting and depends on the hardware and software capabilities of your system.

To work on GPU, the library requires:

Windows: NVIDIA® GPU card with CUDA® 11.2 upd. 1 support.
iOS: Apple GPU A7+.
Android: devices with Vulkan 1.0 support.
Linux/macOS: no support for GPU processing as yet.

FineObj

The NeoML library originates in ABBYY internal infrastructure. For various reasons ABBYY uses a cross-platform framework called FineObj. Because of this, the open library version uses some of this framework primitives. See the common classes description.

C++ interface

NeoML contains two C++ libraries:

Algorithms library NeoML

The library provides C++ objects that implement various high-level algorithms. It consists of several parts:

NeoMathEngine

The math engine used for calculations is a separate module that implements the low-level mathematical functions used in the algorithms library. The user can also call these functions but usually never needs to.

This module has different implementations for different platforms. In particular, there is an implementation that uses a GPU for calculations.

The math engine is also a set of C++ interfaces described here.

Python module

See the extensive documentation of the Python module on readthedocs.io.

Java interface

To work with the inference version of the library in Java and Kotlin we provide a Java interface.

Objective-C interface

To work with the inference version of the library in Swift and Objective-C we provide an Objective-C interface.