1. DingoDB

DingoDB is an open-source distributed multi-modal vector database independently designed and developed by DataCanvas, which integrates real-time strong consistency, relational semantics, and vector semantics into a unified platform, DingoDB positioning itself as a distinctive multi-modal database solution. With exceptional horizontal scalability and elastic scaling capabilities, it effortlessly meets enterprise-grade high availability requirements. Furthermore, DingoDB offers extensive multi-language interfaces and seamless compatibility with the MySQL protocol, delivering unparalleled flexibility and convenience for users. Demonstrating comprehensive excellence in functionality, performance, and user-friendliness, DingoDB stands out as a robust solution for modern data-driven applications.

Key Features

1. Comprehensive access interface

DingoDB provides comprehensive access interfaces, supporting various flexible access modes such as SQL, SDK, and API to meet the needs of different developers. Additionally, it introduces Table and Vector as first-class citizen data models, providing users with efficient and powerful data processing capabilities.

2.Built-in data high availability

DingoDB provides fully functional and highly available built-in configurations without the need to deploy any external components, which can significantly reduce users’ deployment and operation and maintenance costs and significantly improve the efficiency of system operation and maintenance.

3.Fully automatic elastic data sharding

DingoDB supports dynamic configuration of data shard size, automatic splitting and merging, realizing efficient and friendly resource allocation strategies, and easily responding to various business expansion needs.

4.Scalar-vector hybrid retrieval

DingoDB supports both traditional database index types and various vector index types, providing a seamless scalar and vector hybrid retrieval experience, reflecting industry-leading retrieval capabilities. In addition, it also supports fusion of scalars and vectors. Distributed transaction processing.

5.Built-in real-time index optimization

DingoDB can build scalar and vector indexes in real time, providing users with unconscious background automatic index optimization. At the same time, it ensures no delays during data retrieval.

Get Start

Docs

All Documentation Docs

Install

How to install and deploy Docker or Ansible

Usage

How to use DingoDB Usage

Developing DingoDB

VS Code

We recommend VS Code to develop the DingoDB codebase.

Java Profiler tools: YourKit

We recommend YourKit Java Profiler for any preformance critical application you make.

Check it out at https://www.yourkit.com/

2. DingoFS

DingoFS is a cloud-native distributed high-speed file storage system designed and developed by DataCanvas. It integrates multiple features such as elasticity, multi-cloud compatibility, multi-protocol convergence, and exceptional performance.By leveraging its multi-tiered, multi-type, and high-performance distributed multi-level caching architecture, DingoFS accelerates data I/O for AI workflows, effectively addressing burst I/O challenges in AI scenarios. Additionally, it provides local cache storage capabilities to meet the full lifecycle storage requirements of large-scale AI models.

Key Features

1. POSIX Compliance

DingoFS delivers a native file system-like operational experience, enabling seamless system integration.

2. AI-Native Architecture

Deeply optimized for large language model (LLM) workflows, efficiently managing massive training datasets and checkpoint workloads.

3. S3 Protocol Compatibility

DingoFS supports standard S3 interface protocols for streamlined access to filesystem namespace resources.

4. Fully Distributed Architecture

DingoFS’s metadata Service (MDS), data storage layer, caching system, and client components all support linear scalability.

5. Exceptional Performance

Combines SSD-level low-latency responsiveness with object storage-grade elastic throughput capacity.

6. Intelligent Caching Acceleration System

DingFS implements a three-tier caching topology (memory/local SSD/distributed cluster) to deliver high-throughput, low-latency intelligent I/O acceleration for AI workloads.

Get Start

1. Setup Dingo-eureka and Dingo-sdk

If you installed the software using a Docker container, the container already includes pre-integrated Dingo-eureka and Dingo-sdk, no additional installation is required.

2. Install jemalloc

wget https://github.com/jemalloc/jemalloc/releases/download/5.3.0/jemalloc-5.3.0.tar.bz2
tar -xjvf jemalloc-5.3.0.tar.bz2
cd jemalloc-5.3.0 && ./configure && make && make install

3. Download dep

git submodule sync
git submodule update --init --recursive

4. Build Etcd Client

bash build_thirdparties.sh

5. Build

mkdir build
cd build
cmake ..
make -j 32

Developing DingoFS

Install Dependencies

We recommend Rocky and Ubuntu to develop the DingoFS codebase.

GCC 13

We recommend using GCC 13 as the primary compiled language.

Projects about Dingo

The main projects about Dingo are as follows:

• DingoDB: A Unified SQL Engine to parse and compute for both structured and unstructured data.
• DingoFS: A Cloud-native distributed high-speed file storage system.
• Dingo-Store: A strongly consistent distributed storage system based on the Raft protocol.
• Dingo-Deploy: The deployment project of compute nodes and storage nodes.
• Dingo-eureka: A Necessary Service Components for DingoFS.
• Dingo-sdk: A Unified Software Development Kit (SDK) required for DingoFS.

How to make a clean pull request

• Create a personal fork of dingo on GitHub.
• Clone the fork on your local machine. Your remote repo on GitHub is called origin.
• Add the original repository as a remote called upstream.
• If you created your fork a while ago be sure to pull upstream changes into your local repository.
• Create a new branch to work on. Branch from develop.
• Implement/fix your feature, comment your code.
• Follow the code style of Google code style, including indentation.
• If the project has tests run them!
• Add unit tests that test your new code.
• In general, avoid changing existing tests, as they also make sure the existing public API is
  unchanged.
• Add or change the documentation as needed.
• Squash your commits into a single commit with git’s interactive rebase.
• Push your branch to your fork on GitHub, the remote origin.
• From your fork open a pull request in the correct branch. Target the Dingo’s develop branch.
• Once the pull request is approved and merged you can pull the changes from upstream to your local
  repo and delete your branch.
• Last but not least: Always write your commit messages in the present tense. Your commit message
  should describe what the commit, when applied, does to the code – not what you did to the code.

Special Thanks

DataCanvas

Dingo is Sponsored by DataCanvas, a new platform to do data science and data process in real-time.

DingoDB is an open-source project licensed in Apache License Version 2.0, and DingoFS is an open-source project licensed in License Version 3.0, welcome any feedback from the community.
For any support or suggestion, please contact us.

Contact us

If you have any technical questions or business needs, please contact us.

Attach the Wetchat QR Code