Contents

• Contents
• News
  • Introduction
• Architecture
• Quick start
  • Dependencies
  • Start service
    • Start demo
    • Service Startup With Docker-compose
    • Start normal service
• Visit the service
  • Write cache
  • Query cache
  • Clear cache
• Function comparison
• Features
• Todo List
  • Adapter
  • Embedding model&inference
  • Scalar Storage
  • Vector Storage
  • Ranking
  • Service
• Acknowledgements
• Contributing

News

• 🔥🔥[2024.10.22] Added tasks for 1024 developer day.
• 🔥🔥[2024.04.09] Added Redis Search to store and retrieve embeddings in multi-tenant. This can reduce the interaction time between Cache and vector databases to 10ms.
• 🔥🔥[2023.12.10] Integrated LLM embedding frameworks such as ‘llmEmb’, ‘ONNX’, ‘PaddleNLP’, ‘FastText’, and the image embedding framework ‘timm’ to bolster embedding functionality.
• 🔥🔥[2023.11.20] Integrated local storage, such as sqlite and faiss. This enables you to initiate quick and convenient tests.
• [2023.08.26] codefuse-ModelCache…

Introduction

Codefuse-ModelCache is a semantic cache for large language models (LLMs). By caching pre-generated model results, it reduces response time for similar requests and improves user experience.
This project aims to optimize services by introducing a caching mechanism. It helps businesses and research institutions reduce the cost of inference deployment, improve model performance and efficiency, and provide scalable services for large models. Through open-source, we aim to share and exchange technologies related to large model semantic cache.

Architecture

Quick start

You can find the start script in flask4modelcache.py and flask4modelcache_demo.py.

• flask4modelcache_demo.py: A quick test service that embeds SQLite and FAISS. No database configuration required.
• flask4modelcache.py: The standard service that requires MySQL and Milvus configuration.

Dependencies

• Python: V3.8 or above

• Package installation

  pip install -r requirements.txt 
  

Start service

Start demo

1. Download the embedding model bin file from Hugging Face. Place it in the model/text2vec-base-chinese folder.
2. Start the backend service:

cd CodeFuse-ModelCache

python flask4modelcache_demo.py

Service Startup With Docker-compose

1. Download the embedding model bin file from Hugging Face. Place it in the model/text2vec-base-chinese folder.
2. Configure docker network, only need to execute once

cd CodeFuse-ModelCache

docker network create modelcache

3. Execute the docker-compose command

# When the modelcache image does not exist locally for the first time, or when the Dockerfile is changed
docker-compose up --build

# This is not the first run and the Dockerfile has not changed
docker-compose up

Start normal service

Before you start standard service, do these steps:

1. Install MySQL and import the SQL file from reference_doc/create_table.sql.

2. Install vector database Milvus.

3. Configure database access in:

   • modelcache/config/milvus_config.ini
   • modelcache/config/mysql_config.ini

4. Download the embedding model bin file from Hugging Face. Put it in model/text2vec-base-chinese.

5. Start the backend service:

   python flask4modelcache.py
   

Visit the service

The service provides three core RESTful API functionalities: Cache-Writing, Cache-Querying, and Cache-Clearing.

Write cache

import json
import requests
url = 'http://127.0.0.1:5000/modelcache'
type = 'insert'
scope = {"model": "CODEGPT-1008"}
chat_info = [{"query": [{"role": "system", "content": "You are an AI code assistant and you must provide neutral and harmless answers to help users solve code-related problems."}, {"role": "user", "content": "你是谁?"}],
                  "answer": "Hello, I am an intelligent assistant. How can I assist you?"}]
data = {'type': type, 'scope': scope, 'chat_info': chat_info}
headers = {"Content-Type": "application/json"}
res = requests.post(url, headers=headers, json=json.dumps(data))

Query cache

import json
import requests
url = 'http://127.0.0.1:5000/modelcache'
type = 'query'
scope = {"model": "CODEGPT-1008"}
query = [{"role": "system", "content": "You are an AI code assistant and you must provide neutral and harmless answers to help users solve code-related problems."}, {"role": "user", "content": "Who are you?"}]
data = {'type': type, 'scope': scope, 'query': query}

headers = {"Content-Type": "application/json"}
res = requests.post(url, headers=headers, json=json.dumps(data))

Clear cache

import json
import requests
url = 'http://127.0.0.1:5000/modelcache'
type = 'remove'
scope = {"model": "CODEGPT-1008"}
remove_type = 'truncate_by_model'
data = {'type': type, 'scope': scope, 'remove_type': remove_type}

headers = {"Content-Type": "application/json"}
res = requests.post(url, headers=headers, json=json.dumps(data))

Function comparison

We’ve implemented several key updates to our repository. We’ve resolved network issues with Hugging Face and improved inference speed by introducing local embedding capabilities. Due to limitations in SqlAlchemy, we’ve redesigned our relational database interaction module for more flexible operations. We’ve added multi-tenancy support to ModelCache, recognizing the need for multiple users and models in LLM products. Lastly, we’ve made initial adjustments for better compatibility with system commands and multi-turn dialogues.

Module	Function
		ModelCache	GPTCache
Basic Interface	Data query interface	☑	☑
	Data writing interface	☑	☑
Embedding	Embedding model configuration	☑	☑
	Large model embedding layer	☑
	BERT model long text processing	☑
Large model invocation	Decoupling from large models	☑
	Local loading of embedding model	☑
Data isolation	Model data isolation	☑	☑
	Hyperparameter isolation
Databases	MySQL	☑	☑
	Milvus	☑	☑
	OceanBase	☑
Session management	Single-turn dialogue	☑	☑
	System commands	☑
	Multi-turn dialogue	☑
Data management	Data persistence	☑	☑
	One-click cache clearance	☑
Tenant management	Support for multi-tenancy	☑
	Milvus multi-collection capability	☑
Other	Long-short dialogue distinction	☑

Features

In ModelCache, we incorporated the core principles of GPTCache. ModelCache has four modules: adapter, embedding, similarity, and data_manager.

• The adapter module orchestrates the business logic for various tasks, integrate the embedding, similarity, and data_manager modules.
• The embedding module converts text into semantic vector representations, and transforms user queries into vectors.
• The rank module ranks and evaluate the similarity of recalled vectors.
• The data_manager module manages the databases.

To make ModelCache more suitable for industrial use, we made several improvements to its architecture and functionality:

• [x] Architectural adjustment (lightweight integration):
  • Embedded into LLM products using a Redis-like caching mode
  • Provided semantic caching without interfering with LLM calls, security audits, and other functions
  • Compatible with all LLM services
• [x] Multiple model loading:
  • Supported local embedding model loading, and resolved Hugging Face network connectivity issues
  • Supported loading embedding layers from various pre-trained models
• [x] Data isolation
  • Environment isolation: Read different database configurations based on the environment. Isolate development, staging, and production environments.
  • Multi-tenant data isolation: Dynamically create collections based on models for data isolation, addressing data separation issues in multi-model/service scenarios within large language model products
• [x] Supported system instruction: Adopted a concatenation approach to resolve issues with system instructions in the prompt paradigm.
• [x] Long and short text differentiation: Long texts bring more challenges for similarity assessment. Added differentiation between long and short texts, allowing for separate threshold configurations.
• [x] Milvus performance optimization: Adjusted Milvus consistency level to “Session” level for better performance.
• [x] Data management:
  • One-click cache clearing to enable easy data management after model upgrades.
  • Recall of hit queries for subsequent data analysis and model iteration reference.
  • Asynchronous log write-back for data analysis and statistics
  • Added model field and data statistics field to enhance features

Todo List

Adapter

• [ ] Register adapter for Milvus：Based on the “model” parameter in the scope, initialize the corresponding Collection and perform the load operation.

Embedding model&inference

• [ ] Inference Optimization: Optimizing the speed of embedding inference, compatible with inference engines such as FasterTransformer, TurboTransformers, and ByteTransformer.
• [ ] Compatibility with Hugging Face models and ModelScope models, offering more methods for model loading.

Scalar Storage

• [ ] Support MongoDB
• [ ] Support ElasticSearch

Vector Storage

• [ ] Adapts Faiss storage in multimodal scenarios.

Ranking

• [ ] Add ranking model to refine the order of data after embedding recall.

Service

• [ ] Supports FastAPI.
• [ ] Add visual interface to offer a more direct user experience.

Acknowledgements

This project has referenced the following open-source projects. We would like to express our gratitude to the projects and their developers for their contributions and research.
GPTCache

Contributing

ModelCache is a captivating and invaluable project, whether you are an experienced developer or a novice just starting out, your contributions to this project are warmly welcomed. Your involvement in this project, be it through raising issues, providing suggestions, writing code, or documenting and creating examples, will enhance the project’s quality and make a significant contribution to the open-source community.