GPU environment and cluster management with LLM support
Genv - GPU Environment and Cluster Management
Genv is an open-source environment and cluster management system for GPUs.
Genv lets you easily control, configure, monitor and enforce the GPU resources that you are using in a GPU machine or cluster.
It is intendend to ease up the process of GPU allocation for data scientists without code changes 💪🏻
Check out the Genv documentation site for more details and the website for a higher-level overview of all features.
This project was highly inspired by pyenv and other version, package and environment management software like Conda, nvm, rbenv.
❓ Why Genv?
- Easily share GPUs with your teammates
- Find available GPUs for you to use: on-prem or on cloud via remote access
- Switch between GPUs without code changes
- Save time while collaborating
- Serve and manage local LLMs within your team’s cluster
Plus, it’s 100% free and gets installed before you can say Jack Robinson.
🙋 Who uses Genv?
Data Scientists & ML Engineers, who:
- Share GPUs within a research team
- Pool GPUs from multiple machines (see here), and allocate the available machine without SSH-ing every one of them
- Enforce GPU quotas for each team member, ensuring equitable resource allocation (see here)
- Reserve GPUs by creating a Genv environment for as long as you use them with no one else hijacking them (see here)
- Share GPUs between different projects
- Allocate GPUs across different projects by creating distinct Genv environments, each with specific memory requirements
- Save environment configurations to seamlessly resume work and reproduce experiment settings at a later time (see here)
- Serve local open-source LLMs for faster experimentation within the whole team
- Deploy local open-source LLMs for accelerated experimentation across the entire team
- Efficiently run open-source models within the cluster
Admins, who:
- Monitor their team’s GPU usage with Grafana dashboard (see the image below)
- Enforce GPU quotas (number of GPUs and amount of memory) to researchers for a fair game within the team (see here)
Ollama 🤝 Genv
Ready to create an LLM playground for yourself and your teammates?
Genv integrates with Ollama for managing Large Language Models (LLMs). This allows users to efficiently run, manage, and utilize LLMs on GPUs within their clusters.
$ genv remote -H gpu-server-1, gpu-server-2 llm serve llama2 --gpus 1
Check out our documentation for more information.
🏃 Quick Start
Make sure that you are running on a GPU machine:
$ nvidia-smi
Tue Apr 4 11:17:31 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.161.03 Driver Version: 470.161.03 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
...
- Install Genv
- Using pip
pip install genv
- Using conda
conda install -c conda-forge genv
- Verify the installation by running the command:
$ genv status
Environment is not active
- Activate an environment (in this example, we activate an environment named my-env that contains 1 GPU and will have 4GB of memory)
$ genv activate –-name my-env —-gpus 1
(genv:my-env)$ genv config gpu-memory 4g
(genv:my-env)$ genv status
Environment is active (22716)
Attached to GPUs at indices 0
Configuration
Name: my-env
Device count: 1
GPU memory capacity: 4g
- Start working on your project!
📜 Documentation
Check out the Genv documentation site for more details.
💫 Simple Integration & Usage with your fav IDE
| Integration with VSCode (Take me to the installation guide!) |
|---|
 |
| Integration with JupyterLab (Take me to the installation guide!) |
|---|
 |
A PyCharm integration is also in our roadmap so stay tuned!
🏃🏻 Join us in the AI Infrastructure Club
We love connecting with our community, discussing best practices, discovering new tools, and exchanging ideas with makers about anything around making & AI infrastructure. So we created a space for all these conversations. Join our Discord server for:
- Genv Installation and setup support as well as best practice tips and tricks directly for your use-case
- Discussing possible features for Genv (we prioritize your requests)
- Chatting with other makers about their projects & picking their brain up when you need help
- Monthly Beers with Engineers sessions with amazing guests from the research and industry (Link to previous session recordings)
License
The Genv software is Copyright 2022 [Run.ai Labs, Ltd.].
The software is licensed by Run.ai under the AGPLv3 license.
Please note that Run.ai’s intention in licensing the software are that the obligations of licensee pursuant to the AGPLv3 license should be interpreted broadly.
For example, Run.ai’s intention is that the terms “work based on the Program” in Section 0 of the AGPLv3 license, and “Corresponding Source” in Section 1 of the AGPLv3 license, should be interpreted as broadly as possible to the extent permitted under applicable law.