MineStudio

MineStudio: A Streamlined Package for Minecraft AI Agent Development

CraftJarvis

258

Python

Overview

MineStudio contains a series of tools and APIs that can help you quickly develop Minecraft AI agents:

Simulator: Easily customizable Minecraft simulator based on MineRL.
Data: A trajectory data structure for efficiently storing and retrieving arbitray trajectory segment.
Models: A template for Minecraft policy model and a gallery of baseline models.
Offline Training: A straightforward pipeline for pre-training Minecraft agents with offline data.
Online Training: Efficient RL implementation supporting memory-based policies and simulator crash recovery.
Inference: Pallarelized and distributed inference framework based on Ray.
Benchmark: Automating and batch-testing of diverse Minecraft tasks.

This repository is under development. We welcome any contributions and suggestions.

News

2025/05/28 - We have released a big update of MineStudio (v1.1.4) with the following changes:
- Refactored the data component to support more flexible data loading and processing, all the trajectory modals are now decoupled. Users are able to customize their own data processing methods.
- Added detailed code comments and docstrings to all the modules, making it easier to understand and use the code.
- Improved the documentation with more examples, tutorials, and a new API reference section.

Installation

For a more detailed installation guide, please refer to the documentation.

MineStudio requires Python 3.10 or later. We recommend using conda to maintain an environment on Linux systems. JDK 8 is also required for running the Minecraft simulator.

conda create -n minestudio python=3.10 -y
conda activate minestudio
conda install --channel=conda-forge openjdk=8 -y

MineStudio is available on PyPI. You can install it via pip.

pip install MineStudio

To install MineStudio from source, you can run the following command:

pip install git+https://github.com/CraftJarvis/MineStudio.git

[Important] Minecraft simulator requires rendering tools. For users with nvidia graphics cards, we recommend installing VirtualGL. For other users, we recommend using Xvfb, which supports CPU rendering but is slightly slower. Refer to the documentation for installation commands.

After the installation, you can run the following command to check if the installation is successful:

python -m minestudio.simulator.entry # using Xvfb
MINESTUDIO_GPU_RENDER=1 python -m minestudio.simulator.entry # using VirtualGL

Docker

We provide a Docker image for users who want to run MineStudio in a container. The Dockerfile is available in the assets directory. You can build and run the image by running the following command:

cd assets
docker build --platform=linux/amd64 -t minestudio .
docker run -it minestudio

Datasets on 🤗 Hugging Face

We converted the Contractor Data the OpenAI VPT project provided to our trajectory structure and released them to the Hugging Face. (The old dataset is only available in v1.0.6 and earlier versions. From v1.1.0, we have changed the dataset structure to support more flexible data loading and processing.)

from minestudio.data import RawDataset
from minestudio.data.minecraft.callbacks import ImageKernelCallback, ActionKernelCallback

dataset = RawDataset(
    dataset_dirs=['6xx', '7xx', '8xx', '9xx', '10xx'],  # Specify the dataset directories
    modal_kernel_callbacks=[
        ImageKernelCallback(frame_width=224, frame_height=224, enable_video_aug=False), 
        ActionKernelCallback(enable_prev_action=True, win_bias=1, read_bias=-1),
    ],
    win_len=128, 
    split_ratio=0.9,
    shuffle_episodes=True,
)
item = dataset[0]
print(item.keys())

Models on 🤗 Hugging Face

We have pushed all the checkpoints to 🤗 Hugging Face, it is convenient to load the policy model.

from minestudio.simulator import MinecraftSim
from minestudio.simulator.callbacks import RecordCallback
from minestudio.models import VPTPolicy

policy = VPTPolicy.from_pretrained("CraftJarvis/MineStudio_VPT.rl_from_early_game_2x").to("cuda")
policy.eval()

env = MinecraftSim(
    obs_size=(128, 128), 
    callbacks=[RecordCallback(record_path="./output", fps=30, frame_type="pov")]
)
memory = None
obs, info = env.reset()
for i in range(1200):
    action, memory = policy.get_action(obs, memory, input_shape='*')
    obs, reward, terminated, truncated, info = env.step(action)
env.close()

Here is the checkpoint list:

CraftJarvis/MineStudio_VPT.foundation_model_1x, trained by OpenAI
CraftJarvis/MineStudio_VPT.foundation_model_2x, trained by OpenAI
CraftJarvis/MineStudio_VPT.foundation_model_3x, trained by OpenAI
CraftJarvis/MineStudio_VPT.bc_early_game_2x, trained by OpenAI
CraftJarvis/MineStudio_VPT.rl_from_house_2x, trained by OpenAI
CraftJarvis/MineStudio_VPT.rl_from_early_game_2x, trained by OpenAI
CraftJarvis/MineStudio_VPT.bc_house_3x, trained by OpenAI
CraftJarvis/MineStudio_VPT.bc_early_game_3x, trained by OpenAI
CraftJarvis/MineStudio_VPT.rl_for_shoot_animals_2x, trained by CraftJarvis
CraftJarvis/MineStudio_VPT.rl_for_build_portal_2x, trained by CraftJarvis
CraftJarvis/MineStudio_GROOT.18w_EMA, trained by CraftJarvis
CraftJarvis/MineStudio_STEVE-1.official, trained by STEVE-1
CraftJarvis/MineStudio_ROCKET-1.12w_EMA, trained by CraftJarvis
CraftJarvis/MineStudio_ROCKET-2-1x-22w, trained by CraftJarvis
CraftJarvis/MineStudio_ROCKET-2-1.5x-17w, trained by CraftJarvis

Star History

Acknowledgement

The simulation environment is built upon MineRL and Project Malmo.
We also refer to Ray, PyTorch Lightning for distributed training and inference.
Thanks for their great work.

Citation

@inproceedings{MineStudio,
  title={MineStudio: A Streamlined Package for Minecraft AI Agent Development},
  author={Shaofei Cai and Zhancun Mu and Kaichen He and Bowei Zhang and Xinyue Zheng and Anji Liu and Yitao Liang},
  year={2024},
  url={https://api.semanticscholar.org/CorpusID:274992448}
}