README.md

metadata

title: Matrix Game Demo
emoji: 🎮
colorFrom: indigo
colorTo: pink
sdk: gradio
python_version: '3.10'
app_file: app.py
pinned: false

Matrix Game 2.0 Interactive Demo

This folder contains a minimal but fully‑working example for running and interacting with the Matrix‑Game 2.0 model. The goal of this demo is to expose the core mechanics of the model—turning a single input image and a sequence of user actions into a short video—behind a simple web interface.

❗️Hardware requirements

Matrix‑Game 2.0 is a very large model (over 1.8 B parameters) and was designed to run on datacenter GPUs like NVIDIA A100 or H100. You can technically run it on a consumer GPU or even CPU, but inference will be extremely slow and may run out of memory. For the best experience you should launch this demo on a machine with at least 24 GiB of GPU VRAM. The code will gracefully fall back to CPU execution if no GPU is available, but expect generation to take minutes per frame on a CPU.

Setup

1. Create a fresh Python environment (Python 3.10+ is recommended):

   python -m venv .venv
   source .venv/bin/activate
   

2. Install the dependencies listed in requirements.txt:

   pip install --upgrade pip
   pip install -r requirements.txt
   

3. Log in to the Hugging Face Hub using your own access token. You can either export it as an environment variable or pass it directly to the application. To export it, replace YOUR_HF_TOKEN with a valid token generated from your Hugging Face account:

   export HF_TOKEN="YOUR_HF_TOKEN"
   

   Alternatively, run the command interactively:

   huggingface-cli login --token YOUR_HF_TOKEN
   

4. Launch the interactive demo:

   python matrix_game_interface.py
   

   The first time you run the script it will download several gigabytes of model weights from Hugging Face. Subsequent runs will reuse the cached files.

Usage

Once the Gradio interface starts (usually it will open your browser automatically), follow these steps:

1. Select an input image – this image acts as the first frame of the generated video. The model expects images with a 16∶9 aspect ratio. You can upload any photo (for example, a screenshot from a game or a view you want to explore).

2. Choose the number of frames you want to generate. The demo currently supports up to 30 frames (roughly one second of video at 30 fps). Longer videos require more memory and compute.

3. Click “Generate”. The model will synthesize a sequence of frames conditioned on your chosen image. You can watch the result directly in the browser or download the MP4 file to view it offline.

Action control

Matrix‑Game 2.0 normally accepts keyboard and mouse actions at each time step to steer the camera within the scene. The simplified interface provided here does not expose those controls directly—primarily because real‑time interaction requires high‑frequency communication with the model that cannot be reliably handled in a browser without significant latency.

However, the underlying MatrixGame class exposes a generate method that takes optional mouse and keyboard tensors (representing camera and movement commands). Feel free to modify the UI to add your own custom controls if you would like to experiment with the full action‑conditioned generation.

Project structure

matrix_game_app/
├── README.md               – this file
├── requirements.txt        – minimal Python dependencies
└── matrix_game_interface.py – entry point for the interactive demo

Notes

This project is intentionally light‑weight. It does not attempt to replicate the full training or inference pipeline from the official Matrix‑Game repository. Instead, it leverages the diffusers integration for the model to provide a quick way to run inference. If you need the full streaming inference pipeline with mouse/keyboard injection (as available in the original repo), please clone SkyworkAI/Matrix‑Game and follow the instructions in its README.md.

Finally, please be aware that the model weights are released under the MIT license. Make sure you adhere to the license terms when redistributing or using the model.