MoEVM Lab

Expert-aware memory virtualization for sparse MoE inference.

MoEVM Lab investigates whether VRAM can act as a small, predictive working set while colder expert weights live in RAM and NVMe.

A reproducible simulator, real OLMoE routing capture, measured workstation calibration, and a bounded paged-runtime prototype.

Run the demo View on GitHub

Pre-alpha research Apache-2.0 No model weights included

Not a Kimi K3 runtime. No production-serving or universal speedup claim.

VRAM Active working set
Fast · Small
RAM Warm expert cache
Larger · Slower
NVMe Cold expert library
Largest · Slowest

Evidence

Measured on one RTX 3080 Ti.
Reported with limits.

A controlled OLMoE study compared MoEVM with one specific Transformers/Accelerate CPU-offload baseline.

Empty expert cache
1.338×
Immediate retained repeat
1.900×
Lower peak allocated VRAM
21.33%

5 prompts · 16 teacher-forced tokens each · seed 17 · no concurrency

Aggregate wall time lower is better
CPU offload
49.665 s
MoEVM empty
37.114 s
MoEVM retained
26.140 s

Evidence boundary. This is not a general serving claim. The baseline was not a tuned server, the OS file cache was not flushed, and logical traffic is not physical NVMe telemetry.

Read the full evidence

How it works

From routing trace to a hardware decision.

  1. 1

    Capture routing

    Pin the checkpoint and record real expert selections.

  2. 2

    Calibrate hardware

    Measure storage and RAM-to-VRAM transfers.

  3. 3

    Replay policies

    Compare caching, placement, and prefetch under the same workload.

  4. 4

    Validate a prototype

    Run bounded expert paging and preserve the evidence boundary.

The real prototype exercises a bounded OLMoE paging path. It is a research instrument, not a production serving stack.

One command

Try the bounded OLMoE demo.

The launcher checks compatible hardware, prepares an isolated environment when needed, verifies the pinned public checkpoint, and reports timing, memory, and logical storage traffic.

Windows · Python 3.12 · NVIDIA BF16 GPU · CUDA 13-compatible driver · 8 GiB VRAM total / 4 GiB free (6 GiB recommended) · 16 GiB RAM / 8 GiB available · up to 35 GiB disk

.\demo.cmd Open demo guide One prompt and up to two generated tokens: an interactive hardware demonstration, not publishable benchmark evidence.

Design-partner pilot

Need a model and hardware fit answer?

A fixed-scope audit can cover one checkpoint, one hardware profile, one workload family, and one existing baseline. The result may be positive, negative, or conditional—and ends with reproducible evidence plus a go/no-go or next-step recommendation.

Paid work covers scoped engineering and measurement, not a hidden edition of the Apache-2.0 project. No speedup, arbitrary checkpoint compatibility, or production SLA is guaranteed.

Start a public inquiry

GitHub issues are public. Do not include private weights, credentials, customer data, or confidential prompts.

Review scope and introductory ranges