MoEVM Lab

Expert-aware memory virtualization for sparse MoE inference.

MoEVM Lab investigates whether VRAM can act as a small, predictive working set while colder expert weights live in RAM and NVMe.

A reproducible simulator, real OLMoE routing capture, measured workstation calibration, and a bounded paged-runtime prototype.

Run the demo View on GitHub

Pre-alpha research Apache-2.0 No model weights included

Not a Kimi K3 runtime. No production-serving or universal speedup claim. OLMoE-first prototype (public revision-pinned).

VRAM Fast · Small
VRAM expert slots (bounded)
GPU compute Router · attention · non-expert weights · KV state stay outside paged expert slots.
Pinned RAM Bounded · Transfer-only
Bounded pinned RAM staging
NVMe Cold · Backing store
NVMe checkpoint (cold)
Demand (solid) Lookahead (dotted) System layer · unobserved

Evidence

Measured on one RTX 3080 Ti.
Reported with limits.

A controlled OLMoE study compared MoEVM with one specific Transformers/Accelerate CPU-offload baseline.

Empty expert cache
1.338×
Immediate retained repeat
1.900×
Lower peak allocated VRAM
21.33%

5 prompts · 16 teacher-forced tokens each · seed 17 · no concurrency

Aggregate wall time lower is better
CPU offload
49.665 s
MoEVM empty
37.114 s
MoEVM retained
26.140 s

Evidence boundary. This is not a general serving claim. The baseline was not a tuned server, the OS file cache was not flushed, and logical traffic is not physical NVMe telemetry.

Read the full evidence

How it works

From routing trace to a hardware decision.

Sparse by design Route a few experts. Keep the rest cold. MoEVM studies which expert payloads should move, when they should move, and how much memory the working set really needs.
  1. 1

    Capture routing

    Pin the checkpoint and record real expert selections.

  2. 2

    Calibrate hardware

    Measure storage and RAM-to-VRAM transfers.

  3. 3

    Replay policies

    Compare caching, placement, and prefetch under the same workload.

  4. 4

    Validate a prototype

    Run bounded expert paging and preserve the evidence boundary.

The current prototype exercises a bounded OLMoE paging path with a pinned public revision. It is a research instrument, not a production serving stack.

Can we use other sparse MoE models? Yes, in principle. Today we need a compatible expert-routing adapter plus memory-map and transfer profile calibration for the new checkpoint before reliable claims are available.

One command

Try the bounded OLMoE demo.

The launcher checks compatible hardware, prepares an isolated environment when needed, verifies the pinned public checkpoint, and reports timing, memory, and logical storage traffic.

Windows · Python 3.12 · NVIDIA BF16 GPU · CUDA 13-compatible driver · 8 GiB VRAM total / 4 GiB free (6 GiB recommended) · 16 GiB RAM / 8 GiB available · up to 35 GiB disk

.\demo.cmd Open demo guide One prompt and up to two generated tokens: an interactive hardware demonstration, not publishable benchmark evidence.

Design-partner pilot

Need a model and hardware fit answer?

A fixed-scope audit can cover one checkpoint, one hardware profile, one workload family, and one existing baseline. The result may be positive, negative, or conditional—and ends with reproducible evidence plus a go/no-go or next-step recommendation.

Paid work covers scoped engineering and measurement, not a hidden edition of the Apache-2.0 project. No speedup, arbitrary checkpoint compatibility, or production SLA is guaranteed.

Start a public inquiry

GitHub issues are public. Do not include private weights, credentials, customer data, or confidential prompts.

Review scope and introductory ranges