colibri
Assist with Colibri: pure-C LLM inference engine for running GLM-5.2 (744B MoE) on consumer machines with ~25 GB RAM. Use when setting up, building, converting models, running inference, configuring expert streaming and caching, optimizing speculative decoding (MTP), GPU integration, and integrating Colibri into production pipelines. Includes build setup, model download & conversion, chat/inference modes, performance tuning, and API integration patterns.
How do I install this agent skill?
npx skills add https://github.com/akillness/jeo-skills --skill colibriIs this agent skill safe to install?
- Gen Agent Trust Hubpass
Error: AI Analysis failed with no response - please submit the skill again
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
Colibri
Assist with Colibri — the pure-C inference engine for running GLM-5.2 (744B Mixture-of-Experts) on consumer machines with ~25 GB RAM, featuring expert disk-streaming, MLA attention, and native speculative decoding (MTP).
When to use this skill
- Setup & build: Installing C build tools, running
setup.sh, configuring OpenMP, cross-platform compilation (Linux, macOS, Windows via WSL) - Model download & conversion: Running
coli convertto fetch GLM-5.2-FP8 shards, convert to int4 container, prepare MTP heads (int8 quantization) - Inference modes: Running chat mode (
coli chat), API server setup, batch inference, configuring sampling and temperature - Expert caching & storage: Tuning LRU cache sizes, pin-hot-store configuration, disk I/O optimization, thermal management
- Speculative decoding (MTP): Enabling/disabling multi-token prediction, measuring draft acceptance rates, optimizing verification latency
- GPU acceleration: Configuring CUDA pinned experts, GPU vRAM budgeting, hybrid CPU-GPU inference
- Performance profiling: Measuring tokens/second, analyzing expert-load patterns, memory residency tuning
- Integration & deployment: Embedding Colibri in applications, wrapping inference API, local-first LLM pipelines
When not to use this skill
- Python-first LLM frameworks: Use
llama-index,crewai-multi-agent, orpydantic-aifor high-level abstractions - Distributed/cloud inference: Use cloud LLM APIs or
deepspeedfor multi-GPU/multi-node training - LLM fine-tuning: Use
dspy,openrlhf-training, ormoe-trainingfor training workflows - General AI infra: Use
system-environment-setupfor unrelated dev environments
Core architecture
Main components:
- glm.c (~2,400 lines) — Faithful GLM-5.2 (glm_moe_dsa) forward pass with MLA attention, MTP speculative decoding, DeepSeek-V3 sigmoid router
- Expert streaming — 21,504 routed experts (75 MoE layers × 256 experts) live on disk (~370 GB total at int4), streamed on demand with per-layer LRU cache
- Storage hierarchy — Dense layers (attention, shared experts, embeddings) in RAM (~9.9 GB at int4), experts on NVMe/HDD, OS page cache as L2
- MTP speculative decoding — GLM-5.2's native multi-token-prediction head (layer 78, int8 for lossless 39–59% draft acceptance) drafts and main model verifies in one forward
- MLA attention — q/kv-LoRA + interleaved partial RoPE with compressed KV-cache (576 floats/token vs 32,768 dense — 57× smaller)
CLI modes:
coli chat— Interactive chat with streaming, auto-detected RAM/cache/MTPcoli convert— Download GLM-5.2-FP8, quantize to int4, prepare MTP int8 headcoli plan— Inspect storage hierarchy, vRAM budget, expert placement before loading
Key parameters
| Parameter | Scope | Default | Notes |
|---|---|---|---|
COLI_MODEL | env | (required) | Path to converted int4 model directory |
--ram <GB> | plan/chat | auto-detect | RAM budget for dense layers (min 10 GB) |
--vram <GB> | plan/chat | 0 (CPU-only) | GPU vRAM for pinned experts (requires CUDA build) |
--cache <MB> | plan/chat | auto-detect | Per-layer LRU cache size for routed experts |
--mtp on/off | chat | auto-detect | Enable/disable speculative decoding (off = single-token mode) |
--temperature <0–2> | chat | 0.95 | Sampling temperature (0 = greedy) |
--top-p <0–1> | chat | 0.9 | Nucleus sampling threshold |
Performance metrics (community benchmarks)
- Tokens/second (CPU, int4 cache warm):
- Single token: ~1.2 tok/s (minimal cache misses)
- MTP enabled: ~2.2–2.8 tok/s (2.2–2.8× speedup, 39–59% draft acceptance)
- MTP with cold cache: may regress due to extra expert loads (~660 → ~1100/token)
- Residency: Dense layers ~9.9 GB RAM; experts 370 GB disk
- Memory overhead: ~11 GB per-token token from routed experts; 57× smaller KV-cache than dense attention
- Conversion time: ~30–60 min (one-time, depends on disk speed and internet)
Integration patterns
- Local chat UI: Wrap
coli chatin a terminal/web interface - API server: Stream inference output over HTTP/gRPC
- Batch processing: Run multiple prompts with persistent model load
- Hybrid CPU-GPU: Pin hot experts to GPU vRAM, stream others from disk
- Edge/mobile: int4 model size (~370 GB) requires local NVMe; consider pruning/distillation for smaller devices
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/akillness/jeo-skills/colibri">View colibri on skillZs</a>