fframes-video

Making videos with fframes

fframes is a Rust library that renders video from code. A video is a Rust struct with a render_frame(frame) -> Svgr method: for every frame it returns an SVG tree, built with the svgr! macro (SVG markup with {rust expressions}). Scenes split the timeline, timeline! and springs animate values, an audio map places sound, and ffmpeg encodes the result.

Each project gets a command line (from fframes::cli). Use it for everything: render a video, open the real-time preview window, and look at frames, contact sheets and loudness numbers as files you can read. Read references/design.md before designing, references/api.md while writing code and references/audio.md for sound.

1. Install

Rust from https://rustup.rs, then the system libraries ffmpeg is built with:

# macOS
brew install pkg-config ffmpeg x264 x265 opus nasm ninja
# Debian / Ubuntu
sudo apt-get install -y yasm nasm ffmpeg libx264-dev libx265-dev libopus-dev libclang-dev clang ninja-build libvpx-dev libasound2-dev
# Arch
sudo pacman -S ninja yasm nasm ffmpeg x264 x265 opus clang

Windows links a prebuilt FFmpeg 9 shared build instead (set FFMPEG_DIR to the unzipped ffmpeg-n9.0-latest-win64-gpl-shared build from BtbN/FFmpeg-Builds, add its bin to PATH, install LLVM with winget install LLVM.LLVM and set LIBCLANG_PATH). Details are in the fframes README.

2. Create a project

curl -fsSL https://raw.githubusercontent.com/dmtrKovalenko/fframes/main/scripts/new-video.sh | bash -s -- my-video --yes
# or install the generator once and use it directly:
cargo install --locked cargo-fframes
cargo fframes new my-video --format landscape --fps 30 --yes

Options: --template single-scene|multi-scene, --format landscape|portrait|square|uhd, --fps, --title, --backend, --dir. The project depends on the fframes release that matches the installed cargo-fframes; --git uses the repository's main branch instead. Always pass --yes so nothing waits for input.

Templates:

• single-scene (default): one scene. It adapts to every format; start here for single-shot clips, title cards and portrait video.
• multi-scene: two scenes. Start here for anything with several scenes; it only accepts landscape and uhd.

The template content is placeholder code that shows the API. Replace its layout, colors, fonts and decorations with a design made for the video the user asked for.

Use the Skia GPU backend (the default). cargo fframes new picks Skia on Metal (macOS) or Vulkan (Linux, Windows). It renders about 10x faster than the CPU backend and gives you the real-time preview window. On macOS and Linux the first build downloads prebuilt Skia and ffmpeg and only compiles the Rust dependencies (about a minute; other targets, or metal and vulkan together, compile Skia from source for ~20 minutes). Start it right away and write the video while it runs:

cd my-video && cargo build --release        # run in the background, the first build is the slow one

--backend cpu needs no Skia but has no preview window; pick it only when there is no GPU.

The project renders as generated:

my-video/
  src/lib.rs        # the video (from the template)
  src/main.rs       # the command line
  media/            # fonts, images and audio compiled into the binary (one starter font)
  README.md         # the commands below

Every command below is cargo run --release -- <command>. Define a short shell function (a variable like R="cargo run --release --" does not split into words in zsh, the macOS shell):

R() { cargo run --release -- "$@"; }

3. The loop

After every change:

1. $R timeline lists the scenes with their frame and second ranges, and every audio track. Check structure and pacing here before looking at pixels.
2. $R inspect checks a frame every 0.25 s plus the first and last frame of every scene. It reports missing images, fonts or glyphs, text cut off by the edge of the canvas, invalid SVG (zero-sized rectangles, bad radii), broken transforms and panics, each with its time and scene. It exits with code 2 on errors. Fix everything it reports. A warning that only appears on a scene's first frames is usually an entrance; check it with a strip.
3. $R strip <scene or range> -n 12 writes a contact sheet (strip.png): evenly spaced frames in one labelled image. Open it with your image tool. It is the fastest way to judge layout, rhythm and motion.
4. $R frame Intro@end,Outro@50% writes full-size PNGs into frames/ for detail checks (typography, alignment, contrast) and prints problems found in those frames.
5. $R onion "Intro@0..Intro@1s" -n 6 blends frames into one image (onion.png) to show the path and spacing of a movement: easing, overshoot, stagger.
6. $R preview Intro opens a real-time window with sound for the user to watch (space play/pause, h/l seek a second, j/k step a frame, q quit). It blocks until closed, so start it in the background or ask the user to run it. Offer it whenever the user wants to see the video; you review with strips and frames, the user watches in the preview.
7. $R render Intro --draft encodes one scene at half resolution in about a second; $R render writes the final out.mp4.

Rules:

• Look at the PNGs before saying anything looks good.
• Prefer strip to many frame calls; add --scale 0.5 when composition is all you need.
• Add --json when parsing output: the result goes to stdout, progress to stderr.
• Keep --release: debug builds render many times slower.

Addressing time

120 (frame), 3.2s, 500ms, 1:05.5, 50%, start, end; scenes by struct name, Intro (also matches IntroScene, case-insensitive), #3 (scene index), Intro[1] (second scene of that type); inside a scene Intro@1.2s, Intro@12, Intro@50%, Intro@end. Ranges: a..b (end exclusive), a.., ..b, all, or a scene name for the whole scene. Separate several times with commas.

All commands

command	use it to
timeline	see scenes, durations, audio tracks and their mix settings
inspect [RANGE] [--every 0.1s | --all-frames] [--fail-on warning]	find problems without rendering pixels
strip [RANGE] -n N [--columns 4] [--width 480]	review flow and motion in one image
frame TIMES [-o dir] [--svg]	full-size PNGs (and the laid-out SVG) of chosen frames
onion RANGE -n N	see a movement's trajectory and easing
svg TIME	read a frame's final SVG as text (positions, text, colors)
preview [TIME] [--paused] [--mute]	real-time window with sound, for the user
render [RANGE] [-o out.mp4] [--draft]	encode the video or a part of it (audio cut to match)
snapshot TIMES [--update]	compare frames with approved PNGs, .diff.png marks changes
audio analyze [RANGE] [--waveform w.png]	loudness (LUFS), true peak, clipping, silence, per scene
audio at TIMES	which sounds play at a moment, where in their file and how loud
audio render [RANGE] -o a.wav	the mix as a WAV file

Browser editor

fframes also has a web editor with a timeline and scrubbing, useful when a person wants to tweak a video interactively. It runs the video compiled to WebAssembly and needs Node.js plus wasm-pack; the setup is the editor/ folder of the hello-world example in the fframes repository (https://github.com/dmtrKovalenko/fframes/tree/main/examples/hello-world). Mention it when the user asks for a GUI editor. For watching use preview, and for your own review use the CLI.

4. Writing the video

impl Video for MyVideo<'_> {
    const FPS: usize = 30; const WIDTH: usize = 1920; const HEIGHT: usize = 1080;
    fn duration(&self) -> Duration<'_> { Duration::Auto }            // sum of the scenes
    fn audio(&self) -> AudioMap<'_> { AudioMap::none() }              // see references/audio.md
    fn define_scenes(&self) -> Scenes<'_> { Scenes::from(vec![&self.intro as &dyn Scene, &self.main]) }
    fn render_frame<'a>(&'a self, frame: Frame, ctx: &FFramesContext<'a, '_>) -> Svgr<'a> {
        fframes::svgr!(<svg xmlns="http://www.w3.org/2000/svg" width={Self::WIDTH} height={Self::HEIGHT}>
            <rect width={Self::WIDTH} height={Self::HEIGHT} fill={BACKGROUND} />
            {ctx.render_scenes(&frame)}
        </svg>)
    }
}

• One scene per idea, 2-6 s each. Inside a scene frame.seconds() counts from the scene start.
• Animate with frame.animate(&fframes::timeline!(at 0.2 => 0.8, animate 0.0_f32 => 1.0, Easing::EaseOut)) and springs, Easing::Spring { mass: 1.0, stiffness: 180.0, damping: 20.0 }. Leave the end time off a spring so it settles on its own.
• Markup without {} is cached across frames. Keep decoration literal and put animated values on a wrapping <g transform={..} opacity={..}>.
• render_frame runs for every frame on several threads: no panics, file reads or heavy work in it. Prepare data in the constructor or a OnceLock. Media lookups return Option; fall back to Svgr::empty() instead of expect.
• Put every font file in media/ and refer to it by family name, with a numeric font-weight.
• Measure text rather than guess: frame.text_width, frame.text_fit(.., TextOverflow::Ellipsis), frame.text_break_lines for paragraphs. inspect catches text leaving the canvas, not text leaving its own box, so check boxes in a frame PNG.
• GPU shaders (SkSL or Shadertoy GLSL) run on the Skia backend through fframes::Shader; they take images and synced video frames as input (green-screen keying, color grading). See references/api.md.

5. Making it look good

The short version of references/design.md:

• One idea per scene, large type, margins of 8-10% of the width, two or three colors plus neutrals and one accent for emphasis.
• Elements enter with a spring or ease-out over 300-600 ms and leave faster, with ease-in over 200-300 ms. Related items stagger by 60-120 ms. Give the viewer 1-2 s to read after the motion settles, and never move everything at once.
• Cross-fade scenes (fn overlap(&self) -> Overlap { Overlap::Previous(0.4) }; the overlap is added to the scene's duration(), see references/api.md) or carry an element across the cut. Keep a little motion during holds so they do not look frozen.
• At 1920x1080: titles 96-140 px, body 44-60 px, at most about 8 words per line and 3 lines per card. Portrait 1080x1920 is watched on a phone: same pixel sizes or larger, content inside the middle 80% because platform UI covers the top and bottom.
• Check every scene against the checklist in references/design.md with a strip.

6. Sound

Put audio files in media/ (compiled in, mono) or load a folder at runtime with MediaDirectory for stereo, then place them:

AudioMap::from([
    AudioTrack::new("music.mp3", Second(0.)..Eof).gain_db(-18.).fade_in(1.).fade_out(2.).duck_under_voice(),
    AudioTrack::new("vo.wav", Second(0.6)..Eof).voice(),
    AudioTrack::new("whoosh.wav", Second(3.1)..Eof).gain_db(-8.),
])

Time sound effects from the same constants that drive the animation. Check levels with $R audio analyze --waveform w.png (about -14 LUFS integrated for web video, true peak below -1 dBTP, no unintended silence) and $R audio at 3.1s for what plays at an event. Then let the user listen in $R preview.

7. Finish

1. $R inspect --fail-on warning passes, or the remaining warnings are understood entrances.
2. A strip of every scene looks right and key frames are checked at full size.
3. $R audio analyze shows sensible levels.
4. $R render -o out.mp4, then confirm size, frame count and audio with ffprobe -v error -show_entries stream=codec_type,width,height,nb_frames,duration out.mp4.

Tell the user the output path, the duration, the path of a strip image and the preview command to watch it.

Troubleshooting

• Build fails in ffmpeg-sys-fframes: a system library from step 1 is missing (nasm, pkg-config, the codec -dev packages).
• Build fails in the Skia bindings (skia-bindings) with bindgen or libclang errors (Skia is only compiled from source when no prebuilt matches, e.g. metal and vulkan together): point LIBCLANG_PATH at a working libclang (on macOS Xcode's: export LIBCLANG_PATH=$(xcode-select -p)/Toolchains/XcodeDefault.xctoolchain/usr/lib).
• Skia bindings fail to compile on macOS 27 / Xcode 27 with cannot find type `_Traits` in std___hash_table___node_allocator: only happens when Skia is compiled from source (the prebuilt binaries ship their bindings). skia-bindings 0.153.3 lacks the bindgen rule (rust-skia #1335); use one GPU backend so the prebuilt download matches, or patch a copy of skia-bindings-0.153.3 (add "std::__hash_table.*", after "std::__tree.*", in OPAQUE_TYPES in build_support/skia_bindgen.rs) through [patch.crates-io].
• can't find crate for `fframes_media_dir_macro` on macOS 27 while the file exists: the proc macro was linked by an older Rust whose output the macOS 27 loader rejects (a "LINKEDIT string pool" error). Update Rust (rustup update; 1.98 works) and rebuild.
• Text renders in the wrong font or not at all: inspect shows "No match for ... font-family"; add the font file to media/ and use its exact family name.