aip-dynamic-caption

Captions for an AI Producer project

Captions are moments you write, the same way as any other motion graphic: one visual moment per stretch of speech, each a composition under the composition contract whose cues are static text elements shown and hidden on the moment's own paused timeline. Static text in a moment is what the editor lets the user select, retype, recolor and change the font of, so every word a caption shows is written in the composition's HTML, never generated by its script. Never call build_captions, never use the narrator-captions id, and never write compositions/narrator_captions.html or compositions/transcript-src.json: those are the service's own caption layer, which this skill does not use.

Timing

Write the captions once the speaker cut is final. Run caption_cues.py on the render-engine directory:

python3 <loaded-aip-skill>/scripts/caption_cues.py <render-engine-root> --max-words 4

It synchronizes compositions/editing-script.json with the cut in index.html, as preflight's --sync-editing-script does, so that document goes in the same upload, and prints one row per caption moment: id, src, start on the cut, duration, src_anchor (the source second at start), and cues, each with in, out and its words as [text, in, out], in seconds from the moment's start. Take every time from it, never from arithmetic of your own on get_transcript. A moment never spans a speaker cut, a sentence end or a long pause, so when the user later cuts between two moments in the editor, each moves whole with its source anchor. A cut inside a moment leaves its later words early, and one that removes its first word hides the whole moment, because the editor hides a moment whose anchor is cut away; either way the moment is right again once you write it afresh for the new cut. --max-words is how many words a cue shows at once: 4 by default, 1 for a look that shows one word at a time, and 8 to 12 for speech whose transcript words are a character or two each, such as Chinese or Japanese. You may merge two cues or split one at a word boundary, using the words' own times, and correct a word's spelling, casing or punctuation; never change what was said.

Mount each row as a moment host, after every other moment inside the root and all on one track no other moment uses, so the captions draw above every moment:

<div class="visual-host clip" data-composition-id="captions-01" data-title="Captions 1"
  data-composition-src="compositions/captions-01.html" data-start="<start>" data-duration="<duration>"
  data-src-anchor="<src_anchor>" data-track-index="<n>" data-width="W" data-height="H"></div>

Publish the caption moments with their hosts in index.html, the synchronized EditingScript and the font files. One commit carries at most 200 files and at most 256 KiB of inline content in all, so a long video's captions go out over several commits in timeline order: each mounts only the caption hosts whose files it or an earlier commit carried, and only the last one closes authoring.

The EditingScript sync refuses a cut change while caption moments are mounted (editing_script_mounted_captions_require_rebake), because their timing belongs to the old cut. To recut a captioned project, take the caption hosts out of index.html and drop their files from the next commit, make the cut, run the script again and write the captions afresh. A project whose captions the service built earlier, a narrator-captions host, keeps them until the user asks to change its captions or its cut; then replace that layer the same way.

A caption moment

<template>
  <div data-composition-id="captions-01" data-width="1080" data-height="1920">
    <style>
      @font-face { font-family: "Inter"; font-weight: 700; src: url("public/fonts/Inter-700.woff2") format("woff2"); }
      [data-composition-id="captions-01"] .cue {
        position: absolute; left: 8%; right: 12%; top: 70%; transform: translateY(-50%);
        text-align: center; font: 700 64px/1.15 "Inter"; color: #ffffff;
        text-shadow: 0 3px 12px rgba(0, 0, 0, 0.6); visibility: hidden;
      }
    </style>
    <div class="cue" data-in="0" data-out="0.82">What do you like</div>
    <div class="cue" data-in="0.82" data-out="1.9">about using Opus</div>
    <script>
      const root = document.querySelector('[data-composition-id="captions-01"]');
      const tl = gsap.timeline({ paused: true });
      root.querySelectorAll(".cue").forEach((cue) => {
        tl.set(cue, { visibility: "visible" }, Number(cue.dataset.in));
        tl.set(cue, { visibility: "hidden" }, Number(cue.dataset.out));
      });
      window.__timelines["captions-01"] = tl;
    </script>
  </div>
</template>

• One cue element per cue, its text written in the HTML, its data-in and data-out from the script's row. The timeline reads them, so retiming a cue never touches its text, and a cue's entrance or exit tween starts at its data-in or ends at its data-out. A look that treats words one by one wraps each word in its own span, with that word's times, inside the cue.
• One cue on screen at a time: the next appears as the previous leaves.
• Each caption file is at most 64 KiB, a moment's own. Repeat the @font-face and the cue rules in every caption file, so each moment renders on its own.
• A caption moment is not a beat: it takes no sound effect, and it does not count toward the visual moments of the fine cut.

The default look

Without a reference or the user's words about it, every caption is the same plain look: Inter Bold, white, with a soft dark shadow, centred, at about 6% of the canvas's short side (64 px on a 1080-wide canvas), at most two lines, with no emphasized word. Its band is centred at 70% of the frame height on a portrait canvas and at 78% on a landscape or square one while the speaker owns the frame; under a visual moment it sits where the framing skill puts the caption for that moment's layout, set on each cue that plays there.

Download the font from Google Fonts or its Fontsource mirror and stage it under public/fonts/, in the same upload as the first caption file. Stage every file the captions' characters need, and for speech in a script Inter lacks, such as Chinese, Japanese, Korean, Arabic or Hindi, take the Noto Sans family for that script instead. Never name a font you have not staged: the export renders on a machine with its own fonts, so an unstaged name renders there as another face.

With a reference

When the user's edit copies a reference, the reference style skill decides the caption look, which cues show together and how they are laid out; it replaces the default look wholesale. This skill's timing, mounting and legibility still apply.

What keeps captions legible

These hold over the default, a reference and the user's words:

• Every word stays wholly inside the frame's safe area for its whole time on screen, at its largest scale in any entrance or emphasis: the safe area the aesthetic rules set for the canvas, clear of a portrait canvas's platform buttons on the right edge and its caption bar at the bottom.
• A cue wraps inside its box rather than shrinking; split a cue that would take a third line instead of making its type smaller.
• No cue plays under a headline overlay or a statement full cover, which already draw the phrase, under any other graphic that shows the spoken words itself, such as a title set in the speaker's own words, under a logo with text or a quote card, which the aesthetic rules give their beat alone, or under a full cover whose detail must occupy the band; drop those cues, and the whole moment when every cue falls there.