framevideo-voiceover-ssml

FrameVideo Voiceover SSML

Generate narration scripts from existing project content (subtitles, transcripts, AI analysis) and enhance them with SSML-style markup for pronunciation fixes, pauses, and number reading control.

When To Use

• User wants narration generated from project subtitles or transcript
• Need to fix pronunciation of brand names, technical terms, or foreign words
• Need to control pause timing for dramatic effect or pacing
• Need to normalize how numbers, dates, or prices are spoken
• Expanding terse subtitles into natural narration
• Creating voiceover that stays aligned with visible text

Do NOT Use

• For simple text-to-speech without markup → use framevideo-media (tts) directly
• For digital human video synthesis → use chanjing-digital-human
• For choosing TTS voices or providers → use framevideo-media
• When the user provides a complete narration script ready for TTS

---

Quick Start

Minimal workflow:

# 1. Find source text (subtitle file, transcript, or analysis)
cat subtitles.txt
# Output: "Visit chanjing.ai for more info"

# 2. Add SSML markup for pronunciation
echo '<phoneme alphabet="ipa" ph="tʃæn.dʒɪŋ">Chanjing</phoneme> <break time="0.3s"/> Visit chanjing dot AI for more info.' > script-marked.txt

# 3. Generate TTS with markup (if provider supports)
npx framevideo tts script-marked.txt --voice af_heart --output narration.wav

# 4. Transcribe back to get timestamps
npx framevideo transcribe narration.wav --output transcript.json

# 5. Reference in composition
# <audio src="narration.wav" data-start="0" data-duration="5.2"></audio>

---

Core Concepts

Source Text Priority

Find the source text in this order:

1. Project subtitles/transcript — the primary source, already aligned with visible content
2. AI analysis text — scene descriptions or product copy extracted during storyboard
3. Existing narration draft — if the user provided one
4. Manual authoring — last resort when no source exists

Why subtitles first? They're already timed to the visual beats and match what viewers see on screen.

SSML-Style Markup

Three tags for delivery control:

Tag	Purpose	Example
<phoneme>	Fix pronunciation	<phoneme alphabet="ipa" ph="tʃæn.dʒɪŋ">Chanjing</phoneme>
<break>	Insert pause	<break time="0.5s"/>
<ttnumber>	Control number reading	<ttnumber pronounce="twenty twenty-six">2026</ttnumber>

Markup is authoring-layer only. Keep it in your script file, but strip it before sending to TTS providers that don't support SSML.

Fallback Strategy

Not all TTS providers support SSML tags:

• Kokoro (local): Does NOT support SSML — strip tags before generation
• Chanjing TTS: Supports phoneme, break, ttnumber — pass through unchanged
• ElevenLabs: Supports subset of SSML — check their docs

Always save both versions:

• script-marked.txt — original with SSML tags
• script-fallback.txt — clean text for non-SSML providers

---

Workflow

Step 1: Locate Source Text

# Check for existing subtitles
ls subtitles.txt captions.json transcript.json

# Check AI analysis output (if using website-to-framevideo)
cat STORYBOARD.md SCRIPT.md

# Check existing narration drafts
ls narration-draft.txt script.txt

Step 2: Normalize Into Speakable Script

Expand terse subtitles:

Source: "New feature: AI search"
Narration: "Introducing our new feature: AI-powered search that understands your intent."

Keep alignment with visible text:

• If subtitle says "50% faster", narration should say "fifty percent faster" at the same timing
• If product name appears on screen as "FrameVideo", say "FrameVideo" not "frame video"

Text normalization rules:

• Convert & to "and"
• Spell out acronyms on first use: "TTS, or text to speech"
• Convert URLs: "chanjing.ai" → "chanjing dot AI"
• Convert symbols: $99 → "ninety-nine dollars"

Step 3: Add SSML Markup

Use sparingly. Only add markup where TTS misreads or pacing is critical.

Phoneme Tag — Pronunciation Fixes

Chinese names and brands:

<phoneme alphabet="py" ph="chan1 jing4">蝉镜</phoneme>
<phoneme alphabet="py" ph="xi1">茜</phoneme> (name: "Xi", not "qi")

English technical terms:

<phoneme alphabet="ipa" ph="dʒiː.sæp">GSAP</phoneme> (not "guh-sap")
<phoneme alphabet="ipa" ph="weɪ.ˈeɪ.ˈeɪ.ˈpiː">WAAPI</phoneme> (spell it out)

Product names:

<phoneme alphabet="ipa" ph="freɪm.ˈvɪd.i.oʊ">FrameVideo</phoneme>

When to use:

• Brand names mispronounced by TTS
• Technical jargon with non-standard pronunciation
• Foreign words in English narration
• Acronyms that should be spelled vs pronounced

Alphabets:

• ipa — International Phonetic Alphabet (universal)
• py — Pinyin for Mandarin Chinese
• x-sampa — Extended SAMPA (ASCII-safe IPA)

Break Tag — Pause Control

Dramatic pauses:

Introducing our new product. <break time="0.8s"/> Built for creators.

List separation:

Three reasons: speed<break time="0.4s"/>, quality<break time="0.4s"/>, and ease of use.

Transition markers:

In the past<break time="0.3s"/>, this took hours. Now? Seconds.

Timing guidelines:

• 0.2s — natural comma pause
• 0.3-0.5s — sentence boundary, breath
• 0.6-0.8s — dramatic pause, emphasis
• 1.0s+ — scene transition, major shift

When to use:

• Default TTS pacing feels rushed
• Need emphasis before key message
• Narration must sync with on-screen timing

When NOT to use:

• Every comma (let TTS handle natural pauses)
• Replacing proper sentence structure

Ttnumber Tag — Number Reading

Years:

<ttnumber pronounce="twenty twenty-six">2026</ttnumber>
<ttnumber pronounce="nineteen fifty-five">1955</ttnumber>

Product codes:

Model <ttnumber pronounce="A X dash three hundred">AX-300</ttnumber>

Prices:

Only <ttnumber pronounce="ninety-nine dollars">$99</ttnumber>

Dates:

Launch date: <ttnumber pronounce="July seventeenth">7/17</ttnumber>

When to use:

• Default TTS reads numbers awkwardly ("two thousand twenty-six" vs "twenty twenty-six")
• Product codes should be spelled ("A-X-three-hundred" not "ax three hundred")
• Currency needs natural phrasing
• Dates need cultural reading convention

Step 4: Choose TTS Provider

Decision tree:

Does user require Chanjing platform voices?
├─ YES → Use chanjing-digital-human skill (supports SSML)
└─ NO
   ├─ Need offline/free → Kokoro via framevideo-media (strip SSML)
   └─ Need high quality → ElevenLabs (check SSML support)

Provider SSML support:

Provider	Phoneme	Break	Ttnumber	Invoke via
Kokoro	❌	❌	❌	framevideo-media
Chanjing TTS	✅	✅	✅	chanjing-digital-human
ElevenLabs	✅	✅	⚠️	External API

Step 5: Generate Audio

With SSML support (Chanjing):

# Pass marked script directly
npx framevideo chanjing tts script-marked.txt \
  --voice-id <voice-id> \
  --output narration.wav

Without SSML support (Kokoro):

# Strip tags first
sed 's/<[^>]*>//g' script-marked.txt > script-fallback.txt

# Generate clean
npx framevideo tts script-fallback.txt \
  --voice af_heart \
  --output narration.wav

Stripping SSML tags:

# Simple regex (removes all XML-like tags)
sed 's/<[^>]*>//g' input.txt > output.txt

# Or Python for precise control
python3 << 'EOF'
import re
with open('script-marked.txt') as f:
    text = f.read()
# Remove SSML tags but keep inner text
text = re.sub(r'<phoneme[^>]*>(.*?)</phoneme>', r'\1', text)
text = re.sub(r'<break[^>]*/?>', ' ', text)
text = re.sub(r'<ttnumber[^>]*>(.*?)</ttnumber>', r'\1', text)
with open('script-fallback.txt', 'w') as f:
    f.write(text)
EOF

Step 6: Save Both Versions

File structure:

project/
├── script-marked.txt        # Original with SSML
├── script-fallback.txt      # Stripped for non-SSML TTS
├── narration.wav            # Generated audio
└── transcript.json          # Re-transcribed with timestamps

Why save both?

• Marked version preserves authoring intent
• Fallback version for re-generation with different TTS
• Future TTS providers may add SSML support

Step 7: Writeback to Project

Generate transcript from audio:

npx framevideo transcribe narration.wav --output transcript.json

Use in composition:

<audio 
  src="narration.wav" 
  data-start="0" 
  data-duration="12.4"
  data-track-index="10"
  data-volume="1.0">
</audio>

Sync captions (see framevideo skill for caption patterns):

// transcript.json → caption timing
const WORDS = [
  { text: "Introducing", start: 0.2, end: 0.8 },
  { text: "FrameVideo", start: 0.9, end: 1.4 },
  // ...
];

---

Patterns

Pattern 1: Subtitle Expansion

Input subtitles.txt:

🚀 New: AI Search
50% faster
Try it now

Output script-marked.txt:

Introducing our newest feature:<break time="0.4s"/> AI-powered search.
<break time="0.3s"/>
It's fifty percent faster than traditional search.
<break time="0.5s"/>
Try it now at <phoneme alphabet="ipa" ph="tʃæn.dʒɪŋ">chanjing</phoneme> dot AI.

Pattern 2: Brand Name Consistency

DESIGN.md snippet:

## Brand Voice
- Company: "Chanjing" (IPA: tʃæn.dʒɪŋ, NOT "chan-jing")
- Product: "FrameVideo" (one word, capital F, capital V)

Apply in script:

Welcome to <phoneme alphabet="ipa" ph="tʃæn.dʒɪŋ">Chanjing</phoneme>.
<phoneme alphabet="ipa" ph="freɪm.ˈvɪd.i.oʊ">FrameVideo</phoneme> makes video simple.

Pattern 3: Technical Demo Narration

For product walkthroughs:

First, click the Timeline button.<break time="0.6s"/>
Then drag the <phoneme alphabet="ipa" ph="dʒiː.sæp">GSAP</phoneme> animation track.<break time="0.5s"/>
Notice the seek time updates in real-time.

Pattern 4: Multilingual Product Names

English narration with Chinese brand:

This is <phoneme alphabet="py" ph="chan1 jing4">蝉镜</phoneme><break time="0.3s"/>, the AI video platform from China.

Pattern 5: Number-Heavy Content

Pricing and metrics:

Available in three tiers:<break time="0.4s"/>
Starter at <ttnumber pronounce="forty-nine dollars">$49</ttnumber>,<break time="0.3s"/>
Pro at <ttnumber pronounce="ninety-nine dollars">$99</ttnumber>,<break time="0.3s"/>
and Enterprise at <ttnumber pronounce="four ninety-nine">$499</ttnumber>.

---

Rules (Non-Negotiable)

1. Always save the marked script — even if TTS doesn't support SSML today
2. Fallback text must be natural — stripping tags shouldn't break grammar
3. Transcribe generated audio — don't assume timing matches script
4. Source text takes priority — expand subtitles, don't replace them
5. Markup is sparse — only where TTS fails or timing is critical

---

Avoid

• ❌ Adding SSML to every sentence — only fix actual problems
• ❌ Using <break> instead of rewriting awkward phrasing
• ❌ Assuming Kokoro supports SSML — it doesn't, strip tags first
• ❌ Hard-coding phonemes without testing TTS output — verify it helps
• ❌ Ignoring provider limitations — check what tags are supported
• ❌ Discarding the marked script after generation — keep for iteration

---

Integration

With framevideo-media

# This skill: create marked script
echo '<phoneme>...</phoneme> text' > script-marked.txt

# framevideo-media: generate TTS
npx framevideo tts script-fallback.txt --voice af_heart --output narration.wav

# framevideo-media: transcribe
npx framevideo transcribe narration.wav --output transcript.json

# framevideo skill: use in composition

With chanjing-digital-human

# This skill: create marked script with SSML
cat script-marked.txt

# chanjing-digital-human: generate with SSML support
npx framevideo chanjing tts script-marked.txt --voice-id <id> --output narration.wav

With website-to-framevideo

Step 3 (Storyboard) → produces SCRIPT.md
↓
This skill → adds SSML markup to SCRIPT.md content
↓
Step 4 (VO) → generates audio via framevideo-media or chanjing-digital-human
↓
Step 5 (Build) → uses transcript.json for caption timing

---

Validation

1. Check SSML syntax:

# No unclosed tags
grep -o '<[^/>]*>' script-marked.txt | sort | uniq -c

# Phoneme has alphabet attribute
grep '<phoneme' script-marked.txt | grep -v 'alphabet='
# (should return nothing)

2. Test fallback stripping:

sed 's/<[^>]*>//g' script-marked.txt > test-fallback.txt
cat test-fallback.txt
# Verify: readable, no XML artifacts, natural grammar

3. A/B test TTS output:

# Generate without markup
npx framevideo tts script-fallback.txt --voice af_heart -o no-markup.wav

# Generate with markup (if supported)
npx framevideo chanjing tts script-marked.txt --voice-id <id> -o with-markup.wav

# Compare audio quality and pronunciation

4. Verify transcript alignment:

npx framevideo transcribe narration.wav --output transcript.json
# Check: word timing matches intended beats

---

References

• references/workflow.md — step-by-step workflow details
• references/ssml.md — SSML tag specifications
• framevideo-media skill — TTS generation and transcription
• chanjing-digital-human skill — SSML-aware TTS via Chanjing platform
• framevideo skill — audio element conventions and caption timing
• SSML W3C spec: https://www.w3.org/TR/speech-synthesis/
• IPA chart: https://www.internationalphoneticalphabet.org/ipa-charts/