soak-test

!bash "${CLAUDE_SKILL_DIR}/../../hooks/yaml-helper.sh" resolve_config --keys automation,workflow

Automation mode: Resolve modes.automation (project.local.yaml → project.yaml → default collaborative). Every AskUserQuestion call and every file write follows .claude/docs/automation-modes.md (collaborative asks always · guided major-only · autonomous logs and proceeds; automation_always_ask categories always prompt).

Soak Test

A soak test (also called an endurance test) is an extended play session run with specific observation goals. Unlike a smoke check (broad critical path, ~10 min) or a single-feature playtest (~30 min), a soak test runs for 30 minutes to several hours to surface:

• Memory leaks — gradual heap growth that only appears after scene transitions
• Performance drift — frame time degradation that worsens over time
• State accumulation bugs — issues that only appear after N repetitions of a mechanic (inventory full, score overflow, AI state corruption)
• Fun fatigue — mechanics that feel good in a first session but grow repetitive over extended play
• Content exhaustion — the point where players run out of novel content

This skill generates the observation protocol and analysis harness — the human does the actual playing.

Output: production/qa/soak-test-[date]-[duration].md

When to run:

• Polish phase — before /gate-check release
• After fixing a memory or stability issue (regression soak)
• When extended play has not been formally tracked

---

1. Parse Arguments

Duration (default: 1h):

• 30m — short soak; suitable for testing a single mechanic or scene
• 1h — standard soak; covers most common leak categories
• 2h — extended soak; recommended for first full Polish soak
• 4h — deep soak; required for games with long session design (RPGs, sims)

Focus (default: all):

• memory — focus on heap size, object count, leak patterns
• stability — focus on crash/freeze/hang detection
• balance — focus on fun fatigue, content exhaustion, difficulty perception
• all — all of the above

---

2. Load Context

Read:

• project.yaml — engine.name (for engine-specific memory monitoring guidance) and performance.* budgets (memory ceiling, target FPS); for any key absent or empty (including when project.yaml has no performance or engine block), fall back to .claude/docs/technical-preferences.md
• design/gdd/game-concept.md — intended session length (for comparison against soak duration), core loop description (or design/game-brief.md, the one-page brief that replaces it at rigor: minimal — core loop, and session length only if its "Who it's for" line states one)
• Most recent file in production/qa/playtests/ — prior playtest findings (to avoid re-documenting known issues)
• Most recent file in production/qa/qa-plan-*.md — current sprint test coverage (to understand what has been formally tested vs. what the soak covers)

Note any performance budget targets (performance.* from project.yaml, else .claude/docs/technical-preferences.md):

• Memory ceiling: [N MB, or "not set"]
• Target FPS: [N, or "not set"]
• Frame budget: [N ms, or "not set"]

---

3. Define Observation Checkpoints

Based on duration, generate timed checkpoints:

30m soak: T+0, T+10, T+20, T+30 1h soak: T+0, T+15, T+30, T+45, T+60 2h soak: T+0, T+20, T+40, T+60, T+80, T+100, T+120 4h soak: T+0, T+30, T+60, T+90, T+120, T+180, T+240

At each checkpoint, the observer records the observation items defined in Phase 4.

---

4. Generate the Soak Test Protocol

Memory / Stability observation items (if focus = memory or all)

Engine-specific monitoring guidance.

Record the unit the tool shows; never convert, and never assume one. A soak test looks for growth, so every threshold below is a ratio or a delta against this session's own T+0 baseline — which is unit-agnostic and stays correct however the editor reports the number. Write the unit down at T+0 exactly as displayed and use it consistently for the rest of the run.

This replaces a note asserting the return units of Performance.get_monitor — NOT SOURCEABLE from docs/engine-reference/, the identifier appears nowhere in it — a claim that sat one line under a row asking the tester to record "Static Memory (KB)". A wrong units claim in a leak detector is off by 1024× in the one measurement the protocol exists to take, and it would read as a plausible instruction throughout. Deltas need no such claim, so the safest fix was to stop needing it.

NOT SOURCEABLE — the tool, panel and counter names below are not covered by docs/engine-reference/: Godot's Debugger → Monitors and its memory counters, Unity's Memory Profiler and its fields, Unreal's stat memory. Mark them so in the protocol: they are pointers for the tester to confirm in their own editor at T+0, not verified paths. If no memory tool can be found, nothing was measured, and the Verdict section's NOT ASSESSED applies.

Godot 4:

• Open Debugger → Monitors tab; track Memory → Static Memory and Object Count → Objects across checkpoints
• Record: Static Memory (unit as displayed), Object Count, Orphan Nodes count
• Alert threshold: Memory growth > 20% from T+0 after the first 15 minutes (some growth on load is expected; sustained growth indicates a leak)
• Orphan Nodes is the one absolute number worth watching: it should return to its T+0 value after a scene unload. A ratio hides that; a non-zero floor that keeps rising is a leak regardless of units

Unity:

• Open Memory Profiler (Window → Analysis → Memory Profiler)
• Record: Total Reserved Memory, GC Allocated, Object Count at each checkpoint (units as displayed)
• Alert threshold: GC Allocated growing monotonically across 3+ checkpoints — a monotonicity check, deliberately unit-free

Unreal Engine:

• Use stat memory console command at each checkpoint
• Record: Physical Memory Used, Physical Memory Available (units as displayed)
• Alert threshold: Physical Memory Used growth > 20% over the full soak, measured against this run's T+0. The previous threshold was an absolute "> 50MB", which silently assumes both the unit and a project scale — 50MB is a rounding error for one game and a catastrophe for another

Stability observation items (if focus = stability or all)

At each checkpoint, note:

• No crash, hang, or freeze occurred since last checkpoint
• Frame rate still within target budget ([target FPS] fps)
• Audio still playing correctly (no desync or silence)
• All HUD elements still rendering correctly
• Input responding as expected (no input loss or lag spike)

Balance / fatigue observation items (if focus = balance or all)

Collect subjective observations at each checkpoint:

• Core mechanic still feels rewarding (Y/N)
• Perceived difficulty level: [too easy / appropriate / too hard]
• Any "I've seen this before" moments since last checkpoint? (novel content exhaustion)
• Any moment of frustration since last checkpoint? Note cause.
• Any moment of peak engagement since last checkpoint? Note cause.

---

5. Generate the Protocol Document

# Soak Test Protocol

> **Date**: [date]
> **Duration**: [duration]
> **Focus**: [memory | stability | balance | all]
> **Engine**: [engine]
> **Generated by**: /soak-test

---

## Pre-Session Setup

Before starting the soak:

- [ ] Game is running from a **fresh launch** (not resumed from a prior session)
- [ ] All background applications closed (minimise OS memory interference)
- [ ] Performance monitoring tool open and recording:
  - **Godot**: Debugger → Monitors tab → Memory section visible
  - **Unity**: Memory Profiler window open
  - **Unreal**: `stat memory` ready in console
  - Tool actually used, as named in this editor: [record it — the names above
    are NOT SOURCEABLE from `docs/engine-reference/`; none found → NOT ASSESSED]
- [ ] Soak target confirmed: [session design intent from game concept]
- [ ] Prior known issues to watch for: [from most recent playtest / qa-plan]

---

## Baseline (T+0) — Record Before Playing

| Metric | Baseline Value |
|--------|---------------|
| Memory / Heap | [record before first frame of gameplay] |
| Object Count | [record] |
| FPS (first 30 seconds) | [record] |
| [Engine-specific metric] | [record] |

---

## Checkpoint Log

### T+[N] minutes

**Memory / Stability** *(if applicable)*:

| Metric | Value | Δ from Baseline | Alert? |
|--------|-------|-----------------|--------|
| Memory / Heap | | | |
| Object Count | | | |
| FPS | | | |
| Crashes / Hangs | | | |

**Stability checks**:
- [ ] No crash or hang since last checkpoint
- [ ] Frame rate within budget ([N] fps target)
- [ ] Audio correct
- [ ] HUD rendering correctly
- [ ] Input responding correctly

**Balance / Fatigue** *(if applicable)*:
- Core mechanic still rewarding: Y / N
- Difficulty perception: too easy / appropriate / too hard
- Notable moments: [note any peak engagement or frustration]
- Content exhaustion signs: Y / N — [describe]

**Free observations**:
*(Note anything unexpected observed since the last checkpoint)*

---

[Repeat Checkpoint Log section for each timed checkpoint]

---

## Post-Session Analysis

### Memory Trend

| Checkpoint | Memory | Δ/hr extrapolated |
|------------|--------|-------------------|
| T+0 | | |
| [T+N] | | |

**Leak detected?** Y / N
**Estimated time to OOM at current rate**: [N hours / not applicable]

### Stability Summary

Total crashes: [N]
Total hangs: [N]
Worst FPS observed: [N] fps at [checkpoint]
Performance degradation: stable / mild / severe

### Balance / Fatigue Summary

Fun curve: [engaged throughout / fatigue onset at T+N / repetitive from start]
Content exhaustion point: [never / at T+N / early]
Difficulty arc: [appropriate / too easy throughout / difficulty spike at T+N]

### Issues Found

| ID | Severity | Checkpoint | Description |
|----|----------|------------|-------------|
| SOAK-001 | S[1-4] | T+[N] | [description] |

---

## Verdict: PASS / PASS WITH CONCERNS / NOT ASSESSED / FAIL

**PASS**: No leaks detected, stability maintained, fun factor consistent
**PASS WITH CONCERNS**: Minor drift or fatigue noted; addressable in Polish
**NOT ASSESSED**: The soak did not run to a length that could show what it looks
for — say how far it got and which checkpoints were never reached
**FAIL**: Memory leak confirmed, stability breach, or severe fun fatigue

> **A short soak cannot return PASS.** Everything this protocol exists to detect
> — slow leaks, fatigue, late-appearing edge cases — is by definition invisible
> early, so a session that ended before the checkpoints it was built around has
> not shown stability; it has shown nothing yet. Record `NOT ASSESSED — reached
> T+[N] of [duration]; checkpoints [list] not reached`. Ranks **above both pass
> values** and **below FAIL**: a crash observed at T+20 is a real finding no
> matter how short the run, and must not be demoted behind the run's length.
> The same applies when the build crashed for reasons unrelated to the soak, when
> no memory instrumentation was available (nothing was measured, so "no leaks
> detected" means "no leaks could have been detected"), or when the protocol was
> written but never executed — a protocol document is not a result.

---

## Sign-Off

- **Tester**: [name] — [date]
- **QA Lead review**: [name] — [date]

---

6. Write Output

Present the protocol summary in conversation, then ask:

"May I write this soak test protocol to production/qa/soak-test-[date]-[duration].md?"

Write only after approval.

After writing:

"Protocol written. To run the soak:

1. Open the file and follow the Pre-Session Setup checklist
2. Record each checkpoint as you play
3. Complete the Post-Session Analysis section when done
4. File bugs from 'Issues Found' to production/qa/bugs/
5. Run /bug-triage sprint after the session to integrate any S1/S2 issues (at workflow: minimal, which has no sprints: /bug-triage full)

If the verdict is FAIL, run /smoke-check again after fixing the issues."

---

Collaborative Protocol

• This skill generates a protocol — humans run it — never attempt to run a soak test automatically. The observations require a human observer.
• Duration should match the game's session design — a 5-minute game doesn't need a 4h soak; a city-builder might. Use judgment and ask if unclear.
• First soak should be all focus — narrow focus (memory-only) is for regression soaks after a specific fix, not the first pass
• Ask before writing — always confirm before creating the protocol file