skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
visionagents.ai155 installs

Agent

Use when building real-time voice and video AI agents, integrating with multiple AI providers, deploying to production, adding function calling and RAG, or testing agent behavior. Reach for this skill when working with Python agents that need to join calls, process audio/video, call external tools, or scale across multiple servers.

How do I install this agent skill?

npx skills add https://visionagents.ai --skill agent
view source ↗

Is this agent skill safe to install?

No partner audit is available yet. Read the source before installing.

What does this agent skill do?

Vision Agents Skill Reference

Product Summary

Vision Agents is an open-source Python framework for building real-time voice and video AI agents. You write an Agent that joins a call session, connects to AI providers through swappable plugins (LLM, STT, TTS, vision models), and responds in real time. The framework handles call lifecycle, audio/video routing, turn-taking, and deployment. Key files: agent.py (agent definition), .env (API keys), pyproject.toml (dependencies). CLI: uv run agent.py run (console mode), uv run agent.py serve (HTTP server). Plugins: 35+ integrations for LLMs, speech, vision, avatars, and infrastructure. Primary docs: https://visionagents.ai

When to Use

Reach for this skill when:

  • Building voice agents: Custom STT/LLM/TTS pipelines or realtime models (OpenAI, Gemini, Qwen)
  • Building video agents: VLMs, YOLO processors, or realtime video understanding
  • Adding tools and knowledge: Function calling, MCP servers, or RAG (Gemini FileSearch, TurboPuffer)
  • Deploying to production: HTTP server, Docker, Kubernetes, horizontal scaling with Redis
  • Phone integration: Inbound/outbound calls via Twilio or Telnyx
  • Testing agent behavior: Text-only tests without audio/video infrastructure
  • Monitoring: OpenTelemetry metrics, Prometheus, Jaeger tracing
  • Choosing providers: Swapping LLMs, STT, TTS, or vision models without rewriting logic

Quick Reference

Project Setup

TaskCommand
Scaffold new projectuvx vision-agents init my-agent && cd my-agent
Add pluginsuv add "vision-agents[deepgram,elevenlabs,gemini]"
Run in consoleuv run agent.py run
Run HTTP serveruv run agent.py serve --host 0.0.0.0 --port 8000
Run testsuv run pytest tests/ -m integration

Core Agent Structure

from vision_agents.core import Agent, User, Runner, AgentLauncher
from vision_agents.plugins import getstream, gemini

async def create_agent(**kwargs) -> Agent:
    return Agent(
        edge=getstream.Edge(),
        agent_user=User(name="Assistant", id="agent"),
        instructions="You're a helpful voice assistant.",
        llm=gemini.Realtime(),  # or gemini.LLM() + stt + tts
    )

async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        await agent.simple_response("Say hi and introduce yourself.")
        await agent.finish()

runner = Runner(AgentLauncher(create_agent=create_agent, join_call=join_call))
if __name__ == "__main__":
    runner.cli()

Environment Variables

VariableRequiredNotes
STREAM_API_KEYYesFrom getstream.io dashboard
STREAM_API_SECRETYesFor server-side token minting
GOOGLE_API_KEYFor GeminiFrom aistudio.google.com
DEEPGRAM_API_KEYFor Deepgram STTFrom deepgram.com
ELEVENLABS_API_KEYFor ElevenLabsFrom elevenlabs.io
OPENAI_API_KEYFor OpenAIFrom platform.openai.com
TWILIO_ACCOUNT_SIDFor phoneFrom Twilio console
TWILIO_AUTH_TOKENFor phoneFrom Twilio console

Agent Response Methods

MethodUse Case
await agent.simple_response(text, interrupt=True)Send text prompt; LLM generates response and TTS speaks it
await agent.say(text, interrupt=False)Make agent speak text directly (bypasses LLM)
await agent.finish()Wait for call to end gracefully
await agent.close()Clean up resources (called automatically on context exit)

Plugin Categories & Swap Pattern

# Realtime models (speech-to-speech, built-in STT/TTS)
llm=gemini.Realtime()      # Gemini Live
llm=openai.Realtime()      # OpenAI Realtime
llm=qwen.Realtime()        # Qwen OMNI
llm=aws.Realtime()         # AWS Bedrock Nova

# Language models (require separate STT/TTS)
llm=gemini.LLM()           # Gemini 3 Flash
llm=openai.LLM()           # GPT-5.4
llm=anthropic.LLM()        # Claude Sonnet

# Speech-to-text
stt=deepgram.STT(eager_turn_detection=True)
stt=elevenlabs.STT()
stt=cartesia.STT()

# Text-to-speech
tts=elevenlabs.TTS()
tts=cartesia.TTS()
tts=kokoro.TTS()  # Local, no API key

# Vision/Video
llm=nvidia.VLM(model="nvidia/cosmos-reason2-8b", fps=1)
processors=[ultralytics.YOLOPoseProcessor()]

Decision Guidance

Realtime vs Custom Pipeline

ScenarioUse RealtimeUse Custom Pipeline
Fastest path, lowest latency✓
Need specific STT provider✓
Need specific TTS provider✓
Want full control over turn detection✓
Multimodal (audio + video) support✓ (some)✓
Function calling / tool use✓ (LLM mode)

RAG Backend Choice

BackendSetupSearchBest For
Gemini FileSearchSimpleManagedPrototypes, quick setup
TurboPufferMore setupHybrid (vector + BM25)Production, custom needs

Deployment Path

GoalStart Here
Local dev/testingHTTP Server (uv run agent.py serve)
Single containerDocker Deployment
Multiple replicasHorizontal Scaling (Redis session registry)
Full productionKubernetes Deployment (Helm + Prometheus + Grafana)

Workflow

1. Build a Voice Agent

  1. Scaffold: uvx vision-agents init my-agent && cd my-agent
  2. Add keys: Copy .env.example to .env, fill in STREAM_API_KEY, STREAM_API_SECRET, GOOGLE_API_KEY
  3. Review agent.py: Understand create_agent() and join_call() functions
  4. Run: uv run agent.py run — opens browser demo
  5. Customize: Edit instructions, swap LLM/STT/TTS plugins, add function calling

2. Add Function Calling

  1. Register functions on the LLM:
    @llm.register_function(description="Get weather for a location")
    async def get_weather(location: str) -> dict:
        return {"temp": "22C", "condition": "Sunny"}
    
  2. LLM calls automatically when relevant to conversation
  3. Test with: response.assert_function_called("get_weather", arguments={"location": "Tokyo"})

3. Add RAG

  1. Choose backend: Gemini FileSearch (simple) or TurboPuffer (full control)
  2. For Gemini:
    store = gemini.GeminiFilesearchRAG(name="my-kb")
    await store.create()
    await store.add_directory("./knowledge")
    llm = gemini.LLM(tools=[gemini.tools.FileSearch(store)])
    
  3. For TurboPuffer:
    rag = turbopuffer.TurboPufferRAG(namespace="my-kb")
    await rag.add_directory("./knowledge")
    @llm.register_function(description="Search knowledge base")
    async def search(query: str) -> str:
        return await rag.search(query, top_k=5, mode="hybrid")
    

4. Deploy to Production

  1. HTTP Server (local): uv run agent.py serve --host 0.0.0.0 --port 8000
  2. Docker: Create Dockerfile, build image, push to registry
  3. Horizontal Scaling: Add Redis session registry for multi-node deployments
  4. Kubernetes: Use provided Helm chart with Prometheus + Grafana
  5. Monitor: Export OpenTelemetry metrics to Prometheus, view in Grafana

5. Test Agent Behavior

  1. Setup pytest: Add asyncio_mode = auto to pytest.ini
  2. Create test:
    from vision_agents.testing import TestSession, LLMJudge
    async def test_greeting():
        llm = gemini.LLM()
        judge = LLMJudge(gemini.LLM())
        async with TestSession(llm=llm, instructions="Be friendly") as session:
            response = await session.simple_response("Hello")
            assert response.function_calls == []
            verdict = await judge.evaluate(response.chat_messages[0], intent="Friendly greeting")
            assert verdict.success
    
  3. Run: uv run pytest tests/ -m integration

Common Gotchas

  • Do not reuse Agent instances: Create a new agent for each call. Calling join() twice raises RuntimeError.
  • Realtime models disable STT/TTS: When using AudioLLM (Realtime), STT, TTS, and turn detection are automatically disabled. Don't configure them.
  • Turn detection conflicts: If STT plugin has built-in turn detection (stt.turn_detection=True), don't add a separate turn detection plugin.
  • Async functions only: @llm.register_function() requires async functions; sync functions raise ValueError.
  • Environment variables must be set before import: Load .env with load_dotenv() before importing plugins.
  • Session limits: Set max_concurrent_sessions, max_sessions_per_call, agent_idle_timeout to prevent resource exhaustion.
  • Video override path: Set agent.set_video_track_override_path() before calling join(), not after.
  • Metrics broadcast interval: Default is 5 seconds; adjust broadcast_metrics_interval if needed.
  • Custom FastAPI app: When providing ServeOptions(fast_api=app), you must register all endpoints yourself; defaults are not applied.
  • MCP servers timeout: Set session_timeout on MCP servers to prevent hanging connections.

Verification Checklist

Before submitting work:

  • Agent runs locally: uv run agent.py run opens browser demo
  • .env file has all required API keys (check plugin docs for specifics)
  • create_agent() returns an Agent instance with valid edge, LLM, and instructions
  • join_call() creates a call, joins it, and calls finish() or close()
  • Function calls are registered with @llm.register_function() and have descriptions
  • Tests pass: uv run pytest tests/ -m integration
  • HTTP server starts: uv run agent.py serve and responds to /health
  • Metrics are exported if using Prometheus: check http://localhost:9464/metrics
  • Docker image builds: docker build -t my-agent .
  • Session limits are configured for production: max_concurrent_sessions, agent_idle_timeout
  • No hardcoded API keys in code; all in .env or environment variables
  • Logging level is appropriate for environment (DEBUG for dev, INFO for prod)

Resources


For additional documentation and navigation, see: https://visionagents.ai/llms.txt

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/visionagents.ai/agent">View Agent on skillZs</a>