Agent
Use when building real-time voice and video AI agents, integrating with multiple AI providers, deploying to production, adding function calling and RAG, or testing agent behavior. Reach for this skill when working with Python agents that need to join calls, process audio/video, call external tools, or scale across multiple servers.
How do I install this agent skill?
npx skills add https://visionagents.ai --skill agentIs this agent skill safe to install?
No partner audit is available yet. Read the source before installing.
What does this agent skill do?
Vision Agents Skill Reference
Product Summary
Vision Agents is an open-source Python framework for building real-time voice and video AI agents. You write an Agent that joins a call session, connects to AI providers through swappable plugins (LLM, STT, TTS, vision models), and responds in real time. The framework handles call lifecycle, audio/video routing, turn-taking, and deployment. Key files: agent.py (agent definition), .env (API keys), pyproject.toml (dependencies). CLI: uv run agent.py run (console mode), uv run agent.py serve (HTTP server). Plugins: 35+ integrations for LLMs, speech, vision, avatars, and infrastructure. Primary docs: https://visionagents.ai
When to Use
Reach for this skill when:
- Building voice agents: Custom STT/LLM/TTS pipelines or realtime models (OpenAI, Gemini, Qwen)
- Building video agents: VLMs, YOLO processors, or realtime video understanding
- Adding tools and knowledge: Function calling, MCP servers, or RAG (Gemini FileSearch, TurboPuffer)
- Deploying to production: HTTP server, Docker, Kubernetes, horizontal scaling with Redis
- Phone integration: Inbound/outbound calls via Twilio or Telnyx
- Testing agent behavior: Text-only tests without audio/video infrastructure
- Monitoring: OpenTelemetry metrics, Prometheus, Jaeger tracing
- Choosing providers: Swapping LLMs, STT, TTS, or vision models without rewriting logic
Quick Reference
Project Setup
| Task | Command |
|---|---|
| Scaffold new project | uvx vision-agents init my-agent && cd my-agent |
| Add plugins | uv add "vision-agents[deepgram,elevenlabs,gemini]" |
| Run in console | uv run agent.py run |
| Run HTTP server | uv run agent.py serve --host 0.0.0.0 --port 8000 |
| Run tests | uv run pytest tests/ -m integration |
Core Agent Structure
from vision_agents.core import Agent, User, Runner, AgentLauncher
from vision_agents.plugins import getstream, gemini
async def create_agent(**kwargs) -> Agent:
return Agent(
edge=getstream.Edge(),
agent_user=User(name="Assistant", id="agent"),
instructions="You're a helpful voice assistant.",
llm=gemini.Realtime(), # or gemini.LLM() + stt + tts
)
async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
call = await agent.create_call(call_type, call_id)
async with agent.join(call):
await agent.simple_response("Say hi and introduce yourself.")
await agent.finish()
runner = Runner(AgentLauncher(create_agent=create_agent, join_call=join_call))
if __name__ == "__main__":
runner.cli()
Environment Variables
| Variable | Required | Notes |
|---|---|---|
STREAM_API_KEY | Yes | From getstream.io dashboard |
STREAM_API_SECRET | Yes | For server-side token minting |
GOOGLE_API_KEY | For Gemini | From aistudio.google.com |
DEEPGRAM_API_KEY | For Deepgram STT | From deepgram.com |
ELEVENLABS_API_KEY | For ElevenLabs | From elevenlabs.io |
OPENAI_API_KEY | For OpenAI | From platform.openai.com |
TWILIO_ACCOUNT_SID | For phone | From Twilio console |
TWILIO_AUTH_TOKEN | For phone | From Twilio console |
Agent Response Methods
| Method | Use Case |
|---|---|
await agent.simple_response(text, interrupt=True) | Send text prompt; LLM generates response and TTS speaks it |
await agent.say(text, interrupt=False) | Make agent speak text directly (bypasses LLM) |
await agent.finish() | Wait for call to end gracefully |
await agent.close() | Clean up resources (called automatically on context exit) |
Plugin Categories & Swap Pattern
# Realtime models (speech-to-speech, built-in STT/TTS)
llm=gemini.Realtime() # Gemini Live
llm=openai.Realtime() # OpenAI Realtime
llm=qwen.Realtime() # Qwen OMNI
llm=aws.Realtime() # AWS Bedrock Nova
# Language models (require separate STT/TTS)
llm=gemini.LLM() # Gemini 3 Flash
llm=openai.LLM() # GPT-5.4
llm=anthropic.LLM() # Claude Sonnet
# Speech-to-text
stt=deepgram.STT(eager_turn_detection=True)
stt=elevenlabs.STT()
stt=cartesia.STT()
# Text-to-speech
tts=elevenlabs.TTS()
tts=cartesia.TTS()
tts=kokoro.TTS() # Local, no API key
# Vision/Video
llm=nvidia.VLM(model="nvidia/cosmos-reason2-8b", fps=1)
processors=[ultralytics.YOLOPoseProcessor()]
Decision Guidance
Realtime vs Custom Pipeline
| Scenario | Use Realtime | Use Custom Pipeline |
|---|---|---|
| Fastest path, lowest latency | ✓ | |
| Need specific STT provider | ✓ | |
| Need specific TTS provider | ✓ | |
| Want full control over turn detection | ✓ | |
| Multimodal (audio + video) support | ✓ (some) | ✓ |
| Function calling / tool use | ✓ (LLM mode) |
RAG Backend Choice
| Backend | Setup | Search | Best For |
|---|---|---|---|
| Gemini FileSearch | Simple | Managed | Prototypes, quick setup |
| TurboPuffer | More setup | Hybrid (vector + BM25) | Production, custom needs |
Deployment Path
| Goal | Start Here |
|---|---|
| Local dev/testing | HTTP Server (uv run agent.py serve) |
| Single container | Docker Deployment |
| Multiple replicas | Horizontal Scaling (Redis session registry) |
| Full production | Kubernetes Deployment (Helm + Prometheus + Grafana) |
Workflow
1. Build a Voice Agent
- Scaffold:
uvx vision-agents init my-agent && cd my-agent - Add keys: Copy
.env.exampleto.env, fill inSTREAM_API_KEY,STREAM_API_SECRET,GOOGLE_API_KEY - Review agent.py: Understand
create_agent()andjoin_call()functions - Run:
uv run agent.py run— opens browser demo - Customize: Edit
instructions, swap LLM/STT/TTS plugins, add function calling
2. Add Function Calling
- Register functions on the LLM:
@llm.register_function(description="Get weather for a location") async def get_weather(location: str) -> dict: return {"temp": "22C", "condition": "Sunny"} - LLM calls automatically when relevant to conversation
- Test with:
response.assert_function_called("get_weather", arguments={"location": "Tokyo"})
3. Add RAG
- Choose backend: Gemini FileSearch (simple) or TurboPuffer (full control)
- For Gemini:
store = gemini.GeminiFilesearchRAG(name="my-kb") await store.create() await store.add_directory("./knowledge") llm = gemini.LLM(tools=[gemini.tools.FileSearch(store)]) - For TurboPuffer:
rag = turbopuffer.TurboPufferRAG(namespace="my-kb") await rag.add_directory("./knowledge") @llm.register_function(description="Search knowledge base") async def search(query: str) -> str: return await rag.search(query, top_k=5, mode="hybrid")
4. Deploy to Production
- HTTP Server (local):
uv run agent.py serve --host 0.0.0.0 --port 8000 - Docker: Create
Dockerfile, build image, push to registry - Horizontal Scaling: Add Redis session registry for multi-node deployments
- Kubernetes: Use provided Helm chart with Prometheus + Grafana
- Monitor: Export OpenTelemetry metrics to Prometheus, view in Grafana
5. Test Agent Behavior
- Setup pytest: Add
asyncio_mode = autotopytest.ini - Create test:
from vision_agents.testing import TestSession, LLMJudge async def test_greeting(): llm = gemini.LLM() judge = LLMJudge(gemini.LLM()) async with TestSession(llm=llm, instructions="Be friendly") as session: response = await session.simple_response("Hello") assert response.function_calls == [] verdict = await judge.evaluate(response.chat_messages[0], intent="Friendly greeting") assert verdict.success - Run:
uv run pytest tests/ -m integration
Common Gotchas
- Do not reuse Agent instances: Create a new agent for each call. Calling
join()twice raisesRuntimeError. - Realtime models disable STT/TTS: When using
AudioLLM(Realtime), STT, TTS, and turn detection are automatically disabled. Don't configure them. - Turn detection conflicts: If STT plugin has built-in turn detection (
stt.turn_detection=True), don't add a separate turn detection plugin. - Async functions only:
@llm.register_function()requires async functions; sync functions raiseValueError. - Environment variables must be set before import: Load
.envwithload_dotenv()before importing plugins. - Session limits: Set
max_concurrent_sessions,max_sessions_per_call,agent_idle_timeoutto prevent resource exhaustion. - Video override path: Set
agent.set_video_track_override_path()before callingjoin(), not after. - Metrics broadcast interval: Default is 5 seconds; adjust
broadcast_metrics_intervalif needed. - Custom FastAPI app: When providing
ServeOptions(fast_api=app), you must register all endpoints yourself; defaults are not applied. - MCP servers timeout: Set
session_timeouton MCP servers to prevent hanging connections.
Verification Checklist
Before submitting work:
- Agent runs locally:
uv run agent.py runopens browser demo -
.envfile has all required API keys (check plugin docs for specifics) -
create_agent()returns anAgentinstance with valid edge, LLM, and instructions -
join_call()creates a call, joins it, and callsfinish()orclose() - Function calls are registered with
@llm.register_function()and have descriptions - Tests pass:
uv run pytest tests/ -m integration - HTTP server starts:
uv run agent.py serveand responds to/health - Metrics are exported if using Prometheus: check
http://localhost:9464/metrics - Docker image builds:
docker build -t my-agent . - Session limits are configured for production:
max_concurrent_sessions,agent_idle_timeout - No hardcoded API keys in code; all in
.envor environment variables - Logging level is appropriate for environment (DEBUG for dev, INFO for prod)
Resources
- Comprehensive navigation: https://visionagents.ai/llms.txt
- Core API reference: https://visionagents.ai/core/overview
- Integration catalog: https://visionagents.ai/integrations/introduction-to-integrations
- Deployment guide: https://visionagents.ai/guides/deploying-overview
- GitHub repository: https://github.com/GetStream/vision-agents
- Discord community: https://discord.gg/RkhX9PxMS6
For additional documentation and navigation, see: https://visionagents.ai/llms.txt
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/visionagents.ai/agent">View Agent on skillZs</a>