promptlayer
Use when managing LLM prompts, evaluating prompt changes, building multi-step AI workflows, logging and tracing LLM requests, or running evaluations against datasets. Reach for this skill when working with prompt versioning, release management, observability, or building evaluation pipelines.
How do I install this agent skill?
npx skills add https://docs.promptlayer.com --skill promptlayerIs this agent skill safe to install?
No partner audit is available yet. Read the source before installing.
What does this agent skill do?
PromptLayer Skill
Product summary
PromptLayer is a platform for managing, versioning, and evaluating LLM prompts outside your codebase. It provides a Prompt Registry for storing and releasing prompt templates with version control, Tables for building evaluation pipelines and datasets, observability tools for logging and tracing LLM requests, and Workflows for orchestrating multi-step AI systems. Agents use the Python SDK (promptlayer), JavaScript SDK, or REST API (authenticated with X-API-Key header) to fetch prompts, run them against LLM providers, log requests, and execute workflows. Key files: prompt templates stored in Prompt Registry, Tables for evaluations and datasets, Workflows for multi-step orchestration. Primary docs: https://docs.promptlayer.com
When to use
Use this skill when:
- Managing prompts: Creating, versioning, or updating prompt templates without code changes
- Deploying prompts: Using release labels to control which prompt version production uses
- Evaluating changes: Building evaluation pipelines to score prompt outputs before release
- Logging requests: Tracking LLM calls, tokens, costs, latency, and associating them with prompts
- Tracing execution: Following multi-step AI operations across LLM calls, tools, and application code
- Building workflows: Creating multi-step AI systems with prompts, tools, code, and conditionals
- Running evals in CI/CD: Testing agents or prompts against datasets in your pipeline
- Organizing resources: Using folders, external IDs, and workspaces to structure prompts and workflows
Quick reference
Authentication
export PROMPTLAYER_API_KEY="pl_your_key"
export OPENAI_API_KEY="sk_your_key" # or other provider key
Core SDK methods (Python)
| Task | Code |
|---|---|
| Run a prompt | client.run(prompt_name="my-prompt", input_variables={...}) |
| Run with release label | client.run(prompt_name="my-prompt", prompt_release_label="prod") |
| Run a workflow | client.run_workflow(workflow_id_or_name="my-workflow", input_variables={...}) |
| Get a template | client.templates.get("my-prompt", {"label": "prod"}) |
| Log a request | client.log_request(provider="openai", model="gpt-4", input={...}, output={...}) |
| Track metadata | client.track.metadata(request_id, {"key": "value"}) |
| Track score | client.track.score(request_id, score_value) |
REST API endpoints
| Resource | Endpoint |
|---|---|
| Get prompt | GET /prompt-templates/{id} with label or version query param |
| List prompts | GET /prompt-templates |
| Publish prompt | POST /prompt-templates/{id}/publish |
| Run workflow | POST /workflows/{id}/run |
| Log request | POST /log-request |
| Search logs | POST /search-request-logs |
| Get trace | GET /traces/{trace_id} |
Release labels
- Create: hover version in UI → "Add Release Label" → enter name (e.g., "prod", "staging")
- Retrieve: pass
labelparameter totemplates.get()orrun() - Move: click label → select new version
- A/B test: use Dynamic Release Labels to split traffic between versions
Tables workflow
- Create table from prompt, import request history, or start blank
- Add text columns for inputs, expected outputs, metadata
- Add computed columns for prompts, code, assertions, comparisons
- Run sheet, column, or selected cells
- Configure scoring with scorecards
- Review history and analytics
Environment variables
Store API keys and secrets as workspace environment variables (Settings → Environment Variables), then reference them in prompts and MCP configurations with {{env.VARIABLE_NAME}}.
Decision guidance
| Scenario | Use | Why |
|---|---|---|
| Running a single LLM call | client.run() | Simplest, fetches prompt + executes in one call |
| Complex multi-step AI system | Workflows | Visual builder, handles conditionals, tools, code |
| Evaluating prompt quality | Tables | Unified workspace for inputs, outputs, scoring, history |
| Logging custom LLM calls | log_request() | For providers or clients PromptLayer doesn't auto-instrument |
| Following execution flow | Traces + auto-instrumentation | Automatic for OpenAI/Anthropic/Google; manual spans for custom code |
| Deploying without code changes | Release labels | Move label to new version; production code stays the same |
| Organizing large prompt sets | Folders + external IDs | Folders for hierarchy; external IDs for syncing with external systems |
| Testing in CI/CD | SDK evals | Run agents against datasets; fail CI on low scores |
Workflow
Typical prompt management flow
- Create prompt: Dashboard → New → Prompt, add messages with
{{variable}}syntax - Test in Playground: Define input variables, click Run, verify output
- Save version: Click Save Template, add commit message
- Add release label: Hover version → Add Release Label → enter "prod" or "staging"
- Run from code:
client.run(prompt_name="...", prompt_release_label="prod", input_variables={...}) - Monitor logs: Dashboard → Logs, search by prompt name or tags
- Iterate: Edit prompt → Save new version → Run eval → Move label to new version
Typical evaluation flow
- Create table: Dashboard → New → Table → Create from prompt
- Add rows: Import CSV, request history, or add manually
- Add computed columns: Prompt execution, code, assertions, comparisons
- Configure scoring: Scorecard → define criteria and aggregation
- Run evaluation: Click Run or use API to queue operation
- Review results: Check cell status, view scores, inspect failures
- Save version: Checkpoint meaningful states for comparison
Typical observability flow
- Enable auto-instrumentation: Set
PROMPTLAYER_API_KEY, import SDK, callinstrument_openai()or equivalent - Make LLM calls: Your code calls OpenAI/Anthropic/Google SDK normally
- Spans created automatically: Each call becomes a span in a trace
- View in dashboard: Traces → search by trace ID or filter by tags
- Enrich with metadata:
client.track.metadata(request_id, {...})orclient.track.score(request_id, score) - Search and analyze: Use Search to find patterns; Analytics to aggregate metrics
Common gotchas
- API key not set: Always export
PROMPTLAYER_API_KEYbefore running code. Provider keys (e.g.,OPENAI_API_KEY) must also be set; PromptLayer never sends them to its servers. - Template variables not filled: Use
{{variable_name}}syntax (f-string or Jinja2). Verify input_variables dict keys match template variable names exactly. - Release label not found: Labels are case-sensitive and scoped to a single prompt. Verify the label exists on the version you expect.
- Stale cells in Tables: After changing a computed column's source or configuration, cells show "stale" status. Click Run or use the API to recalculate.
- Traces missing spans: Auto-instrumentation only works for supported providers (OpenAI, Anthropic, Google, Bedrock). For custom providers, use manual tracing with
PromptLayer.run()or custom spans. - Deprecated endpoints: Datasets and Evaluations APIs are deprecated; use Tables API for new workflows.
- Provider-specific schema changes:
llm_kwargsandraw_responseobjects have provider-specific structures. Useprompt_blueprint.prompt_templatefor stable, provider-agnostic data. - Cache bypass: Requests with
metadata_filtersormodel_parameter_overridesskip the SDK cache even if enabled. - Workflow timeout: Long-running workflows may timeout. Use
timeoutparameter or check execution results withworkflow_version_execution_results(). - External ID conflicts: External IDs must be unique per source within a resource type. Attempting to attach a duplicate external ID fails.
Verification checklist
Before submitting work with PromptLayer:
- API key is set and valid (
PROMPTLAYER_API_KEYexported) - Provider API key is set (e.g.,
OPENAI_API_KEY) - Prompt template exists in Prompt Registry and has the expected version/label
- Input variables in code match template variable names exactly
- Release label points to the correct version (verify in UI)
- Evaluation pipeline runs without errors; check cell status for failures
- Traces appear in dashboard with expected spans (for auto-instrumented code)
- Request logs show correct metadata, tags, and scores
- Table scorecards are configured and recalculated before reading results
- Workflow nodes have correct dependencies and conditionals
- External IDs are unique per source if used for syncing
- No deprecated Datasets/Evaluations APIs used in new code
Resources
- Comprehensive navigation: https://docs.promptlayer.com/llms.txt
- Quickstart: https://docs.promptlayer.com/quickstart
- Prompt Registry: https://docs.promptlayer.com/features/prompt-registry/overview
- Tables: https://docs.promptlayer.com/features/tables/overview
- Observability: https://docs.promptlayer.com/features/observability/overview
- Python SDK: https://docs.promptlayer.com/sdks/python
- REST API: https://docs.promptlayer.com/reference/introduction
- SDK Evals: https://docs.promptlayer.com/sdks/evals/overview
For additional documentation and navigation, see: https://docs.promptlayer.com/llms.txt
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/docs.promptlayer.com/promptlayer">View promptlayer on skillZs</a>