langsmith-graph-evaluator
Experimental. Build a LangSmith evaluator as a graph of steps with openevals.graph: code checks, LLM judges and decision models wired together, each traced as its own span and drawn as a graph in LangSmith. Use when an evaluator needs several chained checks, or the user asks for a graph, multi-step or composed evaluator. For a single judge or code check, use langsmith-evaluator.
How do I install this agent skill?
npx skills add https://github.com/langchain-ai/langsmith-skills --skill langsmith-graph-evaluatorIs this agent skill safe to install?
- Gen Agent Trust Hubpass
This skill provides a framework for building experimental evaluators for LangSmith. It includes security considerations regarding the handling of untrusted data and the management of environment variables. These patterns are consistent with its purpose as a developer tool.
- Socketpass
No alerts
- Snykwarn
Risk: MEDIUM · 1 issue
What does this agent skill do?
Build a graph evaluator
openevals.graph is experimental; its API may change between releases. Python only.
pip install "openevals==0.2.5" langchain-anthropic # or the provider your judges use; openevals[typesafe] for Jev
Set the provider keys the graph calls (ANTHROPIC_API_KEY, OPENAI_API_KEY, TYPESAFE_API_KEY for Jev). Never write a key into the evaluator file.
Decide what to measure first
Start by reading the agent code in the user's repo and a sample of its recent traces in LangSmith (see langsmith-trace). Reuse any code location and tracing project or trace link already supplied in the conversation, selected target, or local project configuration. If the code location is missing, ask for its repository path or entry point. If the tracing context is missing, ask for a development (Dev) tracing project or representative trace link. Do not assume the project is named Dev or require a separate Dev project when a tracing target is already supplied.
Before writing the evaluator, ask the user what they want to measure and whether to score each run or whole threads. Include requests for any missing code or tracing context in the same question, and wait for their answer before writing the evaluator.
For runs, confirm the exact fields in run.inputs and run.outputs. For threads, choose the input representation needed for the evaluation: raw thread, all_messages, human_ai_pairs, or first_human_last_ai. Inspect representative data in that format. For raw thread, confirm the fields needed within each root run's inputs and outputs in thread.runs. If using a dataset, also confirm its example.outputs. Make a node only for a meaningful step; keep small helpers as plain functions. One node is fine.
Nodes
Every node is a top-level name in the file, so LangSmith can read the graph from the source.
- Code: a function decorated with
@nodethat returns a dict. A parameter namedrun,thread,trajectoryorexample(or annotatedRun,Thread,Trajectory,Example) gets that context. A parameter named after another node gets that node's outputs. - LLM judge:
LLM(model="<provider>:<model>", prompt=..., inputs={...}, output_schema=...).promptis a string or a list of(role, content)messages with{{name}}placeholders.output_schemais a TypedDict or Pydantic class defined in the file, or a JSON schema; without one the node returns{"score": number}, andfeedback_keyrenames that key. - Decision model:
DecisionModel(model=..., inputs={...}, questions={...})for yes/no, scale and choice judgements.modelis required:"typesafe:jev-latest", or anylangchain_core.decisions.BaseDecisionModelinstance. It takes no prompt; its inputs are the state it judges, so each question names the inputs it compares. Each question key is an output of the node.
Questions are langchain_core.decisions types, re-exported by openevals.graph, and take keyword arguments only:
Noul(instructions="Does the answer address the question?") # probability of yes
Score(instructions="How helpful is it?", criteria=["poor", "ok", "great"]) # index into criteria
Choice(instructions="Tone", criteria={"calm": None, "rude": None}) # chosen label
inputs maps each name to a typed reference, never a string: Run.inputs.question, Run.outputs.answer, Example.outputs.reference, or another node's output like tool_usage.calls.
Example
from openevals.graph import Choice, DecisionModel, Graph, LLM, Noul, Run, node
judge = LLM(
model="anthropic:claude-sonnet-5",
prompt="Rate this answer from 0 to 1: {{answer}}",
inputs={"answer": Run.outputs.answer},
)
checks = DecisionModel(
model="typesafe:jev-latest",
inputs={"question": Run.inputs.question, "answer": Run.outputs.answer},
questions={
"on_topic": Noul(instructions="Does the answer address the question?"),
"tone": Choice(instructions="Tone", criteria={"calm": None, "rude": None}),
},
)
@node
def answer(judge, checks):
ok = checks["on_topic"] > 0.5 and checks["tone"] == "calm"
return {"score": judge["score"] if ok else 0}
graph = Graph([judge, checks, answer])
Graph([...]) checks the wiring when it is built: a parameter that names no listed node, an output a node does not produce, or a cycle raises ValueError before any model is called. Nodes run after the nodes they read, whatever order they are listed in.
Run it as an evaluator
graph_evaluator = graph.as_evaluator(answer)
as_evaluator(node) returns a (run, example) evaluator that runs the graph and reports that node's score (and comment, if it returns one) under the node's name; key="..." renames it. A node with exactly one output reports that output instead. run and example can be LangSmith objects or dicts, and example is optional for online evaluators.
Test it on a few real runs before attaching it anywhere: fetch them with langsmith run get '<run-id>' --full --format json, call graph_evaluator(run) locally, and check each node's output with graph.run(run=run). With tracing on, each node appears as its own span under the evaluator's run, so a wrong final score can be traced to the node that produced it.
Use graph_evaluator offline by passing it to evaluate(..., evaluators=[graph_evaluator]) (see langsmith-evaluator for run functions and datasets).
Run it inside LangSmith
Create it through the API rather than langsmith evaluator upload, which can't declare the packages a graph needs. POST <LangSmith API URL>/v1/platform/evaluators with the headers x-api-key and X-Tenant-Id: <workspace id>:
{"name": "Answer quality", "type": "code", "code_evaluator": {
"code": "<the whole Python file>", "language": "python", "advanced_features_enabled": true,
"workspace_secrets_keys": ["ANTHROPIC_API_KEY", "TYPESAFE_API_KEY"],
"dependencies": "openevals[typesafe]==0.2.5\nlangchain-anthropic"}}
dependenciesis one package per line: pinopenevals==0.2.5(openevals[typesafe]==0.2.5when the file has atypesafe:decision model, as in this example), and addlangchain-<provider>for each judge provider other than OpenAI.workspace_secrets_keysnames the provider keys the file calls. LangSmith sends each workspace secret with the model calls, so the file never holds a key; the clients still need some key to start, so set a placeholder at the top of the file:
import os
for key in ("ANTHROPIC_API_KEY", "TYPESAFE_API_KEY"):
os.environ.setdefault(key, "set-by-langsmith")
- Define
evaluate(ctx)and return exactly ascoreand optionalcomment, so the feedback is named after the evaluator:
def evaluate(ctx):
results = graph.run(ctx)
return {"score": results[answer]["score"]}
- Give each
LLMandDecisionModeladescription="...", and a code node a docstring, so LangSmith can label the nodes; adddisplay_name="..."when a name doesn't read well as a label. Write their arguments inline as literals (onlypromptmay name a top-level constant, andoutput_schemaa class in the file), not computed values like f-strings, and list every node in oneGraph([...]).
Runs or threads
An evaluator scores either single runs or whole threads (conversations), fixed when it's created. Choose from what the user wants to measure, and ask if it's unclear.
- Runs (the default): leave out
code_evaluator_input. Nodes readRun/runas above. - Threads: set
code_evaluator_inputincode_evaluatortothread,all_messages,human_ai_pairs, orfirst_human_last_ai. Only the selected representation is populated. For the three folded formats, read the selected string through a code node'sctxparameter (for example,ctx.all_messages).
For raw thread, a code node's thread parameter gets thread_id, session_id and runs, the thread's root runs in order, each with its inputs and outputs. Build what a judge needs in a code node:
@node
def transcript(thread):
"""Joins the thread's turns in order."""
turns = [f"User: {r.inputs.question}\nAssistant: {r.outputs.answer}" for r in thread.runs]
return {"text": "\n\n".join(turns)}
RESOLVED_PROMPT = "Did the assistant resolve the user's request by the end? Score 0 to 1.\n\n{{conversation}}"
resolved = LLM(
model="anthropic:claude-sonnet-5",
prompt=RESOLVED_PROMPT,
inputs={"conversation": transcript.text},
description="Judges whether the conversation ended with the request resolved",
)
A thread evaluator can only be attached to threads, and a run evaluator only to runs.
Then attach the evaluator to a project or dataset in LangSmith. To change it later, PATCH <LangSmith API URL>/v1/platform/evaluators/<id> with {"code_evaluator": {"code": "<the whole Python file>"}}.
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/langchain-ai/langsmith-skills/langsmith-graph-evaluator">View langsmith-graph-evaluator on skillZs</a>