Skip to main content

​
Signature

synkro.simulate(
    agent: Callable[[list[dict]], str],
    policy: str,
    scenarios: int = 10,
    turns: int = 3,
    model: str | None = None,
    grading_model: str | None = None,
) -> SimulationResults

​
Parameters

​
agent
Callable
required
A callable that takes OpenAI-format messages ([{"role": "user", "content": "..."}]) and returns a response string. Can be sync or async — async agents are auto-detected.
​
policy
str
required
Policy document text. Rules are extracted automatically.
​
scenarios
int
default:"10"
Number of test scenarios to auto-generate. Produces a balanced mix of positive, negative, edge-case, and irrelevant scenarios.
​
turns
int
default:"3"
Maximum conversation turns (user-agent exchanges) per scenario. The simulated user may end earlier if the conversation reaches a natural conclusion.
​
model
str
default:"auto"
Model for the simulated user and scenario generation. Auto-detected if not specified.
​
grading_model
str
default:"model"
Model for verification/grading. Defaults to model. A stronger model produces more accurate verification.

​
Returns

SimulationResults — aggregated results with per-scenario details.

​
SimulationResults

PropertyTypeDescription
pass_ratefloatFraction of scenarios that passed (0.0–1.0)
totalintTotal scenarios simulated
passedintNumber that passed verification
failedintNumber that failed verification
resultslist[SimulationResult]Individual scenario results
logic_mapLogicMapExtracted policy rules
datasetDatasetConvert results to a Dataset for export
Methods:
  • save(path) — Save results to JSON
  • __iter__ — Iterate over individual SimulationResult objects
  • __len__ — Number of results

​
SimulationResult

FieldTypeDescription
scenarioGoldenScenarioThe scenario that was tested
messageslist[dict]Full conversation transcript (OpenAI-format)
passedboolWhether the agent passed verification
issueslist[str]Issues found during verification
verificationVerificationResultFull verification details

​
Examples

​
Basic Usage

import synkro

def my_agent(messages):
    resp = openai.chat.completions.create(model="gpt-4o", messages=messages)
    return resp.choices[0].message.content

results = synkro.simulate(
    agent=my_agent,
    policy="All refunds require a receipt. Max refund is $500.",
    scenarios=10,
    turns=3,
)

print(f"{results.passed}/{results.total} passed ({results.pass_rate:.0%})")

​
With Model Selection

results = synkro.simulate(
    agent=my_agent,
    policy=policy,
    scenarios=20,
    model="gpt-4o-mini",
    grading_model="gpt-4o",
)

​
Async

results = await synkro.simulate_async(
    agent=my_async_agent,
    policy=policy,
    scenarios=20,
)

​
Export Results

# JSON with full details
results.save("sim_results.json")

# JSONL dataset
results.dataset.save("sim_traces.jsonl")

# HuggingFace
results.dataset.push_to_hub("my-org/agent-sim-results")

​
Simulator Class

For advanced control (e.g., custom concurrency), use the Simulator class directly: 
from synkro import Simulator

sim = Simulator(
    model="gpt-4o-mini",
    grading_model="gpt-4o",
    concurrency=10,
)

# Sync
results = sim.run(agent=my_agent, policy=policy, scenarios=20, turns=3)

# Async
results = await sim.run_async(agent=my_agent, policy=policy, scenarios=20)

​
Simulator Parameters

ParameterTypeDefaultDescription
modelstr | NoneautoModel for simulated user and scenarios
grading_modelstr | NonemodelModel for verification
concurrencyint5Max parallel scenario executions