Optimize agents | Opik Documentation | Opik Documentation

The Opik Agent Optimizer

The Opik Agent Optimizer can optimize both simple prompts and complex agent workflows. For most use cases, you can optimize prompts directly using ChatPrompt. When you need multi-prompt workflows, agent orchestration, or custom execution logic, you’ll use OptimizableAgent to create a custom agent class.

When to use OptimizableAgent vs ChatPrompt

Use ChatPrompt directly (default approach):

Use OptimizableAgent when you need:

Optimizers work seamlessly with both approaches. The optimizer calls your agent’s invoke_agent() method repeatedly during optimization, passing different prompt candidates to evaluate.

Single-prompt optimization

For most optimization tasks, you can use ChatPrompt directly without creating a custom agent. The optimizer uses a default LiteLLM-based agent under the hood.

from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer.datasets import hotpot

dataset = hotpot(count=300)

def levenshtein_ratio(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item["answer"],
        output=llm_output
    )

prompt = ChatPrompt(
    system="You are a helpful assistant.",
    user="{question}",
    model="openai/gpt-4o-mini"
)

optimizer = MetaPromptOptimizer(model="openai/gpt-4o")
result = optimizer.optimize_prompt(
    prompt=prompt,
    dataset=dataset,
    metric=levenshtein_ratio,
    max_trials=5,
    n_samples=50
)

result.display()

Custom agent for framework integration

When integrating with specific agent frameworks (Google ADK, LangGraph, CrewAI, etc.), you’ll create a custom OptimizableAgent subclass. This allows the optimizer to work with your framework’s execution model.

Here’s an example for Google ADK:

from typing import Any, TYPE_CHECKING
from opik_optimizer import OptimizableAgent

if TYPE_CHECKING:
    from opik_optimizer.api_objects import chat_prompt

class ADKAgent(OptimizableAgent):
    project_name = "adk-agent"

def invoke_agent(
        self,
        prompts: dict[str, chat_prompt.ChatPrompt],
        dataset_item: dict[str, Any],
        allow_tool_use: bool = False,
        seed: int | None = None,
    ) -> str:
        # Single-prompt agents extract the prompt from the dict
        if len(prompts) > 1:
            raise ValueError("ADKAgent only supports single-prompt optimization.")

prompt = list(prompts.values())[0]
        messages = prompt.get_messages(dataset_item)

# Your framework-specific execution logic here
        # ... create ADK agent, run it, return response ...

return response

Multi-prompt optimization

For multi-step agent workflows, you must use OptimizableAgent because ChatPrompt only handles a single prompt. Multi-prompt optimization allows you to optimize multiple prompts that work together in a pipeline.

When to use multi-prompt optimization

Implementing a multi-prompt agent

Here’s a simple example of a two-step workflow that analyzes input and then generates a response:

from typing import Any
from opik_optimizer import ChatPrompt, OptimizableAgent
from openai import OpenAI

class AnalyzeRespondAgent(OptimizableAgent):
    """Two-step agent: analyze input, then respond based on analysis."""

def __init__(self, model: str = "gpt-4o-mini"):
        super().__init__()
        self.model = model
        self.client = OpenAI()

def invoke_agent(
        self,
        prompts: dict[str, ChatPrompt],
        dataset_item: dict[str, Any],
        allow_tool_use: bool = False,
        seed: int | None = None,
    ) -> str:
        # Step 1: Analyze the input
        analyze_prompt = prompts["analyze"]
        analyze_messages = analyze_prompt.get_messages(dataset_item)

analyze_response = self.client.chat.completions.create(
            model=self.model,
            messages=analyze_messages,
            seed=seed,
        )
        analysis = analyze_response.choices[0].message.content

# Step 2: Generate response based on analysis
        respond_prompt = prompts["respond"]
        # Pass analysis result to the respond prompt
        respond_context = {**dataset_item, "analysis": analysis}
        respond_messages = respond_prompt.get_messages(respond_context)

respond_response = self.client.chat.completions.create(
            model=self.model,
            messages=respond_messages,
            seed=seed,
        )

return respond_response.choices[0].message.content

Using the multi-prompt agent

When optimizing, pass a dictionary of prompts instead of a single prompt:

from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio

# Define both prompts in the workflow
prompts = {
    "analyze": ChatPrompt(
        system="You are an analysis assistant. Extract key information from the input.",
        user="{text}",
        model="gpt-4o-mini"
    ),
    "respond": ChatPrompt(
        system="You are a response assistant. Generate a helpful response based on the analysis.",
        user="Analysis: {analysis}\n\nOriginal question: {text}",
        model="gpt-4o-mini"
    ),
}

optimizer = MetaPromptOptimizer(model="openai/gpt-4o")
result = optimizer.optimize_prompt(
    prompt=prompts,  # Pass dict of prompts
    agent_class=AnalyzeRespondAgent,  # Use your custom agent
    dataset=dataset,
    metric=levenshtein_ratio,
    max_trials=5,
    n_samples=50
)

result.display()

Key implementation details

invoke_agent() method signature

All OptimizableAgent subclasses must implement invoke_agent(): 

def invoke_agent(
    self,
    prompts: dict[str, ChatPrompt],
    dataset_item: dict[str, Any],
    allow_tool_use: bool = False,
    seed: int | None = None,
) -> str:
    # Your implementation here
    return response_string

Parameters:

Returns: A single string output that will be scored by your metric function

Extracting messages from prompts

Use ChatPrompt.get_messages() to format the prompt with dataset values:

messages = prompt.get_messages(dataset_item)
# Returns list of message dicts: [{"role": "system", "content": "..."}, ...]

For multi-prompt workflows, pass additional context when calling get_messages():

# Pass intermediate results to subsequent prompts
context = {**dataset_item, "intermediate_result": some_value}
messages = prompt.get_messages(context)

Best practices

Complete examples

Single-prompt with ChatPrompt (default)

from opik_optimizer import ChatPrompt, EvolutionaryOptimizer
from opik_optimizer.datasets import hotpot
from opik.evaluation.metrics import LevenshteinRatio

dataset = hotpot(count=300)

def metric(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item["answer"],
        output=llm_output
    )

prompt = ChatPrompt(
    system="You are a helpful assistant.",
    user="{question}",
    model="openai/gpt-4o-mini"
)

optimizer = EvolutionaryOptimizer(
    model="openai/gpt-4o-mini",
    population_size=5,
    num_generations=3
)

result = optimizer.optimize_prompt(
    prompt=prompt,
    dataset=dataset,
    metric=metric,
    n_samples=50
)

result.display()

Multi-prompt workflow

from typing import Any
from opik_optimizer import ChatPrompt, OptimizableAgent, HRPO
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer.datasets import hotpot
from openai import OpenAI

class TwoStepAgent(OptimizableAgent):
    def __init__(self, model: str = "gpt-4o-mini"):
        super().__init__()
        self.model = model
        self.client = OpenAI()

def invoke_agent(
        self,
        prompts: dict[str, ChatPrompt],
        dataset_item: dict[str, Any],
        allow_tool_use: bool = False,
        seed: int | None = None,
    ) -> str:
        # First step
        step1_prompt = prompts["step1"]
        step1_messages = step1_prompt.get_messages(dataset_item)
        step1_response = self.client.chat.completions.create(
            model=self.model,
            messages=step1_messages,
            seed=seed,
        )
        step1_result = step1_response.choices[0].message.content

# Second step uses result from first step
        step2_prompt = prompts["step2"]
        step2_context = {**dataset_item, "step1_result": step1_result}
        step2_messages = step2_prompt.get_messages(step2_context)
        step2_response = self.client.chat.completions.create(
            model=self.model,
            messages=step2_messages,
            seed=seed,
        )

return step2_response.choices[0].message.content

# Define multi-prompt workflow
prompts = {
    "step1": ChatPrompt(
        system="Analyze the question and identify key information.",
        user="{question}",
        model="gpt-4o-mini"
    ),
    "step2": ChatPrompt(
        system="Answer the question based on the analysis.",
        user="Question: {question}\n\nAnalysis: {step1_result}",
        model="gpt-4o-mini"
    ),
}

dataset = hotpot(count=300)

def metric(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item["answer"],
        output=llm_output
    )

optimizer = HRPO(
    model="openai/gpt-4o-mini",
    n_threads=2,
    max_parallel_batches=3
)

result = optimizer.optimize_prompt(
    prompt=prompts,
    agent_class=TwoStepAgent,
    dataset=dataset,
    metric=metric,
    max_trials=5,
    n_samples=50
)

result.display()

For advanced multi-prompt examples, see sdks/opik_optimizer/benchmarks/agents/hotpot_multihop_agent.py which implements a complex multi-hop retrieval pipeline with Wikipedia search.

Next steps