# The Opik Agent Optimizer

The Opik Agent Optimizer can optimize both simple prompts and complex agent workflows. For most use cases, you can optimize prompts directly using `ChatPrompt`. When you need multi-prompt workflows, agent orchestration, or custom execution logic, you’ll use `OptimizableAgent` to create a custom agent class.

## When to use OptimizableAgent vs ChatPrompt

**Use `ChatPrompt` directly** (default approach):

- Single-prompt optimization - optimizing one prompt template
- Most common use case
- No custom execution logic needed

**Use `OptimizableAgent`** when you need:

- Multi-prompt workflows - orchestrating multiple prompts in sequence
- Agent framework integration - connecting to ADK, LangGraph, CrewAI, etc.
- Custom execution logic - special tool handling, async workflows, etc.

Optimizers work seamlessly with both approaches. The optimizer calls your agent’s `invoke_agent()` method repeatedly during optimization, passing different prompt candidates to evaluate.

## Single-prompt optimization

For most optimization tasks, you can use `ChatPrompt` directly without creating a custom agent. The optimizer uses a default LiteLLM-based agent under the hood.

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer.datasets import hotpot

dataset = hotpot(count=300)

def levenshtein_ratio(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item["answer"],
        output=llm_output
    )

prompt = ChatPrompt(
    system="You are a helpful assistant.",
    user="{question}",
    model="openai/gpt-4o-mini"
)

optimizer = MetaPromptOptimizer(model="openai/gpt-4o")
result = optimizer.optimize_prompt(
    prompt=prompt,
    dataset=dataset,
    metric=levenshtein_ratio,
    max_trials=5,
    n_samples=50
)

result.display()
```

### Custom agent for framework integration

When integrating with specific agent frameworks (Google ADK, LangGraph, CrewAI, etc.), you’ll create a custom `OptimizableAgent` subclass. This allows the optimizer to work with your framework’s execution model.

Here’s an example for Google ADK:

```python
from typing import Any, TYPE_CHECKING
from opik_optimizer import OptimizableAgent

if TYPE_CHECKING:
    from opik_optimizer.api_objects import chat_prompt

class ADKAgent(OptimizableAgent):
    project_name = "adk-agent"

def invoke_agent(
        self,
        prompts: dict[str, chat_prompt.ChatPrompt],
        dataset_item: dict[str, Any],
        allow_tool_use: bool = False,
        seed: int | None = None,
    ) -> str:
        # Single-prompt agents extract the prompt from the dict
        if len(prompts) > 1:
            raise ValueError("ADKAgent only supports single-prompt optimization.")

prompt = list(prompts.values())[0]
        messages = prompt.get_messages(dataset_item)

# Your framework-specific execution logic here
        # ... create ADK agent, run it, return response ...

return response
```

### Multi-prompt optimization

For multi-step agent workflows, you **must** use `OptimizableAgent` because `ChatPrompt` only handles a single prompt. Multi-prompt optimization allows you to optimize multiple prompts that work together in a pipeline.

### When to use multi-prompt optimization

- Sequential reasoning workflows (analyze → respond)
- Multi-hop retrieval pipelines
- Agent orchestration with multiple steps
- Any workflow where one prompt’s output feeds into another

### Implementing a multi-prompt agent

Here’s a simple example of a two-step workflow that analyzes input and then generates a response:

```python
from typing import Any
from opik_optimizer import ChatPrompt, OptimizableAgent
from openai import OpenAI

class AnalyzeRespondAgent(OptimizableAgent):
    """Two-step agent: analyze input, then respond based on analysis."""

def __init__(self, model: str = "gpt-4o-mini"):
        super().__init__()
        self.model = model
        self.client = OpenAI()

def invoke_agent(
        self,
        prompts: dict[str, ChatPrompt],
        dataset_item: dict[str, Any],
        allow_tool_use: bool = False,
        seed: int | None = None,
    ) -> str:
        # Step 1: Analyze the input
        analyze_prompt = prompts["analyze"]
        analyze_messages = analyze_prompt.get_messages(dataset_item)

analyze_response = self.client.chat.completions.create(
            model=self.model,
            messages=analyze_messages,
            seed=seed,
        )
        analysis = analyze_response.choices[0].message.content

# Step 2: Generate response based on analysis
        respond_prompt = prompts["respond"]
        # Pass analysis result to the respond prompt
        respond_context = {**dataset_item, "analysis": analysis}
        respond_messages = respond_prompt.get_messages(respond_context)

respond_response = self.client.chat.completions.create(
            model=self.model,
            messages=respond_messages,
            seed=seed,
        )

return respond_response.choices[0].message.content
```

### Using the multi-prompt agent

When optimizing, pass a dictionary of prompts instead of a single prompt:

```python
from opik_optimizer import ChatPrompt, MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio

# Define both prompts in the workflow
prompts = {
    "analyze": ChatPrompt(
        system="You are an analysis assistant. Extract key information from the input.",
        user="{text}",
        model="gpt-4o-mini"
    ),
    "respond": ChatPrompt(
        system="You are a response assistant. Generate a helpful response based on the analysis.",
        user="Analysis: {analysis}\n\nOriginal question: {text}",
        model="gpt-4o-mini"
    ),
}

optimizer = MetaPromptOptimizer(model="openai/gpt-4o")
result = optimizer.optimize_prompt(
    prompt=prompts,  # Pass dict of prompts
    agent_class=AnalyzeRespondAgent,  # Use your custom agent
    dataset=dataset,
    metric=levenshtein_ratio,
    max_trials=5,
    n_samples=50
)

result.display()
```

## Key implementation details

### invoke_agent() method signature

All `OptimizableAgent` subclasses must implement `invoke_agent()`:  
```
def invoke_agent(
    self,
    prompts: dict[str, ChatPrompt],
    dataset_item: dict[str, Any],
    allow_tool_use: bool = False,
    seed: int | None = None,
) -> str:
    # Your implementation here
    return response_string
```

**Parameters:**

- `prompts`: Dictionary mapping prompt names to `ChatPrompt` objects
- `dataset_item`: Dataset row used to format prompt messages
- `allow_tool_use`: Whether tools may be executed (for tool-calling prompts)
- `seed`: Optional random seed for reproducibility

**Returns:** A single string output that will be scored by your metric function

### Extracting messages from prompts

Use `ChatPrompt.get_messages()` to format the prompt with dataset values:
```
messages = prompt.get_messages(dataset_item)
# Returns list of message dicts: [{"role": "system", "content": "..."}, ...]
```
For multi-prompt workflows, pass additional context when calling `get_messages()`:
```
# Pass intermediate results to subsequent prompts
context = {**dataset_item, "intermediate_result": some_value}
messages = prompt.get_messages(context)
```

### Best practices

- **Error handling**: Return meaningful error messages if execution fails
- **Model parameters**: Respect `prompt.model` and `prompt.model_kwargs` for consistency
- **Reproducibility**: Use the `seed` parameter when making LLM calls
- **Opik tracing**: The base class handles tracing automatically, but you can add custom metadata via `self.trace_metadata`

## Complete examples

### Single-prompt with ChatPrompt (default)
```
from opik_optimizer import ChatPrompt, EvolutionaryOptimizer
from opik_optimizer.datasets import hotpot
from opik.evaluation.metrics import LevenshteinRatio

dataset = hotpot(count=300)

def metric(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item["answer"],
        output=llm_output
    )

prompt = ChatPrompt(
    system="You are a helpful assistant.",
    user="{question}",
    model="openai/gpt-4o-mini"
)

optimizer = EvolutionaryOptimizer(
    model="openai/gpt-4o-mini",
    population_size=5,
    num_generations=3
)

result = optimizer.optimize_prompt(
    prompt=prompt,
    dataset=dataset,
    metric=metric,
    n_samples=50
)

result.display()
```

### Multi-prompt workflow
```
from typing import Any
from opik_optimizer import ChatPrompt, OptimizableAgent, HRPO
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer.datasets import hotpot
from openai import OpenAI

class TwoStepAgent(OptimizableAgent):
    def __init__(self, model: str = "gpt-4o-mini"):
        super().__init__()
        self.model = model
        self.client = OpenAI()

def invoke_agent(
        self,
        prompts: dict[str, ChatPrompt],
        dataset_item: dict[str, Any],
        allow_tool_use: bool = False,
        seed: int | None = None,
    ) -> str:
        # First step
        step1_prompt = prompts["step1"]
        step1_messages = step1_prompt.get_messages(dataset_item)
        step1_response = self.client.chat.completions.create(
            model=self.model,
            messages=step1_messages,
            seed=seed,
        )
        step1_result = step1_response.choices[0].message.content

# Second step uses result from first step
        step2_prompt = prompts["step2"]
        step2_context = {**dataset_item, "step1_result": step1_result}
        step2_messages = step2_prompt.get_messages(step2_context)
        step2_response = self.client.chat.completions.create(
            model=self.model,
            messages=step2_messages,
            seed=seed,
        )

return step2_response.choices[0].message.content

# Define multi-prompt workflow
prompts = {
    "step1": ChatPrompt(
        system="Analyze the question and identify key information.",
        user="{question}",
        model="gpt-4o-mini"
    ),
    "step2": ChatPrompt(
        system="Answer the question based on the analysis.",
        user="Question: {question}\n\nAnalysis: {step1_result}",
        model="gpt-4o-mini"
    ),
}

dataset = hotpot(count=300)

def metric(dataset_item, llm_output):
    return LevenshteinRatio().score(
        reference=dataset_item["answer"],
        output=llm_output
    )

optimizer = HRPO(
    model="openai/gpt-4o-mini",
    n_threads=2,
    max_parallel_batches=3
)

result = optimizer.optimize_prompt(
    prompt=prompts,
    agent_class=TwoStepAgent,
    dataset=dataset,
    metric=metric,
    max_trials=5,
    n_samples=50
)

result.display()
```

For advanced multi-prompt examples, see `sdks/opik_optimizer/benchmarks/agents/hotpot_multihop_agent.py` which implements a complex multi-hop retrieval pipeline with Wikipedia search.

## Next steps

- Explore [optimization algorithms](/content/docs/opik/development/optimization-runs/algorithms/overview/index.html) to choose the right optimizer
- Learn about [defining datasets](/content/docs/opik/development/optimization-runs/optimization/define_datasets/index.html) and [metrics](/content/docs/opik/development/optimization-runs/optimization/define_metrics/index.html)
- Check framework-specific examples in `sdks/opik_optimizer/scripts/llm_frameworks/`