docs/advanced-pipeline-examples.md

# Advanced Pipeline Examples

This guide shows how to implement and validate sophisticated multi-step pipelines for quizbowl agents.

## Goals
Using advanced pipelines, you will:
- Improve accuracy by 15-25% over single-step agents
- Create specialized components for different tasks
- Implement effective confidence calibration
- Build robust buzzer strategies

## Two-Step Justified Confidence Pipeline

### Baseline Performance
Standard single-step agents typically achieve:
- Accuracy: ~65-70%
- Poorly calibrated confidence
- Limited explanation for answers

### Loading the Pipeline Example

1. Navigate to the "Tossup Agents" tab
2. Click "Select Pipeline to Import..." and choose "two-step-justified-confidence.yaml"
3. Click "Import Pipeline"

### Understanding the Pipeline Structure

This pipeline has two distinct steps:

#### Step A: Answer Generator
- Uses OpenAI/gpt-4o-mini
- Takes question text as input
- Generates an answer candidate
- Focuses solely on accurate answer generation

#### Step B: Confidence Evaluator
- Uses Cohere/command-r-plus
- Takes question text AND generated answer from Step A
- Evaluates confidence and provides justification
- Specialized for confidence assessment

### Validation
Test the pipeline and check:
- Is accuracy improved compared to single-step?
- Are confidence scores better calibrated?
- Does the justification explain reasoning clearly?

### Results
Two-step justified confidence typically achieves:
- Accuracy: ~80-85%
- Well-calibrated confidence scores
- Clear justification for answers and confidence
- More strategic buzzing

## Enhancing the Two-Step Pipeline

### Step 1: Upgrade Answer Generator

#### Current Performance
The default example uses gpt-4o-mini which may lack:
- Specialized knowledge in some areas
- Consistent answer formatting

#### Implementation
1. Click on Step A
2. Change model to a stronger option (e.g., gpt-4o)
3. Modify system prompt to focus on answer precision

#### Validation
Test with sample questions and check:
- Has answer accuracy improved?
- Is formatting more consistent?

#### Results
With upgraded answer generator:
- Accuracy increases to ~85-90%
- More consistent answer formats

### Step 2: Improve Confidence Evaluator

#### Current Performance
The default evaluator may:
- Over-estimate confidence on some topics
- Provide limited justification

#### Implementation
1. Click on Step B
2. Enhance the system prompt:
```
You are an expert confidence evaluator for quizbowl answers.

Your task:
1. Evaluate ONLY the correctness of the provided answer
2. Consider question completeness and available clues
3. Provide specific justification for your confidence score
4. Be especially critical of answers with limited supporting evidence

Remember:
- Early, difficult clues justify lower confidence
- Later, obvious clues justify higher confidence
- Domain expertise should be reflected in your assessment
```

#### Validation
Test and verify:
- Are confidence scores better aligned with correctness?
- Does justification include specific clues from questions?
- Is confidence calibrated appropriately for question position?

#### Results
With improved evaluator:
- More accurate confidence calibration
- Detailed justifications citing specific clues
- Better buzzing decisions

## Three-Step Pipeline with Analysis

### Concept
Adding a dedicated analysis step before answer generation:

1. **Step A: Question Analyzer**
   - Identifies key clues, entities, and relationships
   - Determines question category and format
   
2. **Step B: Answer Generator**
   - Uses analysis to generate accurate answers
   - Focuses on formatting and precision
   
3. **Step C: Confidence Evaluator**
   - Assesses answer quality based on analysis and clues
   - Determines optimal buzz timing

### Implementation
Create this pipeline from scratch or modify the two-step example.

### Validation
Compare to the two-step pipeline:
- Does the analysis step improve answer accuracy?
- Does it provide better performance on difficult questions?
- Are there improvements in early buzzing?

### Results
Three-step pipelines typically achieve:
- Accuracy: ~90-95%
- Earlier correct buzzes
- Exceptional performance on difficult questions

## Specialty Pipeline: Literature Focus

### Concept
Create a pipeline specialized for literature questions:

1. **Step A: Literary Analyzer**
   - Identifies literary techniques, periods, and styles
   - Recognizes author-specific clues
   
2. **Step B: Answer Generator**
   - Specialized for literary works and authors
   - Formats answers according to literary conventions
   
3. **Step C: Confidence Evaluator**
   - Calibrated specifically for literature questions

### Implementation
Create specialized system prompts for each step focusing on literary knowledge.

### Validation
Test specifically on literature questions and compare to general pipeline.

### Results
Specialty pipelines can achieve:
- 95%+ accuracy in their specialized domain
- Earlier buzzing on category-specific questions
- Better performance on difficult clues

## Best Practices for Advanced Pipelines

1. **Focused Components**: Each step should have a clear, single responsibility
2. **Efficient Communication**: Pass only necessary information between steps
3. **Strong Fundamentals**: Start with a solid two-step pipeline before adding complexity
4. **Consistent Testing**: Validate each change against the same test set
5. **Strategic Model Selection**: Use different models for tasks where they excel