Meta-reasoning is the practice of reasoning about reasoning. In the context of LLM workflows, it means systematically capturing, analyzing, and optimizing how AI systems solve problems.

Traditional LLM usage follows a simple pattern: prompt in, response out. Meta-reasoning adds a layer of introspection that enables:

A meta-reasoning system consists of four interconnected components:

These components form a feedback loop. Captured metadata supports diagnosis. Evaluation results measure each strategy change.

A trace is a best-effort record of observable runtime events. It is not hidden model reasoning, a complete replay, or an audit ledger.

Current Versalist Boundary

• Capture is off by default and can be limited by Episode run type.
• Captured events include agent turns, agent model calls, and judge calls.
• Events can include status, model identity, latency, token counts, and hashes.
• Trace event rows do not include raw prompts, completions, or evaluator reasoning.
• The current runtime does not emit tool, sandbox, approval, or subagent events.

LLM outputs are inherently variable. Deterministic evaluation adds consistency by measuring outputs against defined criteria. This creates a ground truth for quality that doesn't depend on subjective judgment.

Three Layers of Evaluation

Evaluation Results

Each evaluation produces a result object containing:

• success: Boolean indicating if all rules passed
• failureReasons: Array of rule violations
• qualityMetrics: Scores for each defined metric
• schemaValidationPassed: Whether Zod validation succeeded

Different tasks benefit from different prompting approaches. A "strategy" encapsulates a specific approach: the prompt template, decomposition steps, and applicability conditions.

What is a Strategy?

{
  name: 'Technical Deep-Dive',
  taskType: 'challenge_generation',
  promptTemplate: `You are a Senior AI Engineer...
    Focus on API-level implementation details...`,
  applicabilityConditions: {
    categories: ['AI Development', 'Agent Building'],
    difficulty: ['intermediate', 'advanced']
  },
  isActive: true,
  successCount: 45,
  failureCount: 12
}

How Selection Works

1. Context Matching: Find strategies where applicability conditions match the current task context.
2. Performance Ranking: Among matching strategies, rank by success rate (success_count / total_uses).
3. Selection: Choose the highest-performing strategy, with some exploration for new strategies.

Optimization Loop

The system continuously improves through a simple feedback loop:

This example shows strategy selection and outcome recording. The current product does not expose a meta-reasoning trace API.

import { getMetaReasoning } from '@/lib/meta-reasoning';
import { challengeEvaluator } from '@/lib/evaluators/challenge-evaluator';

export async function generateChallengeWithMetaReasoning(input: {
  title: string;
  category: string;
  difficulty: string;
}) {
  const mr = getMetaReasoning();

  // 1. Select optimal strategy based on context
  const { strategy } = await mr.selectStrategy('challenge_generation', {
    category: input.category,
    difficulty: input.difficulty,
  });

  try {
    // 2. Generate with the selected strategy.
    const prompt = strategy?.promptTemplate || defaultPrompt;
    const result = await llm.generate(prompt, input);

    // 3. Evaluate the output.
    const evaluation = challengeEvaluator.evaluate(result);

    // 4. Record the strategy outcome.
    if (strategy?.id) {
      await mr.recordOutcome(strategy.id, evaluation.success);
    }

    return {
      result,
      evaluation,
      strategyUsed: strategy?.name || 'default',
    };
  } catch (error) {
    throw error;
  }
}

Meta-reasoning enables continuous improvement through data-driven iteration:

Key Metrics to Track

• Evaluation Pass Rate: % of generations passing all rules
• Average Quality Score: Mean of quality metrics across generations
• Strategy Distribution: Which strategies are being selected
• Latency P95: 95th percentile generation time
• Token Efficiency: Output quality per token spent