Skip to main content

Response Auditor

The Response Auditor is a post-generation quality audit pattern. It takes an AI response and runs it through one or more analyzers that detect problems — bias, hallucination, policy violations, whatever you configure. Each analyzer produces a structured result with a score, annotations, and a surface flag. The auditor collects results, computes an overall score, and filters to only surface findings above a configurable threshold.

Use it when:

  • You need to detect and surface quality issues in AI-generated responses
  • You want modular, pluggable analysis that can grow over time
  • You want structured audit data (not just pass/fail) for downstream decisions

If you need to rewrite the response based on findings, chain the auditor's output into a revision step. The auditor itself only analyzes — it doesn't modify.

The factory​

Import: @flow-state-dev/patterns/response-auditor

import { responseAuditor } from "@flow-state-dev/patterns/response-auditor";

const auditor = responseAuditor({
  analyzers: [biasAdapter, toneAnalyzer],
  threshold: 0.3,         // only surface results above this score (default 0.3)
  maxConcurrency: 2,      // parallel analyzer limit (default: all)
});

responseAuditor() returns a sequencer. Pass it an array of analyzer blocks and a threshold. The factory handles the rest: capturing context, fanning out to analyzers, catching failures, aggregating scores, and filtering results.

Pipeline internals​

input { userInput, response }
  → captureContext             (tap: store input in sequencer state, produces no output)
  → map to array               (one copy per analyzer)
  → forEach(safeAnalyzer)      (run analyzers concurrently, with .rescue() per analyzer)
  → filter nulls               (drop failed analyzers)
  → aggregateResults           (average score across all results)
  → applyThreshold             (split into results + surfacedResults)

Each analyzer is wrapped in a mini-sequencer with .rescue(), so individual failures don't crash the auditor. Failed analyzers produce null and are filtered out before aggregation.

Output shape:

{
  results: AnalyzerResult[],           // all analyzer results
  surfacedResults: AnalyzerResult[],   // filtered: score >= threshold OR shouldSurface
  overallScore: number,                // average of all analyzer scores
}

Basic usage​

Wire the auditor as a .sideChain() sidechain so the primary response streams unblocked:

import { defineFlow, sequencer, generator } from "@flow-state-dev/core";
import { responseAuditor } from "@flow-state-dev/patterns/response-auditor";
import { z } from "zod";

const chat = generator({
  name: "chat",
  model: "openai/gpt-5.5",
  prompt: "You are a helpful assistant.",
  user: (input) => input.message,
});

const auditor = responseAuditor({
  analyzers: [myAnalyzer],
  threshold: 0.3,
});

const pipeline = sequencer({
  name: "chat-with-audit",
  inputSchema: z.object({ message: z.string() }),
})
  .step(chat)
  .sideChain(
    (chatOutput, ctx) => ({
      userInput: String(ctx.parent?.input?.message ?? ""),
      response: typeof chatOutput === "string" ? chatOutput : JSON.stringify(chatOutput),
    }),
    auditor,
  );

The audit runs in the background while the user sees the chat response immediately. Downstream blocks or the client can check the audit result via getBlockOutput(auditor).

Rendering the audit​

getBlockOutput(auditor) is the data path for programmatic consumers. To show the audit in the UI, you don't wire anything per response. When at least one result surfaces (score above threshold, or an analyzer sets shouldSurface), the pattern emits an append-only audit-annotation component item carrying { results, surfacedResults, overallScore }. Any renderer registered for that component key draws the card.

The @flow-state-dev/ui registry ships that renderer, and chatAssistantRenderers includes it by default, so a chat surface built on those renderers shows the audit card with no extra code. If you're not using chatAssistantRenderers, install it standalone with fsdev ui add audit-annotation and register it under renderers.component["audit-annotation"].

The emitted item is append-only (no key), so each auditor invocation in a request renders its own card instead of overwriting a previous one. The getBlockOutput(auditor) path stays available for downstream blocks that branch on the score.

The AnalyzerResult contract​

Every analyzer must accept { userInput: string, response: string } and produce output conforming to this schema:

{
  analyzerId: string,          // unique identifier for the analyzer type
  category: string,            // domain: 'metacognition', 'compliance', 'tone', etc.
  score: number,               // 0-1 composite score
  shouldSurface: boolean,      // force-surface even if below threshold
  annotations: Array<{
    type: string,              // annotation category
    label: string,             // human-readable label
    severity: 'info' | 'warning' | 'critical',
    description: string,       // what was found
    evidence?: string,         // supporting text from the response
  }>,
  supplementary?: Record<string, unknown>,  // domain-specific extras
}

The shouldSurface flag lets an analyzer force its result into surfacedResults regardless of score. The supplementary field is a grab bag for domain-specific data that the UI can display (counter-arguments, sycophancy breakdowns, etc.).

Shipped analyzers​

Bias & sycophancy detection​

The biasAnalyzer() from @thought-fabric/core/metacognition detects agreement bias, cognitive biases, and sycophantic patterns. It outputs the thought-fabric AnalyzerResult schema, which is different from the patterns AnalyzerResult. You need an adapter to bridge the two.

The kitchen-sink reference app shows this pattern:

import { sequencer } from "@flow-state-dev/core";
import { biasAnalyzer } from "@thought-fabric/core/metacognition";
import { responseAuditor, auditorInputSchema } from "@flow-state-dev/patterns/response-auditor";
import { z } from "zod";

// Adapter: bridges biasAnalyzer output → patterns AnalyzerResult
const biasAdapter = sequencer({
  name: "bias-adapter",
  inputSchema: z.object({ userInput: z.string(), response: z.string() }),
})
  .map((input: { userInput: string; response: string }) => ({
    userInput: input.userInput,
    aiResponse: input.response,   // biasAnalyzer expects "aiResponse", not "response"
  }))
  .step(biasAnalyzer({ model: "openai/gpt-5.4-mini" }))
  .map((output: Record<string, unknown>) => {
    const annotations = (output.annotations as Array<Record<string, unknown>>) ?? [];
    const severity = output.severity as string;
    return {
      analyzerId: output.analyzerId as string,
      category: output.category as string,
      score: output.score as number,
      shouldSurface: (output.score as number) >= 0.3,
      annotations: annotations.map((a) => ({
        type: a.biasType as string,
        label: (a.biasType as string).replace(/_/g, " "),
        severity: severity as "info" | "warning" | "critical",
        description: a.description as string,
        evidence: a.evidence as string | undefined,
      })),
      supplementary: {
        summary: output.summary,
        label: output.label,
        sycophancyScore: output.sycophancyScore,
        counterArguments: output.counterArguments,
      },
    };
  });

const auditor = responseAuditor({
  analyzers: [biasAdapter],
  threshold: 0.3,
});

The bias analyzer pipeline: biasDetectAgreement → biasClassify → biasScore → biasCounterpoint → biasFormat. Three LLM calls (detect, classify, counterpoint) plus two deterministic handlers (score, format). Counter-argument generation is conditional — skipped below threshold.

Six bias types detected: sycophancy, confirmation bias, anchoring bias, authority deference, recency bias, false consensus.

Building custom analyzers​

Any block that accepts { userInput, response } and returns an AnalyzerResult works. The simplest approach: wrap a generator with a formatting handler.

import { generator, handler, sequencer } from "@flow-state-dev/core";
import { AnalyzerResultSchema, auditorInputSchema } from "@flow-state-dev/patterns/response-auditor";
import { z } from "zod";

const detect = generator({
  name: "tone/detect",
  model: "openai/gpt-5.4-mini",
  inputSchema: auditorInputSchema,
  outputSchema: z.object({
    tone: z.string(),
    confidence: z.number(),
    issues: z.array(z.string()),
  }),
  prompt: "Analyze the tone of the AI response. Flag condescending, dismissive, or unprofessional language.",
  user: (input) => `User: ${input.userInput}\n\nAI: ${input.response}`,
  // No `itemVisibility` — the generator runs silently and returns its typed
  // `block_trace` for the format handler to consume.
});

const format = handler({
  name: "tone/format",
  outputSchema: AnalyzerResultSchema,
  execute: (input) => ({
    analyzerId: "tone-audit",
    category: "communication",
    score: 1 - input.confidence,
    shouldSurface: input.issues.length > 0,
    annotations: input.issues.map((issue) => ({
      type: "tone",
      label: "Tone Issue",
      severity: input.issues.length > 2 ? "warning" as const : "info" as const,
      description: issue,
    })),
  }),
});

const toneAnalyzer = sequencer({ name: "tone", inputSchema: auditorInputSchema })
  .step(detect)
  .step(format);

Then compose it alongside the bias adapter:

const auditor = responseAuditor({
  analyzers: [biasAdapter, toneAnalyzer],
  threshold: 0.3,
});

Exports​

ExportKindPurpose
responseAuditor(config)factoryCreates the full auditor sequencer
AnalyzerResultSchemaschemaZod schema for analyzer output
AuditAnnotationSchemaschemaZod schema for individual annotations
auditorInputSchemaschema{ userInput: string, response: string }
responseAuditorStateSchemaschemaInternal sequencer state
captureContexthandlerStores input in sequencer state (tap — no output)
aggregateResultshandlerComputes average score across results
applyThreshold(n)factoryCreates threshold filter handler

The internal blocks (captureContext, aggregateResults, applyThreshold) are exported for flow authors who want to remix the pipeline with custom steps.

captureContext must be composed with .tap()

captureContext only writes to sequencer state; it returns nothing. Compose it with .tap(), which runs a block for its side effect and passes the untouched pipeline value through to the next step:

sequencer({ name: "my-auditor", inputSchema: auditorInputSchema, stateSchema: responseAuditorStateSchema })
  .tap(captureContext)   // state is written, { userInput, response } flows on
  .step(myCustomAnalyzer)

Composing it as .step(captureContext) hands undefined to the next step, because .step() forwards the block's output and there is no output. Earlier releases echoed the input back, so .step() appeared to work; that echo was removed because it duplicated the payload in the items log for no benefit.

Composability​

The response auditor is a sequencer, so it composes with the other patterns:

import { responseAuditor } from "@flow-state-dev/patterns/response-auditor";

const auditor = responseAuditor({ analyzers: [biasAdapter, toneAnalyzer] });

// Background sidechain — audit while user sees the response
pipeline.sideChain(auditor);

// Chain audit into a conditional revision
const selfCorrectingChat = sequencer({ name: "self-correct", inputSchema: chatInput })
  .step(chat)
  .sideChain(
    (output, ctx) => ({ userInput: ctx.parent?.input?.message, response: output }),
    auditor,
  )
  .stepIf(
    (_output, ctx) => {
      const audit = ctx.getBlockOutput(auditor);
      return audit?.overallScore >= 0.7;
    },
    revisionGenerator,
  );

See also​