> ## Documentation Index
> Fetch the complete documentation index at: https://gladlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi model qa

# Multi-Model QA

**File:** `src/cofounder_agent/modules/content/multi_model_qa.py`
**Tested by:** `src/cofounder_agent/tests/unit/services/test_multi_model_qa.py`
**Last reviewed:** 2026-06-22

## What it does

`MultiModelQA.review()` runs a draft through several reviewers — a
deterministic programmatic validator, an LLM critic that uses a
different model than the writer, and a fan of opt-in gates (citation
verifier, topic-delivery, internal-consistency, image relevance, web
fact-check, URL verification, rendered-preview screenshot). Each
reviewer returns a `ReviewerResult`; the aggregator weighs the scores,
applies validator-warning penalties, and decides `approved=True/False`
based on a configurable threshold (default `qa_final_score_threshold=70`).

The point is adversarial coverage. Different reviewers have different
blind spots: regex catches what LLMs miss, a Claude critic catches what
local Ollama misses, web fact-check catches post-cutoff product claims
the LLM critic would otherwise reject as fabricated. The aggregator
trusts the score over individual `approved` booleans because critics
nitpick approval status more than they nitpick the number.

## Public API

* `MultiModelQA(pool, settings_service=None)` — constructor.
* `await qa.review(title, content, topic="", research_sources=None, preview_url=None) -> MultiModelResult` —
  full review, returns the aggregated decision.
* `MultiModelResult.approved: bool` — final decision.
* `MultiModelResult.final_score: float` — weighted-average score (0-100).
* `MultiModelResult.reviews: list[ReviewerResult]` — per-reviewer detail.
* `MultiModelResult.summary: str` — single-line human summary.
* `MultiModelResult.format_feedback_text(max_chars=4000) -> str` —
  human-readable critique for the approval UI; lands in
  `pipeline_tasks.qa_feedback` and `pipeline_versions.qa_feedback`.
* `format_qa_feedback_from_reviews(qa_reviews, final_score, approved, max_chars)` —
  module-level helper for callers that hold the serialized review
  dicts (e.g. finalize) without reconstructing the `MultiModelResult`.
* `ReviewerResult(reviewer, approved, score, feedback, provider)` —
  per-reviewer record; `provider` is one of `programmatic`, `ollama`,
  `anthropic`, `consistency_gate`, `vision_gate`, `web_factcheck`,
  `url_verifier`, `http_head`.

## Configuration

All settings come from `app_settings` via the injected `settings_service`.

Aggregation:

* `qa_validator_weight` (default `0.4`) — programmatic validator weight.
* `qa_critic_weight` (default `0.6`) — Ollama/Anthropic/Google critic weight.
* `qa_gate_weight` (default `0.3`) — weight for consistency, vision,
  web fact-check, URL gates.
* `qa_final_score_threshold` (default `70`) — approval cut-off.
* `qa_consistency_veto_threshold` (default `50`) — consistency gate is
  advisory unless its score drops below this.
* `content_validator_warning_qa_penalty` (default `3` points/warning) —
  per-warning penalty applied to the final aggregated score (GH-91).

Critic + writer:

* `pipeline_critic_model` (default `gemma3:27b`) — primary critic.
* `qa_fallback_critic_model` (default `gemma3:27b`) — fallback when the
  primary returns empty or errors.
* `qa_thinking_model_max_tokens` (default `8000`) — used for thinking
  models like `glm-4.7`, `qwen3:30b`.
* `qa_standard_max_tokens` (default `1500`) — non-thinking models.
* `qa_temperature` (default `0.3`) — both critic and gates.
* `electricity_rate_kwh` — local critic cost telemetry input.

Gates (mostly opt-in):

* `qa_web_factcheck_enabled` (default `true`) — DuckDuckGo verification
  of product/spec claims.
* `qa_web_factcheck_match_ratio` (default `0.6`) — fraction of a claim's
  key terms that must appear in the search snippets to mark it VERIFIED.
* `qa_web_factcheck_num_results` / `_snippet_chars` / `_min_term_len` /
  `_max_claims` (defaults `3` / `500` / `2` / `3`) — search breadth and the
  claim-matching heuristics (previously hardcoded literals in the rail).
* `qa_citation_verify_enabled` (default `true`) — HTTP HEAD for cited URLs.
* `qa_citation_max_dead_ratio` (default `0.30`).
* `qa_citation_min_count` (default `0`).
* `qa_citation_timeout_seconds` (default `8.0`).
* `qa_vision_check_enabled` (default `false`) — inline image relevance
  via vision model (\~10s/image).
* `qa_vision_model` (default `qwen3-vl:30b`).
* `qa_vision_max_images` (default `3`).
* `qa_vision_pass_threshold` (default `60`).
* `qa_vision_num_predict` (default `1024`) — output-token budget for the
  vision legs (shared by image-relevance + rendered-preview).
* `qa_vision_thinking_num_predict` (default `8000`) — the larger budget a
  *thinking* vision model (e.g. `qwen3-vl`) gets so its `<think>` trace does
  not exhaust the budget before the JSON scores are emitted.
* `qa_preview_screenshot_enabled` (default `false`) — full-page
  screenshot via Playwright + vision review.
* `qa_preview_vision_model` (default `qwen3-vl:30b`).
* `qa_preview_pass_threshold` (default `70`).
* `qa_preview_viewport_width/height` (defaults `1280` × `1024`).
* `qa_gate_max_tokens` / `qa_gate_timeout_seconds` (defaults `600` / `60`).
* `site_url`, `site_domain` — used by URL verifier to distinguish
  external citations from self-links.

## Dependencies

* **Reads from:**
  * `modules.content.content_validator.validate_content` and
    `verify_content_urls` (programmatic layer).
  * `services.citation_verifier` (HTTP HEAD path).
  * `services.web_research.WebResearcher` (DuckDuckGo fact check).
  * `services.preview_screenshot.capture_preview_screenshot` (Playwright).
  * `services.ollama_client.OllamaClient` (deliberately concrete — it
    exposes `configure_electricity` + `check_health` features the
    Provider Protocol does not).
  * `services.cost_lookup` for per-token cost; the critic model is the `pipeline_critic_model` per-step pin (the `cost_tier.*` resolution was removed in PR #1907 — it had replaced the deleted `services.model_router.get_model_router` after the 2026-05-08 Phase 2 cleanup). See [`../cost-tier-routing.md`](../cost-tier-routing.md).
* **Writes to:**
  * `audit_log` via `audit_log_bg` for `critic_fallback` events.
  * Cost rows return through the caller (the `qa.*` rail atoms that
    replaced the `cross_model_qa` stage in #355) which persists them to
    `cost_logs`.
* **External APIs:**
  * Local Ollama HTTP (critic, gates, vision, web-factcheck post-processing).
  * Outbound HTTP HEAD/GET for citation + URL verification + image download.
  * DuckDuckGo via `WebResearcher`.

## Failure modes

* **Programmatic validator critical issue (non-fact)** — short-circuits
  the entire review. Returns `approved=False`. Diagnose via
  `validation.issues`. The only "critical" exemption is
  `known_wrong_fact`, which gets a second chance from the web
  fact-check gate.
* **Ollama unreachable / health check times out** — critic + every gate
  silently skip. `approved` falls back to validator-only score.
  Visible in logs as `[MULTI_QA] Ollama not available` or `health check
  timed out after 5s`. Watch the `critic_fallback` audit event for
  primary-critic failures that triggered the fallback chain.
* **Critic returns unparseable JSON** — logged as warning with first
  200 chars; reviewer skipped. If both primary and fallback
  unparseable, `critic_skipped=True` and the final score uses the
  validator only.
* **Dead links / fabricated URLs** — `url_verifier` returns
  `approved=False, score=max(0, 100 - 20*dead_count)`. Hard-blocks publish.
* **Vision model OOM / timeout** — gates return `None`, no veto
  applied. Increase `qa_gate_timeout_seconds` or disable the gate.
* **Vision gate "passed open" (`vision_scorer_unavailable` finding)** —
  the `qa.vision` atom failed open (didn't block) because neither the
  image-relevance nor the rendered-preview leg produced a *scoreable*
  verdict. This does **not** mean the vision model is down: on the observed
  runs (2026-07-12) it acquired the GPU and spent thousands of tokens, then
  the leg still returned no scores. **Root cause:** `qwen3-vl` is a *thinking*
  model whose `<think>` trace shares the `num_predict` budget with the JSON
  answer; at the 1024 base the trace exhausts the budget before the scores
  are emitted, so the leg returns `None`. Fixed by
  `_maybe_bump_vision_thinking_budget`, which raises the budget to
  `qa_vision_thinking_num_predict` (default 8000) for thinking vision models,
  mirroring the text critic. Every no-verdict path also logs a specific
  `[VISION_QA] …` **WARNING** (shipped to Loki) naming the cause (unreachable
  model, unparseable/empty response, unreadable images, missing handle).
  Defense-in-depth: the cost-logging pool falls back to `site_config._pool`
  via the shared `resolve_pool` helper (with a loud `[POOL]` warning) when the
  threaded `database_service` carries no live pool — so a genuine handle gap is
  caught, not silent. NB: the earlier "pool is None" diagnosis was a
  `cost_logs` NULL-`task_id` artifact (the dispatch did happen). Triage: read
  the `[VISION_QA]` / `[POOL]` WARNING for the run, not just the page.
* **Web fact-check rate-limited** — DuckDuckGo errors are caught and
  logged as `[WEB_FACTCHECK] Failed (non-fatal)`. Reviewer skipped.

## Common ops

* **Soften the critic for a misbehaving niche:** raise
  `qa_final_score_threshold` (e.g. to 60) or lower `qa_critic_weight`.
* **Swap critic model:** `poindexter settings set pipeline_critic_model
  ollama/glm-4.7` (or `anthropic/claude-haiku-4-5` for cloud — note
  cost guard interactions).
* **Disable vision QA:** `qa_vision_check_enabled=false` (default).
* **Enable rendered-preview gate (after Playwright install):**
  `qa_preview_screenshot_enabled=true` and ensure the calling stage
  passes `preview_url=/preview/{hash}`.
* **Audit recent critic fallbacks:**
  `SELECT created_at, payload FROM audit_log WHERE event_type = 'critic_fallback' ORDER BY created_at DESC LIMIT 50;`
* **Read the full feedback for a specific task:**
  `SELECT qa_feedback FROM pipeline_tasks WHERE task_id = '<uuid>';`

## See also

* `docs/architecture/anti-hallucination.md` — full reviewer catalogue +
  pipeline ordering.
* `docs/architecture/services/content_router_service.md` — the calling
  pipeline.
* `project_qa_critic_cutoff` (operator design note)
  — why the web fact-check gate exists.
* `project_multi_model_qa` (operator design note)
  — design vision for adversarial multi-model review.