> ## Documentation Index > Fetch the complete documentation index at: https://gladlabs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Module v1 # Module v1 — making the plugin substrate explicit **Status:** Phases 1, 2, 3-lite, 4-lite shipped 2026-05-13. **Update 2026-06-04 — Phase 3.5 (the physical pipeline-code move into `modules/content/`) shipped**; see CLAUDE.md's content-module section. Phases 4.5 / 5 remain deferred. The forward-looking implementation plan below is the original 2026-05-13 design — read its Phase 3 section as history. **Date:** 2026-05-13 (spec) / 2026-05-13 (implementation) **Author:** brainstormed with Matt 2026-05-13 16:48 UTC **Tracker:** [Glad-Labs/poindexter#490](https://github.com/Glad-Labs/poindexter/issues/490) (umbrella) **Supersedes (when implemented):** the informal "module = collection of contributions across 19 entry-point groups" pattern. ## What shipped 2026-05-13 End-to-end validated against a real second module in the operator-overlay slot (a private business module — see [Visibility](#visibility) below for how `visibility="private"` modules are filtered from the public mirror). | Phase | Status | What landed | | ------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Phase 1 | ✅ Full | `plugins/module.py` Protocol + `ModuleManifest` + `get_modules()` registry accessor with name + manifest validation + duplicate-drop + first-discovered-wins precedence. 5 unit tests pin the contract. | | Phase 2 | ✅ Full | `services/module_migrations.py` runner + `module_schema_migrations` table (compound key on `module_name, migration_name`) + boot wiring in `utils/startup_manager._run_migrations`. 6 unit + 2 integration\_db tests. | | Phase 3 | ✅ Full | `modules/content/` skeleton (`ContentModule`) shipped 2026-05-13 (3-lite); the physical pipeline-code move (**Phase 3.5**) followed 2026-06-04 — `content_validator`, `stages/`, `atoms/`, `multi_model_qa`, `ai_content_generator`, `internal_link_coherence`, `quality_service`, `auto_publish(_gate)` now live under `modules/content/`. The generic engine (`template_runner`, `pipeline_architect`, `prompt_manager`, `llm_text`, `atom_registry`, `canonical_blog_spec`) stays in substrate — content rents it via the DB `graph_def` seam. **Thin-adapter boundary** (`modules/content/api.py`) shipped PR #1389 — all 10 executable imports from substrate files now route through it; 3 string-path registries (`plugins/registry.py` `_SAMPLES`, `atom_registry.py` walk-root, `http_client.py` `WIRED_HTTP_CLIENT_MODULES`) remain out-of-scope (dynamic `importlib`, cannot route via Python import). | | Phase 4 | ⚠️ Lite | Route auto-discovery wired in `utils/route_registration.register_all_routes` — iterates `get_modules()` after substrate routes mount, calls each module's `register_routes(app)`. Grafana dashboard registration, CLI subparser registration, brain-probe registration deferred to **Phase 4.5**. | | Phase 5 | ⏭ Deferred | `visibility` flag drives `scripts/sync-to-github.sh` — currently the sync filter strips private modules via an explicit pattern list (works for n=2 modules; will refactor when n≥3). | **Concrete second module shipped under this pattern:** a `visibility="private"` business module in the operator overlay, filtered from the public mirror by `scripts/sync-to-github.sh`. The substrate landed by writing the second module against the freshly-shipped scaffolding, not by retrofitting old content code. **The cost-benefit takeaway from shipping all four phases in one day:** the Module v1 scaffolding (Phases 1 + 2) is small and high-leverage (\~250 LOC). The "lite" approach to Phases 3 + 4 avoids \~100 file import-path churn by deferring physical moves until a 3rd module concretely demands them. Total ratio: \~700 LOC of scaffolding + glue now supports a 1-day path to "add a new business module" (`modules//` + `_SAMPLES` registration + migrations). ## Why Poindexter's destination is a "cofounder OS run from phone-on-beach" (`project_poindexter_as_business_os`). Content generation is module 1 of N — finance, legal, HR, customer-support, and revenue modules are expected follow-ons (`project_llm_workforce_thesis`). Today's architecture has accreted three pain points that block adding module 2: 1. **Fuzzy module boundaries.** The content pipeline calls into `ai_content_generator.py` (1,331 LOC) which calls back into the stages tree; `brain/` has its own embeddings outside the main migration system (Glad-Labs/poindexter#328); the "Content Module" isn't a thing you can install or describe in one sentence. 2. **OSS / business mix is informal.** `scripts/sync-to-github.sh` filters by a hand-maintained path list (`web/public-site`, `web/storefront`, `mcp-server-gladlabs`, `marketing`, premium dashboards, `writing_samples`, `.shared-context`, `CLAUDE.md`). The line keeps moving; a new private-overlay file requires editing the sync filter, which is easy to forget. 3. **Pipeline orchestration drag.** A new module currently needs plumbing across 6+ surfaces — Prefect flow + Telegram command + MCP tool + Grafana panel + brain probe + CLI subcommand. Each surface has its own registration path. Adding a finance module feels like a five-day plumbing project. The redesign explicitly does **not** reshuffle the substrate (brain, Prefect, Langfuse, cost guard, Telegram bot, MCP server, Grafana, audit\_log, memory client). Those are working and load- bearing. The redesign makes the **module** abstraction first-class on top of that substrate. ## The 70% already built An audit of `src/cofounder_agent/plugins/` on 2026-05-13 found: * **Plugin registry exists** via setuptools `entry_points()` — `plugins/registry.py`, no custom registry code, same mechanism pytest / click / flask use. 19 entry-point groups defined: `taps`, `probes`, `jobs`, `stages`, `reviewers`, `adapters`, `providers`, `packs`, `llm_providers`, `topic_sources`, `image_providers`, `audio_gen_providers`, `video_providers`, `tts_providers`, `caption_providers`, `publish_adapters`. * **Plugin Protocol interfaces** are formalised with `@runtime_checkable` Protocols in `plugins/{tap,stage,reviewer, adapter,probe,job,llm_provider,pack,image_provider, audio_gen_provider,video_provider,tts_provider,caption_provider, media_compositor,publish_adapter,topic_source}.py`. Each has a typed `Result` / `Document` / `Completion` data class. * **Sample plugins** at `plugins/samples/` — `database_probe.py`, `hello_tap.py`, `noop_job.py` — work as reference implementations. * **Substrate services** are wired and load-bearing: `services/cost_guard.py`, `services/prompt_manager.py` (UnifiedPromptManager + Langfuse), `services/site_config.py` DI seam, `services/audit_log.py`, `services/memory_client.py`, the brain daemon (`brain/`). * **Operator surface** is in place: a single Telegram bot (`poindexter-pipeline-bot` container, 24/7), the MCP server (25 tools), the FastAPI worker, the `poindexter` CLI. * **Orchestration** is canonical: Prefect, as of the Stage 3 cutover (2026-05-13). * **Declarative data plane** — 5 tables (`external_taps`, `retention_policies`, `webhook_endpoints`, `publishing_adapters`, `qa_gates`) feed 14 handlers across 5 surfaces. What's missing is the **module-level** layer above these plugin groups. A "Content Module" today is implicit — N stages + M reviewers + K topic\_sources + L probes + … scattered across the registry. There is no manifest that says "these are its parts, this is its version, these are its dependencies, this is whether it's public or overlay-private." ## What changes — Module v1 in five components ### Component 1 — Module manifest + new entry-point group Add a new entry-point group, `poindexter.modules`. Each module is a Python package whose `pyproject.toml` declares **one** entry: ```toml theme={null} [project.entry-points."poindexter.modules"] content = "poindexter_module_content:ContentModule" ``` The target resolves to a `Module` instance. `plugins/module.py` defines the protocol: ```python theme={null} @runtime_checkable class Module(Protocol): """A self-contained business function (content, finance, HR, ...). A Module BUNDLES the lower-level plugin contributions (stages, reviewers, probes, jobs, taps, adapters, providers, packs) plus the things the existing registry doesn't track (DB migrations, Grafana panels, HTTP routes, CLI commands). """ name: str # canonical slug, e.g. "content" version: str # semver visibility: Literal["public", "private"] requires: list[str] # ["substrate>=1.0", "module:memory"] def manifest(self) -> ModuleManifest: ... async def migrate(self, pool: asyncpg.Pool) -> None: """Apply this module's DB migrations. Idempotent.""" def register_routes(self, app: FastAPI) -> None: """Mount this module's HTTP surface on the host app.""" def register_cli(self, parser: CLISubparser) -> None: """Register `poindexter ` entries.""" def register_dashboards(self, grafana: GrafanaProvisioner) -> None: """Contribute Grafana panels under a per-module folder.""" def register_probes(self, brain: BrainProbeRegistry) -> None: """Brain probes specific to this module.""" async def healthcheck(self, pool: asyncpg.Pool) -> ProbeResult: """Module-wide health (sums sub-probe results).""" ``` Importantly, the Module is *just a bundler*. The individual stages, reviewers, probes, etc. continue to be discovered through their existing entry-point groups. A Module's job is to be the unit of install / version / dependency / OSS-visibility. ### Component 2 — Per-module migrations Each module owns a `migrations/` subdirectory inside its package: ``` poindexter_module_content/ ├── __init__.py ├── module.py # ContentModule class ├── migrations/ │ ├── 20260514_120000_init.py │ ├── 20260520_080000_add_dev_diary_template.py │ └── ... ├── prompts/ ├── flows/ ├── routes/ ├── dashboards/ └── probes/ ``` `Module.migrate(pool)` walks the module's `migrations/` directory using the same idempotent runner as `services/migrations/` today, recording applied migrations in a `module_schema_migrations` table keyed on `(module_name, migration_name)` so two modules can have a migration named `init.py` without colliding. Global migrations under `services/migrations/` continue to exist — they apply to substrate tables (`app_settings`, `audit_log`, `embeddings`, `schema_migrations`, …) that all modules share. ### Component 3 — Per-module Grafana dashboards Each module places its dashboard JSON files under `dashboards/` in its package. At boot, the substrate's `GrafanaProvisioner` walks every loaded module, copies the dashboards into a per-module folder in Grafana (e.g. `/folder/content/`, `/folder/finance/`), and preserves a "Substrate" top-level folder for cross-module dashboards (Mission Control, System Health). The existing 7 dashboards under `infrastructure/grafana/` get split: Pipeline + Auto-Publish Gate + QA Rails → `content/` folder; Mission Control + Observability + System Health + Cost stays at the substrate root. ### Component 4 — HTTP route auto-discovery `Module.register_routes(app)` runs at lifespan startup. Each module's routes mount under a per-module path prefix derived from `Module.name`: ```python theme={null} # in poindexter_module_content.module def register_routes(self, app: FastAPI) -> None: from .routes import tasks, approval, posts app.include_router(tasks.router, prefix="/api/content/tasks", tags=["content"]) app.include_router(approval.router, prefix="/api/content/approval", tags=["content"]) app.include_router(posts.router, prefix="/api/content/posts", tags=["content"]) ``` The substrate's `main.py` no longer hardcodes route imports for business modules. It iterates the module registry and calls `register_routes` on each. ### Component 5 — `visibility` flag replaces the sync filter Each module declares `visibility: Literal["public", "private"]` on its manifest. The build process for the public mirror is now: 1. Walk `poindexter.modules` entry-points. 2. Include modules with `visibility="public"` in the OSS sync. 3. Exclude modules with `visibility="private"`. `scripts/sync-to-github.sh` becomes a thin loop over the module manifest rather than a hand-maintained path list. New private code is private by default (a new module is created in a private overlay package), and going public is a one-line manifest change plus a code review for what's in the package. ## The OSS / business split made concrete After Module v1: * **`poindexter` (public OSS)** ships: * the substrate (brain, prefect, langfuse, cost\_guard, prompt\_manager, site\_config, audit\_log, memory\_client, telegram-bot, MCP server, CLI) * the **`content` module** as the reference implementation: canonical\_blog template, multi-model QA, programmatic validator, image-gen/Pexels image stage, publish to a self-hosted Next.js + DB. * the **`base` module** that provides shared utilities every module needs (admin settings UI, audit\_log explorer, healthcheck aggregator). Could also be inlined into substrate; the spec treats it as a module for symmetry. * **`glad-labs-stack` (private overlay)** ships: * the **`gladlabs-business`** module (renamed from the current `mcp-server-gladlabs/` + `web/storefront/` + marketing scripts): Lemon Squeezy customer lookup, premium-prompts seeder, Glad-Labs brand assets, R2 license-delivery API, the storefront Next.js app. * `web/public-site/` stays here too (Vercel deploys from this repo). * the Glad-Labs Postgres + Grafana + Telegram bot instances — deployment artifacts that depend on these modules but aren't distributed. The line "is this poindexter-public or glad-labs-private?" becomes "which module owns it?" — and `Module.visibility` answers in one field. ## Migration path from today The work fits in 5 child issues. Each is independently shippable and provides observable value before the next one starts. Order matters — components 1 and 2 unblock the others. ### Phase 1 — Module manifest + registry (\~1 day) * Define `plugins/module.py` with the `Module` Protocol and a `ModuleManifest` data class. * Add the `poindexter.modules` entry-point group constant. * Extend `plugins/registry.py` with `get_modules() -> list[Module]`. * Pin behaviour with 5 unit tests (module discovery, manifest validation, dependency cycles). No business behaviour changes yet — this is *scaffolding*. ### Phase 2 — Per-module migration runner (\~1 day) * New `services/module_migrations.py` runner. Same shape as `services/migrations/__init__.py` but keyed on `(module_name, migration_name)`. * New `module_schema_migrations` table. * Boot wiring: after substrate migrations apply, iterate registered modules and call `await module.migrate(pool)`. ### Phase 3 — Convert content/ into a Module (\~1-2 days) * Create `poindexter_module_content/` package (could be in-tree initially under `src/poindexter_module_content/` then split out to its own repo when the substrate stabilises). * Move existing content-pipeline code: `services/content_router_service.py`, the stages tree, `multi_model_qa.py`, `content_validator.py`, the content-specific YAML prompts (the `qa.*` / `image.decision` / `topic.ranking` / `narrative.*` keys that landed via Lane A `Glad-Labs/poindexter#450`), the `canonical_blog` LangGraph template, the `content_generation` Prefect flow + deployment. `prompt_manager.py` (UnifiedPromptManager — the loader) stays in substrate; only the prompt files move. * Move content-specific migrations from `services/migrations/` to `poindexter_module_content/migrations/`. * Register the `ContentModule` via entry-point. * All existing tests keep passing; the change is structural, not behavioural. ### Phase 4 — Per-module routes + dashboards (\~1 day) * Refactor `main.py` to iterate `get_modules()` and call `register_routes(app)` for each. * Refactor `infrastructure/grafana/` into per-module folders. * Add `Module.register_dashboards(provisioner)` to seed. * Update Grafana provisioning workflow. ### Phase 4 (closeout) — `register_probes` wire-up (#239, 2026-05-27) * `plugins/probe_registry.py` defines `BrainProbeRegistry` — a worker-side collector that modules write into during `register_probes(registry)`. Rejects duplicate FQIDs per `feedback_no_silent_defaults`. * `main.py` lifespan constructs the shared registry, hands it to every discovered module, then stashes it on `app.state` for request handlers. * `GET /api/modules/probes` returns the registered probe specs (module / name / description / interval). 503 when the registry is missing — never silently empty. * Cross-process bridge to the brain daemon (poll the endpoint, run the probes on the brain's side) is intentionally deferred: no module ships a concrete probe yet, so the discovery half is enough to unblock the next module that needs one. ### Phase 5 — `visibility` + sync rewrite (\~0.5 day) * Add `visibility` field to manifest. * Rewrite `scripts/sync-to-github.sh` (or replace with a Python script) to iterate modules and include only `visibility="public"`. * Document the new sync contract in `docs/operations/oss-sync.md`. Total: \~5 days of focused work. After Phase 5, **adding a finance module is**: create `poindexter_module_finance/`, declare its entry-point, write its `migrate()` / `register_routes()` / `register_dashboards()` / `register_probes()`, ship. ## Non-goals The spec deliberately does NOT include: * **Module sandboxing / process isolation.** Modules run in the same Python process as today. If a future module needs isolation (third-party untrusted code), that's a separate spec. * **Module hot-reload.** Modules are discovered at boot. Adding a module still requires a restart. Hot-reload doesn't compose with Python's import semantics. * **A plugin marketplace.** Distribution is `pip install ` for now. A marketplace can come later if there's demand. * **Cross-module RPC.** Modules continue to communicate through shared substrate (Postgres, audit\_log, brain knowledge graph) — no new RPC layer. * **Versioned migrations across modules.** A module that depends on another module's tables can declare `requires: ["module:content>=1.0"]` but the runner does not yet enforce migration ordering across modules. If module B needs a table from module A, ensure A is installed first. This may need revisiting once we have 3+ modules. ## Testing strategy Each phase ships with tests: * **Phase 1** — `tests/unit/plugins/test_module_registry.py`: 5 cases covering discovery, manifest validation, missing fields, duplicate module names, dependency cycle detection. * **Phase 2** — `tests/integration_db/test_module_migrations.py`: fresh `poindexter_test_` DB, register two test modules, verify both their migrations apply, verify `module_schema_migrations` rows land correctly, verify reruns are no-ops. * **Phase 3** — `tests/integration_db/test_content_module_e2e.py`: with the content module registered, run one pipeline\_tasks row end-to-end through the Prefect flow. Should produce the same awaiting\_approval outcome as today's path. Pin the contract. * **Phase 4** — `tests/unit/routes/test_module_route_discovery.py`: spin up a `TestClient` with two test modules registered, verify both modules' routes are mounted at their declared prefixes. * **Phase 5** — `tests/unit/scripts/test_visibility_filter.py`: feed the sync filter a fake module registry with mixed visibility, verify only `visibility="public"` modules survive the filter. The integration\_db harness (fixed 2026-05-10 via `fix(migrations): reconcile embeddings column drift on stripped DB`, commit 4330e59f) gives us the per-test disposable Postgres we need for Phase 2 + 3. ## Open questions / future work * ~~**Naming.** Is `Module` the right word, or does it collide with Python's existing `module` semantics in confusing ways? Maybe `Workspace` or `Domain` or `Slice`? Defer until Phase 1 review.~~ **Resolved 2026-05-13:** keep `Module`. Matt: "modules in reference to Python files etc. is very broad — we'll know the difference." * **Sub-modules.** Should `gladlabs-business` further decompose into `gladlabs-storefront`, `gladlabs-licenses`, `gladlabs-premium`? Not now — get one private overlay module shipping first. * **In-tree vs out-of-tree packages.** Phase 3 puts the content module package in-tree (`src/poindexter_module_content/`) for convenience. Once the contract is stable, extracting to its own repo + `pip install` is straightforward. Defer the split until there are 2 in-tree modules. ## Ground truth references * `src/cofounder_agent/plugins/registry.py` — existing entry-point discovery, basis for Component 1. * `src/cofounder_agent/plugins/{tap,stage,probe,job,...}.py` — existing plugin Protocols that modules will bundle. * `src/cofounder_agent/services/migrations/__init__.py` — runner shape that Component 2 will mirror. * `scripts/sync-to-github.sh` — sync filter Component 5 replaces. * `CLAUDE.md` — operator-facing description; will need an update when Phase 3 lands.