SterlingIndex Docs

How services/platform-api boots, routes work, runs workers, persists state, and verifies backend behavior.

services/platform-api is the control plane and backend runtime behind Sterling's roster discovery, extraction, validation, scheduling, and product-data surfaces. The codebase is intentionally split between a synchronous HTTP shell, asynchronous orchestration services, Celery workers, browser/transport runtimes, and two Postgres-backed storage domains.

Runtime Map

HTTP clients
  |
  +-- FastAPI app (`apps/api/main.py` -> `rosterscraper.api.app:create_app`)
        |
        +-- Control-plane endpoints (`/`, `/sessions`, `/skills`, `/settings`, `/chat`, `/graphql`)
        +-- Product/auth endpoints (`/v1/...` via `rosterscraper.api.product:router`)
        +-- SSE streams (`/events/control-plane`, chat streams)
        |
        +-- `ControlPlaneService`
              |
              +-- Recon + strategy selection
              +-- Execution plane dispatch
              +-- Browser Use managed execution when selected
              +-- Validation and publish flow
              +-- Scheduled jobs, chat, analytics, press monitoring
              |
              +-- Repositories over intel Postgres
              +-- Core DB access for auth/product session state
              +-- Redis-backed Celery tasks for background execution

Entrypoints and Boot Sequence

API process

services/platform-api/apps/api/main.py exposes the ASGI app object by importing rosterscraper.api.app:app.
services/platform-api/package.json wires local entrypoints:
- pnpm dev:api runs uvicorn rosterscraper.api.app:app --reload
- pnpm dev:worker starts the hosted-style Celery worker on scrape-cloud,validation-publish,compliance
- pnpm dev:worker:local starts the optional macOS-style worker on scrape-local
- pnpm dev:scheduler still starts Celery Beat for compatibility-heavy local workflows
- pnpm scheduler:tick runs the one-shot recurring-job dispatch used by provider cron and the deploy baseline
services/platform-api/src/rosterscraper/api/app.py builds the FastAPI instance in create_app().

Lifespan work

During startup the API does more than register routes:

It loads typed settings from rosterscraper.config.Settings.
It initializes Sentry via rosterscraper.sentry_init.init_sentry(), wiring the FastAPI integration, release/environment tags, app scope tags, and trace sampling from configuration.
It optionally bootstraps the core database when core_db_auto_create is enabled.
It ensures the control-plane event stream storage exists through initialize_control_plane_events().
It runs ControlPlaneService.reconcile_interrupted_state() so abandoned sessions and worker leases are normalized after restarts.
On shutdown it disposes both async SQLAlchemy engines with dispose_engine() and dispose_core_engine().

That startup sequence is why the API process is the coordination boundary for the rest of the backend: even background work relies on tables and session state that the API prepares.

Runtime configuration additions

The observability/security rollout added a few backend-facing settings that now matter across local and hosted environments:

APP_ENV and APP_RELEASE align backend release tagging with the frontend builds
MONITORING_SENTRY_DSN and MONITORING_SENTRY_TRACES_SAMPLE_RATE control Sentry capture volume
SECURITY_TURNSTILE_SECRET_KEY reserves the backend verification secret for future protected browser flows

The checked-in production deployment path for those settings is now the VPS bundle in infra/vps:

infra/vps/docker-compose.yml runs the API, worker, scheduler, Postgres, Redis, and Nginx together on one host
infra/vps/systemd/sterlingindex-platform.service keeps the stack alive across reboots
infra/vps/.env.example documents the expected runtime and backup variables for the single-host path

HTTP Surfaces

Control-plane API

The main FastAPI router in services/platform-api/src/rosterscraper/api/app.py is the operator-facing surface for backend workflows.

Key route families:

Health and runtime:
- /healthz
- /metrics when Prometheus is enabled
- /runtime/readiness
Streaming:
- /events/control-plane for SSE event delivery
- /chat/stream and conversation resume streams
Core orchestration:
- /firms
- /orchestration/discovery
- /orchestration/run
- /orchestration/preview
- /objectives
Run inspection:
- /sessions
- /sessions/{session_id}/runtime
- /sessions/{session_id}/graph
- /sessions/{session_id}/failure-traces
- /execution-sessions
- /sessions/{session_id}/network-captures
- /sessions/{session_id}/execution-artifacts
Control-plane management:
- /skills
- /eips
- /settings
- /settings/proxy/pool
- /analytics/queues
- /schema-registry
- /worker-leases
- /reports/compliance
- /scheduled-jobs
- /press-monitors
Secondary surfaces:
- /graphql
- /exports/attorneys.csv
- /browser-sessions
- /agent-sessions

Dependency injection is used consistently: request handlers depend on an async SQLAlchemy session plus service factories like get_control_plane() and get_chat_service(). That keeps HTTP handlers thin and pushes coordination into service classes.

The API shell now also attaches request-scoped observability metadata in rosterscraper.api.app:

route/method/status tags are added around requests so Sentry issues can be filtered by HTTP surface
the API-level scope is enriched before deeper product/session context is attached downstream

The orchestration routes now support two runtime modes:

ORCHESTRATION_EXECUTION_MODE=inline: local-dev friendly behavior that can still execute discovery and extraction inside the API process.
ORCHESTRATION_EXECUTION_MODE=queue: staging/prod behavior that creates or updates the session, enqueues work, and returns 202 Accepted with session metadata.

In both modes the request payload can include execution_target: "auto" | "cloud" | "local". That target is persisted into the session execution report so operators can tell whether a run was queued for the hosted workers or the optional macOS scrape node.

Product and auth API

services/platform-api/src/rosterscraper/api/product.py mounts a second router under /v1. This is distinct from the control-plane surface:

It owns browser-facing product/auth routes such as /v1/session, /v1/auth/login, /v1/auth/callback, and /v1/auth/logout.
It reads from the core database for organizations, users, sessions, entitlements, saved lists, and usage logging.
It reads from the intel database through rosterscraper.product_read_models to serve dashboard bootstrap data, attorney search, firm search, newsroom data, analytics, and export metadata.

This separation matters operationally:

The control plane is admin-oriented and mutates orchestration state.
The /v1 router is user-session oriented and serves curated read models backed by the same backend.

The /v1 router is also where product-session context is attached to Sentry:

services/platform-api/src/rosterscraper/api/product.py now enriches the active scope with authenticated user identity
organization and role metadata are tagged when a resolved session is available
anonymous product traffic stays anonymous, matching the frontend boundary on www

Domain and Service Layers

Typed models as the boundary contract

services/platform-api/src/rosterscraper/domain/models.py defines the Pydantic models that sit between repositories, services, and HTTP handlers. Examples include:

Firm, Session, Skill, EIP, and ValidationLog
ScheduledJob, AgentSessionGroup, and QueueRuntimeMetric
RuntimeReadinessReport and ControlPlaneSettingsResponse
execution-plane types such as ExecutionSessionRef, StrategyPlan, and VerificationResult

These models keep repository-backed ORM records from leaking straight into the API and make test fixtures cheaper to build.

`ControlPlaneService` is the backend coordinator

services/platform-api/src/rosterscraper/orchestration.py contains ControlPlaneService, which is the dominant application service in this backend.

It is responsible for:

firm CRUD and session lookup
discovery and approved-skill execution
EIP preview, promotion, and rejection flows
validation report access
settings read/write and runtime diagnostics
browser session and execution artifact inspection
agent/chat session aggregation
scheduled extraction dispatch
compliance reporting
snapshot history and delta generation

Internally it composes repositories, runtime helpers, the execution plane, validation, recon, strategy runners, proxy management, LLM routing, skill generation, and product read-model refreshes.

It also owns the managed-execution switch for Browser Use Agent. That switch is deliberately separate from LLM routing: Browser Use Agent can run a browser task end to end, while LLM providers only supply chat-completion-compatible model routes to internal platform workflows.

Recon, strategy, and validation helpers

Several modules support ControlPlaneService but stay focused on one step of the pipeline:

rosterscraper.recon.service
- captures network evidence
- selects browsers
- detects Cloudflare signals
- distills DOM state for downstream reasoning
rosterscraper.strategies
- executes the staged extraction strategies
- prefers transport-only fast paths when a direct JSON/search backend is available
rosterscraper.validation
- runs the Great Expectations write-audit-publish flow
- persists validation artifacts and summaries
rosterscraper.skill_runner
- signs, verifies, and executes approved skill documents inside a restricted sandbox

Discovery and Extraction Flow

The orchestration layer supports two related workflow families and two execution modes.

Discovery and preview

Discovery starts when the API receives /orchestration/discovery, /orchestration/preview, or an objective-creation request. The backend then:

creates or updates a Session
normalizes the requested execution_target and, in queued mode, resolves auto to local or cloud based on worker availability
either enqueues rosterscraper.worker.execute_session onto scrape-local or scrape-cloud, or continues inline for local development
decides whether execution_provider keeps the run on the internal scraper or hands eligible work to Browser Use Agent
runs recon through rosterscraper.recon.service.run_recon() for internal execution
captures candidate endpoints and structural fingerprints
selects a strategy using helper logic in orchestration.py
optionally creates execution-plane records and remote jobs
runs a strategy from rosterscraper.strategies.run_strategy()
normalizes extracted records
validates them with run_wap_validation()
generates or updates skill/EIP artifacts when appropriate

Queued mode is the hosted default because it decouples HTTP request latency from full scrape latency and lets the platform prefer the macOS scrape node when one is online.

Approved skill execution

Approved skill runs skip most of the discovery work:

load the approved skill or promoted EIP
execute through execute_from_skill()
validate/publish the result
update health, recovery, and session state
refresh product read models used by the /v1 router

This is the "reuse over rediscovery" path described in the service blueprint: first runs produce artifacts, later runs consume them.

Execution Plane and Worker Routing

services/platform-api/src/rosterscraper/execution_plane.py adds a persistence-backed layer for remote work dispatch and evidence capture.

What the execution plane tracks

execution sessions
worker leases
network captures
execution artifacts
action-cache entries
lane and provider metadata

Important behaviors:

lane_for_provider() maps providers into transport, lightpanda, browser_stealth, or cpu.
queue_for_lane() still maps those lanes for internal execution semantics, but host placement is now decided first by the top-level scrape-local vs scrape-cloud queue.
begin_execution_session(), claim_worker_lease(), release_worker_lease(), and fail_execution_session() keep remote worker state queryable after the fact.
register_recon_result() persists recon outputs, screenshots, and captured network traffic.

That design lets the API expose post-run evidence such as /execution-sessions, /network-captures, /execution-artifacts, /action-cache, and /worker-leases.

Celery worker topology

services/platform-api/src/rosterscraper/worker.py defines the runtime queues:

Queue	Main responsibility	Representative tasks
`scrape-local`	full scrape runs intended for the optional macOS node	`execute_session` when `execution_target` resolves to `local`
`scrape-cloud`	full scrape runs intended for the hosted workers	`execute_session` when `execution_target` resolves to `cloud`
`validation-publish`	post-processing, scheduled extraction dispatch, and product-read-model refresh flows	`snapshot_delta`, `dispatch_recurring_jobs`, `run_scheduled_extractions`, `run_press_monitor`
`compliance`	privacy/compliance polling	`poll_drop_platform`

Remote execution tasks use:

acks_late=True
reject_on_worker_lost=True
soft/hard time limits derived from execution_plane_task_timeout_seconds

Two worker tasks are especially important for the hybrid runtime:

describe_node advertises queue ownership, platform identity, browser capabilities, and local-model availability for readiness probes.
execute_session is the worker-side handoff that loads a session and runs the same orchestration pipeline the inline API mode would have executed directly.

For deployment, the scheduler contract is no longer "keep Beat alive forever." The deploy baseline is a minutely scheduler_tick invocation. pnpm dev:scheduler remains useful for local compatibility, but the current production bundle runs python -m rosterscraper.scheduler_tick in a dedicated scheduler container loop on the VPS rather than depending on provider cron.

Browser, Transport, and Skill Runtime Boundaries

Browser and recon subsystem

services/platform-api/src/rosterscraper/browser/session.py manages pooled browser contexts, per-domain identities, stealth fingerprints, and vendor-specific launch adapters. Recon and strategy code use BrowserSessionManager instead of talking to Playwright or other runtimes directly.

services/platform-api/src/rosterscraper/recon/service.py injects a recon init script that records fetch/XHR traffic inside the page, then combines that with:

transport-level probes
Cloudflare challenge detection
optional proxy rotation
DOM distillation for later strategy selection

browser-use-cloud is one browser provider inside this subsystem. In that mode the platform still owns navigation intent, extraction strategy, validation, and review, but the browser itself is a remote Browser Use CDP session. Browser Use CDP settings include profile id, proxy country, timeout, recording, diagnostics, live URL metadata when available, and fresh-session retry behavior for disconnects.

Browser Use managed execution

Browser Use Agent is modeled as an execution provider, not as a normal LLM route. The settings literal is:

execution_provider = "internal" | "browser_use_agent" | "auto_browser_use_fallback"

The three modes mean:

Mode	Behavior
`internal`	Keep the existing platform-owned scraper path: recon, browser control, extraction, validation, and staged review.
`browser_use_agent`	Build a structured Browser Use task, stream live agent messages into the session timeline, ingest typed output, and send the result through the same staged review flow.
`auto_browser_use_fallback`	Start with the internal scraper and use Browser Use Agent as the configured fallback path for eligible failures.

The managed client lives in services/platform-api/src/rosterscraper/browser_use_agent.py. It resolves the Browser Use API key from env or encrypted control-plane storage, applies timeout and budget settings, and returns metadata such as provider, session id, task id, live URL, recording URLs, output schema version, model, and retry/heal status when available.

Managed execution keeps workflow shape consistent:

roster extraction and profile enrichment are sent as one combined task: extract the full attorney roster and enrich each attorney from the firm profile page
press/news monitor runs are sent as one structured monitor task that returns publication items, evidence URLs, dates, titles, summaries, and confidence
Browser Use live messages are persisted as provenance events with role, type, summary, data, and screenshot URL fields where available
Browser Use output is staged for Review & QA or Press QA; it does not bypass review into direct publish
recurring targets can use workspace ids and cached-rerun settings so Browser Use can reuse and self-heal scripts when layouts drift

This separation is why Browser Use appears in the admin under Agent Runtime and Browser settings, not under Model Routing.

Strategy subsystem

services/platform-api/src/rosterscraper/strategies.py implements the staged fallback model. A run can stay entirely in transport mode when the backend discovers a direct JSON/search endpoint, or it can escalate into browser-backed strategies when the target site requires DOM interaction or challenge handling.

That strategy execution is unchanged by the hybrid deployment model. The difference is where the run is placed:

the VPS-hosted worker can execute the cloud path safely against the host-local Redis and Postgres services plus shared R2-backed object storage
the macOS scrape node can run the same strategies while using local browser/runtime behavior and local-model hosts such as Ollama or LM Studio

Skill execution boundary

services/platform-api/src/rosterscraper/skill_runner.py is the hard security boundary for generated extraction logic:

skill markdown is HMAC signed
only approved skills execute
AST imports and calls are allowlisted/blocked before runtime
RestrictedPython is used as a compile-time filter
the sandbox command is built with a read-only, no-network, no-capabilities container profile
runtime readiness reports whether Kata/Firecracker is live or simulated on the current host

This is why "approved skill execution" in this service is still a backend runtime concern, not just a file read.

Storage Boundaries

Intel database

The default async session in services/platform-api/src/rosterscraper/db/session.py points at resolved_intel_database_url. Its ORM models in services/platform-api/src/rosterscraper/db/models.py hold runtime and extraction data, including:

firms
sessions
attorneys
skills
eip_registry
validation logs
execution sessions, artifacts, captures, and worker leases
scheduled jobs
press monitoring records
product-facing read-model tables such as attorney_search_mv, firm_search_mv, practice_area_mv, and newsroom_trends_mv

Core database

The separate core session in services/platform-api/src/rosterscraper/core_db/session.py points at resolved_core_database_url. services/platform-api/src/rosterscraper/core_db/models.py keeps product and auth state separate from extraction state:

organizations and memberships
users and identity providers
app sessions and entitlements
saved lists and exports
usage logs and audit logs

The /v1 product router depends on both databases:

core DB for identity and session context
intel DB for attorney/firms/newsroom read models

Read-model refresh boundary

services/platform-api/src/rosterscraper/product_read_models.py rebuilds denormalized read tables from canonical attorney, firm, and press data. refresh_product_read_models() is called after publish-oriented workflows and press-monitor updates so product routes can stay query-friendly without forcing the UI to hit raw extraction tables directly.

Other persisted state

Redis is the Celery broker and result backend (worker.py).
control_plane_events is created lazily by control_plane_events.py and streamed over SSE.
Validation artifacts are written through ValidationArtifactStore.
Skill markdown is persisted as both database state and checked-in/generated file artifacts.
Object-storage settings exist in config.py, and the deploy baseline now expects shared R2-compatible storage so either the VPS-hosted worker or the optional macOS scrape node can write artifacts and attorney media directly into the same cloud-backed persistence path.

Press Monitoring and Product Read Surfaces

Press monitoring is part of the same backend, not a separate service.

services/platform-api/src/rosterscraper/press_monitoring.py fetches source pages, extracts article candidates, ingests article documents, derives mentions/entities, and refreshes product read models afterward.
API endpoints in api/app.py expose /press-monitors and trigger run_press_monitor_job().
Product routes in api/product.py expose newsroom and analytics data built from the resulting intel tables.

That gives the service two outward-facing read experiences:

admin/operator runtime data
end-user product discovery data

Configuration and Readiness

services/platform-api/src/rosterscraper/config.py centralizes runtime configuration with pydantic-settings.

Notable groups:

database and Redis URLs
orchestration execution mode and default scrape target
scrape-node identity, queue advertisement, and readiness probe timing
auth provider and WorkOS settings
public app URLs for redirects/cookies
object storage
LLM routing and model selection for Ollama, LM Studio, OpenRouter, and OpenAI-compatible providers
Browser Use Agent managed execution settings
browser and stealth defaults, including the browser-use-cloud remote CDP browser provider
proxy/challenge settings
validation thresholds
monitoring and alerting
execution-plane timeouts

services/platform-api/src/rosterscraper/runtime.py turns that config into an operational readiness report. It checks:

Docker availability
Ollama and LM Studio reachability
OpenRouter and OpenAI-compatible configuration
Browser Use API key presence plus Agent and Browser capability diagnostics
Kata/Firecracker live vs simulation mode
installed vendor browser runtimes
provider diagnostics from BrowserSessionManager
worker-node availability by probing scrape-cloud and scrape-local with describe_node

The API exposes that report at /runtime/readiness, including a worker_nodes collection that the admin UI uses to show whether the local scrape node is online.

In the current VPS production shape, /runtime/readiness is especially useful for confirming that:

the VPS-hosted scrape-cloud worker is advertising the expected queues
hosted-provider credentials are present even though Ollama is intentionally absent on the VPS
Browser Use Agent and browser-use-cloud diagnostics stay separate from normal LLM provider checks
the optional macOS scrape-local node can still join the same shared cloud state later without changing the API host model

Test and Verification Surfaces

The service has a broad backend test matrix under services/platform-api/tests.

High-signal suites include:

API surface
- tests/api/test_control_plane.py
orchestration and worker routing
- tests/orchestration/test_workflow.py
- tests/orchestration/test_worker.py
- tests/test_execution_plane.py
recon, browser, and strategy behavior
- tests/recon/test_service.py
- tests/browser/test_session_manager.py
- tests/strategies/test_execution.py
runtime and config
- tests/test_runtime_readiness.py
- tests/runtime/test_runtime_planner.py
- tests/runtime/test_runtime_config_chain.py
data quality and security
- tests/validation/test_pipeline.py
- tests/security/test_skill_execution.py
compliance and smoke coverage
- tests/compliance/test_drop_polling.py
- tests/e2e/test_compliance_flow.py
- tests/smoke/test_bootstrap.py

Repository-provided verification commands are documented in:

services/platform-api/README.md
services/platform-api/docs/local-verification.md
services/platform-api/docs/compliance-runbook.md
services/platform-api/scripts/run_local_verification.py

The most important local checks are:

cd services/platform-api
PYTHONPATH=src uv run pytest
pnpm --dir apps/web test
pnpm --dir apps/web build
uv run python scripts/run_local_verification.py --skip-compose-smoke

For hosted deploy prep and hybrid routing changes, the most relevant suites are:

tests/api/test_control_plane.py for 202 orchestration responses and persisted execution_target
tests/orchestration/test_worker.py for scrape-local vs scrape-cloud queue behavior
tests/test_runtime_readiness.py for worker-node readiness reporting
tests/test_browser_use_agent.py and tests/browser/test_selection.py for managed Browser Use Agent behavior and the separate browser-use-cloud CDP path

For the docs app itself, the repository task expects the standalone docs build to pass after content changes:

pnpm --dir apps/docs build

Runtime and Services