Skip to content

Plugin startup and worker health

preloop-sync worker (including --tasks and --exclude-tasks), scheduler and monitor discover plugins before starting their service loops. Installed proprietary extensions are enabled unless DISABLE_RBAC=true or DISABLE_PROPRIETARY_PLUGINS=true (also 1/yes). An absent proprietary package is a normal OSS installation. Import, dependency and service-registration failures inside an installed package log the plugin name and cause at ERROR and terminate startup. A plugin is published only after its components register. Worker startup does not invoke API plugin startup hooks, which can start jobs.

Missing handlers for the five scheduled plugin tasks warn once per task per process and count every skipped invocation. Successful handler results record completion timestamps; skipped or failed shims do not advance them. Schedules and job bodies are unchanged.

Enabled proprietary processes publish a heartbeat every 30 seconds to the preloop_plugin_processes JetStream key-value bucket on the existing NATS service. Its records expire after 120 seconds and storage is bounded. Durable job deadlines/completions use the separate preloop_plugin_status bucket without a TTL. Reporting requires JetStream key-value create/read/write access and retries independently: an unavailable status bucket logs an error but does not prevent task processing. The probe detects missing reporting coverage. Disabled or absent extensions do not add a status connection. Process records contain plugin names and loaded booleans, task counters, task filters and timestamps. They contain no configuration, exception text, task arguments or credentials.

GET /api/v1/health/plugins reads that bucket and requires Authorization: Bearer <operator token>, matching PLUGIN_STATUS_OPERATOR_TOKEN on the API (a secret-typed setting that also loads from native .env files). Both compose API services pass the variable from .env. For Helm, use extraEnv with name: PLUGIN_STATUS_OPERATOR_TOKEN and a valueFrom.secretKeyRef pointing to the operator-managed secret. Keep this credential distinct from account keys. This credential authorizes only this read-only surface; ordinary account credentials do not grant access. Missing server configuration returns 503; invalid credentials return 401. An unreachable broker returns 503; a reachable broker with no process bucket yet returns an empty fleet. Health monitors can use the same endpoint. Process records older than 120 seconds are excluded. Configure a probe's expected worker count to detect partial fleet coverage.

Scheduler status uses the actual registered jobs' next fire times: hourly session digest sweep, configured Stripe reconciliation (six hours by default), daily provider billing ingestion, Monday 07:00 UTC review and Monday 09:00 UTC digest. An unsatisfied expected time persists across scheduler restarts; a successful completion releases it for the next fire. These records survive process restarts in JetStream storage. Fresh installations wait for their first scheduled fire, with five minutes allowed for task delivery/completion. Existing installations begin accumulating evidence when this version first starts.

Policy loading now resolves cel_validation_service through the registry. The former working-directory-dependent import dates to commit 27967afc45 (2026-02-01). Where the top-level plugins package was unavailable, YAML/JSON policy loading skipped the optional CEL syntax checks on tool conditions (explicit cel or detected CEL syntax), and model I/O conditions added by 4920838a1c (2026-08-28). Schema, reference and simple-condition checks still ran; runtime policy evaluation is a separate path. Code establishes the bypass, not which deployed policies used it.

The core drift email persistence primitive uses the existing billing_operation table through CRUD, scoped to account/provider/complete UTC day/material cost fingerprint. The companion Enterprise billing change calls this primitive before email; deduplication is effective only after both changes are integrated. The caller finishes ingestion commits before claiming. Repeated costs then do not send again; changed actuals or estimates can send once. Claims commit before SMTP and are never reclaimed after an uncertain handoff. A definite or uncertain mail failure can therefore need operator investigation rather than automatic retry. Claims also commit the caller session on duplicates. No database migration is required.