Review and status posted by Preloop¶
Design note and threat model for git_clone_config.platform_review: after a
read-only review run finishes and its result passed the flow's result schema,
the control plane posts one pull request review and one commit status on the
commit that was reviewed. The run itself never holds a credential that can
write to GitHub.
User guide: Restricted CI: Review and status posted by Preloop.
Operator surfaces: the flow page card Post review on the pull request
(frontend/src/components/flow-platform-review-card.ts), the execution page
section Posted on the pull request
(frontend/src/components/execution-platform-review.ts) and
preloop flow review-posting (cli/internal/cmd/flow_review_posting.go).
They read and write only the configuration and endpoints described here.
Code: backend/preloop/services/platform_review.py. Tests:
backend/tests/services/test_platform_review.py (every test named below is in
that file).
Route decision¶
Two routes were considered for "a shared review flow leaves a review and a status on the pull request".
- (a) The run stays read-only and the control plane posts. Chosen.
- (b) A write-capable shared flow in which the agent holds a token scoped to review, comment and status.
Reasons for (a):
- GitHub has no permission for "review and comment" that excludes "close,
reopen and edit the pull request". Creating a review and updating a pull
request (
PATCH /repos/{owner}/{repo}/pulls/{pull_number}: state, title, body, base) both needpull_requests: write(permissions required for GitHub Apps). Under (b) the model would hold a token that can close or rewrite a pull request. Under (a) that token exists only inside controller code that can send two write requests and nothing else. - Shared project selection exists only in read-only mode. Route (a) leaves that boundary alone. Route (b) needs a new shared write mode with its own cross-repository analysis.
- The posted text is built by a fixed server template from validated fields. A prompt-injected model cannot choose the target, the action or free-form markup.
- Publishing a finished review again needs no model run: it is the same controller step.
- It is the least-privilege option.
Cost of (a): the model cannot reply in threads or react during the run. Inline comments come only from structured findings in the result.
Who acts, and with what¶
| Actor | Credential | Can write to GitHub |
|---|---|---|
| Container, model, approval hook | read lease: contents: read on one repository |
no |
| CI identity (the caller's CI key) | Preloop CI key | no; it can ask Preloop to publish its own execution |
| Control plane, publication step | publish lease, see below | one review, one commit status |
The publish lease is a GitHub App installation token minted for one publication attempt:
- requested with
repositories: [<the bound repository>]andpermissions: {"pull_requests": "write", "statuses": "write"}. Nocontentspermission is requested, so the token cannot push, merge (PUT .../pulls/{n}/mergeneedscontents: write) or read source; - only what the attempt needs:
statuses: writealone when no review is posted,pull_requests: writealone when the flow sets no status; - accepted only when GitHub's response lists exactly that repository and
exactly those permissions plus the implicit
metadata: read(mint_repository_lease,profile=); - minted after the execution is terminal, never placed in an execution context, an environment variable, a log line or an API response;
- used through one class (
_GitHub) that refuses any request other than:POST .../pulls/{n}/reviews,POST .../statuses/{sha}, and reads of that pull request, its files, its reviews and that commit's statuses; - revoked when the attempt ends, whatever the outcome. GitHub's one hour expiry is the fallback.
What is posted¶
Only fields of the stored result, and only when all of these hold:
- the execution is
SUCCEEDED; - the result passed the schema gate when it was captured;
- the same stored result passes the flow's current
result_schemaagain at publication time; verdictis one of the values the schema's enum admits and the flow maps it explicitly.
Fields read: verdict, summary, findings[] (severity, title, body
or description or message or detail, path or file, line) and
checked[] (or checks[]). Nothing else in the result is read. The template
adds the reviewed commit, a link to the execution built from server
configuration, and a hidden marker.
Escaping rule¶
Every model-written string goes through plain_text:
- A value that is not a string becomes empty.
- Control, format, surrogate, private-use and unassigned characters are dropped. This removes bidirectional overrides and zero-width characters.
- The text is cut to the field's cap.
- A well-formed inline code span (one pair of backticks on one line, no backtick inside) is kept as a code span, so identifiers the model marked as code read as code. GitHub interprets nothing inside a code span.
- Everywhere else, each character that can start or delimit Markdown or
HTML (
\ ` * _ [ ] < > & # @ ~ | ! $) is written as a numeric character reference such as_. GitHub shows the character and never treats a reference as markup: no tag, comment, link, image, emphasis, table, strike-through or math. Ordinary text, dots, dashes and parentheses stay as typed, socalculator.pyreadscalculator.py. - A character that only matters at the start of a line (list markers,
rules, heading underlines, the delimiter of an ordered-list number, a
leading
/) is written as a reference when it is first on its line. - A zero-width space is inserted after
@and#, between a URL scheme's:and//, insidewww.and insideGH-<digit>: no mention, no issue or pull request reference (so no closing-keyword target and no cross-reference on another issue), no bare-URL autolink.
File paths are shown as code spans (code_span): src/calculator.py:12.
A path that contains a backtick cannot be a code span and goes through
plain_text with step 4 off.
Multi-line fields are emitted as block quotes, so no model line starts at column zero.
Caps: body 60,000 characters, summary 4,000, 50 findings in the body (the rest are counted, with a pointer to the execution), title 200, finding body 2,000, 20 inline comments, 30 checked items of 200.
Inline comments¶
A finding becomes an inline comment only when its path equals a file name
in the pull request's file list and its line is a line of the new side
inside a diff hunk. Every finding is also in the body, so a rejected inline
position costs nothing: if GitHub answers 422 to a review with inline
comments, the same review is sent once more without them.
Where¶
Repository, pull request number and head SHA come from the execution's own
ci_review_binding, written by the controller when the trigger was accepted.
They never come from the result, the prompt, the flow text or the caller: the
publish endpoint has no request body.
Before any write the step re-runs the dispatch authorization of the
execution's own identity (authorize_dispatch): the identity is active, its
project is still on the flow's allowlist with the same tracker, its
provisioning snapshot still matches the flow, and its grant still holds
flow:trigger or execution:review:publish. It then reads the pull request
and requires the provider's pull request id, repository id and repository
name to equal the binding.
In a shared flow, execution A cannot post on project B: A's binding names A's project, the lease is minted for A's repository only, and GitHub enforces the repository on the token.
Pinned extra read repositories (a separate feature) are read leases. The publish lease is only ever minted for the bound repository.
Exact commit¶
The review is created with commit_id equal to the bound head. The status is
created on that SHA.
Moved head: the review is skipped and the status is still set on the
reviewed SHA. The status is true for that commit and GitHub evaluates
required statuses on the current head, so it cannot make a newer commit look
reviewed. A review on a commit that is no longer the head would be read as
feedback on the current code, so none is posted
(review_reason: head_moved). A closed or merged pull request also gets no
review (pull_request_closed). A verdict is never attached to a commit that
was not reviewed.
Status semantics¶
status.context: defaultpreloop/review. Keep it stable so a branch protection rule can require it.status.verdict_states: explicit state (success,failure,error) for every verdict in the schema's enum. Saving the flow fails if one is missing.status.incomplete_state(errororfailure, optional): set with the neutral description "Review did not complete; no verdict for this commit" when the run stopped, failed or returned no valid result, so a required status is not left waiting forever. Without it nothing is posted for such runs.- No
pendingstatus is set at trigger time. All writes stay on the completion path; a required context that has not reported shows as expected in GitHub. - The description is fixed text plus the verdict from the enum; the target URL is the execution in the console.
Commit statuses were chosen over check runs: statuses: write is narrower
than checks: write, and a status is what a branch protection rule requires
by context name.
Review action¶
COMMENT unless verdict_actions maps the verdict to approve or
request_changes. Nothing implies an approval.
Idempotency and replay¶
One row per execution in platform_review_publication (unique on
execution_id). An attempt first claims the row with a single statement
(insert, or a conditional update of a failed or abandoned row), so two
workers never both publish. Each provider write is recorded as soon as GitHub
confirms it and is never repeated by a later attempt.
If a worker dies between GitHub accepting the review and the row being updated, the next attempt lists the pull request's reviews and adopts the one that carries this row's random nonce, the bound commit and a bot author. The nonce is unknown to anyone before the first post, so a stranger cannot plant a review that suppresses ours.
After each write the step reads it back from GitHub and compares actor (an
App bot), target (commit, context) and content (body digest, state). A
mismatch is recorded as readback_mismatch and is never cleared by a retry:
a person has to look at the pull request.
Posting modes and caller preconditions¶
platform_review.mode is automatic (the default, and what a block without
mode means) or manual; enabled: false is off. Only automatic posts from
the completion path (FlowExecutionOrchestrator._publish_platform_review)
and from the sweep (unclaimed_execution_ids excludes manual flows, and a
retry left over from automatic mode is closed with manual_mode). A failed
manual attempt is never retried automatically.
The explicit publish call takes optional caller preconditions
(expected_head_sha, expected_base_sha, expected_result_sha256,
expected_flow_revision). They are compared three times: against the stored
binding, result digest and flow revision before the row is claimed; against
the flow and execution re-read inside the claimed attempt; and, for the
commits, against the live pull request after the lease is minted and before
the first write. A mismatch raises PreconditionFailedError before any
provider write, the row ends failed / precondition_failed (not
retryable, reclaimable by the next explicit call) and the API answers 412.
The digest is the existing result_sha256 over the restricted result
projection (review_completion_evidence.result_sha256), so a CI identity
compares exactly the document it read. Who posted (automatic or manual)
is recorded as a completion fact on the execution and projected as
posted_by; every attempt writes an audit entry platform_review_publish.
The record¶
platform_review.status on the execution says what is on the provider:
status |
Review | Commit status | Typical reason |
|---|---|---|---|
pending |
not attempted yet | not attempted yet | none |
publishing |
attempt in flight | attempt in flight | none |
posted |
posted | posted, or not configured | none |
status_only |
none, by rule | posted | run_not_succeeded, result_not_valid, verdict_unmapped, head_moved, pull_request_closed |
skipped |
none | none | same reasons, when no status applies |
failed |
see review |
see commit_status |
the failure |
posted always means a review is on the pull request. A run that did not
succeed and only received the incomplete_state status reads status_only
with review: "skipped", never posted.
Failure isolation¶
A posting failure never changes the execution's status or stored result. It
is recorded on the publication row with a stable reason and is visible on the
execution, its result and the publish endpoint. Retryable failures (GitHub
unavailable, rate limits) are retried by the execution monitor with
exponential backoff (1, 2, 4, ... minutes, capped at one hour) for at most
six attempts. The same sweep takes over a claim that was left publishing
for more than ten minutes (a cancelled attempt, a worker that died), and
publishes finished executions that no worker claimed, for example after an
operator stop. It considers only executions that finished after posting
was turned on (the server sets platform_review.enabled_at when the block
is saved enabled and keeps it across later saves), so turning the opt-in on
never reaches back to earlier runs and an unrelated flow edit drops nothing.
The explicit publish call can still publish an earlier run. A pending retry of a flow that has since opted out is
closed with platform_review_disabled. After the attempts are used, or for
a non-retryable reason, an explicit publish call retries.
Opt-in and identities¶
platform_review is off by default and accepted only with
publication_mode: read_only; every other publication key stays rejected in
that mode. Saving the opt-in also requires validate_result: true, a
result_schema with a required string verdict enum, and a GitHub App
tracker for every repository (a personal access token cannot be limited to
one repository and two permissions).
The flow's posting settings are not part of a CI identity. An identity's
provisioning snapshot records its repository, tracker, publication_mode
and selection mode; platform_review is not in it, and the block is not
validated when an identity authenticates. Turning posting on, changing it or
turning it off leaves every existing identity valid for what it could
already do. This is safe because the identity gains nothing: the write is
made by the control plane with its own per-publication token, on a target
taken from the execution's binding, and only an administrator who may edit
the flow can turn it on. (The first release of this feature did record the
posting capabilities in the snapshot and invalidated identities when they
changed. That key is now ignored when a stored snapshot is compared, so
identities provisioned by that release keep working.)
The only identity-facing addition is the explicit publish call, behind the
separate action execution:review:publish. An administrator adds it to an
existing identity with PATCH /api/v1/ci-identities/{id} (grant with the
added action and extend_keys: true), which widens the grant and the
identity's active keys in place; nothing is reissued.
Broken posting settings (for example validate_result turned off later)
stop posting with configuration_invalid; they do not affect triggering or
reading.
Threats and tests¶
| # | Threat | Control | Test |
|---|---|---|---|
| 1 | The run obtains a write credential | publish lease minted only after the run is terminal, in the control plane; container context holds the read lease only | test_container_context_of_an_opted_in_flow_has_no_write_credential |
| 2 | The lease can push, merge or reach another repository | one repository, no contents; GitHub's returned scope must match exactly |
test_publish_lease_is_one_repository_two_permissions_and_revoked, test_broader_permission_set_returned_by_github_is_refused, test_no_publish_profile_carries_contents, test_lease_for_one_repository_cannot_write_to_the_other |
| 3 | The lease is used for another call (close, edit, merge) | fixed request allowlist; the tests assert the complete list of writes | test_valid_result_posts_one_review_and_one_status_on_the_bound_commit |
| 4 | A prompt-injected result mentions people, links out, closes issues, hides HTML or commands a bot | fixed template, plain_text, caps |
test_plain_text_neutralises_mentions_links_html_and_commands, test_rendered_review_contains_no_model_markup_and_respects_caps, test_paths_render_in_code_spans_without_backslashes, test_path_that_could_leave_a_code_span_is_rendered_as_text, test_ordinary_text_is_left_readable |
| 5 | The result names another pull request, repository or commit | target only from the binding; unknown result fields are not read | test_result_naming_another_pull_request_is_ignored, test_publish_request_cannot_name_a_target |
| 6 | Shared flow: project A's run or key posts on, or reads, project B | binding, per-repository lease, principal-owned queries | test_two_projects_on_one_shared_flow_never_cross, test_ci_identity_cannot_publish_another_principals_execution |
| 7 | A verdict lands on a commit that was not reviewed | commit_id and status SHA from the binding; moved head skips the review |
test_moved_head_skips_the_review_and_keeps_the_status_on_the_reviewed_sha, test_closed_pull_request_gets_no_review, test_pull_request_identity_mismatch_is_refused |
| 8 | An invalid, stopped or failed run posts a verdict | plan requires SUCCEEDED, stored validity and re-validation | test_invalid_stopped_and_failed_runs_post_no_verdict, test_stored_result_that_no_longer_matches_the_schema_is_not_posted, test_incomplete_run_posts_nothing_without_incomplete_state |
| 9 | Retries, restarts or republish calls duplicate the review | claim row, recorded writes, nonce adoption | test_duplicate_delivery_and_republish_post_once, test_live_claim_blocks_a_concurrent_attempt, test_lost_response_after_posting_is_adopted_not_reposted, test_review_with_a_foreign_marker_is_not_adopted |
| 10 | What GitHub holds differs from what was sent | read-back of actor, target and content | test_read_back_mismatch_is_recorded_and_never_cleared_by_a_retry |
| 11 | A posting failure corrupts or hides the review result | isolation, bounded retry, explicit recovery | test_posting_failure_leaves_status_and_result_untouched_then_recovers, test_automatic_retries_are_bounded, test_unexpected_error_during_publication_is_recorded_not_raised, test_completion_path_swallows_a_publication_crash |
| 12 | Enabling posting hands an identity new authority, or breaks it | posting uses the control plane's own token; the snapshot ignores posting settings; republish needs an explicit action added by an administrator | test_existing_identity_survives_adding_changing_and_removing_posting, test_first_run_after_enabling_posts_with_the_existing_identity, test_snapshot_written_by_an_earlier_release_is_honoured, test_what_must_invalidate_an_identity_still_does, test_publish_action_is_added_to_an_existing_key_in_place, test_broken_posting_settings_block_posting_but_not_the_identity |
| 13 | A CI key turns the opt-in on, or publishes without the action | flow routes deny CI keys; execution:review:publish |
test_ci_identity_cannot_enable_the_opt_in, test_publish_needs_the_explicit_action |
| 14 | The opt-in is used with a credential that cannot be narrowed | save-time and publish-time tracker check | test_opt_in_is_rejected_for_personal_token_and_non_github_trackers, test_opt_in_is_accepted_in_read_only_mode_only |
| 15 | Access continues after admission is removed | dispatch authorization re-run before every attempt | test_project_removed_from_the_flow_revokes_posting, test_disabled_identity_revokes_posting, test_identity_with_neither_trigger_nor_publish_cannot_cause_a_post |
| 16 | An approval is posted without anyone asking for it | explicit verdict_actions only |
test_approve_and_request_changes_only_by_explicit_mapping |
| 17 | The App lacks statuses: write and the status is silently dropped |
review-only lease, recorded reason | test_missing_statuses_permission_posts_the_review_and_says_why |
| 19 | A cancelled attempt or dead worker leaves a review unposted forever | interrupted attempts are recorded retryable; stale claims are taken over | test_attempt_cancelled_after_the_claim_is_completed_by_the_next_sweep, test_claim_abandoned_by_a_dead_worker_is_due_and_completed |
| 20 | Stale retries of opted-out flows crowd out real ones; the sweep reaches back before the opt-in | retry closed on opt-out; sweep limited to runs that finished after posting was enabled | test_retry_of_a_flow_that_opted_out_leaves_the_queue, test_sweep_is_bounded_by_when_posting_was_enabled_not_by_later_saves, test_enabled_at_is_set_by_the_server_and_kept_across_saves |
| 18 | With the opt-in off anything changes | no claim, no lease, no call | test_opt_in_off_mints_no_lease_and_makes_no_call, test_completion_path_does_nothing_without_the_opt_in_or_a_ci_binding |
| 21 | A manual-mode flow posts without the caller asking | completion path and sweep post only in automatic mode; manual failures are not retried | test_platform_review_manual.py: test_execution_finish_posts_only_in_automatic_mode, test_sweep_never_posts_for_a_manual_flow, test_failed_manual_attempt_is_not_retried_automatically |
| 22 | A review is posted for a result, commit or flow configuration other than the one the caller validated | caller preconditions checked before the claim, inside the attempt and against the live pull request before any write | test_platform_review_manual.py: test_each_mismatching_precondition_refuses_with_412_and_can_be_corrected, test_commit_moved_between_the_check_and_the_post_writes_nothing, test_result_changed_after_the_first_check_is_caught_at_post_time, test_flow_changed_after_the_caller_reviewed_it_refuses |
GitHub App permissions¶
The App installation that owns the repository needs, under repository permissions:
- Pull requests: Read and write (create the review; read the pull request, its files and its reviews).
- Commit statuses: Read and write (create the status and read it back).
Only needed when the flow sets
status.
Checks is not used. Contents is not requested by the publish lease
(the read-only clone keeps using contents: read).
An installation token cannot exceed what the installation was granted. If
the App does not hold Commit statuses, GitHub refuses the combined lease;
Preloop then mints a review-only lease, posts the review and leaves the
publication failed with reason: statuses_permission_missing (the review
is recorded as posted). An organisation owner has to approve the added
permission on the App installation; a publish call then adds the status
without posting the review again.
Not covered¶
- GitLab and Bitbucket. The opt-in is refused for them.
- Threaded replies, reactions, re-requesting review, dismissing an earlier review, or updating a previous review in place.
- A
pendingstatus at trigger time. - Check runs and annotations.
- Verified against GitHub's documentation and a fake GitHub in tests, not
against a live repository: that GitHub echoes a review body byte for byte
(the read-back compares a digest after normalising line endings), that
reading a pull request works with
pull_requestspermission alone, and that a lease request beyond the installation's permissions answers 422.