Check fix PRs in CI

A pull request that says “Fixes #123”, increasingly one opened by a coding agent, usually passes CI on a machine where the bug never reproduced. The crypticsaiyan/twin@main action checks it where the bug does reproduce: in the reporter’s environment, rebuilt from the capsule attached to the issue.

What it does

On each pull request the action finds the issues the PR closes, looks in each issue’s body and comments for a capsule link, and runs twin verify against the PR’s head commit. It posts the verdict as a comment, or updates its earlier comment, so each push changes one comment instead of adding another. The job passes only when the verdict is fixed, so you can require the check before merging.

The action skips and passes, with verdict=skipped, in two cases: there is no Solari key (pull requests from forks get no secrets), or the PR closes no issue with a capsule attached. The step-by-step behavior and every message are in the Action reference.

Setup

1. Add your Solari key as a repository secret named SOLARI_API_KEY (Settings, then Secrets and variables, then Actions).

2. Add .github/workflows/twin.yml:

   .github/workflows/twin.yml

   name: twin
   on: pull_request
   permissions:
     contents: read
     issues: read
     pull-requests: write
   jobs:
     verify:
       runs-on: ubuntu-latest
       steps:
         - uses: crypticsaiyan/twin@main
           with:
             solari-api-key: ${{ secrets.SOLARI_API_KEY }}

3. Ask reporters for capsules so issues have one attached. See Issue template.

These are the minimum permissions: issues: read to find the capsule and pull-requests: write to post the comment. The repository must be public, because the sandbox clones the PR’s head commit without credentials.

Inputs and output

Input	Required	Default	What it does
solari-api-key	yes		Solari API key. Pass it from a secret.
capsule	no		Path or URL of a capsule. Skips the issue lookup.
attempts	no	3	Runs of the command.
comment	no	true	Post or update the PR comment.
github-token	no	github.token	Token used to read issues and write the comment.

The verdict output is fixed, still-failing, different-failure, flaky, inconclusive or skipped. A later step can use it, for example to label the PR. Give that step if: always(), because the twin step fails on any verdict except fixed.

To check every pull request against one capsule, pass its URL, or commit it and pass its path after checking the repository out:

steps:
  - uses: actions/checkout@v4
  - uses: crypticsaiyan/twin@main
    with:
      solari-api-key: ${{ secrets.SOLARI_API_KEY }}
      capsule: .github/capsules/tz-bug.json

Security

Where the PR code runs

The pull request’s code runs only inside the Solari sandbox, never on the Actions runner. The runner installs and runs twin, which calls the GitHub and Solari APIs, and never checks out or executes the PR’s code. The Solari key stays on the runner and is never copied into the sandbox.

With the plain pull_request trigger, GitHub gives no secrets to workflows for pull requests from forks. The check therefore works for branches in the same repository, which is how coding agents usually open their PRs. On a fork PR the key is empty and the action skips instead of failing the contributor’s PR. Do not switch to pull_request_target to get around this without understanding the risk, because it runs with secrets in the context of the base repository.

Cost

Each check is one verify, which runs two sandboxes for about as long as a replay of the capsule takes. The echarts capsule verifies in under two minutes, a fraction of a cent at about $0.125 per sandbox hour. See How twin uses Solari.