Skip to main content

CI/CD Instructions and Setup

CI scans review every pull request in your repository for security issues. The app generates a workflow file for your project, and you commit it to your repository yourself. We never write to your repository. Once the file is in place:

  • every push to your tracked branch updates the project's memory with the merged code, for free;
  • every push to a pull request into that branch starts a security scan of the code the pull request changes, using the memory as context. You pay per BLoC of that changed code, at the same rate as a full scan.

The generated workflow is for GitHub Actions or GitLab CI. Any other CI system can call the API directly.

The pull request job waits for the scan and comments its findings on the pull request. Findings never fail the job, but the workflow is yours to edit, so you can change that.

Before you start​

  • A Pro, Business or Custom plan. See Subscription Plans.
  • Permission to add a workflow file and a CI secret to the repository.
  • Contracts in Solidity (.sol), Rust (.rs) or Cairo (.cairo).

Set up​

1. Pick the project and the branch to track​

Open CI scans from the sidebar in the app. It lists your organisation's project memories: pick the one CI should follow. You can also open Set up CI on a memory page.

If the project has no memory yet, choose Set up a new repository. Pick the source (GitHub, GitLab or a zip upload) and, for GitHub or GitLab, the branch to track. A new memory is then set up the usual way. Setting up a memory is free, and it uses one of your plan's memory slots.

The CI scans page shows the tracked branch, starting at the branch the memory was built from (or main for a zip), and you can change it there.

Track the branch pull requests merge into

If your repository has both a main branch and a development branch, pick the development branch. That is where pull requests merge, so that is where the memory needs to follow the code.

2. Choose the code in scope​

A scan covers a single language, so first choose the language: Solidity (.sol), Rust (.rs) or Cairo (.cairo). The page starts at the language the memory was built from. Then list the code you want reviewed, one path per line. A folder such as src/ becomes src/*.sol for Solidity, every .sol file under src, subfolders included, and a path ending in *, such as src/Vault*, becomes src/Vault*.sol. In a second box, list any tests, mocks or deploy scripts inside that code to leave out. The page fills both lists into the workflow, as CODE_PATHS and EXCLUDE_PATHS at the top of its script, and shows the workflow only once the language and the code in scope are filled in. See Which files are sent for how paths match.

The page refuses a path that starts with / or holds .. (paths start at the repository root), one in quotes, and a code path for files of another language or type, such as src/*.rs in a Solidity scan. End a folder name with / so it is not taken for a file ending.

3. Store an API key as a CI secret​

Create an API key on the API Keys page in the app (see Set up an API key), then save it in your repository's CI settings as a secret named AUDIT_AGENT_TOKEN.

  • GitHub: Settings, then Secrets and variables, then Actions, then New repository secret.
  • GitLab: Settings, then CI/CD, then Variables. Mark it as masked. If you also mark it as protected, only pipelines on protected branches can read it, so merge request jobs from other branches fail.

4. Add the workflow​

The setup page shows the workflow in three tabs. Each one already contains your memory's id, the tracked branch and our API address.

  • GitHub Actions: save it as .github/workflows/auditagent.yml.

  • GitLab CI: add the jobs to your .gitlab-ci.yml: two jobs, and the hidden .auditagent job that holds the script they share. To let the job comment on merge requests, create a project access token with the api scope and add it as a masked CI variable named AUDITAGENT_GITLAB_TOKEN. Do not mark it as protected, for the same reason as the API key: merge request jobs from unprotected branches would not see it. Without it, the job still runs and prints the result, but does not comment.

  • API only: a short summary of the calls, for other CI systems. See Using the API directly.

5. Record the first commit​

Pull request jobs fail with "not ready" until the memory has recorded a commit of the tracked branch.

  • A new GitHub or GitLab memory set up from the CI scans page records the commit its setup scan read, when that scan finishes. Nothing else to do.
  • A zip memory always needs this step. A zip has no git history, so the memory starts with no commit. Once the workflow is in your repository, run it once by hand on the tracked branch: on GitHub, Actions, then AuditAgent CI scans, then Run workflow; on GitLab, Build, then Pipelines, then Run pipeline. This first run is free: it checks the whole memory against the whole codebase, fixes anything out of date, and records the commit. Every merge after that keeps it current.
  • A GitHub or GitLab memory set up anywhere else, such as one you already had from the memory page, has no recorded commit either. Run the workflow by hand once, the same way.

What happens on each push​

A push to the tracked branch​

The workflow asks us which commit the memory describes, builds a diff from that commit to the new one, and sends it (see Which files are sent). A free scan then updates the memory to match. Its changes apply to the memory directly, with no review step. You can see and edit the result on the memory page as usual. It never adds, edits or removes your custom instructions or excluded findings.

Runs on the tracked branch happen one at a time, so each one starts from the commit the previous one recorded.

A merge that changes no code or documentation file, such as a CI config change or a change only to paths you left out, records the new commit without reviewing the memory.

Instead of a diff, the workflow asks for a full check, where the whole memory is checked against the whole codebase, when:

  • the memory has no recorded commit yet (the first run for a zip memory or one built before CI was set up);
  • the recorded commit is no longer in the branch history, for example after a force-push.

A re-run of an old job, for a commit the memory has already moved past, stops without changing anything.

A push to a pull request​

The workflow runs only for pull requests into the tracked branch. It checks out the pull request's own latest commit, checks that the pull request sits on top of the latest recorded merge (below), builds the diffs and starts the scan. It then prints:

Scan started: https://app.auditagent.nethermind.io/scan-results/<scan id>

A pull request that changes no files of the scanned languages ends the job there, without starting a scan.

The job then waits for the result, checking every 2 minutes. When the scan finishes, it comments on the pull request with the number of findings per severity, each finding's title, and a link to the full results. Later pushes edit the same comment rather than adding new ones.

  • Findings never fail the job. Only a scan that fails does.
  • After 50 minutes the job stops waiting and comments the link instead, so it stays within its 60-minute timeout. The tracked-branch job has a 30-minute timeout.
  • A check that fails because of a network error, a 429 or a 5xx answer is tried again at the next check, rather than failing the job. Any other error fails it.
  • A newer push cancels the older job, and the new job's first step stops the older scan, even if the new job then fails, for example because the branch needs updating.
  • On GitHub the job comments with its own token, which the workflow gives pull-requests: write. On GitLab it needs AUDITAGENT_GITLAB_TOKEN (above). If commenting fails, the job logs why and still passes.

This behaviour is a starting point. Change POLL_MINUTES and WAIT_MINUTES at the top of the script, set WAIT_MINUTES=0 to not wait at all, or edit wait_for_result to fail the job, for example when there are High findings. Waiting uses CI runner minutes.

The scan shows in the app as CI PR Scan, and the memory updates as CI Baseline Update.

Your pull request must sit on top of the latest recorded merge​

A pull request is scanned only when it contains the latest merge the memory has recorded, and nothing newer from the tracked branch. Then its diff is exactly its own changes, scanned against a memory that describes the code it was written against. How you get there is up to you: rebase, or merge the tracked branch in.

The job saysWhat it means
"Update your branch from <branch> (rebase or merge, either is fine)."The pull request is behind the latest recorded merge. Bring it up to date and push again.
"Waiting for the memory to record the latest merge into <branch>..."The pull request contains a merge the memory is still recording, usually because that job is still running. The job asks again every 30 seconds, for up to 5 minutes.
"The latest merge into <branch> is still being recorded; re-run this job shortly."The 5 minutes ran out. Re-run the job once the tracked branch's job has finished.
"The memory is not ready for CI yet: finish setup, or run the workflow on <branch> once."The memory has no recorded commit. See Record the first commit.

Which files are sent​

Each run sends two diffs. The scan also needs the code at the new commit:

  • For a GitHub or GitLab memory, we fetch that commit from the memory's repository through your organisation's GitHub or GitLab connection, so the job sends only the diffs. The connection must be able to read the repository. To upload the code instead, set UPLOAD_CODE=1 at the top of the workflow.
  • For a zip memory, we have no repository to fetch from, so the workflow uploads a zip of the commit (UPLOAD_CODE=1 is set for you).

The two diffs:

  • The code diff covers the code in scope you entered on the setup page (CODE_PATHS in the workflow), as git pathspecs from the repository root. In a pathspec, * also matches inside subfolders, so src/*.sol covers every .sol file under src. Of those, only files of the scanned languages (.sol, .rs and .cairo) are scanned and charged; any other file is ignored.
  • To leave code out of the scans, such as tests, mocks or deploy scripts, list it in the second box on the setup page (EXCLUDE_PATHS in the workflow). Each entry is a git pathspec from the repository root: test/ covers the top-level test folder, and *.t.sol covers every file with that ending, in any folder. Excluded files are left out of pull request scans, so they are never charged, and out of the diffs memory updates read. A full check still reads the whole repository.
  • The docs diff covers documentation files: *.md *.mdx *.rst *.txt by default. Edit DOCS_PATHS in the workflow to match your repository. It tells the scan what a change is meant to do, so code that does something else stands out. On the tracked branch, it can also correct the memory when a spec changes with no code change.

A pull request scan also reads the pull request's title and description, so write down what the change is for. The scan knows the description covers the whole pull request, including files it does not review, and it checks what the description claims against the code rather than taking it on trust. GitLab passes only the first 2700 characters of a merge request's description.

Each diff is capped at 1 MB. A pull request scan is also held to the usual scan limits: at most 100 changed code files, with at most 20,000 BLoC of changed code. A pull request over any of these is refused and not charged, except for a docs diff over 1 MB, which is left out while the scan goes ahead. A tracked-branch update that is too large is not refused; it runs as a full check instead.

Pricing​

  • Pull request scans: added and removed in-scope code is billed per BLoC, at the same rate as a full scan. The changed text is tokenized and converted with ceil(tokens / 10), summed per file. Removed code counts, because deleting a check is a change worth reviewing. The minimum is 200 credits ($2) per scan.
  • Memory updates on the tracked branch are free, and so are the docs diff and the pull request's description.
  • Nothing to scan, nothing charged. When no file of the scanned languages that still exists has changed lines, no scan is created. That covers documentation-only pull requests, pull requests that only delete whole files, and renames with no edits.
  • Credits are checked first. If your balance, including any auto top-up, cannot cover the scan, the job fails and no scan is created.
  • A newer push replaces the older scan on the same pull request. If the older scan had not started, it is cancelled and refunded. If it was already running, it is stopped within about 30 seconds and stays charged. It shows as "Stopped: replaced by a newer push".
  • Re-running a job for the same commit, while its scan is still waiting or running, returns that scan, with no second charge.

Pull requests from forks​

Pull requests from forks cannot be scanned on GitHub. GitHub does not give repository secrets to workflows run for a fork's pull request, so the job has no API key and fails. To scan the change, push the contributor's branch to your repository and open the pull request from there.

Dependabot pull requests are skipped. GitHub does not give Actions secrets to those runs either, so the job would fail with no API key.

Never switch to pull_request_target

pull_request_target runs with your repository's secrets for pull requests from forks. It would give code from a fork access to your API key.

Using the API directly​

The generated workflows are a shell script around four CI calls, plus two existing calls to wait for the result. You can make them from any CI system. The CI calls go to https://api.auditagent.nethermind.io/api/v1/scanner/ci, and every call sends the header X-API-Key: <your API key>. The memory id is the memory's key, for example github:your-org/your-repo or, for an uploaded zip, zip:<name>:<hash>. The generated workflows on the setup page already contain it.

CallWhat it does
GET /baseline?memory=<id>Returns {"commit": "<sha>" or null}: the commit to diff from.
POST /baselineUpdates the memory after a push to the tracked branch. Free. Multipart form: payload, and optionally repo_zip, code_diff and docs_diff. Returns a scan_id, or {"status": "up_to_date"} when there is nothing to do.
POST /pr-scanScans a pull request. Paid. Multipart form: payload, code_diff, and optionally repo_zip, docs_diff and pull_request. Returns a scan_id, or {"status": "nothing_to_scan"} when no scanned-language file has changes (nothing is charged).
POST /pr-scan/stopStops the pull request's unfinished scans of other commits, the way a newer scan does. Multipart form: payload. Returns {"stopped": <how many>}. Send it first, so the older scan stops even if your checks then fail.

The payload field is JSON. head and base must be full 40-character commit ids (git rev-parse <ref>); a branch name or short id is refused.

  • POST /baseline with a diff: {"memory": "<id>", "branch": "<tracked branch>", "head": "<new commit>", "base": "<commit from GET>"}.
  • POST /baseline as a full check, with no diffs: {"memory": "<id>", "branch": "<tracked branch>", "head": "<new commit>", "ignore_diff": true}.
  • POST /pr-scan: {"memory": "<id>", "branch": "<the pull request's own branch>", "base": "<commit from GET>", "head": "<pull request commit>"}. A newer scan with the same branch replaces an older one that has not finished, so send each pull request's own branch, never the tracked one.
  • POST /pr-scan/stop: {"memory": "<id>", "branch": "<the pull request's own branch>", "head": "<pull request commit>"}. Scans of head itself are kept, so re-running a job does not stop its own scan.

Build the rest from your git checkout, with full history:

  • repo_zip: git archive --format=zip <head>. Leave it out to have us fetch head from the memory's GitHub or GitLab repository, through your organisation's connection; a zip memory must send it. When we cannot read the repository, the call answers 422 and asks for repo_zip.
  • code_diff: git diff <base> <head> -- <your code paths>, for example 'src/*.sol'.
  • docs_diff: git diff <base> <head> -- '*.md' '*.mdx' '*.rst' '*.txt'.
  • pull_request (pull request scans only): the pull request's title and description, as plain text or Markdown.

The rules the workflow follows are yours to follow too:

  • For a pull request, base must be the commit GET returned, and the pull request must sit on top of it (above). We only see one commit, not your history, so this check is yours to make.
  • Run tracked-branch calls one at a time, and skip a commit the memory has already moved past (the commit GET returns already has it in its history).
  • Ask for a full check only when GET returns null, or a commit that is not in your branch history.
  • Both POST calls answer 409 when the commit moved between your GET and your POST. Call GET again and retry, once.

To wait for a pull request scan, poll its status, for example every 2 minutes, until status is completed or failed:

GET https://api.auditagent.nethermind.io/api/v1/scanner/direct/status/<scan_id>
-> {"scan_id": "<scan_id>", "status": "pending" | "in_progress" | "completed" | "failed", "info_message": "<why it failed>", ...}

A scan replaced by a newer push to the same branch ends as failed, and says so in info_message. Once it is completed, list its findings, most severe first:

GET https://api.auditagent.nethermind.io/api/v1/scans/<scan_id>/findings
-> [{"index": 0, "title": "<title>", "severity": "High"}, ...]

Both answers come wrapped as {"success": true, "data": ...}, like every call above.

For pricing, see Scan Pricing.