Specialized Cyber Models and the Collapsing Cost of Vulnerability Discovery

Refusal rate is not capability: specialization has a cost

GPT-5.6-Cyber raised "willing to do it" from 1.5% to 95%, yet lost to the general model on OpenAI's own "find the bug + write the report" eval.

// Background and mechanism

OpenAI uses an internal metric, the Advanced Cybersecurity Completion Rate, to measure how often a model completes requests such as exploit-chain development, authentication bypass and privilege escalation: GPT-5.6 Sol under standard safeguards scores 1.5%, 2.0% with Daybreak Blue's loosened safeguards, its predecessor GPT-5.5-Cyber 57.3%, and GPT-5.6-Cyber 95.0%. Note that this metric measures willingness, not accuracy. In the same release, GPT-5.6-Cyber actually scored below Sol on the internal Vulnerability Discovery and Report Writing eval (find a severe, actionable bug, build a working PoC, submit a high-quality report); OpenAI's stated reason is that it "sometimes produc[es] shorter, less detailed vulnerability reports."

The engineering takeaway: willingness and capability are two independent axes. Specialized fine-tuning moved far along the willingness axis and slid back on the "write it completely" axis. That is an old fine-tuning rule: outside the distribution you tuned for, something always gets squeezed out. Before picking a model, decide whether your task is blocked by refusals or by capability. Auditing your own repo, checking dependencies, reading a diff for injection points: a general model with an authorization context rarely refuses. Only exploit-chain writing and bypass validation hit that 1.5% wall.

// Hands-on example

# Defensive audit with a general model: pin down scope, output contract, confidence
You are auditing a repository I own, ./src (scope: this directory only, no network).
Task: find memory-safety / injection / authentication flaws reachable from external input.
For each finding, output JSON:
{"file":..., "line":..., "class":"CWE-xxx",
 "reachability":"how external input reaches this point, hop by hop through the call chain",
 "repro":"minimal reproducing input or test case (required; if you cannot give one, confidence ≤ 0.3)",
 "confidence":0.0-1.0, "fix":"minimal patch"}
Fewer but solid: guesses without a reproducible path go into a separate "hypotheses" array, not findings.

The "cheap to find, expensive to exploit" gap is closing

In March Anthropic said finding is an order of magnitude cheaper than exploiting, but that the gap won't last long; by August a model had chained two V8 bugs into a sandbox escape.

// Background and mechanism

The Firefox numbers are worth a close look: in two weeks Opus 4.6 scanned about 6,000 C++ files and submitted 112 reports; Mozilla confirmed 22 vulnerabilities, 14 of them high severity, fixed in Firefox 148. But the team spent about $4,000 in API credits trying to write PoC exploits, and across several hundred attempts succeeded only twice. Anthropic's conclusion was that finding vulnerabilities is an order of magnitude cheaper than exploiting them, with a caveat: at the current rate of progress, that gap is unlikely to last.

Five months later, OpenAI pointed GPT-5.6-Cyber at V8 and found two previously unknown, chainable vulnerabilities. The optimizing compiler skipped a safety check when converting values to integers, so undefined could become an unexpectedly large number; used as an array index, the compiler assumed it was in bounds and dropped the bounds check, enabling out-of-bounds reads and writes. The second bug was used to escape the V8 heap sandbox. The first was assigned CVE-2026-15903 (CVSS 8.8, fixed in Chrome 150.0.7871.128). For defenders, this means the time from "fix published" to "someone has an exploit" is shrinking, and your patch lag becomes the main risk. Auto-updating browsers are fine, but Electron apps that bundle Chromium, container images pinned to old versions, and lockfiles untouched for six months are all exposed inside that window.

// Hands-on example

# Measure your own patch lag: which known vulns have a fixed version, and how many days the advisory has been out
osv-scanner scan source -r . --format json 2>/dev/null \
| jq -r '.results[].packages[] | .package as $p | .vulnerabilities[]
    | [ $p.name, $p.version, .id,
        ([.affected[]?.ranges[]?.events[]?.fixed // empty] | first // "no-fix"),
        (((now - (.published | sub("\\.[0-9]+"; "") | fromdateiso8601)) / 86400) | floor) ]
    | @tsv' \
| sort -t$'\t' -k5,5nr | column -t
# A version in column 4 plus a large day count in column 5 = handle first; list "no-fix" separately and decide whether to replace the dependency

Split dependency upgrades into two lanes: version updates cool down, security fixes jump the queue

"Upgrade fast" defends against known vulnerabilities; "upgrade slowly" defends against poisoned packages. The two don't conflict, as long as you don't govern them with one rule.

// Background and mechanism

Cheaper vulnerability discovery creates two opposing pressures. On one side, more CVEs and faster exploits demand faster upgrades; on the other, supply-chain poisoning means malicious versions are often caught by researchers and automated scanners shortly after release, which argues for letting new versions sit first. Many teams just do "upgrade everything every Monday" and serve neither goal.

The right move is to split the lanes, and mainstream tools already build the distinction in. Renovate's minimumReleaseAge holds each new version until the specified duration has passed, and the docs state plainly that security updates bypass any minimumReleaseAge checks; PRs raised from vulnerabilityAlerts also ignore schedule and hourly rate limits and go straight to the front of the queue. Dependabot's cooldown option works the same way: it applies only to version updates, not security updates.

// Hands-on example

// renovate.json: routine updates cool for 7 days, majors for 14; security fixes get their own fast lane
{
  "extends": ["config:recommended"],
  "minimumReleaseAge": "7 days",
  "internalChecksFilter": "strict",
  "packageRules": [
    { "matchUpdateTypes": ["major"], "minimumReleaseAge": "14 days" }
  ],
  "vulnerabilityAlerts": { "labels": ["security"], "prConcurrentLimit": 5 },
  "osvVulnerabilityAlerts": true
}

# .github/dependabot.yml: the equivalent (cooldown does not affect security updates)
version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule: { interval: "daily" }
    cooldown:
      default-days: 7
      semver-major-days: 14

The bottleneck has moved from discovery to validation

Discovery got cheap; validation did not. An AI report with no reproduction path just shifts the cost onto the maintainer.

// Background and mechanism

The same year produced two sharply contrasting data points. curl shut down its HackerOne bug bounty on January 31, 2026; Daniel Stenberg's reason was the flood of low-quality, apparently AI-generated reports, with only about 5% of 2025 submissions turning out to be real vulnerabilities. On the other side, every one of Anthropic's 500+ findings was validated by a team member or an outside researcher; even so, the Firefox effort's 112 reports mapped to 22 confirmed vulnerabilities. If a frontier lab's pipeline still needs that much human triage, an individual who runs a model once and submits will produce far more noise.

For a super individual this cuts both ways. As a reporter: when you use AI on someone else's code, "reproducible" is the price of entry; a finding without a reproduction is only a lead. As a maintainer: move AI review into your own CI and find the bugs before others do; and state in SECURITY.md that reports must include a runnable reproduction.

// Hands-on example

# .github/workflows/security.yml: Anthropic's official action, scans the PR diff only
name: Security Review
permissions:
  pull-requests: write
  contents: read
on:
  pull_request:
jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha || github.sha }}
          fetch-depth: 2
      - uses: anthropics/claude-code-security-review@main
        with:
          comment-pr: true
          claude-api-key: ${{ secrets.CLAUDE_API_KEY }}