AI Strategy Training for CISOs: Write the Stop Rule First

Most documents titled "AI strategy" in a security organization are shopping lists: a copilot for the SOC, an AI email gateway, maybe an agent that closes tickets. A shopping list is not a strategy. A strategy says which problems get money, how you will know each bet failed, and what each one adds to your attack surface. AI strategy training for CISOs and security leaders is worth paying for if it leaves you able to write that document for your own program. Below is what the document looks like, with a script that pressure-tests it.

The direct answer: four decisions, not a framework tour

A security leader's AI strategy comes down to four decisions:

• Where AI goes. Pick problems where you can measure the current process and you have labeled outcomes. No baseline, no way to show improvement. No labels, no way to evaluate the model.
• What it adds to the attack surface. Any model that reads email bodies, ticket comments, or process command lines is reading attacker-controlled text. MITRE ATLAS tracks this as AML.T0051.001, indirect prompt injection.
• What ends the bet. A stop rule, written before the pilot, with a date on it.
• Who runs it. The skills to evaluate a model on your own data, which is a different job from buying one.

Frameworks support those decisions. They do not make them. The NIST AI RMF (AI 100-1) asks you to document intended purpose and context (MAP 1.1) and to have a way to disengage a system that underperforms (MANAGE 2.4). It does not tell you whether phishing triage or endpoint auto-isolation should get this year's budget.

A worked example: the portfolio file

Write each candidate use case as a structured entry that a peer can argue with. Here are two, the first fundable and the second not:

# ai_portfolio.yml  (illustrative numbers)
use_cases:
  - name: phishing-report-triage
    problem: user-reported phishing queue sits too long before first touch
    baseline: {median_hours_to_first_touch: 9}
    labeled_positives: 412          # reports closed as malicious, last 12 months
    reads_untrusted_text: true      # the email body is attacker-written
    acts_without_approval: false    # model ranks the queue; analysts decide
    attack_surface: [AML.T0051.001]
    kill_rule: >
      median time to first touch not under 4h by day 60, or any confirmed
      malicious report ranked in the bottom 20% of the queue
    review: 2027-01-15

  - name: auto-isolate-endpoints
    problem: containment is slow on high-severity EDR alerts
    baseline: null                  # nobody has measured time to contain
    labeled_positives: 18
    reads_untrusted_text: true      # command lines, file names, hostnames
    acts_without_approval: true     # isolates hosts on its own
    attack_surface: [AML.T0051.001, OWASP-LLM06]
    kill_rule: null
    review: null

The second entry is the one vendors pitch hardest, and it fails on every line. Nobody measured containment time, so nobody can show it improved. Eighteen confirmed cases cannot support an evaluation. And a model that reads attacker-written command lines and then takes action without a human is the pattern OWASP calls LLM06:2025 Excessive Agency. An attacker who can name a file can try to write instructions into the name.

Score the portfolio before anyone buys anything

A few gates catch most bad bets. This is short enough to run in the meeting where the strategy is debated:

import yaml

GATES = [
    (lambda u: u["baseline"] is None,
     "measure the current process for 30 days before buying anything"),
    (lambda u: u["labeled_positives"] < 30,
     "cannot be evaluated; fund labeling or pick another problem"),
    (lambda u: u["reads_untrusted_text"] and u["acts_without_approval"],
     "require human approval on every action (OWASP LLM06)"),
    (lambda u: not u.get("kill_rule"),
     "not a strategy item until it has a stop rule"),
]

with open("ai_portfolio.yml") as f:
    portfolio = yaml.safe_load(f)["use_cases"]

for u in portfolio:
    problems = [msg for test, msg in GATES if test(u)]
    print(f"{u['name']:<28} {'HOLD' if problems else 'FUND PILOT'}")
    for msg in problems:
        print(f"    - {msg}")

Output for the file above:

phishing-report-triage       FUND PILOT
auto-isolate-endpoints       HOLD
    - measure the current process for 30 days before buying anything
    - cannot be evaluated; fund labeling or pick another problem
    - require human approval on every action (OWASP LLM06)
    - not a strategy item until it has a stop rule

The threshold of 30 is not arbitrary. If a model catches all 18 of 18 malicious cases, the rule of three still puts the 95 percent upper bound on its miss rate near 3/18, about 17 percent. You cannot distinguish that model from a coin that got lucky. Our post on proving SOC automation works covers building the labeled set itself.

The gates also push you toward classical ML more often than the pitch decks suggest. A ranking problem with 412 labeled examples is a gradient-boosted classifier as much as it is an LLM problem, and it is cheaper to evaluate. We cover that tradeoff in classical ML vs. generative AI in the SOC.

The other half: the AI you have to secure

A security AI strategy has a second column: the AI the rest of the company deploys, which your team inherits. That column starts with an inventory, not a policy. The proxy log exercise in where CISOs learn about AI security builds the first version in an afternoon. Give each system found there the same fields as above, with attack_surface and kill_rule filled in. A business unit's customer chatbot needs a stop rule as much as your SOC copilot does.

How to judge a strategy course

Bring the portfolio file to the vendor conversation. Ask whether the course has you score your own candidate use cases, or only walks through capability demos and framework summaries. Ask whether it covers classical ML or treats "AI" as a synonym for LLMs. And ask whether you leave with a draft document your deputy can poke holes in on Monday. Training that sends you home with a maturity model and a vocabulary list is cheaper to deliver than training that makes you defend a budget line. That is why there is more of it.

When this approach is overkill

If your AI footprint is a vendor feature you cannot turn off, such as an assistant built into your EDR console, the portfolio has one row and the work is contract review: data retention, whether your telemetry trains the vendor's model, and whether the feature can be disabled per tenant. And the gates are a filter, not proof. A use case that passes all four still needs a pilot with real measurement; the vendor POC guide covers that step.

We built A Cyber Executive's Guide for Artificial Intelligence around judgment rather than implementation: ROI evaluation, vendor assessment, governance, and what AI can and cannot do in a security program. It runs as a one-day course and as a private executive briefing for a leadership team.