DEV Community

Anoymask
Anoymask

Posted on

Industrial-Scale AI Distillation: Collecting Model Outputs Through Distributed Accounts and Relays

#security #threatintel

1. Basic Information

2. Executive Summary

U.S. agencies assess that China-based AI companies used multiple accounts and access routes to collect billions of tokens across millions of requests for training. Knowledge distillation trains a student model from a teacher model’s outputs; it is distinct from directly stealing model weights.

3. Attack Flow

Collection and training process described in the joint advisory

  1. Obtain access through premium subscriptions, fraudulently created accounts, and shared accounts.
  2. Distribute requests across native APIs, cloud providers, aggregators, and relay proxies that resell access.
  3. Collect teacher-model outputs using large volumes of domain-specific prompts and requests intended to elicit reasoning.
  4. Detect blocking or declining response quality and switch providers or access routes.
  5. Use the collected outputs as synthetic data to train a student model.

4. Attacker Position and Execution Environment

  • The collecting entities obtain accounts or relay access that can call model APIs. Their infrastructure submits requests and stores responses for training. This technique does not require code execution on the provider’s servers.

5. What Users and Administrators May See

Users

  • This is not an attack that requires an ordinary user to click a lure. AI providers’ usage records are the main investigation surface.

Administrators

  • Inference: Immediate heavy use of new subscriptions, similar requests across accounts, and route changes after blocking can provide leads. Compare these with legitimate batch workloads.

6. Success Conditions and Risk Reduction

Success Conditions

  • Access to the target model’s API and sustained retrieval of responses are possible.
  • The collecting entity can operate across accounts or relays and use the output for training. A particular unpatched CVE is not a prerequisite.

Failure Conditions and Risk Reduction

  • Correlate accounts, payment details, IP addresses and ASNs, devices, and prompt similarity.
  • For suspected abusive distillation, apply response adjustments, rate and concurrency limits, and additional verification in line with provider policy.

7. Potential Impact

  • The joint advisory describes transfer of proprietary capabilities through collected outputs, with reduced development time and expense.
  • Distribution across accounts and routes can weaken the effectiveness of usage restrictions and detection.

8. Observable Logs

Inference: These are investigation sources and their collection requirements.

  • Email: Registration or subscription records may support abuse investigations. Do not assume phishing is the entry point.
  • Proxy/SWG/DNS: Review relay destinations and API connections within your visibility. Do not assume access to another provider’s proxy internals or encrypted request bodies.
  • Endpoint/EDR: Inspect request-generating processes and stored datasets only where the collection infrastructure is available for investigation. Provider-side EDR cannot establish training performed on a customer’s systems.
  • Identity/IdP: Link account creation, subscriptions, authentication sources, and signs of account sharing. Authentication success alone does not establish abusive distillation.
  • SaaS/Cloud: Review request counts, token volumes, selected models, rate limits, and response outcomes. Assess prompt similarity only where content is appropriately recorded and available for that purpose.
  • Network: Correlate source IPs and ASNs, connection frequency, and route changes. High traffic volume alone does not establish intent.

9. Assessing Attack Success

The following are evidence criteria for an individual investigation, not claims that each stage occurred. See Facts and Unknowns for the reported scope.

  • Attempt observed (success unconfirmed): Evidence criterion: Observe requests suspected of collection. Heavy use alone does not confirm abusive distillation.
  • User interaction confirmed: Evidence criterion: Victim interaction is not required. Account and subscription operations belong to the collecting entity’s activity.
  • Initial execution confirmed: Evidence criterion: Confirm an operating loop of repeated API collection. This does not mean malware executed at the provider.
  • Malware execution or authentication success confirmed: Evidence criterion: Confirm successful API access and response retrieval. Evaluate account authentication separately from whether the use is authorized.
  • Data theft or session compromise confirmed: Evidence criterion: Confirm collection of model outputs. Do not equate this with theft of model weights or other users’ credentials.
  • Follow-on compromise confirmed: Evidence criterion: Require additional evidence for training a student model or transferring capabilities. Request counts alone do not establish training results.

10. Investigation Playbook

Inference: Operational recommendations based on the described behavior.

  • Trigger: Immediate heavy use of new subscriptions combined with request patterns shared across accounts or access routes.
  • Initial Checks: Establish the target models, period, request and token volumes, subscriptions, and stated use cases.
  • Endpoint: Inspect request-generating processes and stored datasets only where the collection infrastructure is available for investigation. Provider-side EDR cannot establish training performed on a customer’s systems.
  • Identity and Cloud: Link account creation, subscriptions, authentication sources, and signs of account sharing. Authentication success alone does not establish abusive distillation. Review request counts, token volumes, selected models, rate limits, and response outcomes. Assess prompt similarity only where content is appropriately recorded and available for that purpose.
  • Follow-on Activity: Require additional evidence for training a student model or transferring capabilities. Request counts alone do not establish training results.
  • Containment: Restrict accounts or routes assessed as abusive. Apply rate limits, additional verification, or response adjustments under provider policy, assessing effects on legitimate use.
  • Classification: Distinguish contact, execution, abuse of privileges, data collection, and follow-on compromise using evidence. Missing logs do not prove that compromise did not occur.

11. Defense and Detection Ideas

Inference: Operational recommendations based on the described behavior.

  • Single Event: Immediate heavy use of new subscriptions combined with request patterns shared across accounts or access routes.
  • Time-Series Correlation: Link account and subscription groups, concentrated similar requests, enforcement events, and continued activity through other routes.
  • Threat Hunting: Combine available account, payment, IP/ASN, device, and prompt-similarity signals. Do not attribute activity from one indicator alone.
  • Logging Gaps: Inspect request-generating processes and stored datasets only where the collection infrastructure is available for investigation. Provider-side EDR cannot establish training performed on a customer’s systems. Correlate source IPs and ASNs, connection frequency, and route changes. High traffic volume alone does not establish intent.
  • Priority Controls: Correlate accounts, payment details, IP addresses and ASNs, devices, and prompt similarity. For suspected abusive distillation, apply response adjustments, rate and concurrency limits, and additional verification in line with provider policy.

12. Facts / Inference / Hypothesis

Facts

  • CISA, NSA, and the FBI assess that six China-based AI companies obtained billions of tokens across millions of requests since at least late 2024.
  • The advisory describes fraudulent and shared accounts, native APIs, clouds, aggregators, and relay proxies used to bypass regional restrictions.
  • It identifies reasoning extraction, automatic route switching after blocks, and checks for defensive degradation of outputs.
  • Indicators include immediate heavy usage by new accounts, sustained activity with few idle periods, shared access from multiple IPs, and common patterns across access routes.
  • Knowledge distillation is also a legitimate research technique. This report concerns activity the joint advisory assesses as abusive large-scale collection.

Inference

  • The investigation, success-assessment, and defense recommendations are derived from public information. They are not observations of real-world compromise.

Hypothesis

No additional hypotheses. Unresolved points are listed under Unknowns and Further Investigation.

13. MITRE ATT&CK Mapping

  • T1583.006 Acquire Infrastructure: Web Services (medium confidence): A supplementary mapping limited to acquiring web services and relay infrastructure for collection. The joint advisory primarily uses MITRE ATLAS.

14. Unknowns and Further Investigation

  • Per-company request volumes and the underlying provider telemetry.
  • Direct evidence of government involvement and the companies’ responses.

15. Impact on SOCs and Organizations

AI providers need to examine usage across related accounts and access routes, beyond per-account request limits. For an enterprise SOC, shared organizational API keys and unexpected usage volumes are practical starting points. Heavy use can also reflect legitimate work, so volume alone should not establish abusive distillation or attribution to a particular company.

16. Audience Summaries

  • SOC: Link account and subscription groups, concentrated similar requests, enforcement events, and continued activity through other routes.
  • Administrators: Restrict accounts or routes assessed as abusive. Apply rate limits, additional verification, or response adjustments under provider policy, assessing effects on legitimate use.
  • Users: Do not share organizational API keys or accounts; use approved access routes.

Top comments (0)