98.1 Introduction
Technology alone cannot create a secure AI platform.
A platform may have:
- encryption,
- firewalls,
- authentication,
- security monitoring,
- AI safety controls,
- vulnerability scanners,
- automated testing,
- and sophisticated infrastructure,
yet still experience security incidents because people misunderstand, bypass, or incorrectly operate those controls.
Security culture addresses the human and organizational side of security.
The objective is to make secure behavior part of normal engineering and business activity rather than something performed only when a security team requests it.
The core principle is:
Security is everyone's responsibility, but accountability must remain explicit.
98.2 What Is Security Culture?
Security culture is the collection of:
- behaviors,
- expectations,
- knowledge,
- incentives,
- processes,
- leadership practices,
- and organizational norms
that influence how people handle security.
A strong security culture encourages people to:
- report problems early,
- question unsafe behavior,
- use secure defaults,
- protect credentials,
- follow access-control requirements,
- perform security testing,
- and learn from incidents.
98.3 Security Culture vs Security Awareness
These terms are related but different.
Security awareness
People understand security risks.
Security culture
People consistently behave securely because secure behavior is integrated into the organization's systems and incentives.
For example:
Knowing that secrets should not be committed to Git is awareness.
Automatically preventing secrets from being committed and making secure development the normal workflow is culture supported by technology.
98.4 Leadership Responsibility
Security culture starts with leadership.
Leadership should demonstrate that:
- security matters,
- incidents should be reported quickly,
- responsible disclosure is encouraged,
- security work receives resources,
- secure development is valued,
- shortcuts that create unacceptable risk are not rewarded.
If leadership prioritizes speed at any cost, employees will often interpret security as optional.
98.5 Security as an Engineering Quality
Security should be treated similarly to:
- reliability,
- performance,
- availability,
- maintainability,
- accessibility,
- and correctness.
A feature that works but creates unacceptable security risk is not a complete feature.
98.6 Secure-by-Design Culture
The organization should encourage teams to ask security questions during design.
Examples:
- What data does this feature process?
- Who can access it?
- What happens if the user is malicious?
- What happens if the AI model behaves unexpectedly?
- What happens if a dependency is compromised?
- What happens if an external provider becomes unavailable?
These questions should happen before deployment.
98.7 Developer Security Responsibilities
Developers are important security stakeholders.
Their responsibilities may include:
- validating inputs,
- enforcing authorization,
- protecting secrets,
- handling errors safely,
- using dependencies responsibly,
- writing security tests,
- reviewing security findings,
- avoiding unsafe defaults.
Developers should not be expected to become security specialists, but they should understand the security requirements relevant to their work.
98.8 Secure Coding Education
Training should focus on practical problems developers encounter.
Topics can include:
Web security
- XSS,
- CSRF,
- injection,
- authentication,
- authorization.
API security
- object-level authorization,
- rate limiting,
- input validation,
- API abuse.
Cloud security
- IAM,
- secrets,
- storage permissions,
- network controls.
AI security
- prompt injection,
- output validation,
- tool authorization,
- data leakage,
- unsafe model assumptions.
98.9 Security Champions
A security champion is an engineer or team member who helps bring security knowledge into a product or engineering team.
Security champions can:
- participate in threat modeling,
- help review security findings,
- promote secure coding,
- coordinate with security engineers,
- identify risky changes early.
They do not replace professional security teams.
98.10 Security Champion Operating Model
A practical model is:
```text id="k2d6nd"
Security Team
│
↓
Security Champions
│
↓
Engineering Teams
│
↓
Product Development
Champions act as communication bridges between security specialists and product teams.
---
# 98.11 AI Security Champions
AI platforms can benefit from dedicated AI-security champions.
They should understand:
* model limitations,
* prompt injection,
* jailbreak risks,
* RAG poisoning,
* agent permissions,
* tool abuse,
* output validation,
* privacy concerns.
This becomes increasingly important as AI functionality spreads across multiple product teams.
---
# 98.12 Human Factors in AI Security
AI systems introduce new human risks.
Users may:
* trust AI outputs too much,
* assume the model is always correct,
* provide sensitive information,
* approve dangerous actions,
* misunderstand generated content,
* rely on automation without verification.
Security design should therefore consider human decision-making.
---
# 98.13 Automation Bias
Automation bias occurs when people give excessive trust to automated systems.
For example:
> “The AI approved it, so it must be safe.”
This assumption is dangerous.
High-impact AI decisions should have appropriate:
* validation,
* confidence indicators,
* policy checks,
* human review,
* auditability.
---
# 98.14 Human-in-the-Loop
Human approval can be appropriate for high-impact operations.
Examples:
* deleting large amounts of data,
* changing privileged permissions,
* publishing sensitive content,
* executing high-risk administrative actions,
* modifying critical infrastructure.
The objective is not to require humans for every action.
The objective is to use human judgment where the consequences justify it.
---
# 98.15 Human-on-the-Loop
For lower-risk automated operations, humans may supervise the system rather than approve every individual action.
For example:
```text id="x8x0l7"
AI Agent
↓
Automated Action
↓
Monitoring
↓
Human Escalation if Needed
This can provide scalability while retaining oversight.
98.16 Security UX
Security controls should be understandable.
Poor security UX can cause users to:
- reuse passwords,
- ignore warnings,
- bypass security procedures,
- approve actions without reading them.
Good security UX should:
- explain important risks,
- use clear language,
- minimize unnecessary prompts,
- distinguish dangerous actions,
- provide meaningful recovery paths.
98.17 Secure Defaults
The safest configuration should generally be the default.
Examples:
- private storage by default,
- least privilege by default,
- encryption enabled by default,
- secure cookies by default,
- restrictive API permissions,
- disabled dangerous tools until explicitly enabled.
Users should not need advanced security knowledge to avoid common mistakes.
98.18 Dangerous Defaults in AI
AI applications should avoid defaults such as:
- unrestricted tool access,
- unrestricted external network access,
- automatic execution of high-impact actions,
- unlimited file access,
- unrestricted retrieval across tenants.
AI agents should start with the minimum permissions necessary.
98.19 Security Training Program
A mature organization can divide training into levels.
Level 1 — Everyone
- account security,
- phishing,
- sensitive data,
- incident reporting,
- safe AI usage.
Level 2 — Developers
- secure coding,
- dependency security,
- API security,
- secrets,
- testing.
Level 3 — AI Engineers
- AI threat modeling,
- prompt injection,
- agent security,
- model evaluation,
- RAG security.
Level 4 — Security Specialists
- threat hunting,
- incident response,
- AI red teaming,
- forensics.
Level 5 — Leadership
- risk,
- incident decisions,
- business continuity,
- security investment.
98.20 Training Should Be Role-Based
Generic annual security training has limited value if it does not address actual work.
A developer should receive different training from:
- an accountant,
- an AI researcher,
- a system administrator,
- a customer-support employee.
Training should reflect actual responsibilities and risks.
98.21 Secure AI Usage Policy
Organizations using AI internally should establish rules for:
- approved AI tools,
- sensitive-data handling,
- confidential information,
- generated code,
- generated documents,
- customer information,
- model-provider selection.
Employees should understand what information may and may not be entered into AI systems.
98.22 Shadow AI
Shadow AI refers to employees using AI tools without organizational approval or visibility.
Risks can include:
- sensitive data exposure,
- unauthorized external processing,
- uncontrolled model usage,
- intellectual-property concerns,
- inconsistent security controls.
Organizations should provide approved alternatives rather than relying exclusively on prohibition.
98.23 AI Acceptable-Use Governance
An internal AI policy can define:
```text id="8w6q0x"
Approved Tools
Approved Data Classes
Prohibited Data
Required Review
Logging Requirements
Retention Requirements
Human Oversight
Security Reporting
Policies should be understandable and practical.
---
# 98.24 Phishing and Social Engineering
Technical security controls should be supported by awareness of social engineering.
Users may encounter:
* fake login pages,
* fraudulent support requests,
* malicious attachments,
* impersonation,
* urgent payment requests,
* fake security alerts.
Training should emphasize verification rather than fear.
---
# 98.25 AI-Assisted Social Engineering
AI can make deceptive content more convincing.
Organizations should therefore teach employees to verify:
* unexpected requests,
* identity,
* payment instructions,
* account changes,
* sensitive-data requests.
The existence of realistic AI-generated communication makes independent verification increasingly important.
---
# 98.26 Incident Reporting Culture
Employees should know:
* what constitutes a security incident,
* where to report it,
* what information to provide,
* how quickly to report it.
The reporting process should be simple.
A user should not need to determine whether an event is definitely a breach before reporting it.
---
# 98.27 Early Reporting
Early reporting can reduce incident impact.
Examples:
```text id="s7w3p5"
Lost Device
↓
Immediate Report
↓
Account Revocation
↓
Session Invalidation
↓
Investigation
Waiting several days can make containment significantly harder.
98.28 Blameless Reporting
Employees should not be discouraged from reporting mistakes.
For example:
“I accidentally uploaded a sensitive file to the wrong workspace.”
A healthy organization responds by:
- containing the exposure,
- assessing impact,
- correcting the process,
- improving safeguards.
Punitive reactions can cause future incidents to remain hidden.
98.29 Security Incentives
Organizations should reward behavior such as:
- reporting vulnerabilities,
- improving security controls,
- fixing recurring weaknesses,
- completing security reviews,
- identifying architectural risks.
However, incentives should not encourage gaming metrics.
98.30 Developer Security Metrics
Possible measurements include:
- security-training completion,
- security-review participation,
- vulnerability remediation performance,
- recurring vulnerability rate,
- threat-model coverage,
- security-test coverage.
These should be used to improve processes, not punish individual developers.
98.31 Psychological Safety
People should feel able to say:
“I think this design has a security problem.”
without fear of being punished for raising the concern.
Psychological safety improves early detection of architectural weaknesses.
98.32 Security Review Collaboration
Security reviews should ideally be collaborative.
Instead of:
Security rejects the design.
Prefer:
Security identifies the risk, explains the impact, and works with engineering to find an acceptable solution.
This approach reduces conflict and improves security ownership.
98.33 Threat Modeling Workshops
Security champions can help organize threat-modeling sessions.
A basic session can ask:
- What are we building?
- What data does it handle?
- Who can interact with it?
- What can go wrong?
- What controls reduce those risks?
- How will we verify those controls?
98.34 Secure Development Lifecycle Culture
Security should become part of normal development ceremonies.
For example:
Planning
Identify security requirements.
Design
Perform threat modeling.
Development
Use secure coding practices.
Pull request
Perform security review.
CI/CD
Run automated security checks.
Release
Validate security gates.
Production
Monitor and respond.
98.35 Security in Code Review
Developers should review security-sensitive code carefully.
Examples:
- authorization,
- file access,
- payment operations,
- database queries,
- external API calls,
- authentication,
- AI tool execution.
Security review should focus on behavior and trust boundaries rather than only syntax.
98.36 Security Documentation Culture
Important security knowledge should be documented.
Examples:
- why a control exists,
- how to operate it,
- what happens if it fails,
- who owns it,
- how to test it.
Documentation reduces organizational dependence on individual memory.
98.37 Onboarding
New employees and contractors should receive security orientation.
Topics should include:
- account setup,
- authentication,
- access rules,
- data handling,
- approved tools,
- incident reporting,
- AI usage,
- secure development where relevant.
Access should be granted according to job responsibilities.
98.38 Offboarding
When someone leaves or changes roles:
- access should be reviewed,
- credentials should be revoked where appropriate,
- sessions should be invalidated,
- privileged permissions should be removed,
- company data access should end.
Offboarding is an important security control.
98.39 Role Changes
Internal transfers can create hidden privilege accumulation.
Example:
```text id="d0s5h6"
Developer
↓
Security Engineer
↓
Administrator
If previous permissions are never removed, the user may retain excessive access.
Access reviews should therefore include role changes.
---
# 98.40 Contractor Security
Third-party personnel should receive appropriate:
* access restrictions,
* security requirements,
* confidentiality obligations,
* authentication controls,
* offboarding procedures.
Contractor access should be time-bounded where appropriate.
---
# 98.41 Privileged Human Actions
Privileged actions should receive stronger controls.
Examples:
* production database access,
* identity administration,
* infrastructure changes,
* security-policy modification.
Controls can include:
* strong authentication,
* approval workflows,
* session logging,
* least privilege,
* just-in-time access.
---
# 98.42 Human Error Reduction
The best security architecture does not assume humans will never make mistakes.
Instead, it designs systems that make mistakes:
* less likely,
* less severe,
* easier to detect,
* easier to recover from.
Examples:
* confirmation for destructive actions,
* automatic backups,
* secret scanning,
* permission boundaries,
* safe defaults.
---
# 98.43 Defense Against Misconfiguration
Misconfiguration is a common operational risk.
Organizations can reduce it through:
* infrastructure as code,
* policy-as-code,
* configuration validation,
* automated deployment,
* drift detection,
* peer review.
Automation can turn security requirements into enforceable controls.
---
# 98.44 Security Culture and AI Agents
Agents require particular attention because they can combine:
* planning,
* tool use,
* data access,
* external communication,
* autonomous execution.
Humans should understand that an AI agent is not simply another user interface.
It may act as a powerful software principal.
---
# 98.45 Agent Permission Culture
Teams developing agents should routinely ask:
* Does the agent need this permission?
* Can the permission be narrower?
* Does the agent need write access?
* Does it need external network access?
* Does it need access to customer data?
* Which actions require human approval?
Least privilege should apply to agents just as it applies to human users.
---
# 98.46 Security Communication for AI Failures
AI systems may fail in unexpected ways.
Employees should know how to report:
* unexpected model behavior,
* data leakage,
* prompt-injection success,
* unsafe outputs,
* unauthorized tool execution,
* suspicious model activity.
These events should be treated as security signals when appropriate.
---
# 98.47 AI Security Bug Reporting
Organizations can establish a simple reporting format:
```text id="z7o4lq"
What happened?
Which system?
When?
What data was involved?
What action occurred?
What evidence exists?
Is the issue still active?
The goal is rapid triage.
98.48 Security Community of Practice
Larger organizations can establish internal security communities.
Activities may include:
- security workshops,
- threat-modeling sessions,
- lessons learned,
- AI-security discussions,
- secure coding demonstrations.
This helps distribute security knowledge across teams.
98.49 Security Drills
Security awareness should include practical exercises.
Examples:
- incident-reporting drills,
- account-compromise exercises,
- backup recovery exercises,
- AI-agent failure simulations,
- phishing simulations where appropriate.
The goal is learning rather than punishment.
98.50 Measuring Security Culture
Security culture is difficult to measure with a single number.
Useful indicators include:
- vulnerability reporting rate,
- time to report incidents,
- security-review participation,
- security training effectiveness,
- repeated security mistakes,
- security champion activity,
- employee survey results.
Trends are more meaningful than isolated measurements.
98.51 Security Culture Survey
A periodic anonymous survey can ask:
- Do you know how to report a security problem?
- Do you understand which data may be entered into AI tools?
- Do you feel comfortable raising security concerns?
- Are security requirements clear?
- Are security controls practical?
- Do teams receive useful security guidance?
The results can identify organizational weaknesses.
98.52 Security Friction
Not all security friction is bad.
Some friction is appropriate for:
- privileged access,
- sensitive-data operations,
- destructive actions,
- financial transactions.
However, unnecessary friction encourages users to bypass controls.
Security should therefore optimize for:
Appropriate friction at appropriate risk levels.
98.53 Risk-Based User Experience
Low-risk actions can remain simple.
High-risk actions can require:
- stronger authentication,
- additional confirmation,
- human approval,
- increased logging.
This produces a proportional security experience.
98.54 Security Culture and Product Design
Security expectations should influence product design.
Examples:
File sharing
Default to private.
AI generation
Apply policy checks automatically.
Account recovery
Require appropriate verification.
Administrative operations
Use stronger authorization.
Agent execution
Restrict tool permissions.
Security should be built into the product rather than delegated entirely to users.
98.55 Security Culture and Organizational Learning
Every major incident, vulnerability, or security exercise can provide learning opportunities.
The organization should ask:
What should change so that this class of problem becomes less likely?
Possible improvements:
- new automation,
- better training,
- stronger defaults,
- architectural changes,
- additional monitoring.
98.56 Security Culture Maturity Model
Level 1 — Reactive
Security is mostly handled after incidents.
Level 2 — Awareness
Employees receive security training.
Level 3 — Integrated
Security becomes part of development and operations.
Level 4 — Collaborative
Security champions and engineering teams share ownership.
Level 5 — Security-by-Design
Security is embedded into architecture, product development, incentives, automation, and organizational decision-making.
98.57 Practical Implementation Roadmap
Phase 1 — Establish Expectations
Create:
- security principles,
- acceptable-use policies,
- incident-reporting procedures.
Phase 2 — Train Teams
Provide role-specific training.
Phase 3 — Establish Security Champions
Select representatives across engineering teams.
Phase 4 — Integrate Security
Add security to:
- planning,
- architecture,
- development,
- deployment,
- operations.
Phase 5 — Improve Security UX
Reduce unnecessary friction while strengthening high-risk controls.
Phase 6 — Measure
Track:
- reporting,
- training effectiveness,
- vulnerability trends,
- security-review coverage.
Phase 7 — Learn Continuously
Use:
- incidents,
- exercises,
- vulnerabilities,
- employee feedback
to improve the program.
98.58 Security Culture Checklist
Leadership
- [ ] Security expectations are clear.
- [ ] Security has appropriate resources.
- [ ] Leaders support responsible reporting.
- [ ] Security is considered in strategic decisions.
Developers
- [ ] Secure coding training exists.
- [ ] Security reviews are integrated.
- [ ] Developers understand common security risks.
- [ ] Security testing is automated where appropriate.
AI Teams
- [ ] AI-security training exists.
- [ ] Prompt-injection risks are understood.
- [ ] Agent permissions are reviewed.
- [ ] AI failures can be reported easily.
Security Champions
- [ ] Champions are identified.
- [ ] Champions receive training.
- [ ] Champions work with security specialists.
- [ ] Champion activity is measured.
Human Factors
- [ ] Security UX is understandable.
- [ ] Dangerous actions receive stronger controls.
- [ ] Human approval exists for appropriate high-impact operations.
- [ ] Automation bias is addressed.
Organization
- [ ] Incident reporting is simple.
- [ ] Employees can raise concerns safely.
- [ ] Onboarding includes security.
- [ ] Offboarding removes access.
- [ ] Security lessons are shared.
98.59 Final Security Culture Architecture
A mature organization can operate security culture as:
```text id="g7z7x4"
SECURITY LEADERSHIP
│
↓
SECURITY PRINCIPLES
│
┌─────────────┼─────────────┐
│ │ │
Training Processes Technology
│ │ │
└─────────────┼─────────────┘
↓
SECURITY CHAMPIONS
│
↓
ENGINEERING TEAMS
│
↓
PRODUCT TEAMS
│
↓
AI SYSTEMS
│
↓
USERS / OPERATORS
│
↓
SECURITY TELEMETRY
│
↓
INCIDENTS / LESSONS
│
↓
CONTINUOUS LEARNING
│
└────────→ SECURITY CULTURE
---
# 98.60 Conclusion
Security culture is the human foundation of a secure AI platform.
Technology can enforce many controls, but people still:
* design systems,
* write software,
* configure infrastructure,
* approve changes,
* operate AI systems,
* respond to incidents,
* and make security decisions.
A mature organization therefore builds security into everyday behavior.
The strongest model is not:
> Security team protects the organization.
It is:
> Security team provides expertise and governance while every relevant team participates in protecting the platform.
For AI platforms, this becomes particularly important because AI introduces new human risks:
* over-trust in model output,
* unsafe automation,
* excessive agent permissions,
* inappropriate data sharing,
* misunderstanding of AI-generated information.
The solution is not to eliminate humans from the system.
The solution is to design the relationship between humans, automation, and security controls carefully.
The long-term objective is:
**Secure People + Secure Processes + Secure Technology + Continuous Learning**
When these four elements reinforce one another, security becomes a property of the organization rather than merely a collection of security products.
**End of Chapter 98**
Top comments (0)