Protecting Sensitive Data When Using Generative AI

The AI system is larger than the model

A generative AI capability may include user interfaces, identity services, prompts, retrieval indexes, vector stores, orchestration, plugins, agents, external tools, memory, logs, model providers, and human review. Sensitive information can enter or leave through any of these components.

Security decisions should therefore address the full data lifecycle and trust chain rather than relying on a model-level privacy statement.

Classify data and approved uses first

Before connecting mission information to an AI capability, teams should identify data categories, permitted purposes, prohibited content, approved users, hosting boundaries, retention requirements, and disclosure consequences.

• Define which data may appear in prompts, retrieval, training, logs, or outputs.
• Separate experimentation from approved operational use.
• Prevent users and agents from crossing classification or need-to-know boundaries.
• Document human review requirements for consequential decisions.

Control retrieval, tools, and agents

Retrieval-augmented generation and tool-using agents expand both utility and risk. An agent may discover information, call an API, change a system, or transmit content based on manipulated instructions. Every retrieval source and tool should have an explicit trust level, scoped authorization, input validation, output controls, and auditable activity.

• Use service identities with narrowly scoped privileges.
• Filter retrieved content by the requesting user’s authorization.
• Treat external content as untrusted input.
• Require approval for high-impact or irreversible actions.
• Separate system instructions, user input, retrieved data, and tool output.

Test for misuse and failure

Functional testing alone does not establish AI security. Adversarial testing should examine prompt injection, indirect instruction attacks, sensitive-data extraction, authorization bypass, unsafe tool use, model manipulation, insecure output handling, and provider or dependency failure.

Findings should produce specific safeguards, monitored conditions, acceptance criteria, and residual-risk decisions.

Monitor what matters

Operational monitoring should help teams identify unusual access, prohibited data use, repeated policy violations, anomalous tool calls, unsafe outputs, model or configuration changes, and unexpected provider behavior. Logging should support accountability while avoiding unnecessary reproduction of sensitive content.