DEV Community

Cover image for Stop confidential data before it reaches AI, and trace each detection to the conversation: the Prompt Flow Studio Enterprise plan
uehara
uehara

Posted on Edited on

Stop confidential data before it reaches AI, and trace each detection to the conversation: the Prompt Flow Studio Enterprise plan

When a message matches a warning rule in Prompt Flow Studio, the person who sent it sees nothing. The message goes through as usual. What changes is the audit log: one new line recording who sent it, when, which rule it matched, which AI it was going to, and which message in which conversation. The message itself is not written into the log line; whether matched excerpts are kept for investigation is an organization setting.

That record is the core of the Enterprise plan we have launched for Prompt Flow Studio, our multi-AI workspace (the full release in Japanese is on PR TIMES). This post covers one thing only: how your organization's own rules mask, warn on or block a message before it reaches any AI provider, and how each detection traces back to the exact conversation.

The problem: you cannot see what people send

Once a team uses generative AI for real work, two questions stay open for whoever is responsible. Are people pasting customer data or confidential material straight into an AI? And when something goes wrong, can you trace which conversation it happened in?

Per-provider settings do not answer either question. People switch between OpenAI, Anthropic, Google and others in the same day. Unless the same rules apply to every model and every event leaves the same kind of record, the organization has no single view.

Prompt Flow Studio works with more than 50 AI providers and over 400 models on one screen. So we put the controls in front of all of them: the check happens before a message reaches any AI, and it applies to every model.

Three kinds of rules

Administrators add their own rules on the "Security Filter" settings page. There are three kinds.

  • Keywords: internal project names, customer names and in-house terms, with optional fuzzy matching.
  • Regular expressions: fixed formats such as employee or case numbers.
  • AI detection rules: for what keywords cannot capture (next section).

Each rule is set to mask, warn or block. A blocked message is never sent to the AI. Policies apply organization-wide; members cannot switch them off.

The "Custom rules" list on the Prompt Flow Studio Security Filter page. Rules include Antitrust and price coordination, Attorney–client privileged material, Export-controlled technology, Harassment and discriminatory HR decisions, Regulated health and financial data, Departing-employee data exfiltration, Trade secrets and source code, and Material non-public information (MNPI), each of type AI detection, with columns for content, action (Warn or Block), hits in the last 30 days and an Enabled switch

The list shows each rule's type, content and action, plus how many times it matched in the "Last 30 days." That column tells you which rules are doing work and which are not.

AI detection rules catch what never names itself

A keyword only matches when the word is there. Someone can describe a confidential project in detail without ever typing its name. AI detection rules are for that.

In "Add rule," the administrator picks "AI detection" as the type and writes the "Topic to stop (policy)" in plain language. The demo policy reads: "Stop messages that contain details (budget, specifications, partners) of the confidential 'Project Bluewater'. Treat a message as matching when the content clearly concerns this project, even if the name is not mentioned."

Then come "Examples that match" and "Examples that do not match." The demo's second matching example is "Translate the spec sheet for the new ocean sensor prototype (internal code BW-7)…", which never names the project. The non-matching examples are "Explain to a child why the sea looks blue" and "Proofread the text of our published company brochure": similar words, nothing to stop. As the screen puts it, "Adding look-alike examples reduces false detections." The counters show the limits: 200 characters for the rule name, 300 for the policy and 20 examples of each kind.

Detection by AI is probabilistic, so you test before saving. The trial runs your examples against up to three providers and reports, per provider, matches detected, non-matches passed, misses, false positives, cost and latency. You enable the rule only after you have seen the miss and false-alarm counts. AI rules either block or warn.

The lower half of "Add rule" in Prompt Flow Studio. Action is set to "Stop sending." Notes explain that a dedicated detection model from the same provider as the destination makes the decision on masked text only, and that detection costs about 0.1 PFC per send. Under "Accuracy — trial results (required before saving)," a trial on google shows 4 examples, 2/2 detected, 2/2 passed, 0 missed, 0 false positives, 0.30 PFC and 0.7 s

As the screen says, a dedicated detection model from the same provider as the destination makes the decision. Only the masked text is passed, and the original text is never sent to another company. Detection costs about 0.1 PFC per send, added to the cost of the send itself. When a send is stopped, only the detection is charged (minimum 1 PFC). The trial cost comes out of the PFC of the administrator who runs it (minimum 1 PFC). One limit to know: sends to providers without a detection model cannot be checked, so an AI rule does not stop them.

What the sender sees when a message is blocked

When a message matches a block rule, it stops there and never reaches the AI. The sender sees that the organization's rules stopped it, and which rule. The typed text stays on screen, but it has not reached the AI.

Prompt Flow Studio chat. A request to list customers with overdue balances and their account numbers for a collections agency has been typed, and a red bar above the input reads "Not sent due to your organization's rules (Regulated health and financial data)"

A warning rule behaves differently, as in the opening: the sender sees nothing, the message is sent, and the audit log gains a "Warning" line. Use it for messages you do not want to stop but do want to know about. In neither case is the message text written into the audit log line.

An audit trail that leads back to the conversation

Every message the Security Filter acts on is logged with who, when, which rule or detection type, which AI, and which message in which conversation.

The Prompt Flow Studio Audit Log, Security Filter tab. Columns for date, user, result (Warning or Blocked), detected item, rule, AI model and thread list warnings and blocks across many rules. One expanded row shows the path (Chat) and the reason the AI judged the message to match "Regulated health and financial data"

You can filter by type, period and user, or ask an AI, for example "show me the messages the Security Filter blocked this month." Expanding a row for an AI detection rule shows the reason the model gave.

The log line itself never holds the message body or the detected value, and administrators cannot open members' conversations. Saved excerpts can be viewed only by the people your organization designates. From the line, you can trace which message in which conversation.

Whether to keep the matched excerpts for investigation is up to your organization. It is the "Keep matched excerpts for investigation" setting under "Organization policy" on the Security Filter page, next to the retention period and who can view them. The screen notes: "Stored encrypted and deleted automatically after the period. Even with unlimited retention, everything is deleted when the organization contract ends."

Separately, sign-ins, permission and member changes, personal-data deletions, Legal Hold and billing or usage-allocation events are kept for seven years as hash-chained, tamper-evident records.

Personal data is masked before sending

Alongside custom rules, fixed-format personal data is masked before reaching the AI and restored in the response: email addresses, phone numbers, national ID numbers, card and bank account numbers, API keys and more. People write as usual.

Prompt Flow Studio chat. A request for a thank-you email that includes a customer's email address and phone number gets a drafted email back, and below the answer a notice reads "Automatically masked 2 item(s) of personal information in your message before sending, and restored them in the response," highlighted in red with the note "The masking notice appears below the AI response"

Administrators choose the scope: conversation history, attachments, knowledge (RAG), fetched page text and image-generation prompts can be covered too. Names and addresses are not covered by this automatic masking, because they cannot be found by format. Text inside images is not covered either.

Example policies

Rules describe what your organization wants to avoid, in its own words. Our English demo organization uses policies like these:

  • Block: "Trade secrets and source code," "Material non-public information (MNPI)," "Regulated health and financial data" (HIPAA, PCI DSS) and "Export-controlled technology" (ITAR/EAR).
  • Warn: "Attorney–client privileged material," "Harassment and discriminatory HR decisions," "Antitrust and price coordination" and "Departing-employee data exfiltration."

The product does not identify departing employees or classify topics on its own. A departing-employee rule warns only on what the administrator's rule describes, as a line in the log.

Dashboard and risk signals

The organization dashboard rolls the records up by period: active users, messages, PFC, PII detections, denials, late-night use and custom rule hits, with an hour-by-weekday heatmap and per-member activity. Repeated PII detections or denials and heavy late-night use appear as risk signals, computed from records, never from message bodies. Viewing the dashboard is itself logged.

Pricing and what else is included

Seats and AI usage are priced separately (USD, before tax). Seats include no AI usage. The organization picks one shared monthly PFC pack, and administrators allocate it per member on the "PFC Management" page, with limits, and reclaim what goes unused.

  • Seats: $135 per seat monthly, or $1,440 paid yearly ($120 per month). Minimum 100 seats.
  • PFC pack (required, one per organization): from 10,000 PFC for $160 a month.
  • Example: 100 seats plus 10,000 PFC costs $13,660 monthly, or $145,920 yearly.

Outside this post's topic, the plan also lets you restrict models by developer country, place data under Legal Hold, erase a member's personal data under GDPR Article 17, and distribute standard personas. Sign up at the Enterprise page. Smaller teams can use Team, Business or Business+, which do not include Enterprise features; see the pricing page.

The short version

  • Per-provider settings cannot tell you what people send to AI. Prompt Flow Studio checks every message against your rules before it reaches any model.
  • Rules are keywords, regular expressions or AI detection rules, each set to mask, warn or block. AI rules are tested before saving.
  • Each detection records who, when, which rule and which message in which conversation. The log line holds no message text; keeping matched excerpts is an organization setting.

Terms

  • Security Filter: the Prompt Flow Studio feature that checks outgoing text against your rules and built-in personal-data detection. Also the name of its settings page and audit log tab.
  • Masking: replacing a detected value before sending, so the original never reaches the outside AI.
  • Audit log: the record of who did what and when.
  • PFC: Prompt Flow Studio's usage credit. It puts different provider costs on one scale.
  • Legal Hold: suspending deletion of an organization's data to preserve it for litigation or audits.
  • Risk signal: a dashboard indicator, built from records only, for patterns such as repeated detections or heavy late-night use.

This series shares practical knowledge from building and operating our flagship SaaS, Prompt Flow Studio, and other AI products. If you are interested, please like and follow.

Prompt Flow Studio is a multi-AI workspace where you use, compare, and debate multiple AIs on one screen. Learn more at https://prompt-flow.studio
Our other products are listed at https://www.eln.ne.jp/products


EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.

Top comments (0)