Data Shield

Data Shield is TurfAI's governance layer for sensitive data. It tokenizes PII before a workflow, agent, or extraction call reaches the LLM provider — the model sees opaque tokens (EMAIL_84d0a133, SSN_c95eb795); your caller sees the original values restored in the response. Every tokenization event is recorded in an immutable audit log holding entity counts only — never the raw values.

How tokenization works

Tokenization is deterministic within a request scope — the same email maps to the same token across every message in one call, so the model reasons coherently about "the customer" as a consistent entity. It is isolated across scopes — the same email in two different calls (or two different tenants) maps to two different tokens.

The scope key is {tenant_id}:{scope_type}:{scope_id}, so tenant isolation holds by construction and there is no cross-request linkage of real values. In v3.0 every scope is ephemeral: the token map lives in memory for the duration of one request and is destroyed when the request frame ends. There is no on-disk token map.

Before and after: what the model actually sees

Say a workflow sends this to an llm_task:

Email [email protected] a payment reminder. His SSN on file is 123-45-6789.

Data Shield rewrites the payload before it leaves the service mesh, so the LLM provider receives only tokens:

Email EMAIL_84d0a133 a payment reminder. His SSN on file is SSN_c95eb795.

The model replies in terms of those same tokens (deterministic within the scope, so a second mention of the email is the same token):

Drafted a reminder to EMAIL_84d0a133. Note SSN_c95eb795 is masked in the message body.

On the egress path the gateway restores the original values, so your caller sees:

Drafted a reminder to [email protected]. Note 123-45-6789 is masked in the message body.

The LLM provider's logs and any fine-tuning pipeline contain tokens only — never [email protected] or 123-45-6789.

What's covered

Eight L1 entity types, detected by regex + format validation, run unconditionally when Data Shield is enabled on a node (detection target: < 5 ms p95):

Detectors are recall-biased: a 9-digit string resembling an SSN may be tokenized even if it isn't one. Over-tokenization is recoverable on the return path; under-tokenization is a leak. Treat tokenized values as PII candidates, not a precise classification.

L2 gazetteer — tenant word lists (competitor names, product codes, banned vocabulary), case-insensitive and word-boundary matched.

L3 NER — Microsoft Presidio (spaCy) for free-text PERSON, LOCATION and ORGANIZATION, plus URL, NRP, MEDICAL_LICENSE, US driver-licence / passport / bank / ITIN numbers and CRYPTO addresses. DATE_TIME is opt-in only — list it explicitly in entity_types. L1 always wins an overlap with L3. NER uses a small English-only spaCy model by default (operators can set DATA_SHIELD_SPACY_MODEL to the larger model for higher precision).

Custom detectors and packs — each tenant can add its own L1 regex patterns, L2 word lists and L3 Presidio recognizers (regex + context words) without a deploy; they are stored tenant-scoped. Three domain packs — Insurance, HR and Financial essentials — install and uninstall in one click. Manage them in /settings/data-shield (Simple or Advanced mode) and try them on sample text at /data-shield/demo.

Entry points covered:

• POST /api/v1/chat — direct LLM chat; used by llm_task (text mode) and the agent_task ReAct loop (every chat turn).
• POST /api/v1/extract — structured extraction; used by extraction_task and the agent_task file-mode extract.
• agent_task ReAct loop — both the chat and extract call sites inside an agent's reasoning loop.
• Doc-batch action paths — classification, summarization and extraction.

With the shield enabled, execution history stores the tokenised prompt in _prompt_diagnostics, never the raw one.

Not covered until v3.5 — plan accordingly, do not assume coverage:

• RAG chat path — questions and retrieved context are sent unmodified. See Knowledge Base & RAG — RAG is not tokenized until v3.5.
• Chatbot public chat — the public widget's round-trips do not pass through Data Shield.
• Prompt Lab preview — the dev-only preview surface is not shielded.
• Document file contents — the shield never reads files. With Data Shield enabled, POST /api/v1/extract with non-empty file_urls fails fast with HTTP 400 and a clear message: extract the text first and send it through content (it will be shielded), or disable Data Shield for that workflow. In-boundary OCR lands in v3.5.
• Customer-held KMS / persistent document scope and evidence-pack export are v3.5.

Turn on Data Shield

Data Shield is opt-in — it is not on by default. Enable it at one of three levels.

Per node — turn on the Data Shield switch on an llm_task, extraction_task, classification_task, summarization_task or agent_task node. Optionally, restrict it to a subset of entity types. The node's activity row carries the policy:

{
  "data_shield": {
    "enabled": true,
    "entity_types": ["EMAIL", "SSN", "CREDIT_CARD"]
  }
}

Omit entity_types to run every detector.

Per workflow — on the Resilience tab, turn on Require Data Shield on every LLM node. The executor then fails any run whose LLM nodes haven't opted in, terminating with DATA_SHIELD_POLICY_VIOLATION:

{
  "data_shield_policy": { "required": true }
}

Per pack — declare data_shield_policy in a Solution Pack manifest. Every workflow instantiated from a pack template inherits the policy onto its activity row; the workflow author can refine it per instance:

{
  "data_shield_policy": {
    "required": true,
    "entity_types": ["EMAIL", "PHONE", "SSN", "CREDIT_CARD", "AADHAAR", "PAN"]
  }
}

Audit log

Every shielded call writes one immutable row under the action data_shield.tokenise. It holds entity counts by type (never raw values), the layers invoked (["L1"] in practice today), per-stage latencies, a retention timestamp (default 30 days, tunable via DATA_SHIELD_AUDIT_RETENTION_DAYS), and a correlation ID back to the run. A sample row:

{
  "action": "data_shield.tokenise",
  "entity_counts": { "EMAIL": 1, "SSN": 1 },
  "layers": ["L1"],
  "latencies_ms": { "tokenise_ms": 1, "roundtrip_ms": 1461 },
  "provider": "anthropic",
  "model": "claude",
  "correlation_id": "wfexec_5f1c…",
  "retention_until": "2026-07-20T00:00:00Z"
}

Rows are immutable after write (update/delete rejected at the database lifecycle layer). End users see their own activity at /account/activity; operators search across tenants in the admin audit dashboard at /audit.

Failure modes

Two off-by-default kill switches exist for incident response only: DATA_SHIELD_DISABLED=true bypasses the gateway entirely (raw payload to LLM, no audit row); DATA_SHIELD_AUDIT_DISABLED=true tokenizes normally but suppresses the audit emit. Either one active is an attestation gap for that period.

Test Data Shield

You can't see the tokenized payload directly (it's restored before it reaches you), so verify two things: the response came back with real values restored, and the audit row recorded the expected entity counts. Send a payload with known PII through a shielded llm_task, then read the audit log. The base URL is https://apisandbox.turfai.in/api.

BASE="https://apisandbox.turfai.in/api"

# 1. Run a workflow whose llm_task has Data Shield enabled, with known PII in the input.
EXEC=$(curl -s -X POST "$BASE/workflow-executions" \
  -H "Authorization: Bearer $TURFAI_JWT" \
  -H "Content-Type: application/json" \
  -d '{ "data": { "activity": 77, "inputs": { "text": "Email [email protected]; SSN 123-45-6789." } } }' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["data"]["id"])')

curl -s -X POST "$BASE/workflow-executions/$EXEC/execute" \
  -H "Authorization: Bearer $TURFAI_JWT" -H "Content-Type: application/json" \
  -d '{ "inputs": { "text": "Email [email protected]; SSN 123-45-6789." } }'

# 2. Confirm the response restored real values (you should see [email protected], not a token).
curl -s "$BASE/workflow-executions/$EXEC/status" -H "Authorization: Bearer $TURFAI_JWT"

# 3. Inspect your audit activity: the row should show {EMAIL:1, SSN:1} and layers ["L1"].
curl -s "$BASE/account/activity?action=data_shield.tokenise" -H "Authorization: Bearer $TURFAI_JWT"

import os, time, requests

BASE = "https://apisandbox.turfai.in/api"
HEAD = {"Authorization": f"Bearer {os.environ['TURFAI_JWT']}", "Content-Type": "application/json"}
PII  = {"text": "Email [email protected]; SSN 123-45-6789."}

# 1. Run a shielded workflow with known PII.
exec_id = requests.post(f"{BASE}/workflow-executions", headers=HEAD,
                        json={"data": {"activity": 77, "inputs": PII}}).json()["data"]["id"]
requests.post(f"{BASE}/workflow-executions/{exec_id}/execute", headers=HEAD,
              json={"inputs": PII}).raise_for_status()

# 2. The restored response must contain the real value, never a token.
for _ in range(30):
    data = requests.get(f"{BASE}/workflow-executions/{exec_id}/status", headers=HEAD).json()["data"]
    if data["status"] in ("completed", "failed"):
        break
    time.sleep(2)
assert "[email protected]" in str(data["outputs"]), "detokenize failed — token leaked to caller"

# 3. The audit row proves tokenization happened, with counts only.
audit = requests.get(f"{BASE}/account/activity", headers=HEAD,
                     params={"action": "data_shield.tokenise"}).json()
print(audit)   # expect entity_counts {"EMAIL": 1, "SSN": 1}, layers ["L1"]

const BASE = "https://apisandbox.turfai.in/api";
const HEAD = {
  Authorization: `Bearer ${process.env.TURFAI_JWT}`,
  "Content-Type": "application/json",
};
const PII = { text: "Email [email protected]; SSN 123-45-6789." };

// 1. Run a shielded workflow with known PII.
const created = await fetch(`${BASE}/workflow-executions`, {
  method: "POST", headers: HEAD,
  body: JSON.stringify({ data: { activity: 77, inputs: PII } }),
});
const execId = (await created.json()).data.id;
await fetch(`${BASE}/workflow-executions/${execId}/execute`, {
  method: "POST", headers: HEAD, body: JSON.stringify({ inputs: PII }),
});

// 2. The restored response must contain the real value, never a token.
let data: any;
for (let i = 0; i < 30; i++) {
  data = (await (await fetch(`${BASE}/workflow-executions/${execId}/status`, { headers: HEAD })).json()).data;
  if (data.status === "completed" || data.status === "failed") break;
  await new Promise((r) => setTimeout(r, 2000));
}
if (!JSON.stringify(data.outputs).includes("[email protected]"))
  throw new Error("detokenize failed — token leaked to caller");

// 3. The audit row proves tokenization happened, with counts only.
const audit = await (await fetch(`${BASE}/account/activity?action=data_shield.tokenise`, { headers: HEAD })).json();
console.log(audit); // expect entity_counts {EMAIL: 1, SSN: 1}, layers ["L1"]

Roadmap

Forward-looking — not part of the v3.0 contract above.

• v3.5 (coming soon) — RAG path coverage (RAG chat, Chatbot public chat, Prompt Lab), in-boundary OCR for shielded file content, customer-held KMS vault, persistent document scope, hash-chained evidence-pack export.
• v4.0 — L4 commercial detector adapters, L5 local-LLM detector, sensitivity-based LLM routing, image redaction for pixel-based media.

Related

• Knowledge Base & RAG — the RAG path is not tokenized until v3.5.
• Solution Packs — declare a data_shield_policy that workflows inherit.
• API reference — synced contract for POST /api/v1/chat and POST /api/v1/extract, the entry points Data Shield covers.