AI Feature Contractor Brief — Prompts, Metrics & Guardrails to Ship Fast

Start the contractor relationship with a single-file brief: role, goal, constraints, input schema, output contract, unit tests, and monitoring hooks. A disciplined brief reduces back-and-forth and gives the contractor a deterministic target to implement and test against.

Use a layered system prompt that sets role, objective, and strict constraints. Example structure (to include in the brief): 1) system role and high-level goal, 2) required output fields and JSON schema, 3) disallowed behaviors (fabrication, legal advice, PII retention), 4) uncertainty rule (how to say I don’t know), and 5) response length and format constraints. Include one canonical example (input -> expected JSON).

bullets',['Include: role, goal, explicit

Deliver prompt templates, not one-off text. Provide: a system prompt (immutable baseline), a user prompt template with placeholders, and a short 'few-shot' example set used in unit tests. This makes A/Bing prompts or prompt tuning repeatable.

Make prompts deterministic where possible: require explicit output format (JSON schema), instruct the model to list sources for factual claims, and add a final validation step in the prompt (e.g., “If any field is unverifiable, set value null and add reason”). That lets automated validators detect violations reliably.

bullets',['Provide 3–5 few-shot examples covering normal, edge, and failure cases.','Require exact JSON schema and a final short rationale field for audits.','Add a “confidence” numeric field (0–1) the model must fill when it answers factual queries.'],

sourceIds

Specify unit tests: (A) correctness tests where the expected JSON is known, (B) hallucination tests where the model must respond with null + reason for unverifiable claims, and (C) adversarial prompts that probe confident false assertions. Provide a small labeled dataset (10–30 cases) the contractor must pass locally before integration.

Measure with simple, practical metrics: precision for factual claims (fraction of claimed facts that match ground truth), refusal rate on out-of-scope questions (target a minimum), and LLM-judge agreement (use an LLM-based judge for scalable checks). Note: automatic hallucination detectors are imperfect — include manual spot checks for early releases.

bullets',['Include 10–30 labeled test cases: 50% in-domain truth, 30% unverifiable, 20% adversarial falsehoods.','Track precision (factual claims correct), false-positive hallucination rate, and human spot-check pass percentage.','Use an LLM-as-judge step for CI, but require human review for initial rollouts.'],

sourceIds

Define what user data the feature can send to the model and what must be filtered or redacted. Include explicit PII rules: detect and mask names, emails, SSNs, payment details, and any content flagged by your privacy policy before sending to an API. Provide the contractor with a PII-detection step (pre-send filter) and a post-response scrubber.

Mandate a conservative default: if there’s any doubt, redact or return a ‘can’t answer due to sensitive data’ response. Link to a provider-side PII detector if available (for example, OpenAI’s Privacy Filter or similar), but require you own logging of redaction decisions for audits. Also require data retention rules and telemetry that records only metadata for debugging (no raw PII storage).

bullets',['Pre-send PII detection + redaction required.','Post-response scrubber with provenance tags when returning user-visible text.','Log redactions and auditor-visible reasons; do not store raw PII.'],

sourceIds

Set concrete budgets the contractor can measure: 95th percentile latency target (e.g., < 800ms for UI micro-interactions, or < 2s for richer features), and per-request token or cost budget (e.g., max n prompt tokens and m response tokens). Provide an emulator of your chosen model or a usage profile so cost estimations are reproducible.

Define acceptance criteria as pass/fail checks the contractor runs: unit tests pass, hallucination test precision >= threshold (e.g., 90% for in-domain claims), refusal rate within bounds, PII-redaction tests all pass, latency and error-rate thresholds met under a small load test. Require an integration smoke test that runs the labeled dataset through the production pipeline and produces a short results report.

bullets',['Specify P95 latency target and per-request token/cost caps.','Require a CI job that runs labeled tests and returns a pass/fail report.','Include a short rollout plan: internal beta (2–4 weeks) with human review before public release.'],

sourceIds