AI vendor security questionnaire: 120 weighted questions
Score an AI vendor across data handling, model governance, residency, subprocessors, incident response and evaluations. Export the answers, track coverage, and record the conditions.
Your assessment
| Section | Answered | Points | Score |
|---|
Answers are saved in this browser's local storage. Nothing is submitted to Plugsky. The score is a review aid, not a certification.
Six sections, 120 questions, one score
The questionnaire is built for AI procurement, not generic SaaS review. It asks about model versions and provenance, prompt and completion retention, inference residency, tool permissions and evaluation evidence. The generic controls are still there — SSO, encryption, incident response — because they are prerequisites, not substitutes.
| Section | Questions | Max points | What the evidence should show |
|---|---|---|---|
| Data handling and privacy | 20 | 54 | How customer content, personal data and secrets are protected, accessed and deleted. |
| Model governance and provenance | 20 | 49 | How models are selected, documented, evaluated, changed and retired. |
| Residency, transfer and retention | 20 | 50 | Where data is processed and stored, how it moves, and when it is deleted. |
| Subprocessors and supply chain | 20 | 47 | Who else touches the data, how they are contracted, and how concentration is managed. |
| Incident response and resilience | 20 | 47 | How failures, breaches and outages are detected, notified, recovered and reviewed. |
| Evaluation, monitoring and assurance | 20 | 42 | How quality, safety and drift are measured after launch, and what evidence is available. |
Scoring bands
The bands are deliberately simple. They are not industry benchmarks and they do not replace a risk appetite set by your own security or legal team. Use them to structure a go/no-go conversation, then record the exceptions.
| Score | Band | What it means in practice |
|---|---|---|
| 90–100 | Strong | Evidence is complete for the critical questions. Remaining gaps are low-weight or have compensating controls. |
| 75–89 | Acceptable with conditions | Material controls exist, but at least one important question is partial or unanswered. Add contractual commitments and a review date. |
| 50–74 | Material gaps | Several weight-3 questions are “No” or unverified. Do not deploy sensitive data until the gaps are closed or accepted by the risk owner. |
| 0–49 | High risk | Core controls are missing or unverified. Treat the vendor as unsuitable for regulated or high-impact data. |
What good looks like
Use the maturity levels to calibrate an answer. A vendor that says “yes” but cannot produce evidence is at level 1, not level 3. The evidence column in each question tells you what to ask for.
| Level | Label | Observable test |
|---|---|---|
| 1 | Asserted | The vendor says the control exists, but there is no document, screenshot, report or test result. |
| 2 | Documented | A policy or design document exists and is current, but it has not been independently tested. |
| 3 | Tested | There is recent evidence of operation: a test result, audit sample, restore drill, red-team report or access review. |
| 4 | Continuous | The control is monitored, alerted and reviewed on a defined cadence, with a named owner and a remediation record. |
The questions that decide the deal
Not all 120 questions carry equal risk. In most AI reviews, five clusters decide the outcome:
- Training use. Does the vendor train on customer content by default, and can you turn it off? A “yes” without a contractual opt-out is a data-leakage issue, not a feature choice.
- Retention and deletion. Prompts and completions can contain regulated data. Ask for the retention matrix, the deletion SLA, and evidence that backups follow it. “Deleted from the primary store” is not the same as “deleted everywhere”.
- Residency and routing. “We have a region” is not “inference stays in the region”. Ask whether region pinning is contractual or best-effort, and what happens during capacity failover.
- Subprocessor change. A model provider or observability vendor can be added after signature. Ask for notice, objection rights and a current list with countries.
- Tool authority. If the vendor's agent can write, send, refund or delete, the questionnaire needs the tool permission model, approval workflow and audit log. A model that is accurate but over-privileged is still a high-risk system.
These five clusters map to weight-3 questions in the data, residency, subprocessor and model sections. If you have limited time, answer those first and leave the weight-1 questions for the second pass.
How to run the review
- Set the threshold first. Decide what score and which critical questions are mandatory before you see the answers. This prevents the score from being rationalised after the fact.
- Send the questionnaire with the evidence request. The evidence line under each question is what makes the answer auditable. A vendor that returns only “yes/no” has not completed the review.
- Score with coverage visible. A 95% score on 20 of 120 questions is not a 95% vendor. Use the coverage KPI and the unanswered filter.
- Verify the critical claims. Read the SOC 2 scope, the ISO certificate, the pen-test summary and the DPA. Do not accept a logo or a trust-centre page as the only evidence.
- Record conditions. For every partial or no, write the compensating control, the owner and the review date. Export the JSON and attach it to the contract file.
- Re-run after material change. A new model, a new subprocessor or a new region is a new review, not an addendum.
Contract language and limits
Several questions cannot be answered by a security team alone. Training use, deletion, residency, subprocessor notice and audit rights belong in the contract. Where the answer depends on the agreement, the correct entry is “verify in the contract”, not “yes” because the sales engineer said so. Have counsel review the current terms, including the DPA, the acceptable-use policy and any AI-specific annex.
Certifications are claims about a scope and a period. A SOC 2 Type II report covers a defined system and observation window; an ISO/IEC 27001 certificate covers a defined scope. Check the scope, the period and the audit firm. Do not treat a certification as coverage for a service or region it does not include. If a vendor cannot produce the report, record that as evidence missing, not as a failure of the control.
The questionnaire does not cover sector-specific rules (HIPAA, PCI DSS, financial supervision) or every local data-protection requirement. Add those questions separately. Related: the AI vendor risk assessment checklist, the enterprise AI security checklist, and the enterprise article index.
The 120 questions
Answer what you can. Every question includes the evidence to request; the score only counts what you have answered, and coverage is reported separately. N/A removes a question from the denominator, so use it only when the question genuinely does not apply.
Data handling and privacy 20 questions · 54 points
How customer content, personal data and secrets are protected, accessed and deleted.
Is a data processing agreement (DPA) available, and does it cover the AI processing, not just the website?
Evidence to request: Executed DPA, annexes, and the list of processing activities.
Can you confirm that customer content is not used to train shared or foundation models without explicit opt-in?
Evidence to request: Contract clause, product setting, and a written statement of default behaviour.
Can customer data be classified and segregated by sensitivity or tenant?
Evidence to request: Data model, tenant isolation design, and configuration screenshots.
Are PII redaction, secret detection or DLP controls available before content reaches the model?
Evidence to request: Feature documentation, test results, and the list of supported detectors.
Is data encrypted at rest with a documented algorithm and key-management model?
Evidence to request: Encryption standard, KMS design, and key rotation policy.
Is data encrypted in transit with TLS 1.2 or later, and are weak cipher suites disabled?
Evidence to request: TLS configuration, certificate policy, and an external scan.
Is bring-your-own-key (BYOK) or customer-managed encryption available for the relevant data stores?
Evidence to request: Key-management guide and the scope of BYOK coverage.
Can customer content be deleted on request, and what is the maximum deletion time?
Evidence to request: Deletion runbook, API or console evidence, and a recent deletion test.
Do backups containing customer content follow the same deletion and retention rules?
Evidence to request: Backup schedule, retention matrix, and deletion procedure.
Is there a documented data-minimisation and retention schedule for prompts, completions and logs?
Evidence to request: Retention matrix by data type, with default and maximum periods.
Is access to customer content restricted by least privilege and reviewed periodically?
Evidence to request: Role matrix, access-review evidence, and break-glass procedure.
Is SSO supported through SAML or OIDC, and can the customer enforce it?
Evidence to request: SSO configuration guide and enforcement settings.
Is SCIM provisioning and de-provisioning supported?
Evidence to request: SCIM documentation and a test of leaver access removal.
Is MFA required for administrative and support access?
Evidence to request: MFA policy and enforcement evidence.
Are audit logs of data access and administrative actions available to the customer?
Evidence to request: Log schema, retention period, and export method.
Can log retention be configured or shortened by the customer?
Evidence to request: Retention settings and the minimum enforceable period.
Can customer data be exported in a machine-readable format on request?
Evidence to request: Export format, scope, and turnaround time.
Are sensitive data categories (health, biometric, government ID, children's data) prohibited or specially handled?
Evidence to request: Acceptable-use policy and the technical controls that enforce it.
Do you support data-subject requests such as access, correction and erasure?
Evidence to request: DSR workflow, SLA, and the identity-verification method.
Is there a documented lawful transfer mechanism for cross-border data?
Evidence to request: Transfer impact assessment, SCCs or equivalent, and the country list.
Model governance and provenance 20 questions · 49 points
How models are selected, documented, evaluated, changed and retired.
Is there a maintained inventory of models, versions and deployment endpoints?
Evidence to request: Model register with owner, version, endpoint and status.
Does every production model have a model card covering intended use, limits and known failure modes?
Evidence to request: Model cards and the review date for each production model.
Is the provenance of training data documented, including licensed and synthetic data?
Evidence to request: Data provenance summary and licence inventory.
Are model weights, fine-tuning data and training code licensed for the vendor's commercial use?
Evidence to request: Licence review and any restrictions that flow to customers.
Is there a pre-release evaluation gate for quality, safety and regressions?
Evidence to request: Eval plan, acceptance thresholds, and a recent release record.
Is red teaming performed before major model or prompt changes?
Evidence to request: Red-team scope, cadence, and remediation tracker.
Is prompt injection tested across direct, indirect, RAG and tool surfaces?
Evidence to request: Test suite, recent results, and the remediation owner.
Are hallucination and grounding risks measured for retrieval and tool use?
Evidence to request: Groundedness metric, sample size, and the latest result.
Are bias and toxicity evaluated for the intended use and user population?
Evidence to request: Evaluation methodology, disaggregated results, and mitigations.
Is human oversight defined for high-impact or irreversible actions?
Evidence to request: Approval workflow, role definitions, and audit trail.
Will customers be notified before a model change that could alter outputs or behaviour?
Evidence to request: Change-notification policy with lead time and channels.
Is there a deprecation policy with migration support?
Evidence to request: Deprecation timeline, notice period, and migration guide.
Can a model or prompt change be rolled back quickly?
Evidence to request: Rollback procedure, tested time, and version pinning.
Are determinism controls such as seed and temperature exposed and documented?
Evidence to request: API reference and the limits of determinism.
Is fine-tuning data isolated per customer, and is it deleted when the job ends?
Evidence to request: Isolation design, retention period, and deletion evidence.
Does the contract address ownership of model outputs and customer fine-tunes?
Evidence to request: IP clause and the treatment of derivative weights.
Is there an IP indemnity for model outputs, and what does it exclude?
Evidence to request: Indemnity clause and exclusions. Verify in the contract.
Is the model open-weight, proprietary, or a hosted third-party model, and is that disclosed?
Evidence to request: Model provenance statement and the dependency list.
Are safety system cards or acceptable-use enforcement details available?
Evidence to request: Safety documentation and enforcement examples.
Which third-party model providers are in the inference path, and can they change without notice?
Evidence to request: Dependency map, routing policy, and change-notification terms.
Residency, transfer and retention 20 questions · 50 points
Where data is processed and stored, how it moves, and when it is deleted.
Can the customer choose the region where inference and storage occur?
Evidence to request: Region list, control plane, and the default region.
Is region pinning enforceable, or is it a best-effort routing preference?
Evidence to request: Contract language and a routing test.
Are the locations of all subprocessors disclosed, including support and observability?
Evidence to request: Subprocessor list with countries and purposes.
Has a transfer impact assessment been completed for each cross-border flow?
Evidence to request: TIA summary and the legal basis.
Are Standard Contractual Clauses or an equivalent mechanism in place?
Evidence to request: Signed SCCs and the module used.
What is the default retention period for prompts, completions, logs and backups?
Evidence to request: Retention matrix with default and maximum.
Is deletion certified or evidenced, and can the customer obtain a certificate?
Evidence to request: Deletion certificate template and a recent example.
Where are backups stored, and do they cross borders?
Evidence to request: Backup topology and country list.
Can vendor support access customer content from another country, and is that access logged?
Evidence to request: Support-access policy and audit evidence.
Where are logs and observability data stored?
Evidence to request: Log pipeline diagram and storage regions.
Is inference performed in the selected region, or can it be routed elsewhere for capacity?
Evidence to request: Routing policy and failover behaviour.
Is there a disaster-recovery region, and where is it?
Evidence to request: DR plan and the recovery-region list.
Are there data-sovereignty commitments that survive a change of control?
Evidence to request: Contract clause and assignment terms.
What is the policy for government or law-enforcement requests for customer data?
Evidence to request: Transparency policy, legal-review process, and notification practice.
Is a transparency report published, and how often?
Evidence to request: Latest report and publication cadence.
Are there country-specific localisation commitments?
Evidence to request: Localisation annex and the countries covered.
Are cross-border support sessions time-limited and approved by the customer?
Evidence to request: Access workflow and approval evidence.
Is customer data returned in a usable format at termination?
Evidence to request: Exit annex, format, and timeline.
Is transition assistance available, and at what cost?
Evidence to request: Exit assistance clause and pricing.
How are legal holds handled without breaking deletion commitments?
Evidence to request: Legal-hold procedure and customer notice.
Subprocessors and supply chain 20 questions · 47 points
Who else touches the data, how they are contracted, and how concentration is managed.
Is a current subprocessor list published with purpose and location?
Evidence to request: Public list URL and last-updated date.
Will the customer receive advance notice of new subprocessors?
Evidence to request: Notice period and channel.
Does the customer have a right to object to a new subprocessor?
Evidence to request: Objection process and remedy.
Are data-protection and security obligations flowed down to every subprocessor?
Evidence to request: Contract template and flow-down clause.
Are subprocessors assessed before onboarding and re-assessed periodically?
Evidence to request: Assessment questionnaire and cadence.
Are fourth-party dependencies identified where they affect customer data?
Evidence to request: Dependency map and criticality rating.
Is there concentration risk from a single cloud, model or identity provider?
Evidence to request: Concentration analysis and mitigation plan.
Is the hardware supply chain reviewed for provenance and tampering?
Evidence to request: Supplier list and hardware attestation.
What happens if a model provider withdraws or changes terms?
Evidence to request: Fallback model list and contractual protection.
Is there an open-source dependency policy and a licence review process?
Evidence to request: Policy, SBOM, and licence exceptions.
Is a software bill of materials (SBOM) available for the agent or gateway components?
Evidence to request: SBOM format, cadence, and access.
Is there a vulnerability disclosure programme?
Evidence to request: Security.txt, disclosure policy, and response SLA.
Is penetration testing performed by an independent party, and how often?
Evidence to request: Latest executive summary and remediation status.
Is a SOC 2 Type II report available, and what is the observation period?
Evidence to request: Report and bridge letter. Verify scope and period.
Is an ISO/IEC 27001 certificate available, and what is its scope?
Evidence to request: Certificate and statement of applicability. Verify with the certification body.
Are other certifications or attestations claimed, and what exactly do they cover?
Evidence to request: Certificates and scope statements. Do not accept a logo as evidence.
Is cyber-liability insurance in place, and what is the limit?
Evidence to request: Certificate of insurance and exclusions.
Is the vendor financially viable for the contract term?
Evidence to request: Audited accounts or a financial-health summary.
Is there an exit plan if the vendor fails or is acquired?
Evidence to request: Exit plan and escrow arrangement.
Is source-code or data escrow available for critical components?
Evidence to request: Escrow agreement and release conditions.
Incident response and resilience 20 questions · 47 points
How failures, breaches and outages are detected, notified, recovered and reviewed.
Is there 24/7 on-call coverage for security and availability incidents?
Evidence to request: On-call rota and escalation policy.
Is there a documented incident-response plan with roles and decision rights?
Evidence to request: Plan, roles, and last review date.
What is the contractual notification timeline for a personal-data breach?
Evidence to request: Clause and the internal detection-to-notification target.
Is there a named customer notification channel and a template?
Evidence to request: Contact process and sample notification.
Is a post-incident review performed, and will the customer receive a summary?
Evidence to request: RCA template and a recent redacted example.
Is the breach playbook tested, and when was the last test?
Evidence to request: Test record and lessons learned.
Are tabletop exercises run with engineering, legal and communications?
Evidence to request: Exercise schedule and participants.
Is forensic capability available in-house or through a retainer?
Evidence to request: Forensics provider and scope.
How is customer communication handled during a prolonged outage?
Evidence to request: Status-page process and update cadence.
Is there a public status page with historical uptime?
Evidence to request: Status-page URL and history.
Are RTO and RPO defined for the service, and do they match the customer's needs?
Evidence to request: RTO/RPO table by service tier.
Is disaster recovery tested at least annually?
Evidence to request: Test report and date.
Is backup restore tested, and what is the measured restore time?
Evidence to request: Restore test record.
Are single points of failure identified for model serving, data and identity?
Evidence to request: Architecture review and mitigation list.
How are abuse, prompt flooding and denial-of-wallet attacks handled?
Evidence to request: Rate limits, quotas, and abuse-detection rules.
Is DDoS protection in place at the edge?
Evidence to request: Edge configuration and scrubbing capacity.
Are vulnerability remediation SLAs defined by severity?
Evidence to request: SLA table and recent performance.
Is there a patch cadence for model servers, gateways and dependencies?
Evidence to request: Patch policy and maintenance windows.
Are logs retained long enough to investigate an incident?
Evidence to request: Log retention and search capability.
How are law-enforcement and government requests handled during an incident?
Evidence to request: Legal escalation and customer-notification policy.
Evaluation, monitoring and assurance 20 questions · 42 points
How quality, safety and drift are measured after launch, and what evidence is available.
Are quality evaluations run continuously in production or on a fixed cadence?
Evidence to request: Eval dashboard and cadence.
Is there a regression suite for every model, prompt or retrieval change?
Evidence to request: Suite definition and release gate.
Is production drift monitored for input distribution and output quality?
Evidence to request: Drift metrics and alert thresholds.
Is hallucination or groundedness measured, and what is the current result?
Evidence to request: Metric definition, sample size, and latest value.
Are safety classifiers or guardrails deployed, and how are they evaluated?
Evidence to request: Guardrail stack and false-positive/negative testing.
Is human review sampled for high-impact outputs?
Evidence to request: Sampling rate, reviewer training, and escalation.
Is user feedback collected and fed back into evaluation?
Evidence to request: Feedback volume, labelling process, and closed-loop evidence.
Is bias testing repeated after model or data changes?
Evidence to request: Test set, disaggregation, and latest result.
Is external red teaming performed at a defined cadence?
Evidence to request: Provider, scope, and report date.
Are benchmark claims accompanied by methodology, version and date?
Evidence to request: Benchmark report with harness and dataset version.
Is the provenance of evaluation data documented, including contamination checks?
Evidence to request: Dataset cards and contamination analysis.
Can an evaluation be reproduced from the published harness and settings?
Evidence to request: Code, config, seeds, and model version.
Has an independent party audited the evaluation methodology?
Evidence to request: Audit report and scope.
Are performance SLAs defined for latency, availability and error rate?
Evidence to request: SLA table and measurement method.
Is accuracy or task success reported to customers, with confidence intervals?
Evidence to request: Report format and sample size.
Are incident and quality metrics reviewed by leadership?
Evidence to request: Review minutes and action tracker.
Is cost monitored per tenant, model and feature?
Evidence to request: Cost dashboard and anomaly alerts.
Is token and usage reporting transparent enough to reconcile invoices?
Evidence to request: Usage API, granularity, and reconciliation.
Is A/B testing governed to avoid unintended user harm?
Evidence to request: Experiment policy, review, and rollback.
Can the customer run its own evaluations against the production endpoint?
Evidence to request: Eval access, rate limits, and data handling.
FAQ
How is the questionnaire scored?
Each question has a weight from 1 to 3. Yes earns the full weight, Partial earns half, No earns zero, and N/A removes the question from the denominator. The overall score is earned points divided by available points for answered questions. Coverage is shown separately so a high score on ten answers is not mistaken for a completed review.
What score is good enough?
There is no universal threshold. The page's bands are a discussion aid: 90–100 strong, 75–89 acceptable with conditions, 50–74 material gaps, below 50 high risk. A regulated workload may require 90+ with no open critical questions; a low-risk internal tool may accept 75 with compensating controls. Set the threshold before you send the questionnaire.
Does a SOC 2 report replace this questionnaire?
No. SOC 2 covers controls over security, availability, processing integrity, confidentiality or privacy, depending on scope. It does not tell you which model version you are buying, where inference runs, whether prompts are retained, or how a subprocessor change is notified. Use the report as evidence for the controls it covers and ask the AI-specific questions separately.
Can I export the answers?
Yes. The export buttons produce JSON for a system of record and CSV for a spreadsheet. Both include the question ID, category, weight, answer and the evidence requested. The page does not send the answers anywhere; the export happens in your browser.
Is this legal or compliance advice?
No. It is a diligence structure compiled from public frameworks and common AI procurement practice. Contract terms, data-protection obligations and sector rules require counsel review. Where a question depends on the contract, the answer is “verify in the contract”, not a vendor claim.
Sources and verification
Primary sources were opened and checked on 20 November 2026. Product terms and datasets change; the linked source is authoritative.
- NIST AI Risk Management Framework — govern, map, measure and manage functions checked 20 November 2026
- ISO/IEC 42001: AI management systems — the management-system standard and its scope checked 20 November 2026
- OWASP Top 10 for LLM Applications — LLM01 prompt injection and the wider risk list checked 20 November 2026
- NIST Privacy Framework — privacy-risk structure used in the data section checked 20 November 2026
Need the vendor's side of the questionnaire?
Plugsky publishes its plan caps, model catalogue and API reference. Start with a scoped key and test the endpoint before you send production data.
Read the API reference → See plans