Free tool · no sign-up

AI vendor security questionnaire: 120 weighted questions

Score an AI vendor across data handling, model governance, residency, subprocessors, incident response and evaluations. Export the answers, track coverage, and record the conditions.

Your assessment

Overall score—
BandUnscored
Coverage0 / 120
Answered points0 / 0
Critical “No” (weight 3)0
N/A0
Category score. A category with no answered questions shows “—”.
SectionAnsweredPointsScore

Answers are saved in this browser's local storage. Nothing is submitted to Plugsky. The score is a review aid, not a certification.

Six sections, 120 questions, one score

The questionnaire is built for AI procurement, not generic SaaS review. It asks about model versions and provenance, prompt and completion retention, inference residency, tool permissions and evaluation evidence. The generic controls are still there — SSO, encryption, incident response — because they are prerequisites, not substitutes.

Section weights. Points are the maximum available before N/A exclusions.
SectionQuestionsMax pointsWhat the evidence should show
Data handling and privacy2054How customer content, personal data and secrets are protected, accessed and deleted.
Model governance and provenance2049How models are selected, documented, evaluated, changed and retired.
Residency, transfer and retention2050Where data is processed and stored, how it moves, and when it is deleted.
Subprocessors and supply chain2047Who else touches the data, how they are contracted, and how concentration is managed.
Incident response and resilience2047How failures, breaches and outages are detected, notified, recovered and reviewed.
Evaluation, monitoring and assurance2042How quality, safety and drift are measured after launch, and what evidence is available.

Scoring bands

The bands are deliberately simple. They are not industry benchmarks and they do not replace a risk appetite set by your own security or legal team. Use them to structure a go/no-go conversation, then record the exceptions.

Suggested bands. A score is provisional until coverage is at least 80%.
ScoreBandWhat it means in practice
90–100StrongEvidence is complete for the critical questions. Remaining gaps are low-weight or have compensating controls.
75–89Acceptable with conditionsMaterial controls exist, but at least one important question is partial or unanswered. Add contractual commitments and a review date.
50–74Material gapsSeveral weight-3 questions are “No” or unverified. Do not deploy sensitive data until the gaps are closed or accepted by the risk owner.
0–49High riskCore controls are missing or unverified. Treat the vendor as unsuitable for regulated or high-impact data.

What good looks like

Use the maturity levels to calibrate an answer. A vendor that says “yes” but cannot produce evidence is at level 1, not level 3. The evidence column in each question tells you what to ask for.

A four-level rubric for any question. The level is about evidence, not confidence.
LevelLabelObservable test
1AssertedThe vendor says the control exists, but there is no document, screenshot, report or test result.
2DocumentedA policy or design document exists and is current, but it has not been independently tested.
3TestedThere is recent evidence of operation: a test result, audit sample, restore drill, red-team report or access review.
4ContinuousThe control is monitored, alerted and reviewed on a defined cadence, with a named owner and a remediation record.

The questions that decide the deal

Not all 120 questions carry equal risk. In most AI reviews, five clusters decide the outcome:

  1. Training use. Does the vendor train on customer content by default, and can you turn it off? A “yes” without a contractual opt-out is a data-leakage issue, not a feature choice.
  2. Retention and deletion. Prompts and completions can contain regulated data. Ask for the retention matrix, the deletion SLA, and evidence that backups follow it. “Deleted from the primary store” is not the same as “deleted everywhere”.
  3. Residency and routing. “We have a region” is not “inference stays in the region”. Ask whether region pinning is contractual or best-effort, and what happens during capacity failover.
  4. Subprocessor change. A model provider or observability vendor can be added after signature. Ask for notice, objection rights and a current list with countries.
  5. Tool authority. If the vendor's agent can write, send, refund or delete, the questionnaire needs the tool permission model, approval workflow and audit log. A model that is accurate but over-privileged is still a high-risk system.

These five clusters map to weight-3 questions in the data, residency, subprocessor and model sections. If you have limited time, answer those first and leave the weight-1 questions for the second pass.

How to run the review

  1. Set the threshold first. Decide what score and which critical questions are mandatory before you see the answers. This prevents the score from being rationalised after the fact.
  2. Send the questionnaire with the evidence request. The evidence line under each question is what makes the answer auditable. A vendor that returns only “yes/no” has not completed the review.
  3. Score with coverage visible. A 95% score on 20 of 120 questions is not a 95% vendor. Use the coverage KPI and the unanswered filter.
  4. Verify the critical claims. Read the SOC 2 scope, the ISO certificate, the pen-test summary and the DPA. Do not accept a logo or a trust-centre page as the only evidence.
  5. Record conditions. For every partial or no, write the compensating control, the owner and the review date. Export the JSON and attach it to the contract file.
  6. Re-run after material change. A new model, a new subprocessor or a new region is a new review, not an addendum.

Contract language and limits

Several questions cannot be answered by a security team alone. Training use, deletion, residency, subprocessor notice and audit rights belong in the contract. Where the answer depends on the agreement, the correct entry is “verify in the contract”, not “yes” because the sales engineer said so. Have counsel review the current terms, including the DPA, the acceptable-use policy and any AI-specific annex.

Certifications are claims about a scope and a period. A SOC 2 Type II report covers a defined system and observation window; an ISO/IEC 27001 certificate covers a defined scope. Check the scope, the period and the audit firm. Do not treat a certification as coverage for a service or region it does not include. If a vendor cannot produce the report, record that as evidence missing, not as a failure of the control.

The questionnaire does not cover sector-specific rules (HIPAA, PCI DSS, financial supervision) or every local data-protection requirement. Add those questions separately. Related: the AI vendor risk assessment checklist, the enterprise AI security checklist, and the enterprise article index.

The 120 questions

Answer what you can. Every question includes the evidence to request; the score only counts what you have answered, and coverage is reported separately. N/A removes a question from the denominator, so use it only when the question genuinely does not apply.

Data handling and privacy 20 questions · 54 points

How customer content, personal data and secrets are protected, accessed and deleted.

D01 · Data handling and privacy · weight 3

Is a data processing agreement (DPA) available, and does it cover the AI processing, not just the website?

Evidence to request: Executed DPA, annexes, and the list of processing activities.

D02 · Data handling and privacy · weight 3

Can you confirm that customer content is not used to train shared or foundation models without explicit opt-in?

Evidence to request: Contract clause, product setting, and a written statement of default behaviour.

D03 · Data handling and privacy · weight 3

Can customer data be classified and segregated by sensitivity or tenant?

Evidence to request: Data model, tenant isolation design, and configuration screenshots.

D04 · Data handling and privacy · weight 2

Are PII redaction, secret detection or DLP controls available before content reaches the model?

Evidence to request: Feature documentation, test results, and the list of supported detectors.

D05 · Data handling and privacy · weight 3

Is data encrypted at rest with a documented algorithm and key-management model?

Evidence to request: Encryption standard, KMS design, and key rotation policy.

D06 · Data handling and privacy · weight 3

Is data encrypted in transit with TLS 1.2 or later, and are weak cipher suites disabled?

Evidence to request: TLS configuration, certificate policy, and an external scan.

D07 · Data handling and privacy · weight 3

Is bring-your-own-key (BYOK) or customer-managed encryption available for the relevant data stores?

Evidence to request: Key-management guide and the scope of BYOK coverage.

D08 · Data handling and privacy · weight 3

Can customer content be deleted on request, and what is the maximum deletion time?

Evidence to request: Deletion runbook, API or console evidence, and a recent deletion test.

D09 · Data handling and privacy · weight 2

Do backups containing customer content follow the same deletion and retention rules?

Evidence to request: Backup schedule, retention matrix, and deletion procedure.

D10 · Data handling and privacy · weight 2

Is there a documented data-minimisation and retention schedule for prompts, completions and logs?

Evidence to request: Retention matrix by data type, with default and maximum periods.

D11 · Data handling and privacy · weight 3

Is access to customer content restricted by least privilege and reviewed periodically?

Evidence to request: Role matrix, access-review evidence, and break-glass procedure.

D12 · Data handling and privacy · weight 3

Is SSO supported through SAML or OIDC, and can the customer enforce it?

Evidence to request: SSO configuration guide and enforcement settings.

D13 · Data handling and privacy · weight 2

Is SCIM provisioning and de-provisioning supported?

Evidence to request: SCIM documentation and a test of leaver access removal.

D14 · Data handling and privacy · weight 3

Is MFA required for administrative and support access?

Evidence to request: MFA policy and enforcement evidence.

D15 · Data handling and privacy · weight 3

Are audit logs of data access and administrative actions available to the customer?

Evidence to request: Log schema, retention period, and export method.

D16 · Data handling and privacy · weight 2

Can log retention be configured or shortened by the customer?

Evidence to request: Retention settings and the minimum enforceable period.

D17 · Data handling and privacy · weight 2

Can customer data be exported in a machine-readable format on request?

Evidence to request: Export format, scope, and turnaround time.

D18 · Data handling and privacy · weight 3

Are sensitive data categories (health, biometric, government ID, children's data) prohibited or specially handled?

Evidence to request: Acceptable-use policy and the technical controls that enforce it.

D19 · Data handling and privacy · weight 3

Do you support data-subject requests such as access, correction and erasure?

Evidence to request: DSR workflow, SLA, and the identity-verification method.

D20 · Data handling and privacy · weight 3

Is there a documented lawful transfer mechanism for cross-border data?

Evidence to request: Transfer impact assessment, SCCs or equivalent, and the country list.

Model governance and provenance 20 questions · 49 points

How models are selected, documented, evaluated, changed and retired.

M01 · Model governance and provenance · weight 3

Is there a maintained inventory of models, versions and deployment endpoints?

Evidence to request: Model register with owner, version, endpoint and status.

M02 · Model governance and provenance · weight 3

Does every production model have a model card covering intended use, limits and known failure modes?

Evidence to request: Model cards and the review date for each production model.

M03 · Model governance and provenance · weight 2

Is the provenance of training data documented, including licensed and synthetic data?

Evidence to request: Data provenance summary and licence inventory.

M04 · Model governance and provenance · weight 3

Are model weights, fine-tuning data and training code licensed for the vendor's commercial use?

Evidence to request: Licence review and any restrictions that flow to customers.

M05 · Model governance and provenance · weight 3

Is there a pre-release evaluation gate for quality, safety and regressions?

Evidence to request: Eval plan, acceptance thresholds, and a recent release record.

M06 · Model governance and provenance · weight 2

Is red teaming performed before major model or prompt changes?

Evidence to request: Red-team scope, cadence, and remediation tracker.

M07 · Model governance and provenance · weight 3

Is prompt injection tested across direct, indirect, RAG and tool surfaces?

Evidence to request: Test suite, recent results, and the remediation owner.

M08 · Model governance and provenance · weight 2

Are hallucination and grounding risks measured for retrieval and tool use?

Evidence to request: Groundedness metric, sample size, and the latest result.

M09 · Model governance and provenance · weight 2

Are bias and toxicity evaluated for the intended use and user population?

Evidence to request: Evaluation methodology, disaggregated results, and mitigations.

M10 · Model governance and provenance · weight 3

Is human oversight defined for high-impact or irreversible actions?

Evidence to request: Approval workflow, role definitions, and audit trail.

M11 · Model governance and provenance · weight 3

Will customers be notified before a model change that could alter outputs or behaviour?

Evidence to request: Change-notification policy with lead time and channels.

M12 · Model governance and provenance · weight 2

Is there a deprecation policy with migration support?

Evidence to request: Deprecation timeline, notice period, and migration guide.

M13 · Model governance and provenance · weight 3

Can a model or prompt change be rolled back quickly?

Evidence to request: Rollback procedure, tested time, and version pinning.

M14 · Model governance and provenance · weight 1

Are determinism controls such as seed and temperature exposed and documented?

Evidence to request: API reference and the limits of determinism.

M15 · Model governance and provenance · weight 3

Is fine-tuning data isolated per customer, and is it deleted when the job ends?

Evidence to request: Isolation design, retention period, and deletion evidence.

M16 · Model governance and provenance · weight 2

Does the contract address ownership of model outputs and customer fine-tunes?

Evidence to request: IP clause and the treatment of derivative weights.

M17 · Model governance and provenance · weight 2

Is there an IP indemnity for model outputs, and what does it exclude?

Evidence to request: Indemnity clause and exclusions. Verify in the contract.

M18 · Model governance and provenance · weight 2

Is the model open-weight, proprietary, or a hosted third-party model, and is that disclosed?

Evidence to request: Model provenance statement and the dependency list.

M19 · Model governance and provenance · weight 2

Are safety system cards or acceptable-use enforcement details available?

Evidence to request: Safety documentation and enforcement examples.

M20 · Model governance and provenance · weight 3

Which third-party model providers are in the inference path, and can they change without notice?

Evidence to request: Dependency map, routing policy, and change-notification terms.

Residency, transfer and retention 20 questions · 50 points

Where data is processed and stored, how it moves, and when it is deleted.

R01 · Residency, transfer and retention · weight 3

Can the customer choose the region where inference and storage occur?

Evidence to request: Region list, control plane, and the default region.

R02 · Residency, transfer and retention · weight 3

Is region pinning enforceable, or is it a best-effort routing preference?

Evidence to request: Contract language and a routing test.

R03 · Residency, transfer and retention · weight 3

Are the locations of all subprocessors disclosed, including support and observability?

Evidence to request: Subprocessor list with countries and purposes.

R04 · Residency, transfer and retention · weight 3

Has a transfer impact assessment been completed for each cross-border flow?

Evidence to request: TIA summary and the legal basis.

R05 · Residency, transfer and retention · weight 3

Are Standard Contractual Clauses or an equivalent mechanism in place?

Evidence to request: Signed SCCs and the module used.

R06 · Residency, transfer and retention · weight 3

What is the default retention period for prompts, completions, logs and backups?

Evidence to request: Retention matrix with default and maximum.

R07 · Residency, transfer and retention · weight 2

Is deletion certified or evidenced, and can the customer obtain a certificate?

Evidence to request: Deletion certificate template and a recent example.

R08 · Residency, transfer and retention · weight 2

Where are backups stored, and do they cross borders?

Evidence to request: Backup topology and country list.

R09 · Residency, transfer and retention · weight 3

Can vendor support access customer content from another country, and is that access logged?

Evidence to request: Support-access policy and audit evidence.

R10 · Residency, transfer and retention · weight 3

Where are logs and observability data stored?

Evidence to request: Log pipeline diagram and storage regions.

R11 · Residency, transfer and retention · weight 3

Is inference performed in the selected region, or can it be routed elsewhere for capacity?

Evidence to request: Routing policy and failover behaviour.

R12 · Residency, transfer and retention · weight 2

Is there a disaster-recovery region, and where is it?

Evidence to request: DR plan and the recovery-region list.

R13 · Residency, transfer and retention · weight 2

Are there data-sovereignty commitments that survive a change of control?

Evidence to request: Contract clause and assignment terms.

R14 · Residency, transfer and retention · weight 3

What is the policy for government or law-enforcement requests for customer data?

Evidence to request: Transparency policy, legal-review process, and notification practice.

R15 · Residency, transfer and retention · weight 1

Is a transparency report published, and how often?

Evidence to request: Latest report and publication cadence.

R16 · Residency, transfer and retention · weight 2

Are there country-specific localisation commitments?

Evidence to request: Localisation annex and the countries covered.

R17 · Residency, transfer and retention · weight 2

Are cross-border support sessions time-limited and approved by the customer?

Evidence to request: Access workflow and approval evidence.

R18 · Residency, transfer and retention · weight 3

Is customer data returned in a usable format at termination?

Evidence to request: Exit annex, format, and timeline.

R19 · Residency, transfer and retention · weight 2

Is transition assistance available, and at what cost?

Evidence to request: Exit assistance clause and pricing.

R20 · Residency, transfer and retention · weight 2

How are legal holds handled without breaking deletion commitments?

Evidence to request: Legal-hold procedure and customer notice.

Subprocessors and supply chain 20 questions · 47 points

Who else touches the data, how they are contracted, and how concentration is managed.

S01 · Subprocessors and supply chain · weight 3

Is a current subprocessor list published with purpose and location?

Evidence to request: Public list URL and last-updated date.

S02 · Subprocessors and supply chain · weight 3

Will the customer receive advance notice of new subprocessors?

Evidence to request: Notice period and channel.

S03 · Subprocessors and supply chain · weight 3

Does the customer have a right to object to a new subprocessor?

Evidence to request: Objection process and remedy.

S04 · Subprocessors and supply chain · weight 3

Are data-protection and security obligations flowed down to every subprocessor?

Evidence to request: Contract template and flow-down clause.

S05 · Subprocessors and supply chain · weight 2

Are subprocessors assessed before onboarding and re-assessed periodically?

Evidence to request: Assessment questionnaire and cadence.

S06 · Subprocessors and supply chain · weight 2

Are fourth-party dependencies identified where they affect customer data?

Evidence to request: Dependency map and criticality rating.

S07 · Subprocessors and supply chain · weight 2

Is there concentration risk from a single cloud, model or identity provider?

Evidence to request: Concentration analysis and mitigation plan.

S08 · Subprocessors and supply chain · weight 2

Is the hardware supply chain reviewed for provenance and tampering?

Evidence to request: Supplier list and hardware attestation.

S09 · Subprocessors and supply chain · weight 3

What happens if a model provider withdraws or changes terms?

Evidence to request: Fallback model list and contractual protection.

S10 · Subprocessors and supply chain · weight 2

Is there an open-source dependency policy and a licence review process?

Evidence to request: Policy, SBOM, and licence exceptions.

S11 · Subprocessors and supply chain · weight 2

Is a software bill of materials (SBOM) available for the agent or gateway components?

Evidence to request: SBOM format, cadence, and access.

S12 · Subprocessors and supply chain · weight 2

Is there a vulnerability disclosure programme?

Evidence to request: Security.txt, disclosure policy, and response SLA.

S13 · Subprocessors and supply chain · weight 3

Is penetration testing performed by an independent party, and how often?

Evidence to request: Latest executive summary and remediation status.

S14 · Subprocessors and supply chain · weight 3

Is a SOC 2 Type II report available, and what is the observation period?

Evidence to request: Report and bridge letter. Verify scope and period.

S15 · Subprocessors and supply chain · weight 3

Is an ISO/IEC 27001 certificate available, and what is its scope?

Evidence to request: Certificate and statement of applicability. Verify with the certification body.

S16 · Subprocessors and supply chain · weight 2

Are other certifications or attestations claimed, and what exactly do they cover?

Evidence to request: Certificates and scope statements. Do not accept a logo as evidence.

S17 · Subprocessors and supply chain · weight 1

Is cyber-liability insurance in place, and what is the limit?

Evidence to request: Certificate of insurance and exclusions.

S18 · Subprocessors and supply chain · weight 2

Is the vendor financially viable for the contract term?

Evidence to request: Audited accounts or a financial-health summary.

S19 · Subprocessors and supply chain · weight 2

Is there an exit plan if the vendor fails or is acquired?

Evidence to request: Exit plan and escrow arrangement.

S20 · Subprocessors and supply chain · weight 2

Is source-code or data escrow available for critical components?

Evidence to request: Escrow agreement and release conditions.

Incident response and resilience 20 questions · 47 points

How failures, breaches and outages are detected, notified, recovered and reviewed.

I01 · Incident response and resilience · weight 3

Is there 24/7 on-call coverage for security and availability incidents?

Evidence to request: On-call rota and escalation policy.

I02 · Incident response and resilience · weight 3

Is there a documented incident-response plan with roles and decision rights?

Evidence to request: Plan, roles, and last review date.

I03 · Incident response and resilience · weight 3

What is the contractual notification timeline for a personal-data breach?

Evidence to request: Clause and the internal detection-to-notification target.

I04 · Incident response and resilience · weight 3

Is there a named customer notification channel and a template?

Evidence to request: Contact process and sample notification.

I05 · Incident response and resilience · weight 2

Is a post-incident review performed, and will the customer receive a summary?

Evidence to request: RCA template and a recent redacted example.

I06 · Incident response and resilience · weight 2

Is the breach playbook tested, and when was the last test?

Evidence to request: Test record and lessons learned.

I07 · Incident response and resilience · weight 2

Are tabletop exercises run with engineering, legal and communications?

Evidence to request: Exercise schedule and participants.

I08 · Incident response and resilience · weight 2

Is forensic capability available in-house or through a retainer?

Evidence to request: Forensics provider and scope.

I09 · Incident response and resilience · weight 2

How is customer communication handled during a prolonged outage?

Evidence to request: Status-page process and update cadence.

I10 · Incident response and resilience · weight 2

Is there a public status page with historical uptime?

Evidence to request: Status-page URL and history.

I11 · Incident response and resilience · weight 3

Are RTO and RPO defined for the service, and do they match the customer's needs?

Evidence to request: RTO/RPO table by service tier.

I12 · Incident response and resilience · weight 2

Is disaster recovery tested at least annually?

Evidence to request: Test report and date.

I13 · Incident response and resilience · weight 2

Is backup restore tested, and what is the measured restore time?

Evidence to request: Restore test record.

I14 · Incident response and resilience · weight 2

Are single points of failure identified for model serving, data and identity?

Evidence to request: Architecture review and mitigation list.

I15 · Incident response and resilience · weight 2

How are abuse, prompt flooding and denial-of-wallet attacks handled?

Evidence to request: Rate limits, quotas, and abuse-detection rules.

I16 · Incident response and resilience · weight 2

Is DDoS protection in place at the edge?

Evidence to request: Edge configuration and scrubbing capacity.

I17 · Incident response and resilience · weight 3

Are vulnerability remediation SLAs defined by severity?

Evidence to request: SLA table and recent performance.

I18 · Incident response and resilience · weight 2

Is there a patch cadence for model servers, gateways and dependencies?

Evidence to request: Patch policy and maintenance windows.

I19 · Incident response and resilience · weight 3

Are logs retained long enough to investigate an incident?

Evidence to request: Log retention and search capability.

I20 · Incident response and resilience · weight 2

How are law-enforcement and government requests handled during an incident?

Evidence to request: Legal escalation and customer-notification policy.

Evaluation, monitoring and assurance 20 questions · 42 points

How quality, safety and drift are measured after launch, and what evidence is available.

E01 · Evaluation, monitoring and assurance · weight 3

Are quality evaluations run continuously in production or on a fixed cadence?

Evidence to request: Eval dashboard and cadence.

E02 · Evaluation, monitoring and assurance · weight 3

Is there a regression suite for every model, prompt or retrieval change?

Evidence to request: Suite definition and release gate.

E03 · Evaluation, monitoring and assurance · weight 2

Is production drift monitored for input distribution and output quality?

Evidence to request: Drift metrics and alert thresholds.

E04 · Evaluation, monitoring and assurance · weight 2

Is hallucination or groundedness measured, and what is the current result?

Evidence to request: Metric definition, sample size, and latest value.

E05 · Evaluation, monitoring and assurance · weight 2

Are safety classifiers or guardrails deployed, and how are they evaluated?

Evidence to request: Guardrail stack and false-positive/negative testing.

E06 · Evaluation, monitoring and assurance · weight 2

Is human review sampled for high-impact outputs?

Evidence to request: Sampling rate, reviewer training, and escalation.

E07 · Evaluation, monitoring and assurance · weight 2

Is user feedback collected and fed back into evaluation?

Evidence to request: Feedback volume, labelling process, and closed-loop evidence.

E08 · Evaluation, monitoring and assurance · weight 2

Is bias testing repeated after model or data changes?

Evidence to request: Test set, disaggregation, and latest result.

E09 · Evaluation, monitoring and assurance · weight 2

Is external red teaming performed at a defined cadence?

Evidence to request: Provider, scope, and report date.

E10 · Evaluation, monitoring and assurance · weight 3

Are benchmark claims accompanied by methodology, version and date?

Evidence to request: Benchmark report with harness and dataset version.

E11 · Evaluation, monitoring and assurance · weight 2

Is the provenance of evaluation data documented, including contamination checks?

Evidence to request: Dataset cards and contamination analysis.

E12 · Evaluation, monitoring and assurance · weight 3

Can an evaluation be reproduced from the published harness and settings?

Evidence to request: Code, config, seeds, and model version.

E13 · Evaluation, monitoring and assurance · weight 1

Has an independent party audited the evaluation methodology?

Evidence to request: Audit report and scope.

E14 · Evaluation, monitoring and assurance · weight 2

Are performance SLAs defined for latency, availability and error rate?

Evidence to request: SLA table and measurement method.

E15 · Evaluation, monitoring and assurance · weight 2

Is accuracy or task success reported to customers, with confidence intervals?

Evidence to request: Report format and sample size.

E16 · Evaluation, monitoring and assurance · weight 2

Are incident and quality metrics reviewed by leadership?

Evidence to request: Review minutes and action tracker.

E17 · Evaluation, monitoring and assurance · weight 1

Is cost monitored per tenant, model and feature?

Evidence to request: Cost dashboard and anomaly alerts.

E18 · Evaluation, monitoring and assurance · weight 2

Is token and usage reporting transparent enough to reconcile invoices?

Evidence to request: Usage API, granularity, and reconciliation.

E19 · Evaluation, monitoring and assurance · weight 2

Is A/B testing governed to avoid unintended user harm?

Evidence to request: Experiment policy, review, and rollback.

E20 · Evaluation, monitoring and assurance · weight 2

Can the customer run its own evaluations against the production endpoint?

Evidence to request: Eval access, rate limits, and data handling.

FAQ

How is the questionnaire scored?

Each question has a weight from 1 to 3. Yes earns the full weight, Partial earns half, No earns zero, and N/A removes the question from the denominator. The overall score is earned points divided by available points for answered questions. Coverage is shown separately so a high score on ten answers is not mistaken for a completed review.

What score is good enough?

There is no universal threshold. The page's bands are a discussion aid: 90–100 strong, 75–89 acceptable with conditions, 50–74 material gaps, below 50 high risk. A regulated workload may require 90+ with no open critical questions; a low-risk internal tool may accept 75 with compensating controls. Set the threshold before you send the questionnaire.

Does a SOC 2 report replace this questionnaire?

No. SOC 2 covers controls over security, availability, processing integrity, confidentiality or privacy, depending on scope. It does not tell you which model version you are buying, where inference runs, whether prompts are retained, or how a subprocessor change is notified. Use the report as evidence for the controls it covers and ask the AI-specific questions separately.

Can I export the answers?

Yes. The export buttons produce JSON for a system of record and CSV for a spreadsheet. Both include the question ID, category, weight, answer and the evidence requested. The page does not send the answers anywhere; the export happens in your browser.

Is this legal or compliance advice?

No. It is a diligence structure compiled from public frameworks and common AI procurement practice. Contract terms, data-protection obligations and sector rules require counsel review. Where a question depends on the contract, the answer is “verify in the contract”, not a vendor claim.

Sources and verification

Primary sources were opened and checked on 20 November 2026. Product terms and datasets change; the linked source is authoritative.

Need the vendor's side of the questionnaire?

Plugsky publishes its plan caps, model catalogue and API reference. Start with a scoped key and test the endpoint before you send production data.

Read the API reference → See plans