This checklist is for IT, security, legal and procurement teams reviewing an AI tool before the organization buys it, and for department leaders who want to know what that review will involve. It applies whether you are evaluating a company-wide assistant, a meeting notetaker, an automation platform or an AI feature added to software you already use.
Most of the questions are familiar from any SaaS review. What makes AI vendors different is the data flow: prompts, files and connected content are sent to models that may be run by a different company, outputs may be retained or used to improve the product, and agents may take actions in your other systems. A vendor can pass a standard security questionnaire and still leave those questions unanswered.
Use this guide in two ways: read the explanations to understand why each question matters, then copy the printable checklist near the end into your intake process.
What matters at organization scale
- Where your data goes, and who else touches it. Many AI products are built on models from a small number of providers. Your contract is with the tool vendor, but your data may also be processed by the model provider and cloud host.
- What the vendor may do with it. Training, product improvement, human review for abuse monitoring and retention periods vary widely, and the defaults often differ between consumer, team and enterprise tiers of the same product.
- Identity and control. SSO, SCIM, admin roles and audit logs determine whether IT can actually manage the tool once staff start using it. These are frequently reserved for the top tier.
- Legal protection. Data processing terms, IP indemnity, output ownership and liability caps determine who carries the risk when something goes wrong.
- Exit. You need to know you can get your data out, and have it deleted, if you switch vendors or the vendor shuts down. Tools do shut down: Relay.app, an automation tool, closed in September 2026 and deleted all accounts and workflows.
- Proportionality. A notetaker that records customer calls deserves a deeper review than an AI grammar checker used on public marketing copy. Tier your reviews by data sensitivity.
The main options
“AI vendor” covers very different kinds of company. Knowing which kind you are dealing with tells you where to focus.
Frontier model providers’ own apps
Examples include ChatGPT, Claude and Google Gemini. The vendor runs its own models, so there are fewer parties in the data flow, and enterprise tiers typically offer SSO, SCIM, audit logs, published certifications and no-training commitments.
Limitations: features change quickly, so each new capability (agents, browsing, connectors) needs a fresh look. Consumer and enterprise terms differ sharply, so staff on personal accounts are not covered.
Best for: Company-wide assistants where you want the shortest data chain and mature enterprise controls.
Suite-embedded assistants
Examples include Microsoft 365 Copilot (Microsoft Copilot is our page for Microsoft’s assistant) and Gemini in Google Workspace. These inherit your existing identity, permissions and compliance tooling, which simplifies review.
Limitations: they expose any oversharing already present in your file stores, and licensing is often layered, with basic features included and organizational data access sold separately.
Best for: Organizations standardized on one productivity suite.
Application vendors built on third-party models
Most AI meeting notetakers, writing tools, chatbot builders and automation platforms fall here, such as Fireflies.ai, Notion or Zapier. The vendor builds the product and calls one or more model providers behind the scenes.
Limitations: the sub-processor chain is longer, and the vendor’s no-training promise only helps if the model providers it uses are bound by equivalent terms. Controls vary a lot between vendors and tiers.
Best for: Department-specific workflows, provided the vendor can document its model providers and their terms.
AI features added to software you already use
CRMs, helpdesks, design tools and project management apps increasingly add AI features, sometimes switched on by default. These are easy to miss because no new purchase is involved.
Limitations: the AI feature may be governed by an addendum rather than your original contract, and may send data to a model provider not listed in your original review.
Best for: Low-friction adoption, once you have checked the updated terms and admin toggles.
Self-hosted and open-source tools
Tools such as n8n and Dify can be self-hosted, which keeps orchestration and stored data on your infrastructure.
Limitations: you take on patching, access control and monitoring, and the models you connect may still be external APIs with their own terms.
Best for: Technical teams with strict data-location requirements and the capacity to operate the software.
Side-by-side
| Vendor type | Best for | Enterprise controls | Deployment | Main trade-off |
|---|---|---|---|---|
| Frontier model provider apps | Company-wide assistants | Usually strong on enterprise tiers | Cloud | Fast feature changes to re-review |
| Suite-embedded assistants | Single-suite organizations | Inherit suite identity, DLP and retention | Cloud (your suite) | Surfaces existing oversharing |
| Apps on third-party models | Department workflows | Varies widely by vendor and tier | Cloud | Longer sub-processor chain |
| AI features in existing software | Low-friction adoption | Depends on host product | Cloud | Easy to miss; new terms apply |
| Self-hosted / open source | Strict data location | You build and run them | Self-hosted, cloud | Operational burden on your team |
Security and compliance questions to ask
Data retention
Ask how long prompts, outputs, uploaded files, recordings and connector content are kept by default; whether admins can shorten retention; and whether “delete” in the interface actually deletes data from backups and logs, and on what schedule. Ask separately about retention for abuse monitoring, which some providers keep for a fixed period even when other retention is off.
Training on customer data
Get a written statement that your data is not used to train or fine-tune models, including the vendor’s own models and those of its model providers. Check how feedback (thumbs up or down, bug reports) is treated, since it is sometimes excluded from the no-training commitment. Confirm whether the commitment applies on the tier you are buying, not only on the enterprise plan.
Sub-processors and underlying model providers
Request the sub-processor list, and specifically which model providers process your data, in which regions, and under what terms. Ask how you will be notified when the list changes, and whether you can object. If the product routes requests between several models, ask whether you can restrict which ones are used.
SSO and SCIM
Confirm SAML or OIDC single sign-on, enforcement (so users cannot bypass SSO with a password), and SCIM provisioning so that accounts are removed automatically when people leave. Ask whether domain capture is available to bring existing personal accounts on your company domain into the managed workspace.
Certifications: SOC 2 and ISO 27001
Ask for the current SOC 2 Type II report and ISO 27001 certificate, and check that their scope covers the product you are buying, not just the parent company or another product line. Note the audit period and any exceptions. For AI-specific governance, some vendors also hold ISO 42001. A Type I report or a “compliant with” statement is weaker than a Type II report.
DPA and data residency
Review the data processing agreement for GDPR and other applicable privacy laws, including transfer mechanisms for international data flows. If you need data stored or processed in a particular region, confirm which features honor that choice; residency sometimes covers stored data but not model processing, or excludes newer features.
HIPAA BAA
If staff may enter protected health information, you need a business associate agreement. Ask whether one is offered, for which plans, and which features it covers, since BAAs often exclude certain features such as web browsing or third-party integrations.
Audit logs
Ask what events are logged (sign-ins, admin changes, sharing, file uploads, connector access, agent actions), how long logs are retained, and whether they can be exported to your SIEM through an API. For regulated industries, ask whether conversation content can be exported for eDiscovery and legal hold.
Admin controls
Check whether admins can restrict features, connectors, web access, file uploads, agent building and external sharing by user group, and whether they can see usage by team.
IP indemnity and output ownership
Ask whether the vendor indemnifies you against third-party IP claims arising from outputs, what conditions apply (for example, using built-in safety filters), and the liability cap. Confirm that your organization owns inputs and outputs, and that the vendor claims no license beyond what is needed to provide the service.
Incident response
Ask for the breach notification timeline in the contract, who your contact is, and how the vendor handles AI-specific issues such as prompt injection through connected content, or a model producing another customer’s data. Ask whether they run a vulnerability disclosure or bug bounty program.
Exit and data export
Confirm you can export conversations, files, custom assistants, workflows and configurations in a usable format, and that the vendor will delete your data and confirm deletion within a stated period after termination. Ask what happens to your data if the vendor is acquired or shuts down.
Shadow AI policy
The best vendor review is wasted if staff keep pasting data into personal accounts. Pair each approval with a policy that lists approved tools, what data each may be used with, and how to request a new one. Make the approval path fast; a slow process pushes people back to unsanctioned tools.
Printable checklist
Copy this into your intake form. A “no” or “unknown” is not automatically disqualifying, but it should be a documented, approved risk.
Data use
- Written commitment: our data is not used to train or improve models (vendor and model providers)
- Commitment applies to the tier we are buying
- Treatment of feedback submissions is documented
- Default retention periods documented for prompts, outputs, files and recordings
- Admin-configurable retention, including zero or short retention if needed
- Deletion timeline documented, including backups and abuse-monitoring logs
Third parties
- Sub-processor list received, including underlying model providers and regions
- Change-notification process and right to object
- Ability to restrict which models are used (if multi-model)
Identity and access
- SAML/OIDC SSO with enforcement
- SCIM provisioning and deprovisioning
- Domain capture for existing personal accounts
- Role-based admin; feature, connector and sharing controls by group
- Connectors respect source-system permissions; ability to exclude sources
Assurance
- SOC 2 Type II report (current, scope covers this product)
- ISO 27001 certificate (scope covers this product)
- Penetration test summary or equivalent
- Vulnerability disclosure program
Legal and privacy
- DPA signed; international transfer mechanism in place
- Data residency meets our requirements, including for AI processing
- HIPAA BAA available, if applicable, with covered features listed
- IP indemnity for outputs; conditions and cap understood
- Customer owns inputs and outputs
Operations
- Audit logs cover sign-ins, admin actions, sharing, uploads, connectors and agent actions
- Log export via API to our SIEM; retention period acceptable
- eDiscovery / legal hold support, if required
- Contractual breach notification timeline
- Named security contact
Exit
- Export of conversations, files, assistants, workflows and settings
- Post-termination deletion with written confirmation
- Plan for vendor acquisition or shutdown
Internal
- Data classification: which data types may be used with this tool
- Acceptable-use policy published; human-review requirements defined
- Tool added to the approved-tools list; owner assigned
- Review date set (at least annually, and when major features launch)
Rolling it out
Pilot. Run the security review in parallel with a small pilot rather than after it, so problems surface before users are attached to the tool. Limit the pilot to data classes the vendor has been cleared for.
Evaluation criteria. Score each vendor on the checklist sections above, weighted by your risk profile. A healthcare provider will weight the BAA and audit logs heavily; a marketing agency may weight IP indemnity and output ownership more.
Rollout. Tie approval to configuration: SSO enforced, SCIM connected, retention set, risky features disabled until reviewed. Publish the approved-tools list and the request process at the same time.
Measurement. Track the number of AI tools in use (from SSO logs, expense reports and network monitoring), the share that are sanctioned, time to approve a new tool request, and policy exceptions. These tell you whether governance is working without inventing productivity figures.
Common mistakes
- Reviewing the parent company instead of the product. Certifications and no-training commitments may not cover every product or tier.
- Stopping at the vendor. The model provider behind an AI app is part of your data flow.
- Ignoring AI features in existing tools. New AI features can arrive through a product update with new terms.
- One-time reviews. AI products add agents, browsing and connectors frequently; set a re-review trigger.
- Blocking without an alternative. Banning AI tools without offering an approved one pushes usage onto personal accounts you cannot see.
FAQ
Is a SOC 2 report enough?
No. It shows the vendor runs sound security controls, but it says nothing about whether your data is used for training, which model providers see it, or what indemnity you get. Treat it as necessary, not sufficient.
Do we need a separate review for every AI feature in existing software?
Not a full review, but a lightweight check: what data the feature sends, to which provider, under what terms, and whether admins can turn it off. Record the outcome alongside the original vendor review.
How do we find shadow AI?
Look at SSO and OAuth grant logs, expense reports, browser extension inventories and network logs for AI domains. Then talk to teams: most shadow AI exists because people needed something the approved tools didn’t do.
Should we require self-hosting?
Only where regulation or contracts demand it. Self-hosting shifts risk to your team’s ability to operate the software, and the models you connect may still be external.
The bottom line
AI vendor reviews build on standard SaaS due diligence, with extra attention to training, retention, the model providers behind the product, and what agents and connectors can reach. The printable checklist above covers the questions that matter; the harder part is applying it consistently, re-checking when products change, and giving staff a fast path to approved tools so they don’t work around you.
For platform-specific comparisons, see our guide to choosing a company-wide AI assistant, the operations and IT use-case page, and the enterprise hub.