AI Governance in Public Sector Software Procurement

Why AI governance is now part of procurement

When a software vendor's system includes components that use machine learning, automated classification, or AI-assisted data extraction, county governments face questions that don't arise with conventional software. Whose data trained the model? How confident is the system in its outputs? Who reviews decisions the system gets wrong? What happens to county records that are processed by the AI?

These questions aren't hypothetical. Courts, auditors, and state oversight bodies are increasingly asking government agencies to document how automated systems are used in their operations — particularly for systems that handle public records or support government decisions. Some states have enacted laws requiring disclosure of AI use in government functions, with more under consideration.

AI governance review is now a standard part of the IT procurement process in a growing number of county governments, especially for DMS vendors that include AI-assisted indexing, document classification, or records extraction.

What an AI governance form typically asks

AI governance questionnaires — sometimes called AI fact sheets, AI transparency disclosures, or algorithmic impact assessments — vary in depth, but commonly cover the following areas:

• System description. What AI or machine learning components does the system include? What tasks do they perform?
• Training data. What data was the model trained on? Does it include government or county records? Can the vendor confirm that county data submitted during use will not be used to retrain the model without consent?
• Accuracy and performance. How is accuracy measured? What are the system's error rates? How does performance vary across document types, age, and condition?
• Human oversight. What is the human review process for AI-generated outputs? What happens when the system's confidence is low?
• Explainability. Can the system explain or document why it produced a particular output? Is there an audit trail for AI-generated field values?
• Data handling. Where is county data processed? Who has access to it? How long is it retained by the vendor?
• Vendor accountability. What contractual protections does the county have if AI outputs cause downstream errors? What is the process for reporting and correcting systematic errors?

Data residency and model transparency

Two questions that frequently come up in AI-specific procurement reviews are data residency and model transparency.

Data residency refers to where county records are physically stored and processed. When an AI component is involved, county data may be sent to a third-party model API or processed in a cloud environment that the county's standard data residency policy wasn't designed to cover. Vendors should be able to specify where AI processing occurs, whether it involves any subprocessors, and what data protections apply.

Model transparency refers to whether the vendor can describe in meaningful terms how the AI model works. This doesn't require a technical explanation of model architecture — but it does require the vendor to explain, in plain language, what inputs the model uses, what it outputs, and how outputs are validated before they affect county records. A vendor who can only describe their system as "proprietary AI" without further detail is providing less information than most government IT departments now require.

Evaluating AI accuracy claims

Vendors regularly lead with accuracy statistics. These figures are worth scrutinizing carefully, because context matters significantly.

An accuracy figure for a DMS vendor's AI indexing capability is only meaningful if you know: what document types were tested, what condition the documents were in (clean scanned PDFs vs. aged paper records), what volume was used, and how accuracy was measured (character-level, field-level, or document-level). A system that performs well on clean, modern printed documents may perform significantly worse on faded or handwritten historical records.

The most reliable way to evaluate AI accuracy for your use case is to run a structured pilot against a representative sample of your office's actual documents — not documents provided by the vendor. Define success criteria in advance: which fields matter most, what minimum accuracy threshold is acceptable, and how exceptions will be handled.

What to look for in vendor AI disclosures

A vendor's AI governance documentation should be able to answer, in writing:

• What the AI component does, specifically — not just that AI is used
• Whether county documents submitted for processing are used to train or improve the model
• What the human review process looks like for low-confidence AI outputs
• How errors in AI-generated outputs are identified, reported, and corrected
• What accuracy benchmarks exist for the document types your office processes

Vendors who can answer these questions directly and in writing are demonstrating a level of transparency that makes the procurement review process faster. Vendors who cannot or will not answer them are creating uncertainty that typically results in longer review timelines and more scrutiny.

State-level AI transparency requirements

A growing number of states have enacted or proposed legislation requiring government agencies to disclose their use of automated decision systems. These laws vary in scope — some apply only to high-stakes decisions affecting individuals, while others have broader applicability to systems that handle government records.

County offices considering a DMS with AI components should check whether their state has existing or pending AI transparency requirements that would affect vendor selection or documentation obligations. Your state's records management agency or county counsel can help determine what applies.