Federal agencies are acquiring dozens of proprietary AI algorithms for tasks that can affect people’s physical safety and civil rights without having access to detailed information about how the systems work or were trained, according to newly released data.
Customs and Border Protection and the Transportation Security Administration don’t have documentation about the quality of the data used to build and evaluate algorithms that scan travelers’ bodies for threats, according to the agencies’ 2024 AI inventory reports.
The Veterans Health Administration is in the process of acquiring an algorithm from a private company that is supposed to predict chronic diseases among veterans, but the agency said it is “unclear how the company obtained the data” about veterans’ medical records it used to train the model.
And for more than 100 algorithms that can impact people’s safety and rights, the agency using the models didn’t have access to source code that explains how they work.
As the incoming Trump administration prepares to scrap recently enacted rules for federal AI procurement and safety, the inventory data shows how heavily the government has come to rely on private companies for its riskiest AI systems.
“I’m really worried about proprietary systems that wrestle democratic power away from agencies to manage and deliver benefits and services to people,” said Varoon Mathur, who until earlier this month was a senior AI advisor to the White House responsible for coordinating the AI inventory process. “We have to work hand in hand with proprietary vendors. A lot of the time that’s beneficial, but a lot of the time we don’t know what they’re doing. And if we don’t have control over our data, how are we going to manage risk?”
Internal studies and outside investigations have found serious problems with some federal agencies’ high-risk algorithms, such as a racially biased model the IRS used to determine which taxpayers to audit and a VA suicide prevention algorithm that prioritized white men over other groups.
The 2024 inventories provide the most detailed look yet at how the federal government uses artificial intelligence and what it knows about those systems. For the first time since the inventorying began in 2022, agencies had to answer a host of questions about whether they had access to model documentation or source code and whether they had evaluated the risks associated with their AI systems.
Of the 1,757 AI systems agencies reported using throughout the year, 227 were deemed likely to impact civil rights or physical safety and more than half of those highest-risk systems were developed entirely by commercial vendors. (For 60 of the high-risk systems, agencies didn’t provide information on who built them. Some agencies, including the Department of Justice, Department of Education, and Department of Transportation have not yet published their AI inventories, and military and intelligence agencies are not required to do so).
For at least 25 safety or rights-impacting systems, agencies reported that “no documentation exists regarding maintenance, composition, quality, or intended use of the training and evaluation data.” For at least 105 of them, agencies said they did not have access to source code. Agencies didn’t answer the documentation question for 51 of the tools or the source code question for 60 of the tools. Some of the high-risk systems are still in the development or acquisition phase.
Under the Biden administration, the Office of Management and Budget (OMB) issued new directives to agencies requiring them to perform thorough evaluations of risky AI systems and to ensure that contracts with AI vendors grant access to necessary information about the models, which can include training data documentation or the code itself.
The rules are more vigorous than anything AI vendors are likely to encounter when selling their products to other companies or to state and local governments (although many states will be considering AI safety bills in 2025) and government software vendors have pushed back on them, arguing that agencies should decide what kind of evaluation and transparency is necessary on a case-by-case basis.
“Trust but verify,” said Paul Lekas, head of global public policy for the Software & Information Industry Association. “We’re wary of burdensome requirements on AI developers. At the same time, we recognize that there needs to be some attention to what degree of transparency is required to develop that kind of trust that the government needs to use these tools.”
The U.S. Chamber of Commerce, in comments submitted to OMB about the new rules, said “the government should not request any specific training data or data sets on AI models that the government acquires from vendors.” Palantir, a major AI supplier, wrote that the federal government should “avoid overly prescribing rigid documentation instruments, and instead give AI service providers and vendors the needed leeway to characterize context-specific risk.”
Rather than access to training data or source code, AI vendors say that in most cases, agencies should feel comfortable with model scorecards—documents that characterize the data and machine learning techniques an AI model employs but don’t include technical details that companies consider trade secrets.
Cari Miller, who has helped develop international standards for buying algorithms and co-founded the nonprofit AI Procurement Lab, described the scorecards as a lobbyist’s solution that is “not a bad starting point, but only a starting point” for what vendors of high-risk algorithms should be contractually required to disclose.
“Procurement is one of the most important governance mechanisms, it’s where the rubber meets the road, it’s the front door, it’s where you can decide whether or not to let the bad stuff in,” she said. “You need to understand whether the data in that model is representative, is it biased or unbiased? What did they do with that data and where did it come from? Did all of it come from Reddit or Quora? Because if it did, it may not be what you need.”
As OMB noted when rolling out its AI rules, the federal government is the largest single buyer in the U.S. economy, responsible for more than $100 billion in IT purchases in 2023. The direction it takes on AI—what it requires vendors to disclose and how it tests products before implementing them—is likely to set the standard for how transparent AI companies are about their products when selling to smaller government agencies or even to other private companies.
President-elect Trump has strongly signaled his intention to roll back OMB’s rules. He campaigned on a party platform that called for a “repeal [of] Joe Biden’s dangerous Executive Order that hinders AI Innovation, and imposes Radical Leftwing ideas on the development of this technology.”
Mathur, the former White House senior AI advisor, said he hopes the incoming administration doesn’t follow through on that promise and pointed out that Trump kick-started efforts to build trust in federal AI systems with his executive order in 2020.
Just getting agencies to inventory their AI systems and answer questions about the proprietary systems they use was a monumental task, Mathur said, that has been “profoundly useful” but requires follow-through.
“If we don’t have the code or the data or the algorithm we’re not going to be able to understand the impact we’re having,” he said.
+ There are no comments
Add yours