Skip to content
Rush job

AI ruling on jobless claims could make mistakes courts can’t undo, experts warn

Nevada’s plan to let AI rule on unemployment claims is risky, experts warn.

Ashley Belanger | 175

Nevada will soon become the first state to use AI to help speed up the decision-making process when ruling on appeals that impact people's unemployment benefits.

The state's Department of Employment, Training, and Rehabilitation (DETR) agreed to pay Google $1,383,838 for the AI technology, a 2024 budget document shows, and it will be launched within the "next several months," Nevada officials told Gizmodo.

Nevada's first-of-its-kind AI will rely on a Google cloud service called Vertex AI Studio. Connecting to Google's servers, the state will fine-tune the AI system to only reference information from DETR's database, which officials think will ensure its decisions are "more tailored" and the system provides "more accurate results," Gizmodo reported.

Under the contract, DETR will essentially transfer data from transcripts of unemployment appeals hearings and rulings, after which Google's AI system will process that data, upload it to the cloud, and then compare the information to previous cases.

In as little as five minutes, the AI will issue a ruling that would've taken a state employee about three hours to reach without using AI, DETR's information technology administrator, Carl Stanfield, told The Nevada Independent. That's highly valuable to Nevada, which has a backlog of more than 40,000 appeals stemming from a pandemic-related spike in unemployment claims while dealing with "unforeseen staffing shortages" that DETR reported in July.

"The time saving is pretty phenomenal," Stanfield said.

As a safeguard, the AI's determination is then reviewed by a state employee to hopefully catch any mistakes, biases, or perhaps worse, hallucinations where the AI could possibly make up facts that could impact the outcome of their case.

Ars Video

 

Google's spokesperson Ashley Simms told Gizmodo that the tech giant will work with the state to "identify and address any potential bias" and to "help them comply with federal and state requirements.” According to the state's AI guidelines, the agency must prioritize ethical use of the AI system, "avoiding biases and ensuring fairness and transparency in decision-making processes."

If the reviewer accepts the AI ruling, they'll sign off on it and issue the decision. Otherwise, the reviewer will edit the decision and submit feedback so that DETR can investigate what went wrong.

Gizmodo noted that this novel use of AI "represents a significant experiment by state officials and Google in allowing generative AI to influence a high-stakes government decision—one that could put thousands of dollars in unemployed Nevadans’ pockets or take it away."

Google declined to comment on whether more states are considering using AI to weigh jobless claims.

Courts may not be able to undo AI mistakes

Experts, including judicial scholars, lawyers, and a former US Department of Labor official, told Gizmodo that there's a risk that employees might feel pressured to quickly process the AI decisions on appeals because of the DETR's "emphasis on speed."

If the system works perfectly, AI could ensure that people in Nevada get access to unemployment benefits sooner. But if employees rush and fail to carefully monitor for hallucinations or mistakes, the backlog of appeals could get worse as claimants protest any errant AI rulings. There's also a chance that AI rulings could "undermine the claimant’s ability to appeal that wrong decision in a civil court case," experts warned, because a "district court cannot substitute its own judgment for the judgment of the appeal referee." That means that a lazy human reviewer could rubber-stamp an AI ruling that a court can't overturn, experts suggested.

It's still unclear which Google model that Nevada will use, Google's spokesperson told Gizmodo, but Google can't guarantee complete accuracy of any of its AI models. Google's responsible AI guidelines recommend continued monitoring of models after deployment, warning, "issues will occur: any model of the world is imperfect almost by definition."

Research has shown Google's models provide "incorrect or misleading answers to questions between 17 and 33 percent of the time and returned incomplete responses between 18 and 63 percent of the time, Gizmodo reported.

This risk seemingly gave Nevada pause before signing up for the AI service, at least at first. When DETR started talking to Google about the potential AI use during a sales call last year, DETR director Christopher Sewell told The Nevada Independent that he was "hesitant" to launch the AI system. And then there were "a couple of sleepless nights" after the contract was signed, Sewell said.

But Sewell ultimately joined other Nevada officials in advocating for AI, insisting to The Nevada Independent in June that "it's going to work the way we want it," even though the June budget document revealed that a "security impact analysis" had yet to be conducted. That document further warned that although Google's technology was "well-established," it also raised "several cybersecurity concerns" that must be considered prior to deployment of the AI system.

Sensitive information is shared during appeals hearings that the state must ensure remains private, including tax information, social security numbers, and "information about a claimant’s health, family, and finances," Gizmodo noted.

As a reassurance of Nevadans' right to privacy, DETR spokesperson Valentina Bonaparte confirmed to Gizmodo that Google won't be able to access that data or use confidential data that the AI system processes "for other purposes."

While ethical and legal questions swirl ahead of Nevada's AI launch, Sewell told The Nevada Independent that DETR is already dreaming about other ways that AI could be used to streamline processes at the agency. In the future, Sewell suggested, DETR might use AI to answer basic call center questions or "to help people find jobs."

“This is our first go-around with AI,” Sewell said. “But it's not going to be our last.”

And his department won't be the only one turning to AI more and more, The Nevada Independent suggested, noting that "state officials emphasized the technology is all but ensured to become more prevalent across state government."

As experts raise concerns over the state's Google contract, Nevada's chief information officer Timothy Galluzi promised in the state's AI guidelines that Nevada was determined to implement AI safely.

"The judicious use of AI can significantly enhance the efficiency and effectiveness of state government operations," Galluzi said. "It is crucial, however, to balance innovation with responsibility, ensuring that these powerful tools are used securely and ethically."

Photo of Ashley Belanger
Ashley Belanger Senior Policy Reporter
Ashley is a senior policy reporter for Ars Technica, dedicated to tracking social impacts of emerging policies and new technologies. She is a Chicago-based journalist with 20 years of experience.
175 Comments