David Yanagizawa-Drott (University of Zurich)

Learning About AI

Abstract:The rise of generative AI has opened up new possibilities for humans to complete cognitive tasks more quickly and better. As with any new technology, however, it is unclear how well it will perform in any given setting and for any task. Do people learn from personal experience, adopting generative AI when it turns out to be useful? We study the usefulness and adoption of generative AI within the context of an important labor market function: screening for the quality of workers in a hiring process. We embed a two-month long experiment into the process of a non-profit organization hiring university graduates to teach children in rural schools in Ghana. In the standard operating procedure, experienced teachers follow criteria across multiple quality dimensions to score applicants, consisting of essay answers to questions. We give the same task to a new large language model that is state-of-the-art at the time of the experiment (GPT-4), and randomly offer evaluators the AI recommendation before they provide their final scores for potential advancement of a candidate to in-person interviews. We find relatively high adoption initially, but it decreases rapidly over time and there are no significant effects on hiring outcomes from having access to the AI-assistant. AI-assistance appears to reduce productivity, if anything, by slowing down completions of the task. By contrast, when decisions are solely based on the AI, successful hires increase significantly, by about 11 percentage points (70 percent). We examine the signal in the scores and find it is the highest under AI-only, significantly weaker under AI-assistance, and the weakest for Human-only. The signal under human decision-making depends on the complexity of the text (input), but not under AI-only. Our results suggest that the lack of adoption of generative AI assistance can arise due to frictions in learning about the capabilities of the technology, potentially because the learning process itself is complex and noisy. Full automation of the task appears viable, especially when costs for the organization are taken into account.