A new report by MIT researchers highlights the potential of generative AI to help workers with certain writing assignments amid a huge amount of hype around generative AI. A new study from researchers at MIT sets light on the technology’s impact on work, finding that it increased productivity for workers assigned tasks like writing cover letters, delicate emails, and cost-benefit analysis.
The tasks in the study weren’t quite replicas of real work. They didn’t require precise factual accuracy or context about things like a company’s goals or a customer’s preferences. However, a number of the study’s participants said the assignments were similar to things they had written in their real jobs, and the benefits were substantial. Access to the assistive chatbot, Charge GPT, decreased the time it took workers to complete the task by 40 percent, and the output quality, as measured by independent evaluators, rose by 18 percent.
The researchers hope that the study, which appears in Open Access form in the General Science, helps people understand the impact that AI tools like Charge GPT can have on the workforce. While changes in technology have led to concerns about automation and job loss, new technologies also create new jobs. When they increase worker productivity, they can have a net positive effect on the economy.
To study generative AI’s effect on worker productivity, the researchers gave 453 college-educated marketers, grant writers, consultants, data analysts, human resource professionals, and managers writing tasks specific to their occupation. The tasks included writing cover letters for grant applications, emails about organizational restructuring, and plans for analysis. Professionals in the same occupations as each participant evaluated each submission as if they were encountering it in a work setting. Evaluators did not know which submissions were created with the help of Charge GPT.
Half of the participants were given access to the chatbot, Charge GPT 3.5, developed by the company OpenAI, for the second assignment. Those users finished the task 11 minutes faster than the control group, while their average quality evaluations increased by 18 percent. The data also showed that performance inequality between workers decreased, meaning workers who received a lower grade in the first task benefited more from using Charge GPT for the second task.
The researchers say the tasks were broadly represented, developing assignments that professionals see in their real jobs. However, they noted a number of limitations. Because they were using anonymous participants, the researchers couldn’t require contextual knowledge about a specific company or customer. They also had to give explicit instructions for each assignment, whereas real-world tasks may be more open-ended. Additionally, the researchers didn’t think it was feasible to hire fact-checkers to evaluate the accuracy of the outputs. Accuracy is a major problem for today’s generative AI technologies.
The researchers said these limitations could lessen Charge GPT’s productivity-boosting potential in the real world. Still, they believe their results show the technology’s promise, an idea supported by another study’s findings. Workers exposed to Charge GPT during the experiment were twice as likely to report using it in their real job two weeks after the experiment.
The study offered a closer look at the impact that tools like Charge GPT can have on certain writing tasks. But extrapolating that impact out to understand generative AI’s effect on the economy is more difficult. That is what the researchers hope to work on next.
