When it comes to Large Language Models (LLMs) like OpenAI’s ChatGPT, Google’s Bard, and Microsoft’s Bing Chat, the current debate in the field of business goes something like this: Supporters point out LLMs can outperform any single human on the breadth of topics in which they are capable. On the flip side, critics highlight the instances in which LLMs produce work that is remarkably shoddy and even downright wrong.
To help accountants effectively and ethically use LLMs, Culverhouse’s Zach Chism along with collaborators from Bucknell University, Drs. Daniel Street (2020) and Joe Wilck, wrote an article that is forthcoming in The CPA Journal. The article suggests six guiding principles for CPAs:
- Develop specific questions and avoid broad requests.
LLMs are basically complicated autocomplete machines. They calculate the probability that words will be associated with each other, but can quickly veer off course if given vague or broad prompts. Err on the side of specificity. - Break down complex tasks into verifiable subtasks.
This is to verify the correctness of the task before moving into a new one. LLMs may not have the most current information about things like accounting standards or tax codes, or may simply misapply them. It is best to catch these things early in the process before they compound. - Do not input private, sensitive, or proprietary information.
To assume that information you input into LLMs is private is probably a bad idea. It is possible that the platform will incorporate your information into its data, and it could reappear later in ways that compromise your client’s information security. - Mistrust and recalculate quantitative responses.
ChatGPT and its competitors are much better at words than math. Even fairly simple calculations can be incomplete or simply wrong. Leave the math to calculators and spreadsheet software. - Rely on other sources for factual information, especially for less prominent topics.
The authors note that ChatGPT’s “training encouraged it to communicate facts as well as to communicate with human-like creativity.” They go on, “Unfortunately, ChatGPT does not articulate to the user whether it is conveying facts or displaying creativity; hence, it may lie.” Any LLM could invent sources or even facts, so handle with care.” - Use LLMs to enhance rather than replace human expertise.
The authors do not expect LLMs to replace human accountants anytime soon, but still think the technologies can offer some value, particularly in routine, lower-stakes tasks like composing memos or emails, gathering background information, or authoring simple explanatory statements.
The authors conclude: “Some academics and professionals that we speak with wonder if LLMs like ChatGPT are a panacea—after all, there have been widespread reports about its remarkable capabilities—while others dismiss LLMs as unreliable, unhelpful, and overblown tools. We believe that neither extreme perspective is accurate.” By following these six principles, the authors help CPAs leverage LLM technology’s strengths and avoid its pitfalls.