Chatbots to replace lawyers?
Professor Markus Conrads and Professor Sascha Schweitzer investigated the legal competence of chatbots. The use of AI in the courts is still a long way off, but the potential is there.
Who can make better decisions on legal questions: qualified lawyers or artificial intelligence (AI)? Professor Markus Conrads and Professor Sascha Schweitzer from the ESB Business School at Reutlingen University studied this subject in their research project. They used AI-based chatbots to resolve simple legal problems and it emerged that ChatGPT-4 in particular is already producing usable results.
In the course of the study, three chatbots – ChatGPT-4, ChatGPT-3.5 and Google Bard – were given 200 exam questions in the fields of German contract, commercial and employment law. To assess the quality and reliability of the AI systems, the researchers posed the questions several times in different formats.
The chatbots produced surprisingly good results, particularly in the case of expert solutions. ChatGPT-4 achieved an average mark of 54 percent and would therefore be able to pass a law exam. In almost 30 percent of the cases, the chatbot provided answers that were well above-average by human standards. In the multiple choice exam, ChatGPT-4 answered around half of the questions correctly. However, ChatGPT-3.5, which is available free of charge, and Google Bard both performed much worse.
“A good metaphor is that ChatGPT-4 is taking driving lessons, but definitely doesn't yet have a legal driving licence. But it will continue to learn quickly,” said Markus Conrads, Professor of German and International Business Law at the ESB Business School, in his assessment of the results.
He also provided reassurance and explained that no one need be worried about AI in the courtroom: “We investigated cases that were designed to solve a specific legal problem. The discussion is definitely not about whether AI can process complex judicial issues independently.”
Sascha Schweitzer, Professor of Data Science at the ESB Business School, explained that when we reach the point of involving artificial intelligence in legal cases, we need to take ethical perspectives into consideration: “A lot of work is currently being done on explainable AI, which will enable people to understand AI processes. The law is definitely an area where the use of transparent algorithms will play a decisive role.”
The detailed results of the study have been published in “Neue Juristische Wochenschrift”, Germany’s highest circulation journal for legal theory and practice.
