A recent study published in the American Journal of Gastroenterology1 found that the natural language processing model Chat Generative Pretrained Transformer (ChatGPT) did not pass the American College of Gastroenterology (ACG) Self-Assessment Tests. The study was conducted by Kelly Suchman, M.D., from the Zucker School of Medicine at Hofstra/Northwell and the Long Island Jewish Medical Center in New Hyde Park, New York, and colleagues. They assessed ChatGPT-3 and ChatGPT-4 performance on the 2022 and 2021 ACG Self-Assessment Tests. The results showed that ChatGPT-3 scored a 65.1% on 455 included questions, and GPT-4 scored 62.4%, both of which were below the required 70% or higher to pass the assessment2.
What is ChatGPT?
Chat Generative Pretrained Transformer (ChatGPT) is a natural language processing model that uses machine learning to generate human-like responses to text input. It is a type of artificial intelligence that has been used in various applications, including chatbots, language translation, and content creation. ChatGPT is a deep learning model that uses a transformer architecture to generate text. It is trained on a large corpus of text data and can generate responses to text input that are often indistinguishable from human-generated responses2.
What are the American College of Gastroenterology Self-Assessment Tests?
The American College of Gastroenterology (ACG) Self-Assessment Tests are designed to evaluate the knowledge and skills of gastroenterologists in various areas of gastroenterology. The tests are multiple-choice and cover a range of topics, including inflammatory bowel disease, colorectal cancer screening, and gastroesophageal reflux disease (GERD). The tests are used by gastroenterologists to assess their knowledge and skills and to identify areas where they may need further education or training345.
Implications of the Study
The study’s findings suggest that ChatGPT is not suitable for medical education in gastroenterology in its current form. The study’s authors recommend that ChatGPT should not be used for medical education until it can pass the ACG self-assessment test. The study’s findings also highlight the limitations of natural language processing models in medical education and the need for further research to develop more accurate and reliable models12.
Conclusion
The study’s findings suggest that ChatGPT is not suitable for medical education in gastroenterology in its current form. The study’s authors recommend that ChatGPT should not be used for medical education until it can pass the ACG self-assessment test. The study’s findings also highlight the limitations of natural language processing models in medical education and the need for further research to develop more accurate and reliable models. The American College of Gastroenterology provides a range of resources for patients and healthcare professionals, including guidelines on the diagnosis and management of various gastrointestinal conditions6345.