Studies Show AI Chatbots Provide Inconsistent Accuracy for Musculoskeletal Health Information
Retrieved on:
Monday, February 12, 2024
Three new studies presented at the 2024 Annual Meeting of the American Academy of Orthopaedic Surgeons (AAOS) analyzed the validity of the information chatbots gave to patients for certain orthopaedic procedures, assessing the accuracy of how chatbots present research advancements and clinical decision making.
Key Points:
- Three new studies presented at the 2024 Annual Meeting of the American Academy of Orthopaedic Surgeons (AAOS) analyzed the validity of the information chatbots gave to patients for certain orthopaedic procedures, assessing the accuracy of how chatbots present research advancements and clinical decision making.
- While the studies found that certain chatbots provide concise summaries across a wide spectrum of orthopaedic conditions, each demonstrated limited accuracy depending on the category.
- This study, led by Branden Sosa, a fourth-year medical student at Weill Cornell Medicine, assessed the accuracy of Open AI ChatGPT 4.0, Google Bard and BingAI chatbots to explain basic orthopaedic concepts, integrate clinical information and address patient queries.
- Each chatbot was prompted to answer 45 orthopaedic-related questions spanning categories of "Bone Physiology," "Referring Physician," and "Patient Query" and then assessed for accuracy.