Foot and Ankle Surgery Institute Editorial Team

As artificial intelligence (AI) continues to emerge and evolve in various spaces, what is the impact on foot and ankle research? Authors of a recent poster shared that AI could be appealing to accommodate the demand for high-quality research in a quickly advancing field. However, significant challenges exist with concerns of bias, accuracy issues, and AI “hallucinations” that could compromise research integrity. The authors contend that being able to distinguish human authors from AI authors is vital, but is it actually possible? Their study sought to begin to answer this question.¹

Their Level IV survey study took 21 total abstracts in the foot and ankle research space (15 from PubMed and 6 from the Journal of Foot and Ankle Surgery) and used them to train ChatGPT 3.5. The AI system then generated 6 of its own fictional abstracts. All 12 abstracts then comprised a survey, asking the respondent to indicate if the abstract was human- or AI-generated, along with a confidence scale of 0-100. Each participant also provided answers about specific characteristics and answered the survey twice, each 2 weeks apart.¹

The respondents were 9 foot and ankle surgeons, with a mean of 12 years in practice. They had averages of 65 articles each and 10 years reviewing articles among them. Their average AI familiarity score was 32 out of 100.1 Of the 216 total abstracts reviewed in this survey (12 abstracts, 2 surveys, 9 respondents), the participants identified just over 50% of them correctly.¹

There was also a significant correlation between confidence scores and selecting the correct human-generated abstract in the second survey. However, that was the only characteristic with any significance noted among the respondents. They found “moderate” inter-rater reliability in the first survey, but this was rated “poor” for the second survey, as was intra-rater reliability for both surveys.¹

In summary, the authors felt that the ability to accurately discern between AI- and human-generated abstracts in foot and ankle research essentially came down to a coin toss. They note that this aligns with similar previous studies, both regarding human identification and AI detection tools. Limitations the authors shared from this study include the length of abstracts, proficiency with training AI models, the AI model itself, and participants expecting AI-generated content.¹ Overall, good research stewardship is recommended to mitigate concerns related to these challenges.

Reference