Agentic Artificial Intelligence for the Automated Generation of Accurate Summary Podcasts of Radiology Research Papers

Clinical Snapshot

Key Findings

• P Value: Primary hallucination count: P < 0.001; hallucination-free proportion: P = 0.013; QAEP composite score: P < 0.001; mispronunciations: P < 0.001; flow disruptions: P < 0.001; listener preference: P = 0.003

• Effect Size: Absolute reduction of 0.61 hallucinations per episode (65.6% relative reduction); absolute difference of 14.8 percentage points in hallucination-free episode rate

• Primary Outcome: P5 produced significantly fewer hallucinations per episode than NBLM (mean 0.32 vs. 0.93; P < 0.001) and a higher proportion of hallucination-free episodes (71.6% [63/88] vs. 56.8% [50/88]; P = 0.013)

• Nnt Or Sensitivity: Number needed to treat (NNT) equivalent: approximately 7 episodes processed with P5 instead of NBLM to yield one additional hallucination-free episode (based on 14.8 percentage point absolute risk difference). Listener preference for P5: 72.7% (64/88 comparisons)

• Confidence Interval: Not reported

Clinical Application

Limited immediate feasibility beyond the originating institution. The P5 pipeline is a custom multi-agent system requiring institutional infrastructure, technical expertise, and ongoing maintenance. Google NotebookLM is commercially accessible but demonstrated inferior accuracy. Widespread adoption of P5 would require open-source release, validation across diverse institutions, and regulatory consideration for medical education content. No TGA regulatory pathway currently governs AI-generated medical education content in Australia. The RACGP and RANZCR do not yet have specific guidelines for AI-generated CME materials. Australian radiologists using commercial tools such as Google NotebookLM for research summarisation should be aware of the hallucination rates demonstrated in this study (mean 0.93 per episode). The PBS and Medicare implications are indirect — if AI-generated educational content influences clinical practice, accuracy is paramount. Australian institutions considering development of similar pipelines should note the significant institutional resource investment required. The study's findings are relevant to the broader Australian digital health strategy and the TGA's emerging AI as a Medical Device (AIaMD) framework, though podcast-format CME tools may fall outside current AIaMD scope. Radiologists and radiology trainees seeking podcast-format continuing medical education (CME) summaries of peer-reviewed research. Potentially applicable to other medical specialties using similar AI-generated educational content pipelines.

Abstract

OBJECTIVE: To evaluate whether a custom agentic artificial intelligence (AI) pipeline can overcome the limitations of general-purpose large language model tools, when compared with a generic commercial tool (Google NotebookLM [NBLM]), for generating podcast-style summaries of radiology research articles. MATERIALS AND METHODS: Twenty-two PDF-format original research articles published in the April 2025 issue of Radiology were processed using our Programmable, Phoneme-Aware PDF-to-Podcast Pipeline (P5) and NBLM to generate 44 audio episodes. P5 utilizes a multi-agent workflow for script generation, quality assurance, pronunciation enhancement, and audio synthesis. Four radiologists from a pool of 25 (7 generalists and 18 specialists) were randomly assigned to evaluate each blinded audio episode, yielding 176 total evaluations. The primary outcomes were the number of hallucinations (factual errors) per episode and the percentage of hallucination-free episodes. Secondary outcomes included the number of inappropriate statements, mispronunciations, and flow disruptions; the composite quality score (Quality Assessment of Educational Podcasts [QAEP]); the key results coverage score; and overall listener preference. Data were analyzed using generalized linear mixed models. RESULTS: The P5 method produced significantly fewer hallucinations per episode compared with NBLM (mean, 0.32 vs. 0.93; P < 0.001) and a higher proportion of hallucination-free episodes (71.6% [63/88] vs. 56.8% [50/88]; P = 0.013), consistently across generalists and specialists. P5 demonstrated significantly fewer mispronunciations (mean, 0.11 vs. 1.62; P < 0.001) and flow disruptions (mean, 0.19 vs. 1.06; P < 0.001) per episode, as well as higher mean QAEP composite scores (4.62 vs. 4.28; P < 0.001), compared with NBLM, consistently across generalists and specialists. Raters preferred P5 over NBLM in 72.7% (64/88) of comparisons (P = 0.003). CONCLUSION: Our custom agentic AI pipeline generated podcast-style summaries of radiology research articles with significantly higher quality and greater listener preference than the generic commercial tool.

References

1. 1.Lee, J. H., Gu, K., Shin, J., Min, J. H., Hwang, J. A., Ban, J. Y., Choi, J., Han, T., Jeong, D. Y., Kang, K. A., Kim, H., Kim, H., Kim, H. S., Kim, M. J., Kim, M. K., Kim, S., Kwak, M. H., Lee, J. H., Lee, S. Y., Oh, C. H., Park, S. G., Roh, Y. H., Sim, Y., Sohn, B., Sung, S., Yu, M.-H., & Park, S. H. (2026). Agentic artificial intelligence for the automated generation of accurate summary podcasts of radiology research papers. Korean Journal of Radiology. https://doi.org/10.3348/kjr.2026.0203