Benchmarking performance of an automatic polysomnography scoring system in a population with suspected sleep disorders - Choo, Bryan Peide; Mok, Yingjuan; Oh, Hong Choon; Patanaik, Amiya; Kishan, Kishan; Awasthi, Animesh; Biju, Siddharth; Bhattacharjee, Soumya; Poh, Yvonne; Wong, Hang Siang (Frontiers in Neurology, 2023)
Aims: The current gold standard for measuring sleep disorders is polysomnography (PSG) which is manually scored by a sleep technologist. Scoring a PSG is time consuming and tedious, with substantial inter-rater variability. A deep learning-based sleep analysis software module can perform autoscoring of PSG. The primary objective of the study is to validate the accuracy and reliability of the autoscoring software. The secondary objective is to measure workflow improvements in terms of time and cost via a time motion study. Methodology: The performance of a cloud-based automatic PSG scoring software was benchmarked against the performance of two independent sleep technologists on PSG data collected from patients with suspected sleep disorders. The technologists at the hospital clinic and a third-party scoring company scored the PSG records independently. The scores were then compared between the technologists and the automatic scoring system. An observational study was also performed where the time taken for sleep technologists at the hospital clinic to manually score PSGs were tracked, along with time taken by the autoscored software to assess for potential time savings. Results: The Pearson’s correlation between manually scored apnea-hypopnea index (AHI) and automatically scored AHI was 0.962, demonstrating near-perfect agreement. The autoscoring system demonstrated similar results in sleep staging. The agreement between automatic staging and manual scoring was higher in terms of accuracy and Cohen’s Kappa than the agreement between experts. The autoscoring system took an average 42.7 seconds to score each record compared to 4243 seconds for manual scoring. Following a manual review of the auto scores, an average time savings of 38.6 minutes per PSG was observed, amounting to 0.25 Full Time Equivalent (FTE) savings per year. Conclusion: The findings indicate potential for a reduction in the burden of manual scoring of PSGs by sleep technologists and may be of operational significance for sleep laboratories in the healthcare setting.