Cost-effectiveness of AI for pediatric diabetic eye exams from a health system perspective

Model design and target population

Using decision analysis²², we evaluated the cost effectiveness of implementing an autonomous AI strategy for the diagnosis of pediatric DRD. Our model considers patients under the age of 21 with Type 1 (T1D) and Type 2 diabetes (T2D) who are cared for by a primary care physician, pediatric endocrinologist, or other licensed provider and who are eligible for DRD screening according to AAO/ADA guidelines^2,3.

Our analysis was conducted from the perspective of a health system providing care to patients. We examined cost-effectiveness during the first year that a health system considers adding capacity via AI eye exams at the point of care. Since most youth with diabetes receive care in an endocrine practice, the focus of analysis is either a single or a conglomeration of pediatric endocrine practices operating under one organization.

The two DRD screening strategies are as follows: (1) Autonomous AI: The autonomous AI diabetic eye exam is performed at the pediatric endocrine site during a diabetes care visit. The operator is guided by the AI system to capture retinal images, and the AI algorithm provides an immediate diagnosis of whether DRD or diabetic macular edema (DME) is present. (2) ECP: The standard of care proceeds with a referral to an ECP (either an optometrist or ophthalmologist) who performs indirect ophthalmoscopy and stereo biomicroscopy under pharmacologic dilation within a health system. In both strategies, patients with positive (i.e., referable DR, ETDRS 35 or greater) or insufficient results would be referred to a retina specialist or ophthalmologist for further management and treatment ^2,3 (Supplementary Fig. 1).

This study did not require authorization from the institutional review board as it did not involve human subjects.

Parameters estimates and ranges

To comprehensively assess the cost effectiveness of the two alternatives, our model incorporates specific population-level parameters (Table 5) including the prevalence of pediatric DRD, the diagnostic accuracy of the screening modalities including sensitivity and specificity, human behavioral factors as reflected by the probabilities of adherence to follow-up recommendations, health system costs associated with the AI strategy, and reimbursement levels for ECP-conducted DRD exams as a proxy for the costs of the ECP strategy. While Medicare, Medicaid, and private insurance reimbursement exist for AI-based DRD screening, because of the variability in reimbursement rate and the fact that many health systems operate through value-based care mechanisms, we excluded reimbursement from our analysis²⁰. Parameter values were obtained from peer-reviewed published literature, empirical observations, and stakeholder interviews with vendors and hospital administrators. Pediatric data were used when available; otherwise, parameter values were taken from adult data. In choosing our base-case estimates, we opted for a conservative approach, with each parameter estimate tilting the analysis against the AI strategy. We also acknowledge that our model incorporates a variety of cost considerations that may not be applicable to every health system; however, these costs were incorporated to evaluate AI against rigorous criteria.

The SEARCH trial estimated the prevalence of DRD among youths with T1D and T2D as 5.6% and 9.1%, respectively²³. For the base case, we used the weighted average (6.8%) of these estimates. Other studies have reported divergent prevalence values, ranging as low as 3.4% for T1D and 6% for T2D²⁴ to as high as 20.1% for T1D²⁵ and 51.0% for youth onset T2D²⁶; these varying figures were used for the low and high estimates in sensitivity analyses.

ECPs performing adult eye exams achieve a sensitivity of 33% (estimated 95% CI: 20-50%)^27,28,29. In comparison, the three FDA-cleared autonomous AI systems have demonstrably higher sensitivity in detecting DRD, ranging from 87.2% – 93.0% (95% CI: 81.8-97.2%) against a prognostic standard, a proxy for clinical outcome, depending on the sample evaluated^30,31,32. Even when applied to pediatric cases, autonomous AI systems have demonstrated higher sensitivity at 85.7% (95% CI: 42.1-99.6%)³³.

The specificity of ECP-conducted exams in adults averages 95%²⁸, and the specificity of autonomous AI systems in pediatrics and adults ranges from 79.3-91.36% (95% CI: 74.3-93.72%)^30,31,32,33. We used the mean specificity values for the base case and the minimum and maximum values of the 95% CI for the sensitivity analysis.

Diagnosability of the autonomous AI system, defined as the percentage of assessed patients with an interpretable image, ranges from 96.1% and 97.5% for adult and pediatric cases, respectively, and were averaged for base-case estimates^30,33. Low and high estimates were derived from the 95% CI of the adult study³⁰.

The model accounts for patient behavioral factors that influence screening acceptance and completion. Among youths, ECP exam acceptance rate averages 52% and can be as high as 72% ³⁴. While some adult cohorts exhibit a screening acceptance probability as low as 15.3%⁴. Acceptance rates for autonomous AI eye exams average between 95-96.4%^33,35. The probability that a patient will follow up with an ECP after receiving a positive result differs between modalities. After a positive result from an ECP exam, the probability of the patient following-up with an in-person ECP visit is 29-95%³⁶, while the probability of follow-up after a positive AI recommendation is 55.4-64%^35,37. The parameter ranges reflect the low and high values observed in the literature and in practice.

FDA De Novo- authorized or cleared AI systems incur a range of expenses that vary by device. Most AI vendors offer pricing models that are based on a per patient usage fee or a subscription model with associated setup fees. Furthermore, health systems take on additional expenses to integrate AI into clinical workflows related to change management, process redesign, and IT system integration. Due to limited availability of such proprietary data, we used the geometric mean of $10,000 as the base case, and we assumed a per practice AI acquisition and ongoing vendor support cost range of $1000-100,000. Since the AI acquisition cost is a one-time up-front fee, a 20% amortization rate to account for the depreciation of assets over time was also applied³⁸. Based on stakeholder interviews, we estimated the cost of integrating an AI system with existing clinical workflows and health IT systems to be a geometric mean of $3000 per site (with a range of $1000 to $20,000). All start-up costs and maintenance costs are included in this one-year time horizon.

We assumed an hourly wage of $33 for the AI operator based on Centers for Medicare & Medicaid Services estimates³⁹ and varied this parameter from $7.25 (the federal minimum wage)⁴⁰ to $42 (the mean hourly wage of office-based nurses as they may be the ones to operate AI systems in pediatric endocrine practices)⁴¹. Frequently, in adult DRD screening, nurse practitioners (with a substantially higher wage), operate the AI; however, that is less common in pediatric endocrine clinics. Based on clinical experience, it is estimated that such operators can complete a maximum of 4 exams per hour³³. Another significant cost consideration is the space required to house the AI system. Current AI DRD systems are desktop- based and may be placed in an existing room designed for ancillary services, but in some cases endocrine practices may need additional space for the AI system. Using an average exam room size of 100 sq. ft⁴² at $28/sq ft⁴³ which is the rental cost based on commercial real estate rates in Baltimore where this analysis was conducted, we estimated an annualized opportunity cost of $2800 for the space required (range of the parameter over $0-8500).

We used the mean CMS reimbursement of $172 as a base case proxy for the cost of an initial DRD exam and follow-up exam with an ECP and the 10th and 90th percentiles ($110 and $240, respectively) for the sensitivity analysis range. Thus, while we did not account for revenue from insurers or patients for providing screening services, our rationale for using the CMS reimbursement as the basis for the cost estimate of the ECP strategy was that it represents the allowed cost of providing the service, including ophthalmic equipment, eye care professional salaries, clinic space and ancillary costs⁴⁴. Although we recognize there are setup costs associated with training and credentialing ECPs and integrating them into clinical workflows, these costs are assumed to be built into the system and thus are not accounted for in the model, which also tilts the analysis against the AI strategy.

Outcomes and willingness to pay threshold

The primary outcome measure of interest is the incremental cost-effectiveness ratio (ICER) of implementing the AI strategy compared to the ECP strategy. ICER is used in decision-analysis models and cost-effectiveness studies to gauge the additional cost associated with a unit increase in effectiveness of one diagnostic strategy compared with another. We defined effectiveness as 1) the additional number of DRD screenings completed and 2) the additional number of patients who followed up with an ECP for further evaluation and treatment. The cost was the total burden to the health system before reimbursement, resulting from the cost and probability estimates in Table 5, so the two ICERs are interpreted as the “marginal financial cost of completing one additional screening with AI, compared with ECP,” and “the marginal financial cost to complete one additional follow up with the ECP among patients screened positive by AI, with respect to ECP.” An ICER value of 0 indicates the point at which the costs of the AI and ECP strategies are equivalent (i.e., the break-even point) and an ICER value less than 0 indicates that the AI strategy is less expensive than the ECP strategy, resulting in cost savings to the health system.

To further rigorously assess the cost-effectiveness of AI strategy, we established a willingness to pay (WTP) benchmark of $413 per averted case of DRD, far below the standard $50,000 threshold⁴⁵. This conservative ceiling was calculated using the lowest Medicaid reimbursement rate of $28.08 for AI DRD exams⁴⁶ divided by the most probable prevalence of diabetes-related disease (6.8%)²³.

Scenarios

Since the costs associated with the two screening strategies scale with the size of the health system, we modeled a series of scenarios based on practice size estimates of pediatric diabetes volumes: (1) a small endocrine practice with an annual volume of 100-200 pediatric patients due for DRD screening, (2) a medium-sized health system consisting of 1-2 pediatric endocrine sites with a total annual screening volume of 400-600 patients (200-600 patients per site), and (3) a large health system consisting of 3-4 pediatric endocrine sites with a total annual screening volume of 1000-4000 patients (250- ~1333 patients per site)¹⁸.

For patients that are identified to have an abnormal diabetic eye exam (by AI or ECP) or whose images are insufficient and thus require referral to an ECP, we examined the cost-effectiveness of follow-up with an ECP and considered two additional scenarios for each sized health system—whether the follow-up with the ECP takes place 1) within the same health system or 2) outside the health system. For example, if the follow-up visit occurs within the health system, then the total cost would include both the initial screening cost based on the modality and the cost of the ECP follow-up visit itself. If the follow-up visit occurs outside the health system, such as at another private practice or hospital system, the total cost would include only the initial screening.

To account for sampling and parameter uncertainties, deterministic and PSA were performed within each of the scenarios using the ranges shown in Table 5. The deterministic sensitivity analysis informs the thresholds where one strategy is preferrable over the other. The PSAprovides a measure of confidence in the outputs of the base-case analysis. To translate the PSA into meaningful terms, we also report the minimum WTP at which the AI strategy achieves 95% cost-effectiveness probability within each scenario. Modeling and sensitivity analyses were performed using the TreeAge Pro software version (TreeAge Pro 2023, Williamstown, MA, USA). The study was conducted from June 2023 to August 2024.