Validation of the DIGIROP-birth model in a Chinese cohort

Background We aimed to validate the predictive performance of the DIGIROP-Birth model for identifying treatment-requiring retinopathy of prematurity (TR-ROP) in Chinese preterm infants to evaluate its generalizability across countries and races. Methods We retrospectively reviewed the medical records of preterm infants who were screened for retinopathy of prematurity (ROP) in a single Chinese hospital between June 2015 and August 2020. The predictive performance of the model for TR-ROP was assessed through the construction of a receiver-operating characteristic (ROC) curve and calculating the areas under the ROC curve (AUC), sensitivity, specificity, and positive and negative predictive values. Results Four hundred and forty-two infants (mean (SD) gestational age = 28.8 (1.3) weeks; mean (SD) birth weight = 1237.0 (236.9) g; 64.7% males) were included in the study. Analyses showed that the DIGIROP-Birth model demonstrated less satisfactory performance than previously reported in identifying infants with TR-ROP, with an area under the receiver-operating characteristic curve of 0.634 (95% confidence interval = 0.564–0.705). With a cutoff value of 0.0084, the DIGIROP-Birth model showed a sensitivity of 48/93 (51.6%), which increased to 89/93 (95.7%) after modification with the addition of postnatal risk factors. In infants with a gestational age < 28 weeks or birth weight < 1000 g, the DIGIROP-Birth model exhibited sensitivities of 36/39 (92.3%) and 20/23 (87.0%), respectively. Conclusions Although the predictive performance was less satisfactory in China than in developed countries, modification of the DIGIROP-Birth model with postnatal risk factors shows promise in improving its efficacy for TR-ROP. The model may also be effective in infants with a younger gestational age or with an extremely low birth weight. Supplementary Information The online version contains supplementary material available at 10.1186/s12886-021-01952-0.


Background
Retinopathy of prematurity (ROP) is a retinal vasoproliferative disease that affects preterm infants. Although it is avoidable through early diagnosis and timely treatment, it is a leading cause of childhood blindness [1]. The risk of blindness from ROP is as high as 40% in developing countries, while it is less than 10% in developed countries [2]. Insufficient neonatal and ophthalmologic care, as well as variation in clinical practice, may account for the varying neonatal outcomes associated with ROP [3]. Further, noncompliance with follow-up examinations is one of the major causes of blindness from ROP, particularly in developing countries [4]. Under the current screening criteria, less than 10% of screened infants require treatment for ROP [5]. Effective implementation of ROP screening is hindered by limited medical resources in developing countries or regions and over-screening under varying recommended guidelines [1][2][3]5]. Thus, there is a pressing need for prediction models for ROP that safely ease the workload of screening.
Low gestational age and birth weight are the major risk factors for ROP, and form the basis of most established screening guidelines [1]. Prediction models have also been developed that incorporate additional clinical parameters, such as weight gain rate, race, and respiratory distress syndrome [6][7][8][9]. However, according to a 2016 report by the American Academy of Ophthalmology, none of these prediction models are clinically applicable due to limited generalizability and small sample size [10]. Although some models have been validated and exhibited excellent predictive performance, the validation cohorts consisted mainly of infants in highly developed countries [11][12][13][14][15][16][17][18][19]. Nevertheless, the data gathered from less-developed countries are scarce but indispensable. These data indicate that racial variation exists, with Asian infants being at greater risk of ROP than White infants, suggesting a genetic predisposition to ROP in addition to the underlying socioeconomic factors associated with low birth weight, small for gestational age, and preterm birth [20,21]. For example, the WINROP model, based on weekly weight gain [6], has been widely validated around the globe. It displayed favorable utility in predicting severe ROP in developed countries [14][15][16], but performed less satisfyingly in moderately or less-developed countries such as China, Mexico, and Turkey [17][18][19]. Furthermore, the implementation of higher oxygen saturation targets (i.e., 91-95%) in neonatal intensive care units may reduce the impact of poor weight gain as a risk factor for ROP and has been shown to diminish the WINROP model's predictive ability [22]. Thus, ROP risk prediction models should be validated across different races and countries or regions with various clinical settings.
Recently, Pivodic et al. proposed an individualized risk prediction model for treatment-requiring retinopathy of prematurity (TR-ROP), called DIGIROP-Birth. The model was based on gestational age, birth weight, and sex and applicable for infants with a gestational age of 24 to 30 weeks [23]. It has several strengths that suggest its potential for clinical application [24]. First, the model was accessible online as a risk calculator, without any input of postnatal factors. This easy-to-use model enabled early identification of high-risk infants and early planning of ROP examination and follow-up schedules, thereby improving compliance with ROP screening examinations and decreasing visual impairment from ROP. Second, the model was internally and externally validated in large multicenter cohorts consisting of infants born in developed countries and exhibited promising predictive performance. However, no validation has been reported in low-or middle-income countries or in racespecific cohorts.
Therefore, the present study aimed to validate the DIGIROP-Birth model in Chinese preterm infants to evaluate its generalizability across countries and races.

Study design
Preterm infants who received ROP screening examination between June 2015 and August 2020 in the neonatal intensive care unit of Zhujiang Hospital of Southern Medical University, Guangzhou, China, were retrospectively recruited. The study was approved by the hospital's institutional ethics committee and adhered to the tenets of the Declaration of Helsinki. All parents or guardians of the recruited infants provided written informed consent prior to participation. Data were anonymized and de-identified before analysis. According to the Chinese guideline for ROP screening, eligible participants were preterm infants with gestational ages less than 32 weeks, birth weight less than 2000 g, or with risk factors for ROP as determined by a neonatologist [25]. Infants with incomplete data or any other ocular diseases besides ROP were excluded. Also, infants with gestational ages less than 24 weeks or beyond 30 weeks were further excluded as the DIGIROP-Birth model was not developed for application in these gestational ages. The diagnosis of ROP and indication of treatment for ROP followed the International Classification of ROP Revisited and the Early Treatment for ROP Study, respectively [26,27].
Clinical data collected for our study related to maternal factors, neonatal factors, and neonatal interventions. These data were extracted from medical records by SZ and R. Specifically, maternal factors of interest included maternal age, mode of delivery (cesarean or vaginal), multiple gestations, in vitro fertilization, gestational hypertension, gestational diabetes mellitus, reproductive tract infection during pregnancy, and use of antenatal steroids. Neonatal factors included gestational age, birth weight, sex, apnea, respiratory distress syndrome, bronchopulmonary dysplasia, sepsis, necrotizing enterocolitis, intraventricular hemorrhage, patent ductus arteriosus, anaemia, and hyperbilirubinemia. Finally, neonatal interventions included invasive mechanical ventilation and blood transfusions.

Statistical analysis
Categorical variables were expressed as numbers and percentages and analyzed using chi-squared tests. Continuous variables were expressed as means and standard deviations and compared using Wilcoxon's rank-sum tests. P < 0.05 (two-tailed) was considered statistically significant. To identify the independent risk factors for TR-ROP, univariate logistic regression analyses were performed for each variable. Variables with a P < 0.1 in univariate logistic analyses were included in multivariate logistic regression analyses. To assess the predictive performance of the model for TR-ROP, a receiver-operating characteristic (ROC) curve was constructed. In the ROC analysis, the areas under the ROC curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. The cutoff value that resulted in the maximal sum of sensitivity and specificity was chosen and assessed. In addition, the cutoff values previously tested by Pivodic et al. were also selected for validation [23]. The performance of the DIGI ROP-Birth model for identifying TR-ROP infants was also evaluated by gestational age and birth weight. Specifically, the study cohort was divided into gestational age < 28 weeks (extremely preterm infant) and gestational age ≥ 28 weeks subgroups, and birth weight < 1000 g (extremely low birth weight) and birth weight ≥ 1000 g subgroups. Statistical analyses were performed using SPSS 20.0 (IBM Corp., Armonk, NY, USA).

Results
Of the 732 infants who underwent ROP screening, 11 had incomplete data and five had ocular diseases besides ROP. Among the remaining 716 infants, 274 were further excluded due to a gestational age less than 24 weeks or beyond 30 weeks, resulting in 442 infants being included in the study. Two hundred thirty-seven of 442 participants (53.6%) developed ROP of any stage, of whom 93 of 442 (21.0%) required treatment. The mean (SD) gestational age at birth was 28.8 (1.3) weeks, the mean (SD) birth weight was 1237.0 (236.9) g, and the sex distribution was 286 (64.7%) male and 156 (35.3%) female. Clinical characteristics of the study population are summarized in Table 1. Compared with infants without TR-ROP, gestational age and birth weight were significantly lower (both P < 0.001) among TR-ROP cases. A significant difference in sex was also observed between the groups (P = 0.046), where the proportion of males was significantly higher in infants with TR-ROP. Finally, there was a higher occurrence of apnea (P = 0.041) and intraventricular hemorrhage (P = 0.034) in infants with TR-ROP.
Univariate and multivariate logistic regression analyses were conducted to identify risk factors for TR-ROP (Supplementary Table 1). Among the maternal factors, only antenatal steroid use was associated with TR-ROP (P = 0.082). Among the neonatal factors, infants with TR-ROP had a younger gestational age and lower birth weight (both P < 0.001) than infants without TR-ROP. In addition, male sex (P = 0.047), apnea (P = 0.043), bronchopulmonary dysplasia (P = 0.077) and intraventricular hemorrhage (P = 0.035) were associated with TR-ROP.
The DIGIROP-Birth model demonstrated unsatisfactory performance for identifying infants with TR-ROP, with an AUC of 0.634 (95% CI, 0.564-0.705) (Fig. 1). The predictive performances of the DIGIROP-Birth model with different cutoff values are displayed in Table 2. When applying a cutoff value of 0.0084, which resulted in a maximal sum of sensitivity and specificity, the DIGIROP-Birth model demonstrated a sensitivity of 48/93 (51.6%) and an NPV of 263/308 (85.4%). Among those who were determined to be at risk of TR-ROP below 0.84% by the DIGIROP-Birth model, 45 of 299 (15.1%) infants developed ROP that required treatment. Forty-one of these 45 (91.1%) infants had at least one premature birth complication, including apnea and intraventricular hemorrhage. Characteristics of the 45 infants are listed in Table 3. Thus, apnea and intraventricular hemorrhage were subsequently applied as additional risk factors of TR-ROP to improve the sensitivity of the DIGIROP-Birth model. Infants reported to be at risk below 0.84% by the DIGIROP-Birth model were reclassified as high-risk for TR-ROP. The sensitivity of the modified DIGIROP-Birth model that included these additional risk factors increased to 89/93 (95.7%), and the NPV was 263/267 (98.5%).

Discussion
Visual loss from ROP may be prevented by early diagnosis and timely treatment, which emphasizes the importance of ROP screening in routine clinical practice [1]. Recently, Pivodic et al. developed an individual risk prediction model, DIGIROP-Birth, using only birth characteristics to describe a continuous hazard function for identifying TR-ROP [23]. This easy-to-use prediction model was built using Swedish National Patient Registry data, and validated in US and European cohorts, yielding satisfactory results. The present study validated the DIGI ROP-Birth model in Chinese preterm infants, and found that the model had less satisfactory performance than previously reported (AUC = 0.634 in this study vs. AUC = 0.85 in the study by Pivodic et al.) [23].
Several reasons could account for the discrepancy. First, there were only 83 (5.4%) Asians among the 1535 infants that comprised the US validation group in Pivodic et al.'s study [23], while our study cohort consisted of 442 Chinese infants. Asian infants appear to be at higher risk of developing TR-ROP than white infants due to differences in ethnic ancestry and underlying genetic predisposition [20,21]. Second, compared with the DIGIROP-Birth model's training cohort, our cohort of Chinese infants had an older mean gestational age (28.8 weeks vs. 28.1 weeks) and lower mean birth weight (1119 g vs. 1237 g). This might be explained by the fact that in less-developed countries, severe ROP occurs in more mature and larger infants [3]. Third, the quality of neonatal care is one of the most critical factors for ROP development and progression [28]. Besides a lack of neonatologists and nurses with neonatal care expertise, neonatal units are in short supply of enough equipment for continuous monitoring of preterm infants on  [29][30][31]. The timing and duration of supplemental oxygen, oxygen concentration, and prolonged mechanical ventilation are among the most crucial risk factors for TR-ROP [32]. Consequently, preterm infants born in less-developed countries or regions are more likely to be exposed to postnatal risk factors for TR-ROP that are better controlled in industrialized countries [29,31]. Similar characteristics of ROP and the corresponding clinical settings have been reported in other developing countries in Asia and Latin America [29][30][31]. Finally, the DIGIROP-Birth model did not consider postnatal risk factors for ROP, which could also account for its decreased predictive performance in our Chinese cohort. Modification of the DIGIROP-Birth model through the incorporation of postnatal risk factors might improve its applicability in less-developed countries.
Several studies have shown that complications of prematurity, such as bronchopulmonary dysplasia, apnea, intraventricular hemorrhage, and sepsis, are associated with the development of ROP [32][33][34][35]. Gestational age, birth weight, male sex, apnea, and intraventricular hemorrhage were found to be independent risk factors for TR-ROP in our cohort of Chinese infants with       Previous studies also revealed that apnea of prematurity and intraventricular hemorrhage were independently associated with a higher risk of ROP [34][35][36]. Infants with apnea are more likely to require oxygen therapy, which could induce ROP development due to immature antioxidant systems. Oxygen-related factors play a crucial role in TR-ROP, including the duration of supplemental oxygen, oxygen concentration, and prolonged mechanical ventilation [32]. Although several large randomized-controlled studies have compared different oxygen saturation target ranges, the ideal range that could reduce ROP occurrence without increasing preterm infants' mortality remains controversial [37][38][39][40]. Intraventricular hemorrhage occurs in 25-30% of preterm infants with birth weights < 1500 g, often causing neurodevelopmental impairment [41]. Early control of intracranial pressure secondary to intraventricular hemorrhage may prevent TR-ROP development in infants with a combined diagnosis of ROP and intraventricular hemorrhage. This is because the progression of ROP may associate with reduced ocular circulation secondary to high intracranial pressure [42]. Thus, apnea and intraventricular hemorrhage, two important premature birth complications, could greatly improve the predictive ability of the DIGIROP-Birth model for TR-ROP in Chinese preterm infants. However, no significance was observed in other important risk factors included in our study, such as respiratory distress syndrome, anaemia, or invasive mechanical ventilation. This could be explained by the small sample size.
The sensitivity of the DIGROP-Birth model in infants with a gestational age < 28 weeks or a birth weight < 1000 g was satisfactory. This suggests that the DIGROP-Birth model may also be valuable as an auxiliary tool for ROP screening in extremely preterm infants and infants with extremely low birth weight. In infants with a gestational age ≥ 28 weeks or birth weight ≥ 1000 g, the DIGI ROP-Birth model was less effective, but could be modified with postnatal risk factors. Thus, the DIGIROP-Birth model still has the potential to decrease the frequency of ROP examinations in less-developed countries.
There was another factor impeding the generalization of the DIGROP-Birth model to Chinese preterm infants.
The DIGROP-Birth model was developed in Swedish infants with gestational ages less than 31 weeks, thus this model could not be applied to infants with gestational ages of 31 weeks or more [23], resulting in 38.2% of infants eligible for ROP screening to be excluded from our validation study. Nevertheless, TR-ROP could occurs in these Chinese infants with higher gestational ages [4], prediction models are needed to identify these subset of infants at high risk. The performance of the DIGIROP-Birth model is worthy of validation in infants with gestational ages beyond 30 weeks, and might be also modified with postnatal risk factors.
This study has several limitations. First, this validation study was conducted retrospectively. Despite the retrospective nature, however, the clinical data included in our analyses are routinely recorded in the neonatal intensive care units and could be collected reliably. Second, the single-center study had a relatively small sample size compared with other validation studies of ROP prediction models. Future multicenter prospective studies with large cohorts will enhance our findings.

Conclusions
This study validated the DIGIROP-Birth model in a cohort of Chinese preterm infants. Although the predictive performance of the model was lower than in developed countries, modification of the model with postnatal risk factors showed promise for improving its efficacy for TR-ROP. The DIGIROP-Birth model was also effective in infants with a younger gestational age or with extremely low birth weight. We believe that more studies in less-developed countries and other races will improve the efficacy and generalizability of the DIGIROP-Birth model, and subsequent adjustment of the easy-to-use model will help reduce the frequency of ROP screening examinations.
Abbreviations AUC: Area under the receiver-operating characteristic curve; CI: Confidence interval; NPV: Negative predictive value; OR: Odds ratio; PPV: Positive predictive value; ROC: Receiver-operating characteristic; ROP: Retinopathy of prematurity; TR-ROP: Treatment-requiring retinopathy of prematurity diagnosis for the included patients with ROP and critically revised the manuscript. The author(s) read and approved the final manuscript.

Funding
None.

Availability of data and materials
The datasets generated during and analyzed during the current study are not publicly available due to the reason that the data also forms part of an ongoing study but are available from the corresponding author on reasonable request.

Declarations
Ethics approval and consent to participate The study was approved by Zhujiang hospital's institutional ethics committee and adhered to the tenets of the Declaration of Helsinki. All parents or guardians of the recruited infants provided written informed consent prior to participation.