Diagnostic Test Accuracy of Diabetic Retinopathy Screening by Physician Graders Using a Hand Held Non-Mydriatic Retinal Camera at a Tertiary Level Medical Clinic in a Lower-Middle Income Country Setting

Background The evidence on diagnostic test accuracy (DTA) of diabetic retinopathy (DR) screening utilising photographic studies by non-ophthalmologist personnel in low and middle-income country (LMIC) settings is scarce. We aimed to assess DTA of DR screening using a nonmydriatic hand-held digital camera by trained general physicians in a non-ophthalmic setting. Methods This study is a validation of a screening intervention. We selected 700 people with diabetes (PwDM) > 18 years of age, not previously screened or treated for DR, presenting at a tertiary medical clinic in Sri Lanka. Two-field retinal imaging was used to capture fundus images before and after pupil dilatation, using a hand-held non-mydriatic (Visuscout 100®-Germany) digital retinal camera. The images were captured and graded by two trained, masked independent physician graders. The DTA of different levels of DR was assessed comparing physician’s grading with a retinologist’s clinical examination by mydriatic bio-microscopy, according to a locally adopted guideline. Results Seven hundred eligible PwDM were screened by physician graders. The mean age of participants was 60.8 years (SD ±10.08) and mean duration of DM was 9.9 years (SD ±8.09). Ungradable image proportion in non-mydriatic imaging was 43.4% (either eye-31.3%, both eyes 12.1%). This decreased to 12.8% (either eye-11.6%, both eyes-1.2%) following pupil dilatation. In comparison to detection of any level of DR, a referable level DR (moderate non-proliferative DR and levels above) showed a higher level of DTA. The sensitivity of the defined referable DR was 88.7% (95% CI 81.7-93.8%) for grader 1 (positive predictive value PPV 59.1%) and 92.5% (95% CI 86.4-96.5%) for grader 2 (PPV 68%), using mydriatic imaging, after including ungradable images as screen positives. The specificity was 94.9% (95%

medical clinic, with dilatation of pupil of those who have ungradable images, provides a valid modality to identify referable level of DR. This could be a feasible alternative modality to the existing opportunistic screening to improve the access and coverage.

Background
Diabetic retinopathy (DR) is a common complication of diabetes mellitus (DM), leading to sight loss if not detected and treated in time [1]. The International Diabetes Federation (IDF) estimated that cases of DM will increase to 629 million by 2045, with a significant burden (80%) in low and middle income countries (LMIC) [2]. Systematic DR screening (DRS) is a challenge in many of the LMICs due to limited resources [3]. The St Vincent declaration stated that all nations should make efforts to reduce DM related complications, including DR blindness [4]. These recommendations were followed by most of the high-income countries (HICs). The LMICs would also be able to achieve this aim with the adaptation and use of existing technologies according to the local contextual requirements.
The most common method of detecting DR in LMICs is direct ophthalmoscopy and slit-lamp bio-microscopy. The direct ophthalmoscopy has a low sensitivity and specificity even at the hands of experienced eye care specialists [5]. The mydriatic bio-microscopic examination by an ophthalmologist is not practical in these countries due to the low number of ophthalmologists and eye clinics which are already over-subscribed with more common blinding conditions such as cataract [6]. In these circumstances, DR is detected through opportunistic case detection only. Insufficient capacity and lack of screening infrastructure hampers efforts to implement DRS programs (DRSP) in these settings, and there is a lack of evidence of what works in LMICs [7]- [9].
Different models of DRS have been implemented in many parts of the world. In resource poor LMICs development of a DRS model is complex [10]. The lack of trained human resources and infrastructure has outstripped the capacity to deliver systematic DRS in low income settings [11]. There are also poor recording systems to identify the people with DM (PwDM). Therefore, a comprehensive population-based DRSP may not be feasible in LMICs in the near future [8], [10]. DRS also requires appropriate integration into routine care for sustainability [10]. It was shown that public health integration of DRS is a feasible strategy to control avoidable blindness [12]. As such, one feasible model of systematic DRS in LMICs could be screening of PwDM when they attend for routine medical care. This can provide a participant list of PwDM, who can be offered screening at regular intervals.
Integrated DRS at medical care clinics can also facilitate risk stratification and prioritisation of referrals to busy eye clinics. In these circumstances, a key consideration would be the availability of skilled human resources facilitating task shifting and sharing and efficient, cost effective and valid technology for DRS.
Retinal fundus photography is the most common DRS method used globally [13] and digital systems are mostly preferred [14]. Conventional desk-top digital cameras require significant physical space, skilled photographers and large image storage devices which incur high capital investment but are cost effective [15]. Hand-held digital cameras are portable, require less space, minimum power consumption and less skills and training [16]. Non-mydriatic retinal imaging is more popular considering the convenience for both service user and provider due to absence of procedures such as pupil dilatation [17].
However, this may have an impact on image gradability and screening coverage [18].
Hand held retinal cameras use for DRS in various settings and outcomes mainly depend on the image quality. Yogesan et al., (2000) reported that images captured by a hand-held camera were not suitable for tele-screening due to poor quality (only 24% in good quality). However in this study sample size was very low (n=25 participants, 49 eyes) [19]. A study conducted in France, concluded that hand-held retinal imaging system was less efficient with poor image quality. However, in this study the photographer had undergone training only on 10 patients before the study, which is a highly inadequate for a hand-held camera [20]. In contrast, A study conducted in China reported that 63% of the images were in excellent quality, however the age of the participants was started as low as 9 years (age range 9-84 years) [21]. A review by Cuadros et al., (2017) concluded that hand-held cameras are practically convenient but do not provide sufficient image quality [22]. Therefore, quality of the images is a major concern in hand-held devices, though they are easy to use.
To the authors' knowledge there is no evidence on DRS using digital retinal imaging from Sri Lanka. A situational analysis conducted in the Western province showed a large gap in DRS services delivery compared to the estimated need [23]. This study aims to demonstrate the functional and technical feasibility of using a hand-held non-mydriatic digital camera in a LMIC non-ophthalmic setting. We assessed the DTA of DR detection by general physicians using this method compared to the local clinical reference standard of mydriatic indirect ophthalmoscopy and bio-microscopic examination by a retinologist.

Methods
Ethics approval was obtained from both ethics review committees of the London School of Hygiene & Tropical Medicine-United Kingdom and the National Eye Hospital-Sri Lanka. This study adhered to the tenets of the 'Declaration of Helsinki'. A prospective screening intervention validation study was conducted between May 2017 and May 2018 at a tertiary level, public sector out-patient medical clinic in the Western province of Sri Lanka. The main outcome measure was detection of signs of DR (any DR or a referable level) by physician graders using captured digital images, according to a locally adopted guideline.
The protocol of this validation study has been published in Journal of Medical Internet Research (JMIR-doi:10.2196/10900) and a summary is outlined below [24].

Summary of the methods
Nine general physicians from a tertiary level institution underwent a competency-based training programme delivered by two retinologists. The training included the following: capturing retinal fields using a hand-held non-mydriatic fundus camera (Zeiss-Visuscout100®-Germany), identification of signs of DR using images and DR grading according to an adopted classification system based on the United Kingdom -National Screening System [25] (Additional File 1- Table 1). The hand-held imaging system has the ability to capture colour and red free retinal images in a range of +20 diopters (D) to -20 D, at 40 0 field of view angle. The camera comprised of 9 fixation targets and resolution of the camera is 800 x 480 (5 megapixels). Guidelines were used to standardize reporting of image quality, and ungradable images were classified based on the proportion of the retina visible for grading (Additional File 1- figure 1). Physicians were tested using a set of standard images of DR and the two who reached the required level of agreement with the retinologist (k=0.8-0.9) were selected as graders in the validation study.
A sample size of n=506 participants was chosen, in order to estimate the sensitivity within a margin of error 10% (based on 95% confidence intervals), with an expected sensitivity of 70% and prevalence of moderate NPDR among PwDM of 20%. This included an additional 25% to take account of ungradable images (i.e., <50% of the retina visible). Interim analysis was undertaken to ascertain the level of ungradable images and, to take account of a higher than expected proportion of ungradable images, the sample size was increased to 700 PwDM. A consecutive sample of diagnosed PwDM (>18years) without previous DRS at an eye clinic who were able to provide written informed consent were eligible to participate.
Participants were identified at a tertiary level medical clinic, in the Western province of Sri Lanka.
In the index test imaging, two-field (1st field-macula cantered, 2nd field-disc centred) (Additional File 1-figure 2), 45-degree retinal images were captured in each eye before and after pupillary dilatation, using 2% phenylephrine, following adequate mydriasis (5-6 mm) by each physician grader. During grading, the non-mydriatic images were graded first. We calculated DTA at 3 levels for the non-ophthalmic settings: i.e., 1) any DR (detection of R1, R2, R3 and R4), 2) referable DR (R2 and above) and 3) detection of referable level and maculopathy combined with a visual acuity cut off (worse eye >6/18 Snellen visual acuity) (see Additional File 1- Table 1). The graders were masked to the history and clinical examination findings and pupil status of the images. The clinical reference test entailed a detailed, dilated fundus examination by an experienced trainer retinologist using slit-lamp bio-microscopy with a 90D lens and indirect ophthalmoscopy using a 20D lens. The reference test was conducted by one retinologist with more than 15 years of clinical experience in vitreo-retina field. The 7-field 'Early Treatment diabetic Retinopathy Study' references test was logistically not feasible in this resource poor setting. This reference examination took place as soon after imaging as possible in all 700 PwDM that were included in the index test. The retinologist was masked to the clinical status and physician graders' findings.
For quality assurance, 15% of each non-mydriatic and mydriatic image sets were evaluated by the retinologist for technique, ability to image the required field and gradability. Fifteen percent of each hundred image sets were given back to the physician graders for double grading to assess the repeatability and intra-grader agreement in the 1st and 2nd attempts of grading images. A sample of the same image sets (n=212) were graded by the retinologist to calculate inter-grader agreement. DTA variables of sensitivity, specificity and predictive values and agreement analyses (kappa statistics) were calculated with 95% confidence intervals, for each grader and each pupil status compared to the reference standard using individual eyes as the unit of analysis, considering each gradable eye as a separate case. Two approaches were used in the calculations to examine the impact of ungradable images on the outcomes. i.e., by excluding the ungradable images and by including ungradable as test positive in the analysis. As ungradable images indicate a requirement for referral to an eye clinic, we analysed ungradable images as screen positives to examine the sensitivity and specificity of detecting a need for referral. This also allows comparisons with previous studies, which have used both methods.

Analysis
Subgroup analysis conducted for identification of presence/absence of DR (any DR), moderate NPDR and above with / without macular signs, to make recommendations for a referable criterion for the local context. We used different referable criteria i.e., by pupil status, level of DR, level of visual acuity and presence of macular signs in the analysis to understand the variation in DTA to assess the most suitable and accurate cut off level of DR without overloading the eye clinic and also facilitating safe practice at a non-ophthalmic setting.

Participants' characteristics
Of the 826 eligible PwDM identified from medical clinical records, response rate was 84.7% (700/826). Mean age of the participants was 60.8 years (SD ±10.08) and majority were women (66%, 462/700). Only 27.9% (195/700) of the participants were employed and 79.1% (554/700) lived in the capital city of Colombo and hailed from low income families (88%, 616/700, monthly income < £150). Of these, 98.4% (689/700) had type 2 DM and 1.6% (11/689) were diagnosed with DM at age < 30 years and were on insulin. The mean age at diagnosis of DM was 50.9 years (SD ±11.03) and mean duration of diabetes was 9.9 years (SD ±8.09). Mean fasting plasma glucose with in the last 3 months was 140.4 mg/dl (SD ±55.43). Additional co-morbidities included; hypertension (70%), hyperlipidaemia (57.3%), ischaemic heart disease (31.9%), nephropathy (9%) and neuropathy (35%). The Table 1 shows the characteristics of the PwDM in this study. The maximum time interval between index and reference test was 4 weeks.  Table 2 and Additional Files 2 and 3. In addition, a very good gradability agreement (range k=0.72-0.96) was observed for physician graders in comparison to retinologist's findings using a sample of images.  Table 3. We proposed grading the same images by the retinologist would provide a fair comparison for the physician graders in agreement analysis. However, here the concerns were limitations in the degree of view of a hand-held retinal camera and skills of capturing images by the physicians. In this analysis, in the grading of DR and macular signs, we found that inter-grader agreement was mostly >0.8 except for the grader 2 non-mydriatic images (any DR k=0.80-0.89, referable DR k=0.77-0.85, macular signs k=0.77-0.85). We observed a satisfactory level of agreement of the physician graders findings using a 2-field modality.
The overall concordance of the results is described in Table 5.  [25]. We assumed that inclusion of ungradable images in the DTA analysis is a pragmatic approach for a non-ophthalmic setting, considering the requirement of referring those PwDM to the eye clinic for further assessment and treatment. The proposed imaging strategy could act as a filter minimizing the number of referrals at eye clinic, thereby reducing the strain on the system. The United Kingdom prospective diabetes study group (UKPDS) reported that 15.3% of those with signs of DR at baseline, required laser at 3 years [26]. Therefore, identification of even minor levels of DR will be beneficial to stratify the risk groups early.
Digital retinal imaging showed promising results in DRS [25]. The digital imaging systems have the advantage of instant availability of images for quality assessment and convenient storage and retrieval. Several studies have compared digital fundus photography with 7fields used in early treatment diabetic retinopathy study (ETDRS) [27]- [29] or mydriatic ophthalmoscopy [30], [31]  The optimum number of retinal fields in a DRS strategy is a key factor that affects accuracy.
The ETDRS 7-field strategy is considered to be the gold standard for DR detection but is not practical in a screening program [32]. Previous studies showed that single retinal field is inadequate to achieve required standards [33]- [37]. Studies have also demonstrated that 3fields would not improve DTA of detection of any referable DR [38]. A non-mydriatic two field strategy in detection of sight threatening DR (STDR) in a HIC showed a sensitivity of 92% (95% CI 90-94%) and specificity of 96% (95% CI 95-98%) (proportion of ungradabilitynon-mydriatic 15.3-17.6%, mydriatic 1.4-2.1%) [27]. In our study, sensitivity was 71.7-73.5% and specificity 98.9-99.6% for detection of referable DR using non-mydriatic imaging.
We could not achieve a higher level of sensitivity comparable with the studies done in HICs, due to poor image quality. The main causes of poor image quality are dark iris colour, poor pupil dilation status and lens opacity [17], [39]. In HICs prevalence of cataract is less compared to LMICs like Sri Lanka [40]- [42]. We observed that sensitivity increased to 81.8%-89.3% when pupils were dilated. In addition, specificity was high irrespective of the pupil status, because physician graders were confident in grading in the absence of any signs. The study by Henricsson, M. et al. (2000) showed that dilatation and increasing number of fields to 5, the DTA improved to sensitivity of 93% and specificity of 91% [43]. It is apparent that one or more fields to the two central fields in DRS, has increased DTA minimally [28]. Therefore, a 2-field DRS strategy is justifiable for this context. In addition, slit-lamp examination by the retinologist is a justifiable reference test for this context.  showed that there was no significant difference in the assessment of DTA between using 7-field ETDRS and slit lamp examination by ophthalmologists [31].
In some settings, several non-ophthalmological personnel had been employed in DRS. In our study we proposed DRS by trained general physician at medical clinic following assessment of barriers. The estimates from previous studies are comparable with the finding of our study [39]. In a study from the United Kingdom, DRS by general practitioners using 35mm colour images shown that detecting any level of DR was increased from 62.6% (95% CI 55. ophthalmic health professionals and an overall specificity of >91% [45]. The sensitivity of detection of any level of DR increased to 87-100% for general practitioners and >91% for optometrists with pupil dilatation [45]. As a first line, this study has shown that physician graders are capable of DRS in a non-ophthalmic setting in Sri Lanka. However, we will have to study the effectiveness of this modality in a larger number to make specific recommendations to implement a population-based program. In our study 75.6% of the participants had no DR, 16.7% mild DR (R1), 3.6% moderate NPDR (R2), 0.4% severe NPDR (R3) and only 1% had PDR (R4). A study conducted among the slum populations (age > 40 years, known PwDM) in India, using a hand-held nonmydriatic camera reported 8.1% severe NPDR and 6.8% PDR which are higher prevalence than our study [46].
One reason for higher prevalence could be poor diabetes management among the slum populations. However, in this study, relatively a higher gradability of images (89.4% gradable) was observed even in non-mydriatic mode, probably due to images were graded at the site after directly visualising on the display of the hand-held camera. We have noticed that image quality is apparently higher on a small screen compared to displaying on a traditional viewing monitor. In another study conducted in India, among 500 PwDM at an endocrinology clinic, proportion ungradable was 30.6% and 31% among two observers which is comparable to our results [39]. In comparison we observed that studies conducted in HICs reported high proportions of gradability compared to our study. A study conducted in USA 86-94% images were gradable before pupil dilatation in hand-held retinal imaging [47]. Similarly a study conducted in a upper middle income setting (China) reported a low ungradable proportion of 4.75% (19/400) using a hand-held camera [21]. The low prevalence of DR in our study could be attributed to many factors. One reason for this would be excluding those who had undergone previous DRS and treatment. In Sri Lanka about 50% of the PwDM in clinics had DRS [48]. Forty percent (572/1398) of the PwDM had previous DRS or DR treatment in our study. The high proportion poor image quality in our study could be due to smaller pupil size and presence of lens opacities.
Non-mydriatic imaging has lower resolution and lower image quality leading to poorer detection of DR [17], [49]. However, digital imaging has lower technical failure rates than imaging using colour slides [50]. The hand-held non-mydriatic camera used in this study required a minimum of 3.5 mm pupil diameter and average pupil diameter in this study population was 2.01 mm at presentation. When pupils were dilated, proportion of ungradable images was reduced from 43.4% to 12.8%. Even at the reference test 37 eyes were ungradable due to lens opacity. The improvement of image quality in people with dark irises by pupil dilatation has been demonstrated in a previous study [17]. The referral of ungradable images to an ophthalmologist's clinic is in the best interest of patient safety.

Conclusion
In this study we demonstrated that the diagnostic test accuracy of the physician graders was closer to the standard practice of national level screening programs in other settings.
We conclude that 2-field retinal imaging using a hand-held digital camera at a medical clinic, by physician graders, with dilatation of pupil of those who have ungradable images, provides a valid modality to identify referable level of diabetic retinopathy. This strategy is an accurate screening method of detection of a referable level in a health care facilitybased people with diabetes who are at risk of developing sight threatening diabetic retinopathy.

Declarations
Ethics approval and consent to participate -The ethics approval was obtained from the London School of Hygiene and Tropical Medicine -United Kingdom and from the National Eye Hospital of Sri Lanka.
Consent for publication -Authors provide consent for publication in the BMC-Ophthalmology -Open source.
Availability of data and material -Any of the data and material produced in this study will be available under a reasonable request to the corresponding author, for non-commercial purposes.
Competing interests -No conflict of interest.