Intra- and inter-rater agreement between an ophthalmologist and mid-level ophthalmic personnel to diagnose retinal diseases based on fundus photographs at a primary eye center in Nepal: the Bhaktapur Retina Study
© The Author(s). 2016
Received: 14 February 2016
Accepted: 7 July 2016
Published: 18 July 2016
Early detection can reduce irreversible blindness from retinal diseases. This study aims to assess the intra- and inter-rater agreement of retinal pathologies observed on fundus photographs between an ophthalmologist and two-mid level ophthalmic personnel (MLOPs).
A population-based, cross-sectional study was conducted among subjects 60 years and above in the Bhaktapur district of Nepal. Fundus photographs of 500 eyes of 500 subjects were assessed. The macula-centered 45-degree photographs were graded twice by one ophthalmologist and two MLOPs. Intra-rater and inter-rater agreements were assessed for the ophthalmologist and the MLOPs.
Mean age was 70.22 years ± 6.94 (SD). Retinal pathologies were observed in 55.6 % of photographs (age-related macular degeneration: 34.2 %; diabetic retinopathy: 4.2 %; retinal vein occlusion: 3.8 %). Twelve (2.4 %) fundus pictures were non-gradable. The intra-rater agreement for overall retinal pathologies, retinal hemorrhage, and maculopathy were substantial both for the ophthalmologist as well as for the MLOPs. There was moderate inter-rater agreement between the ophthalmologist and the first MLOP on second rating for overall retinal pathologies, [kappa (k); 95 % CI = 0.59 (0.51–0.66)], retinal hemorrhage [k; 95 % CI = 0.60 (0.41–0.78)], and maculopathy [k; 95 % CI = 0.52 (0.43–0.60)]. Inter-rater agreement between the ophthalmologist and the second MLOP for second rating was moderate for overall retinal pathologies [k; 95 % CI = 0.52 (0.44–0.60)], substantial agreement for retinal hemorrhage [k; 95 % CI = 0. 68 (0.52–0.84)], moderate agreement for maculopathy [k; 95 % CI = 0.59 (0.50–0.67)].
There is moderate agreement between the MLOPs and the ophthalmologist in grading fundus photographs for retinal hemorrhages and maculopathy.
KeywordsFundus photograph grading Retinal pathology Ophthalmologist Mid-level ophthalmic personnel
Retinal disorders are among the leading cause of visual impairment and avoidable blindness, not only in developed but also in developing countries. Increases in life expectancy along with changes in lifestyle and dietary habits have resulted in increases in blindness from diabetic retinopathy (DR) and hypertension-related retinal problems in addition to age-related macular degeneration (AMD) [1–4]. Retinal disease has been identified as the major cause of blindness after cataracts in a population-based study conducted recently in Nepal [5, 6].
Several studies have shown strong agreement between ophthalmoscopy and fundus photograph grading for detecting DR as well as digital fundus imaging and stereoscopic fundus pictures for detecting AMD [7–11]. Similarly, there is good agreement between grading conventional 45° images and ultra-wide-angle images in detecting the macular pathology . Therefore, digital imaging of the retina could be a useful tool for screening of retinal diseases in large epidemiological studies in both developed and developing counties.
The majority of retinal diseases are asymptomatic until there has been significant damage to the retina. Nepal has one ophthalmologist per 200,000 inhabitants and limited vitreo-retinal services, centered mainly in the urban areas. For this reason and lack of awareness, patients usually present late in the course of disease [13, 14]. Preservation of vision mainly depends on the timely detection and prompt treatment of sight threatening retinal conditions [15, 16]. Most districts in Nepal have district community eye centers which are primary level eye health facilities run by mid-level ophthalmic personnel (MLOP) who are trained to provide treatment for simple ocular problems. Training MLOPs to obtain and interpret fundus photographs depicting normal and abnormal retinal conditions could facilitate screening for visually threatening retinal diseases. This could lead to early diagnosis and timely referral of patients to an ophthalmologist.
This study will compare the agreement between MLOPs and an ophthalmologist regarding the grading of fundus photographs for the assessment of retinal pathologies. To the best of our knowledge, this is the first population-based study of its kind in Nepal. We hope, the findings from this study will be useful in setting up a system of screening of patients with retinal diseases by MLOPs, especially in remote areas of Nepal, and the results of this study could also be of importance and interest in developed countries.
The Bhaktapur Retina Study is a population-based, cross-sectional study conducted to estimate the prevalence of vitreo retinal diseases among residents 60 years and older in the Bhaktapur district of Nepal. This is the second survey on the same cohort that was enrolled for the Bhaktapur Glaucoma Study conducted between 2007–2010. The study subjects were enrolled from August 2013 to December 2014 at Bhaktapur District Community Eye Center, a primary eye care facility. Detailed demographics of the study subjects have been described elsewhere . Briefly, the second survey was conducted on the same population after a period of 5 years, primarily to study retinal disorders. Study participants were recruited from 30 clusters of the district, and a sample size of 2100 participants was calculated for the retinal study. All subjects attended the community eye centre in Bhaktapur district and underwent an ocular examination. A detailed history, biomicroscopic examination of the eye, including measurement of the intraocular pressure and a dilated fundus examination were conducted. Fundus photograph of the right eye of 500 cases were enrolled in the study for grading. The number of subjects calculated to reach 80 % power for assessing the intra-rater and inter-rater agreement was at least 475 . The fundus picture was taken after dilating the pupil and comprised of macula-centered 45-degree single photographs using a Canon digital fundus camera (CRX, Canon Company). Trained ophthalmic photographers were involved in fundus photography.
A special questionnaire to rate retinal pathology was designed with reference to some studies conducted on diabetic retinopathy, age related macular degeneration, cataract [19–21] and also after consultation with ophthalmologists, public health experts and statisticians. The questionnaire was pre-tested by rating 50 fundus photographs. There was no problem while rating the photographs by an ophthalmologist and mid level ophthalmic personnel.
An ophthalmologist and two MLOPs independently graded the fundus photographs twice at three weekly intervals using a 17-inch computer monitor. No additional training was given between the first and second reading of fundus photographs. MLOPs are trained to deliver primary eye care at the district level. Before getting involved in the study, the MLOP received training regarding the grading of fundus photographs. The training conducted by the ophthalmologist was based upon diagnosing retinal diseases by interpreting several fundus photographs.
A questionnaire was developed in which the graders were asked to assess fundus photographs as normal or abnormal, presence or absence of retinal hemorrhages, and presence or absence of maculopathy and other retinal abnormalities. They were also asked to state when the fundus photo could not be graded. Details have been given in Additional file 1. DR was graded according to the Early Treatment Diabetic Retinopathy Study Criteria . Likewise, AMD was categorized according to the International classification developed by the International ARM Epidemiological Study Group . Both the intra-rater (the agreement between one grader) and inter-rater agreement (agreement between two different graders) were assessed in the study through the use of the kappa coefficient (k). The agreement was less than chance if kappa was <0.01, slight agreement if kappa was 0.01–0.20, fair agreement if kappa was 0.21–0.40, moderate agreement if kappa was 0.41–0.60, substantial agreement if kappa was 0.61–0.80 and almost perfect agreement if kappa was 0.81–0.99 . The study was approved by the Institutional Review Board and Ethics Committee of Tilganga Institute of Ophthalmology (TIO) and conducted in accordance with the Declaration of Helsinki. Informed consent was written in the vernacular and was read to those unable to read. The subjects were asked to sign the consent form, and thumb impressions were taken for those unable to sign before enrollment into the study. Statistical analysis was performed using the STATA 9.0 (Stata Corp LD, College Station, Texas, USA).
Five hundred single fundus photographs of right eyes of 500 subjects were graded. The age ranged from 60 to 93 years with an average age of 70.22 ± 6.94 S.D. years. Females (54.8 %) outnumbered males. Retinal pathologies were observed in 55.6 % of study eyes on clinical examination by the ophthalmologist. AMD was the predominant retinal problem (34.2 %), followed by various grades of hypertensive retinopathy (4.6 %), diabetic retinopathy (4.2 %), retinal vein occlusion (3.8 %), and other retinal pathologies (6.6 %). Twelve (2.4 %) fundus pictures were not gradable due to media haze.
Intra rater agreement of the ophthalmologist and MLOPs
Intra-rater agreement for fundus photograph reading by the ophthalmologist and by two MLOPs
Ophthalmologist (Kappa value; 95 % CI)
MLOP 1 (Kappa value; 95 % CI)
MLOP 2 (Kappa value; 95 % CI)
Overall retinal pathologies
Inter rater agreement between ophthalmologist and MLOP on first rating
Inter-rater agreement between the ophthalmologist and two MLOPs at the first rating
Ophthalmologist Vs MLOP 1 (Kappa value; 95 % CI)
Ophthalmologist Vs MLOP 2 (Kappa value; 95 % CI)
MLOP 1 Vs MLOP 2 (Kappa value; 95 % CI)
Overall retinal pathologies
Inter-rater agreement between the ophthalmologist and two MLOPs at the second rating
Ophthalmologist Vs MLOP 1 (Kappa value; 95 % CI)
Ophthalmologist Vs MLOP 2 (Kappa value; 95 % CI)
MLOP 1 Vs MLOP 2 (Kappa value; 95 % CI)
Overall retinal pathologies
There was substantial agreement on non-gradable fundus photographs for both intra- and inter-rater grading between the ophthalmologists and the MLOPs at both the first and second readings.
This is the first population-based study to compare agreement between an ophthalmologist and MLOPs grading fundus photographs for the assessment of retinal pathologies in Nepal. Visual impairment and blindness resulting from retinal diseases such as AMD, DR and retinal vein occlusion (RVO) are mainly due to the involvement of the macular area [13, 14, 24, 25]. AMD was the most common retinal condition among the study subjects, while DR and RVO were the next common retinal pathologies.
In our study, the intra-rater agreements for retinal hemorrhage and maculopathy were substantial both for the ophthalmologist as well as for one of the MLOPs.
This proves that proper training of MLOPs could be beneficial for screening of retinal diseases. There was moderate inter-rater agreement between the ophthalmologist and first MLOPs on the first rating and second rating of retinal hemorrhage and maculopathy. The inter-rater agreement between ophthalmologist and second MLOP for retinal hemorrhage was substantial in both of the ratings. The difference in rating agreement between the two MLOPs could be explained by individual levels of understanding of the retinal condition. The inter-rater agreement between the two MLOPs was also moderate for overall retinal pathologies and maculopathy at first rating except for retinal hemorrhage. For retinal hemorrhage, there was substantial inter-rater agreement between the MLOPs in the first as well as the second rating. Maculopathy had substantial agreement only in the second rating. Better agreement in the second rating was probably due to a better understanding of the retinal conditions by repeating the tests. Likewise, the obvious retinal lesions such as retinal hemorrhages were better identified while interpreting the fundus photographs.
Developed countries have adequate human resources and facilities for tele-ophthalmology, and testing of automated detection of AMD and DR screening of retinal disease is underway [26, 27]. The scenario is still very different in the developing world. In Nepal, there are only 175 ophthalmologists for a population of 28 million. The majority of the ophthalmologists are providing service in the urban areas, whereas more than two-thirds of the population resides in rural parts of the country. Even more so, tele ophthalmology is not a well-established model of screening for eye diseases. With the increase of diabetes and hypertension as a serious public health problem in Nepal, these models of screening need to be prioritized. AMD has been established as the most common retinal pathology and leading cause of blindness among elderly people in Nepal [5, 25]. Preservation of vision largely depends upon the early detection of such retinal conditions by using cost-effective screening methods conducted by MLOPs in countries with inadequate human resources.
There was exact agreement between ophthalmoscopy and fundus photo grading for detecting all grades of DR in studies conducted in the United States . In another study conducted in the United States, a single field non mydriatric camera was superior to screen diabetic retinopathy over ophthalmoscopy . Similarly, in another study, there was perfect agreement between digital fundus imaging and stereoscopic fundus pictures for detecting AMD which led the authors to conclude that digital imaging of the retina was useful for epidemiological studies . Perfect agreement in detecting retinal hemorrhages by one of the MLOPs in our study proved that retinal diseases like DR and RVO may not have to be screened by an ophthalmologist. MLOPs providing services at the primary eye care level can screen retinal diseases and refer patients. This could help in reducing the irreversible blindness caused by retinal diseases.
Non mydriatric fundus cameras can be used for the screening of the majority of sight-threatening retinal conditions which affect the central part of the retina [29–32]; however, in our study, retinal examination was done after dilating the pupil in order not to miss any retinal pathology during clinical examination. Non mydriatic fundus photography is relatively easy and can be conducted to screen large number of patients in a short duration of time.
Due to the limited number of trained ophthalmologists, the government of Nepal has developed a course to certify MLOPs to provide primary eye care in ophthalmology. The curriculum for the MLOPs consists of a three-year training program after secondary school. It encompasses both general basic science and ophthalmology. They are also trained to carry out all related investigations in ophthalmology. Further enhancing their skill to interpret posterior segment eye diseases will help in screening of retinal conditions. The burden of diabetes and hypertension is expected to increase in the future, and by utilizing MLOPs in the screening of retinal diseases, this burden can be addressed. This will help in early detection of retinal diseases and prevention of blindness.
The moderate to fair agreement between one MLOP and the ophthalmologist warrants caution that some cases of retinal diseases could be missed. We hope that further training could improve the agreement between the two observers and can lead to a better rate of detection.
There are several strengths of the study. The large number of subjects and the setting where the study was conducted are major strengths. Primary eye care centres therefore can function as centres of screening in countries with limited resources. Comparing the results from two MLOPs was also the strength of the study.
In our study, elderly people with some media haze could have resulted in non-gradable pictures when the pupil had not been dilated for fundus photography. The use of a trained photographer and mydriatic photographs could have resulted in a larger number of gradable pictures.
There is moderate to almost perfect agreement between MLOPs and ophthalmologist in grading fundus photographs for retinal hemorrhages and maculopathy. Utilization of MLOPs to screen retinal diseases at the primary eye care level can help in early detection and referral.
We would like to acknowledge Tilganga Institute of Ophthalmology, Kathmandu, Nepal and Vrije University Medical Center, Amsterdam, The Netherlands for funding of this study. Likewise, we would like to acknowledge all the patients that participated in this study and the Bhaktapur Municipality. Professor Paul S. Bernstein MD, PhD of the Moran Eye Center of the University of Utah, USA edited the manuscript.
The study was funded by the Vrije University Medical Center, Amsterdam, The Netherlands and Tilganga Institute of Ophthalmology, Kathmandu, Nepal.
Availability of data and materials
All data analyzed during this study are included and available in this manuscript.
RT carried out the Bhaktapur Retina Study, participated in co-ordination, study concept, study design, data collection, data analysis, and prepared the manuscript. SB participated in data collection, and grading of fundus pictures. RB was involved in study design and critical analysis. GP was involved in critical analysis of the manuscript. SK participated in study design and performed statistical analysis and critical review of the manuscript. SHT was involved in study design and critical analysis of the manuscript. SST participated in co-ordination, study design, manuscript preparation, and critical analysis. GVR was involved in study concept, study design, and critical analysis of the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
The study was approved by the Institutional Review Board and Ethics Committee of Tilganga Institute of Ophthalmology (TIO) and conducted in accordance with the Declaration of Helsinki. Informed consent was written in the vernacular and was read to those unable to read. The subjects were asked to sign the consent form, and thumb impressions were taken for those unable to sign before enrollment into the study.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Woo JH, Sanjay S, AuEong KG. The epidemiology of age-related macular degeneration in the Indian subcontinent. Acta Ophthalmol. 2009;87(3):262–9.View ArticlePubMedGoogle Scholar
- Report of a WHO Consultation in Geneva. Prevention of Blindness from Diabetes Mellitus. 2005, Switzerland.Google Scholar
- Wild S, Roglic G, Green A, Sicree R, King H. Global Prevalence of Diabetes. Estimates for the year 2000 and projections for 2030. Diabetes Care. 2004;27:1047–52.View ArticlePubMedGoogle Scholar
- Nirmalan PK, Robin AL, Katz J, et al. Prevalence of vitreoretinal disorders in a rural population of southern India: the Aravind Comprehensive Eye Study. Arch Ophthalmol. 2004;122:581–6.View ArticlePubMedGoogle Scholar
- Brilliant LB, Pokhrel RP, Grasset NC, et al. Epidemiology of blindness in Nepal. Bull World Health Organ. 1985;63(2):375–86.PubMedPubMed CentralGoogle Scholar
- Thapa SS, Khanal S, Paudyal I, Twyana SN, Ruit S, Van Rens GHMB. Outcome of cataract surgery: a population based developing world study in the Bhaktapur district, Nepal. Clin Exp Ophthalmol. 2011;39:851–7.View ArticlePubMedGoogle Scholar
- Moss SE, Klein R, Kessler SD, Richie KA. Comparison between Ophthalmoscopy and fundus photography in determining severity of diabetic retinopathy. Ophthalmology. 1985;92(1):62–7.View ArticlePubMedGoogle Scholar
- Diamond JP, McKinnon M, Barry C, Geary D, McAllister IL, House P, Constable IJ. Non-mydriatric fundus photography: a viable alternative to fundoscopy for identification of diabetic retinopathy in an Aboriginal population in rural Western Australia? Aust N Z J Ophthalmol. 1998;26(2):109–15.View ArticlePubMedGoogle Scholar
- Gupta SC, Sinha SK, Dagar AB. Evaluation of the effectiveness of diagnostic and management decision by teleophthalmology using indigenous equipment in comparison with in-clinic assessment of patients. Indian J Med Res. 2013;138:531–5.PubMedPubMed CentralGoogle Scholar
- Klein R, Meuer SM, Moss SE, Klein BEK, Neider MW, Reinke J. Detection of Age-related macular degeneration using a non mydriatric digital camera and a standard film fundus camera. Arch Ophthalmol. 2004;122(11):1642–6.View ArticlePubMedGoogle Scholar
- Jain S, Hamada S, Membrey WL, Chong V. Screening for age related macular degeneration using non stereo digital fundus photographs. Eye. 2006;20:471–5.View ArticlePubMedGoogle Scholar
- Csutak A, Lengyel I, Jonasson F, Leung I, Geirssdottir A, Xing W, Peto T. Agreement beween image grading of conventional 45 degree and ultra wide angle 200 degree digital image in the macula in the Reykjavik eye study. Eye. 2010;24:1568–75.View ArticlePubMedGoogle Scholar
- Thapa R, Paudyal G, Maharjan N, Bernstein PS. Demographics and awareness of diabetic retinopathy among diabetic patients attending the vitreo-retinal service at a tertiary eye care center in Nepal. Nepal J Ophthalmol. 2012;4(7):10–6.PubMedGoogle Scholar
- Thapa R, Paudyal G, Shrestha MK, Gurung R, Ruit S. Age-Related Macular Degeneration in Nepal. Kathmandu Univ Med J. 2011;35(3):165–9.Google Scholar
- Early Treatment Diabetic Retinopathy Study Research Group. Early Photocoagulation for Diabetic Retinopathy: ETDRS Report 9. Ophthalmology. 1981;98:766–85.Google Scholar
- Loewenstein A. The significance of early detection of age-related macular degeneration. Richard and Hinda Rosenthal Foundation Lecture, The Macula Society 29th Annual Meeting. Retina. 2007;27:873–8.View ArticlePubMedGoogle Scholar
- Thapa SS, Rana PP, Twyana SN, Shrestha MK, Paudyal I, Paudyal G, Gurung R, Ruit S, Hewitt AW, Craig JE, Van Rens GHMB. Rationale, methods and baseline demographics of the Bhaktapur Glaucoma Study. Clin Exp Ophthalmol. 2011;39(2):126–34.View ArticlePubMedGoogle Scholar
- Sim J, Wright CC. The Kappa Statistic in Reliability Studies: Use, interpretation, and sample size requirements. Phys Ther. 2005;85:257–68.PubMedGoogle Scholar
- Bryan CA, Hooi AW, Stephen CT, Rajagopalan R. A comparison of the 1-, 2-, and 3-Field Retinal Photography Protocols in the Screening and Grading of Diabetic Retinopathy by Ophthalmology Healthcare Providers with Various Levels of Training. Invest Ophthalmol Vis Sci. 2012;53:2858.Google Scholar
- Maberley DAL, Isbister C, Mackenzie P, Aralar A. An evaluation of photographic screening for neovascular age related macular degeneration. Eye. 2005;19:611–6.View ArticlePubMedGoogle Scholar
- Chew EY, Kim J, Sperduto R, et al. Evaluation of the Age-related eye disease study clinical lens grading system AREDS Report No. 31. Ophthalmology. 2010;117(11):2112–9.e3. doi:https://doi.org/10.1016/i.ophtha.2010.02.033.View ArticlePubMedPubMed CentralGoogle Scholar
- International ARM Epidemiological Study Group. An international classification and grading system for age-related maculopathy and agerelated macular degeneration. Surv Ophthalmol. 1995;39:367–74.View ArticleGoogle Scholar
- Viera AJ, Garrett JM. Understanding interobserver agreement: The Kappa Static. Fam Med. 2005;37(5):360–3.PubMedGoogle Scholar
- Thapa R, Paudyal G, Bernstein PS. Demographic characteristics, patterns and risk factors for retinal vein occlusion in Nepal: a hospital-based case–control study. Clin Experiment Ophthalmol. 2010;38(6):583–90.View ArticlePubMedGoogle Scholar
- Thapa SS, Thapa R, Paudyal I, Khanal S, Aujla J, Paudyal G, Rens G. Prevalence and pattern of vitreo-retinal diseases in Nepal: The Bhaktapur glaucoma study. BMC Ophthalmol. 2013;13:9.View ArticlePubMedPubMed CentralGoogle Scholar
- Bhuiyan A, Xiao D, Yogesan K. A review of disease grading and remote diagnosis for sight threatening eye condition: Age Related Macular Degeneration. J Comput Sci Syst Biol. 2014;7(2):062–71.View ArticleGoogle Scholar
- Perumalsamy N, Prasad NM, Sathya S, Ramasamy K. Software for reading and grading diabetic retinopathy Aravind diabetic retinopathy screening 3.0. Diabetes Care. 2007;30:2302–6.View ArticlePubMedGoogle Scholar
- Lin DY, Blumenkranz MS, Brothers RJ, Grosvenor DM. The sensitivity and specificity of single field nonmydriatric monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: A comparison with ophthalmoscopy and standardized mydriatric color photography. Am J Ophthalmol. 2002;134:204–13.View ArticlePubMedGoogle Scholar
- Penman AD, Saaddine JB, Hegazy M, Sous ES, Ali MA, Brechner RJ, Herman WH, Engelgau MM, Klein R. Screening for diabetic retinopathy: the utility of non mydriatric retinal photography in Egyptian adults. Diabet Med. 1998;15(9):783–7.View ArticlePubMedGoogle Scholar
- Farley TF, Mandava N, Prall FR, Carsky C. Accuracy of primary care clinicians in screening for diabetic retinopathy using single-image retinal photography. Ann Fam Med. 2008;6:428–34.View ArticlePubMedPubMed CentralGoogle Scholar
- Saari JM, Summanen P, Kivela T, Saari KM. Sensitivity and specificity of digital retinal images in grading diabetic retinopathy. Acta Ophthalmol Scand. 2004;82(2):126–30.View ArticlePubMedGoogle Scholar
- Klein R, Klein BE, Neider MW, Hubbard LD, Meuer SM, Brothers RJ. Diabetic retinopathy as detected using ophthalmoscopy, a non mydriatric camera and a standard fundus camera. Ophthalmology. 1985;92(4):485–91.View ArticlePubMedGoogle Scholar