Using structured progress to measure competence in flexible bronchoscopy

Introduction

Flexible bronchoscopy is an essential part of the diagnostic workup of patients suspected of lung cancer (1). Correct procedural performance is a prerequisite for accurate diagnosis and allocation of correct treatment of the patient (2). A lower yield of positive biopsy material, increased patient discomfort, and higher complication rates are seen in the early part of a trainee’s learning curve (3-5) and a satisfactory educational level must be ensured before independent/unsupervised practice on patients. Traditionally, certification has been based on an arbitrary number of procedures performed as suggested by The American College of Chest Physicians (100 procedures) (6), and the European Respiratory Society (7). These suggestions are not evidence-based since trainees learn at different paces and procedural experience does not ensure competency (8,9). A CHEST expert panel report suggested a move from volume-based certification system to skill acquisition and knowledge-based competency assessment for trainees (10). Mastery learning is a training modality, where the trainee practices until proficiency targets are met (11), and thereby relies on assessment tools that can determine if the desired level of competence is achieved (12). Several tools have been developed for the assessment of bronchoscopy performance (13,14). These assessment tools allow for the evaluation of diagnostic completeness and procedure time which have been identified as useful measures of competence when performing flexible bronchoscopy (14-17). However, there are no measures which examine whether the bronchial segments are inspected in a random or structured order. A fundamental element in performing bronchoscopy is that the operator should be able to identify all segments without overlooking any of these, since each segment may contain a tumor or other important pathology even in a patient not suspected of lung cancer (18). We hypothesize that following a systematic ascending path through the bronchial tree ensures a higher fraction of examined bronchial segments and limits the procedure time, and therefore can be regarded as an assessment measure of operator competency.

The aim of the study was to explore whether a new outcome measure for structured progression through the bronchial tree could be used to evaluate competency in flexible bronchoscopy.

We present the following article in accordance with the STROBE reporting checklist (available at http://dx.doi.org/10.21037/jtd-20-2181).

Methods

Study design

The study was designed as a prospective comparative study performed in a simulated setting at the Copenhagen Academy for Medical Education and Simulation. Messick’s framework of validity was used to gather validity evidence from four sources: content, response process, internal structure and relations to other variables (Table 1) (19). It is the gold standard when evaluating validity as recommended by the American Educational Research Association (20,21).

Table 1 Different sources of validity evidence for Structured Progress based on Messick’s validity framework (19)
Full table

Simulation equipment

The simulated set-up included an endoscopy tower with a flexible bronchoscope (EVIS Exera II and Q180 flexible bronchoscope, Olympus, Japan). The phantom was a bronchoscopy model equipped with a three-dimensional bronchial tree (CLA Broncho Boy, CLA, Coburg, Germany). The right lung has ten segments (segments #1–#3 in the upper lobe; #4 and #5 in the middle lobe; #6–#10 in the lower lobe), while the left lung has eight segments (#1–#3 in the upper lobe with #1–#2 fused together to form one segment; #4–#5 in lingula; #6–#10 in the lower lobe without a segment #7).

Performance checklist

A checklist that recorded specific checkpoints corresponding to each bronchial segment of the phantom was developed for the purpose of the study. This was set up in a graphical user interface written in Python [3.7.6] using the Tkinter library included in the standard package.

Participants

The participants were recruited and divided into three groups based on levels of experience: The novice group included physicians with no prior experience in flexible bronchoscopy, the intermediate group included physicians who had performed 5–100 procedures, and the expert group included physicians with experience in >500 procedures. For the study size each group should contain at least ten participants. The inclusion period was from 1st of March to 31st of May 2019.

Ethics

Participation was voluntary; informed consent was gathered before inclusion. In accordance with Danish law, an educational study without participation of patients does not require ethical approval.

Assessment procedure

The participants were invited to participate in the study through email. The purpose of the study and the included parameters were not disclosed to the participants in order not to influence their performance. Prior to coming to the simulation center for testing, they received a basic handbook in flexible bronchoscopy to use as a reference, which included a description of the anatomy of the bronchial tree and how to systematically locate all bronchial segments (22). This handbook included a description of the anatomy of the bronchial tree and how to find all bronchial segments.

Testing was performed individually. Each participant initially practiced using the flexible bronchoscope by locating four hidden marks inside a makeshift box. Afterwards, they were asked to perform three bronchoscopy procedures on the phantom.

During the bronchoscopy, the operators could not see the outside of the tracheobronchial tree inside the chest (Figure 1). They were asked to enter all segmental bronchi and were not given any feedback during or in-between the trials.

Figure 1 Live rating using a checklist of bronchial segments while the participants performs a full bronchoscopy on an inanimate bronchoscopy model, Broncho Boy (CLA, Germany).

Rating

The procedures were witnessed and monitored by the primary investigator (KMC) who has extensive experience in performing and rating bronchoscopy in a simulated setting. Each participant was rated during the procedure using the checklist. When the button representing a bronchial segment was pressed, the time and position of the scope were saved automatically in a text file.

All recordings were blinded and verified for the correct amount of visited bronchi by KMC.

Outcome measures

A new outcome measure, Structured Progress (SP) was developed by two pulmonary consultants (PFC and UB) with more than 10 years’ experience in bronchoscopy, a thoracic surgeon and professor of medical education (LK) and a biomedical engineer with more than 5 years’ experience in simulation (MBSS). SP indicates the number of times the operator progressed from one segment to the immediate succeeding segment. Passing from segment #1 to #2 gives one point but going from segment #2 to #1 gives zero point. SP ranges from 0–18 as the potential first point is awarded by passing from the carina to segment #1 either on the right or left side depending on which lung is inspected first. Progression from the carina to segment #1 to segment #2 to segment #3 gives 3 out of three points, while progressing from the carina to segment #1 to segment #3 to segment #2 gives 1 out of three points.

Other measures include diagnostic completeness (DC) defined as the thoroughness of examination measured as the fraction of total bronchial segments entered as a total of 18 segments and average intersegmental time (AIT), which is the average time passed between each segmental visit, calculated based on the difference between timepoints registered.

These three outcome measures were automatically calculated by the graphical interface after each performance.

Data analysis and statistics

The performances of the three groups with regards to SP, DC, and AIT were collected through the checklist software and compared using Kruskal-Wallis with a post hoc Dunn-Bonferroni test. The correlations between SP, DC, and AIT were calculated as Pearson’s r value.

The three groups were compared according to sex, age, medical specialty, individual and supervised bronchoscopies in a clinical setting, bronchoscopies in a simulated setting, bronchoscopies within the last 12 months and days since the last bronchoscopy by ANOVA for data showing normal distribution or by Kruskal-Wallis for data that did not show normal distribution for continuous variables, and by χ²-test for categorical variables.

For data with normal distribution, mean and standard deviation were used. For data that did not show normal distribution, median and range were used.

All data analysis was completed in Python (3.6.6, Python Software Foundation), using libraries Numpy (23), Scipy (24), and Matplotlib (25). Statistical testing was completed in Statistical Package for the Social Sciences version 25 (PASW v25.0; SPSS Inc., Chicago, Ill., USA). A P <0.05 was considered statistically significant.

Results

A total of 33 participants were enrolled: 12 novices, 11 intermediates, and 10 experts (Table 2). All participants performed three procedures each resulting in 98 trials. The first trial data of one of the intermediates was lost due to system breakdown.

Table 2 Participants details and comparison
Full table

A summary of the results based on validity evidence for the four sources are presented in Table 1.

Content

Structured progress is a progressive score ranging from 0–18, indicating the number of times the operator progressed from one segment to the immediate succeeding segment.

Relations to other variables

All three outcome measures (SP, DC, and AIT) were significantly different between the groups using Kruskal-Wallis (P<0.001) (Table 3, Figure 2). For the expert group, DC was not correlated with individual bronchoscopies in a clinical setting (Pearson correlation, r=0.23, P=0.42), bronchoscopies in a simulated-setting (Pearson correlation, r=0.50, P=0.14), bronchoscopies within the last 12 months (Pearson correlation, r=−0.25, P=0.48) or days since last bronchoscopy (Pearson correlation, r=0.21, P=0.56).

Table 3 Trial details and comparison
Full table

Figure 2 Left: DC as a fraction according to the score of SP in points from 0 to 18. Center: AIT in seconds according to the score of SP in points 0 to 18. Right: AIT in seconds according to the score of DC as a fraction. For all three figures bards indicate reference interval of each group. DC, diagnostic completeness; SP, structured progress; AIT, average intersegmental time.

Internal structure

There was a significant correlation between all three outcome measures (Table 4): SP and DC (Pearson correlation, r=0.62, P<0.001), SP and AIT (Pearson correlation, r=−0.52, P<0.001) and DC and AIT (Pearson correlation, r=−0.52, P<0.001). Sub-analysis of each group showed moderate correlation for all three measures except for the novice group at the correlation between SP and AIT (Pearson correlation, r=−0.22, P=0.20) (Table 4).

Table 4 Correlation between outcome variables
Full table

Discussion

This is the first study to explore the importance of a structured navigational path in bronchoscopy. We found that following a structured navigational path in ascending order between the bronchial segments (SP) was correlated with a higher DC and a lower AIT.

More specific guidelines that govern training in bronchoscopy are needed (26). Previous studies in simulation-based training in bronchoscopy have proposed DC as a measure of competence but did not give any suggestions towards a training method to improve this measure (17,27). DC was not attributable to recent or total experience with the procedure for the expert group. This finding is in line with the conclusion that the amount of performed bronchoscopies act as a surrogate marker for competence when assessing bronchoscopy skills (28). Our study shows that performance of a certain number of bronchoscopies does not ensure competency in reaching all segments. Based on our findings we suggest that following a structured navigational path of the procedure could instead be used as an indicator for competence.

Our study showed a positive correlation between DC and SP (Pearson correlation, r=0.62, P<0.001). Several findings in our study indicated this correlation to be a causal relation. It seems obvious that operators should visualize as much of the bronchial tree as possible; to ensure this, a bronchoscopy should optimally be completed in a systematic approach. This finding holds an obvious confounder, as the groups were divided by experience level. When analyzing only the expert group there was no significant correlation between DC and experience but the correlation between DC and SP was still significant (Pearson correlation, r=0.63, P<0.001), indicating that the most systematic experts managed to visualize more of the bronchial tree than their colleagues.

The segments are named after an intended orderly progression in bronchoscopy (29). Yet our study shows that the systematic approach is absent to a high degree when observed in a simulation setting. It has therefore been proposed for decades that bronchoscopy should be navigated through a structured progression between the bronchial segments (29). This study is the first to provide evidence for this assumption.

It is crucial to be systematic when performing an endoscopic pulmonary procedure (27). This has been shown for endobronchial ultrasound guided biopsy (EBUS-TBNA) for the staging of lung cancer. When comparing a systematic approach to a targeted, the former resulted in a higher diagnostic yield (30). Inspection of all segments is important as bronchoscopy can reveal endobronchial tumors that are not visible with computer tomography (31) and in patients not suspected for lung cancer (18). Similar results have been found in the field of colonoscopy (32,33). Therefore, a systematic approach is important when performing flexible bronchoscopy to ensure no segments are missed, and our measure of SP can be used as a valid assessment tool for user competence. The scoring system for SP is also adaptable for example in the case of a known abnormality where bronchoscopic inspection should start in the contralateral side of the lesion. One point is awarded for each systematic progression such that if the inspection starts in the left lung, the first point is given when progressing from carina to segment #1 on the left side.

A systematic progression in this study was associated with lower procedure time, as SP correlated with a decreasing AIT (Pearson correlation, r=−0.52, P<0.001). Procedure time has been identified as a measure of competence (14,15). Reducing the procedure time is an important aspect of bronchoscopy as a time effective procedure limits the amount of patient sedation, discomfort, and costs (34,35).

Simulation continues to play an important role for skills assessment (36). Virtual reality simulators combined with expert raters are currently used to asses competence in bronchoscopy (13). However, the use of a phantom can reliably be used to assess skill level (37) and could be superior to virtual reality simulators (26,38). The disadvantage of training on a phantom is that they do not provide any direct feedback as compared to virtual reality simulators. Computerized feedback improved the performance of trainee endoscopists as compared to no feedback (39).

Based on this study we aim in the future to develop a computerized feedback system that is feasible when training on phantoms to ensure a higher learning outcome. This could also work as an automatic assessment tool, which will be able to set an unbiased pass/fail criterion that is not susceptible to rater bias. Future studies should compare our automatic, objective scores with existing more subjective scoring systems from expert raters (40).

Our study is the first to examine the importance of a structured navigational path for flexible bronchoscopy and has several limitations. It was a single-center study, but participants were from 11 different hospitals in Denmark. Our sample size was relatively small but comparable to average educational studies (41). All data collection was done in a simulated setting to ensure full standardization and avoid ethical concerns. We used a phantom which is static, with no possibilities to train complexities and only exposes the learner to one iteration of the bronchial three. However, this is a first step towards developing an assessment tool of navigational path and it would be useful to repeat the study in real patients and in complicated bronchoscopy procedures. The study only focused on a limited amount of outcome measures. DC and AIT have been identified as useful measures of competence when performing flexible bronchoscopy (14,15,17).

This study identified SP as a useful outcome measure to assess competency in bronchoscopy. SP can be used alongside existing measures of competence (DC and AIT) (14,15,17), adding new useful nuances to the assessment of bronchoscopy performance that are not captured by existing measures—it was the only of the three outcome measures that was able to differentiate between intermediates and experts. We aim in the future to further develop the assessment tool to generate automatic feedback regarding the outcome measures DC, SP and AIT and to set predefined pass/fail standards for competent performance.

Acknowledgments

Funding: None.

Footnote

Reporting Checklist: The authors have completed the STROBE reporting checklist. Available at http://dx.doi.org/10.21037/jtd-20-2181

Data Sharing Statement: Available at http://dx.doi.org/10.21037/jtd-20-2181

Peer Review File: Available at http://dx.doi.org/10.21037/jtd-20-2181

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jtd-20-2181). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Participation was voluntary; informed consent was gathered before inclusion. In accordance with Danish law, an educational study without participation of patients does not require ethical approval.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Andolfi M, Potenza R, Capozzi R, et al. The role of bronchoscopy in the diagnosis of early lung cancer: a review. J Thorac Dis 2016;8:3329-37. [Crossref] [PubMed]
2. Paradis TJ, Dixon J, Tieu BH. The role of bronchoscopy in the diagnosis of airway disease. J Thorac Dis 2016;8:3826-37. [Crossref] [PubMed]
3. Hsu LH, Liu CC, Ko JS. Education and experience improve the performance of transbronchial needle aspiration: a learning curve at a cancer center. Chest 2004;125:532-40. [Crossref] [PubMed]
4. Ouellette DR. The safety of bronchoscopy in a pulmonary fellowship program. Chest 2006;130:1185-90. [Crossref] [PubMed]
5. Stather DR, MacEachern P, Chee A, et al. Trainee impact on procedural complications: an analysis of 967 consecutive flexible bronchoscopy procedures in an interventional pulmonology practice. Respiration 2013;85:422-8. [Crossref] [PubMed]
6. Ernst A, Silvestri GA, Johnstone D. Interventional pulmonary procedures: Guidelines from the American College of Chest Physicians. Chest 2003;123:1693-717. [Crossref] [PubMed]
7. Bolliger CT, Mathur PN, Beamis JF, et al. ERS/ATS statement on interventional pulmonology. European Respiratory Society/American Thoracic Society. Eur Respir J 2002;19:356-73. [PubMed]
8. Barsuk JH, Cohen ER, Feinglass J, et al. Residents' Procedural Experience Does Not Ensure Competence: A Research Synthesis. J Grad Med Educ 2017;9:201-8. [Crossref] [PubMed]
9. Leong S, Shaipanich T, Lam S, et al. Diagnostic bronchoscopy--current and future perspectives. J Thorac Dis 2013;5 Suppl 5:S498-510. [PubMed]
10. Ernst A, Wahidi MM, Read CA, et al. Adult Bronchoscopy Training: Current State and Suggestions for the Future: CHEST Expert Panel Report. Chest 2015;148:321-32. [Crossref] [PubMed]
11. McGaghie WC, Issenberg SB, Cohen ER, et al. Medical education featuring mastery learning with deliberate practice can lead to better health for individuals and populations. Acad Med 2011;86:e8-9. [Crossref] [PubMed]
12. Siddaiah-Subramanya M, Smith S, Lonie J. Mastery learning: how is it helpful? An analytical review. Adv Med Educ Pract 2017;8:269-75. [Crossref] [PubMed]
13. Konge L, Clementsen P, Larsen KR, et al. Establishing pass/fail criteria for bronchoscopy performance. Respiration 2012;83:140-6. [Crossref] [PubMed]
14. Konge L, Larsen KR, Clementsen P, et al. Reliable and valid assessment of clinical bronchoscopy performance. Respiration 2012;83:53-60. [Crossref] [PubMed]
15. Konge L, Arendrup H, von Buchwald C, et al. Using performance in multiple simulated scenarios to assess bronchoscopy skills. Respiration 2011;81:483-90. [Crossref] [PubMed]
16. Konge L, Arendrup H, von Buchwald C, et al. Virtual reality simulation of basic pulmonary procedures. J Bronchology Interv Pulmonol 2011;18:38-41. [Crossref] [PubMed]
17. Colt HG, Crawford SW, Galbraith O 3rd. Virtual reality bronchoscopy simulation: a revolution in procedural training. Chest 2001;120:1333-9. [Crossref] [PubMed]
18. Gottlieb M, Clementsen PF, Hakansson K, et al. Surprising squamous cell carcinoma in the right upper lobe at bronchoscopy in a heavy smoker. Ugeskr Laeger 2014;176:V05140262. [PubMed]
19. Validity MS. In: Linn RL. editor. Educational measurement. 4th ed. New York: American Council on Education and Macmillan, 1989:13-104.
20. Downing S, Yudkowsky R. Assessment in Health Professions Education. New York: NY Routledge, 2009.
21. Borgersen NJ, Naur TMH, Sørensen SMD, et al. Gathering Validity Evidence for Surgical Simulation: A Systematic Review. Ann Surg 2018;267:1063-8. [Crossref] [PubMed]
22. Clementsen PF, Nayahangan LJ, Konge L. Bronchoscopy. Basisbook: Copenhagen Academy for Medical Education and Simulation. 2015. Available online: https://www.nvalt.nl/aios/bronchoscopie-cursus/_/Studiemateriaal/handbook%20bronchoscopy%20CAMES_EvdH.pdf
23. Svd Walt, Colbert SC, Varoquaux G. The NumPy Array: A Structure for Efficient Numerical Computation. Comput Sci Eng 2011;13:22-30. [Crossref]
24. Jones E, Oliphant T, Peterson P. SciPy: Open Source Scientific Tools for Python. 2001. Available online: http://www.scipy.org/
25. Hunter JD. Matplotlib: A 2D Graphics Environment. Comput Sci Eng 2007;9:90-5. [Crossref]
26. Kennedy CC, Maldonado F, Cook DA. Simulation-based bronchoscopy training: systematic review and meta-analysis. Chest 2013;144:183-92. [Crossref] [PubMed]
27. Naur TMH, Konge L, Nayahangan LJ, et al. Training and certification in endobronchial ultrasound-guided transbronchial needle aspiration. J Thorac Dis 2017;9:2118-23. [Crossref] [PubMed]
28. Gaisl T, Bratton DJ, Heuss LT, et al. Sedation during bronchoscopy: data from a nationwide sedation and monitoring survey. BMC Pulm Med 2016;16:113. [Crossref] [PubMed]
29. Sealy WC, Connally SR, Dalton ML. Naming the bronchopulmonary segments and the development of pulmonary surgery. Ann Thorac Surg 1993;55:184-8. [Crossref] [PubMed]
30. Sanz-Santos J, Serra P, Torky M, et al. Systematic Compared With Targeted Staging With Endobronchial Ultrasound in Patients With Lung Cancer. Ann Thorac Surg 2018;106:398-403. [Crossref] [PubMed]
31. Colice GL. Detecting lung cancer as a cause of hemoptysis in patients with a normal chest radiograph: bronchoscopy vs. CT. Chest 1997;111:877-84. [Crossref] [PubMed]
32. Kaminski MF, Wieszczy P, Rupinski M, et al. Increased Rate of Adenoma Detection Associates With Reduced Risk of Colorectal Cancer and Death. Gastroenterology 2017;153:98-105. [Crossref] [PubMed]
33. Corley DA, Jensen CD, Marks AR, et al. Adenoma detection rate and risk of colorectal cancer and death. N Engl J Med 2014;370:1298-306. [Crossref] [PubMed]
34. Leiten EO, Martinsen EM, Bakke PS, et al. Complications and discomfort of bronchoscopy: a systematic review. Eur Clin Respir J 2016;3:33324. [Crossref] [PubMed]
35. Roth K, Hardie JA, Andreassen AH, et al. Cost minimization analysis for combinations of sampling techniques in bronchoscopy of endobronchial lesions. Respir Med 2009;103:888-94. [Crossref] [PubMed]
36. Cook DA, Hatala R. Validation of educational assessments: a primer for simulation and beyond. Adv Simul (Lond) 2016;1:31. [Crossref] [PubMed]
37. Steinfort DP, Yong YH, Byrne T, et al. Assessment of Bronchoscopic Dexterity and Procedural Competency in a Low-fidelity Simulation Model. J Bronchology Interv Pulmonol 2018;25:198-203. [Crossref] [PubMed]
38. Fielding DI, Maldonado F, Murgu S. Achieving competency in bronchoscopy: challenges and opportunities. Respirology 2014;19:472-82. [Crossref] [PubMed]
39. Vilmann AS, Norsk D, Svendsen MBS, et al. Computerized feedback during colonoscopy training leads to improved performance: a randomized trial. Gastrointest Endosc 2018;88:869-76. [Crossref] [PubMed]
40. Pietersen PI, Laursen CB, Petersen RH, et al. Structured and evidence-based training of technical skills in respiratory medicine and thoracic surgery. J Thoracic Dis 2019. [Crossref]
41. Cook DA, Hatala R. Got power? A systematic review of sample size adequacy in health professions education research. Adv Health Sci Educ Theory Pract 2015;20:73-83. [Crossref] [PubMed]