Ysabella Esteban, MD MSc RhMSUS*¹; Jackeline Rodriguez-Smith, MD MSc RhMSUS²; Marie Tominna, DO³; Amy Cassedy, PhD⁴; Arthur B. Meyers, MD⁵; Michael Henrickson, MD MPH RhMSUS²; Tracy V. Ting, MD MSc RhMSUS²; Patricia Vega-Fernandez, MD MSc RhMSUS²

(1) Division of Rheumatology, Nationwide Children’s Hospital, Columbus, Ohio, USA

(2) Department of Pediatrics, University of Cincinnati, Division of Rheumatology, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA

(3) Beaumont Hospital, Royal Oak, Michigan, USA

(4) Department of Pediatrics, University of Cincinnati, Division of Biostatistics and Epidemiology, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA

(5) Department of Radiology, University of Cincinnati, Division of Radiology and Medical Imaging, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA

---

Download article PDF – POCUS Journal 2023; 8(2):142-145.

DOI: https://doi.org/10.24908/pocus.v8i2.16229

---

Abstract

Point of care pediatric musculoskeletal POCUS scanning and scoring protocols for childhood arthritis have emerged in recent years. However, pediatric musculoskeletal POCUS curricula in rheumatology fellowship programs are limited due to availability of trained faculty and resources. This proof-of-concept study investigated the effectiveness of educational methods for a pediatric musculoskeletal POCUS scoring protocol among fellows and physicians of differing subspecialties. Educational methods assessed included recorded videos and virtual review sessions. Effectiveness was assessed by calculating interrater reliability for the musculoskeletal POCUS scoring systems using the intra-class correlation coefficient (ICC). Following training sessions, participants then underwent scoring exercise(s) until the goal of an excellent ICC ≥ 0.75 was reached. Four participants completed two rounds of virtual education, review, and scoring sessions. Excellent interrater reliability was achieved for most views. This proof-of-concept study demonstrated virtual education covering advanced concepts of pediatric musculoskeletal POCUS provides a knowledge base for physicians from different subspecialties and various experience.

Introduction

In recent years, point of care ultrasound (POCUS) is becoming more prevalent in medical education and in patient care [1]. After the onset of the COVID-19 pandemic, there are examples of successful virtual adaptation of POCUS in medical student and radiological resident education [2,3]. From a practical standpoint, the use of musculoskeletal POCUS is becoming more widespread and ubiquitous within medicine, specifically within the field of rheumatology. In addition, a previous study demonstrated successful teaching of sonographic findings of inflammatory arthritis to medical students, based on utilization of a multiple-choice exam, practical skills assessment, and overall score determined by the educators [4].

Advancing a POCUS skillset relies mostly on self-directed education and/or enrollment in specific curricula [5-8]. In recent years, musculoskeletal POCUS training has been integrated into many adult rheumatology fellowship programs, with over 100 programs in the United States providing education on this imaging modality. However, the availability of standardized curricula varies between fellowship programs [5].  In an effort to support musculoskeletal POCUS  training, the American College of Rheumatology (ACR) introduced a musculoskeletal POCUS educational curriculum for adult patients, with related appendices for pediatric patients [6]. In order to quantify the availability of pediatric musculoskeletal POCUS curricula, one recent cross-sectional study showed that 20 of the 36 ACGME-accredited pediatric rheumatology fellowship programs in the United States and Canada offer some level of musculoskeletal POCUS training [8]. However, the dissemination of pediatric musculoskeletal POCUS curricula can be limited due to the availability of trained faculty, constraints on time and resources to perform these studies, and decreased awareness of available curricula [8].

Within the context of clinical applications of pediatric musculoskeletal POCUS, there exists definitions for pediatric sonographic findings of healthy joints as well as sonographic findings of inflammatory arthritis [9-11]. Recent studies within the field of pediatric rheumatology and musculoskeletal POCUS have focused on the development of pediatric-specific scanning protocols for the assessment of synovitis in juvenile idiopathic arthritis(JIA) and corresponding scoring systems, including those proposed by the Outcome Measures in Rheumatology (OMERACT) and Childhood Arthritis and Rheumatology Research Alliance (CARRA) ultrasound work groups [12-17]. However, the establishment of curricula for advanced concepts including consensus-based scoring systems for pediatric synovitis within the context of rheumatic disease has not been well explored. 

This proof-of-concept study aims to explore this knowledge gap, and to provide further insight into educational opportunities within the field. Here, we provided an initial assessment of 1) the educational methods used to teach and 2) the reproducibility of a pediatric-specific musculoskeletal POCUS scoring system among fellows-in-training and a pediatric radiology attending.

Materials and Methods

This was a proof-of-concept study, and participants included those with various degrees of training and experience in musculoskeletal POCUS (from 1 year to >10 years), including three fellows (two pediatric rheumatology fellows, one pediatric radiology fellow) and one radiology attending with expertise in musculoskeletal imaging and 10 years of post-training experience. The education provided for the group included two modalities: 1) recorded educational videos and 2) follow-up review sessions via virtual meet space.  These videos and review sessions were guided by an expert pediatric rheumatologist and ultrasonographer (PVF, 10 years of musculoskeletal POCUS experience). The methods are outlined in Figure 1.

The recorded educational videos were led by the musculoskeletal POCUS expert (PVF) and facilitator (YE), who helped guide discussion points during the recording and review sessions. The videos reviewed the sonographic anatomy as well as pathologic findings for B-mode and PD-mode of the elbow, wrist, finger, knee, and ankle joints. During these videos, the scoring system for each joint was also taught, using static images obtained from patients 2-17 years of age. Briefly, the definitions of pathology used by the scoring systems followed the recommendations proposed by the OMERACT Pediatric Ultrasound Group [7]. The scoring systems used a semi-quantitative scale (0-normal to 3-severe) to categorize the findings of synovitis in B-mode and power Doppler (PD) images of the aforementioned joints (Table 1) [13, 15-17].

Table 1. Anatomical Views for Elbow, Wrist, Finger, Knee, and Ankle Joints [13,15-17]

Joint	Anatomical Views (B-mode and PD-mode)
Elbow	Anterior humeroulnar joint recess
	Anterior humeroradial joint recess
	Posterior humeroulnar joint recess
Wrist	Radiocarpal and midcarpal joint recess – midline
	Radiocarpal and midcarpal joint recess – ulnar
	Distal radioulnar joint recess
	Extensor tendons
Finger	MCP dorsal joint recess in longitudinal
	MCP volar joint recess in longitudinal
	PIP volar joint recess in longitudinal
	PIP dorsal joint recess in longitudinal
Knee	Suprapatellar joint recess
	Medial parapatellar joint recess
	Lateral parapatellar joint recess
Ankle	Anterior tibiotalar joint recess
	Talonavicular joint recess
	Anterior subtalar joint recess (from medial aspect)
	Posterior subtalar joint recess (from lateral aspect)
	Anterior, medial, and lateral tendons
Anatomical views for the scoring protocol as determined by previous studies. [13,15-17]

The duration of each video ranged from 24 to 46 minutes in length. In addition, electronic handouts were provided to participants, which detailed the anatomy, pathology, and scoring system of each joint.  Participants were encouraged to complete the videos prior to the virtual review sessions.

The two review sessions then took place via a virtual meet space, led by YE and PVF, which lasted approximately 1.5 – 2 hours total. During this review, participants asked questions regarding the anatomy, pathology, and scoring system. Then, participants were able to practice the scoring system, and real-time feedback and discussion were provided based on the score each participant chose. Following the review sessions, participants had more opportunities to individually review the material on their own time, and one-on-one question and answer sessions regarding the scoring system via phone or email were provided as needed. 

All participants then underwent a scoring exercise which was conducted electronically, using B- and PD-mode static images from a previously existing pediatric musculoskeletal POCUS image bank obtained from children 2 – 17 years and representing the spectrum of pathology described by the scoring systems for each joint. The images were selected based on the quality of the image, specifically considering clear delineation of bony/cartilage landmarks and the least amount of anisotropy or artifact. The images selected for the review sessions were different than those selected for the scoring exercises. No identifiable patient information was presented with the images save for the age of the subject. The images were scored independently by each participant; statistical analysis followed each exercise. Based on the results from the first scoring exercise, participants underwent a second review session with emphasis on the areas that did not reach excellent reliability as defined by an intra-class correlation coefficient (ICC) ≤0.75 [18,19].  During this live virtual review session, additional sample images for scoring practice were provided, with participants providing scores followed by discussion as to why specific scores were chosen. Then, a second scoring exercise was performed which focused on those particular areas utilizing B- and PD-mode static images.

The interrater reliability was estimated using the two-way single score ICC, along with the 95% confidence intervals (CI). Excellent ICC was defined to be between 0.75 – 1.00, good 0.60 – 0.74, fair 0.40 – 0.59, and poor <0.40 [18,19]. ICC estimates and their 95% confidence intervals were calculated using SAS v9.4©, Cary, NC.

This study was submitted to the authors’ Institutional Review Board (Cincinnati Children’s Hospital Medical Center) and received exemption status: IRB# 2020-0700.

Results

A total of four raters participated in this proof-of-concept study. A total of two rounds of educational and scoring sessions were completed. The results from the first educational and scoring exercise are listed in Table 2. For this scoring exercise, a total number of 588 images representing both normal sonographic anatomy as well as varying degrees of pathology related to the scoring system (including 352 B-mode and 236 PD-mode still images) were scored. Based on these results, excellent interrater reliability (ICC ≥ 0.75) was achieved for most of the B-mode and PD mode views of the elbow, wrist, finger, knee and ankle as delineated in Table 2.

Table 2. Interrater Reliability of Pediatric-Specific musculoskeletal POCUS Scoring System– Exercise 1 and 2.

Joint View		Exercise 1		Exercise 2
		B-mode	Power Doppler-Mode	B-mode	Power Doppler-Mode
	ICC (95% CI)	ICC (95% CI)	ICC (95% CI)	ICC (95% CI)
Elbow	Anterior humeroulnar and humeroradial joint recesses	0.93 (0.89 – 0.95)	0.88 (0.77 – 0.94)	‡	‡
	Posterior humeroulnar joint recess	0.93 (0.89 – 0.95)	0.77 (0.61 – 0.87)	‡	‡
Wrist	Radiocarpal and midcarpal joint — midline	0.86 (0.80 – 0.90)	0.96 (0.94 – 0.97)	‡	‡
	Radiocarpal and midcarpal joint — ulnar	0.80 (0.65 – 0.89)	0.90 (0.71 – 0.97)	‡	‡
	Distal radioulnar joint recess	0.87 (0.78-0.92)	0.93 (0.94 – 0.97)	‡	‡
	Tendons — extensor	0.74* (0.58 – 0.84)	0.66* (0.35 – 0.84)	0.5* (0.19 – 0.72)	0.52* (0.19 – 0.74)
Finger	MCP dorsal joint recess in longitudinal	0.94 (0.90 – 0.96)	0.98 (0.97 – 0.99)	‡	‡
	MCP volar joint recess in longitudinal	0.82 (0.73 – 0.88)	0.89 (0.83 – 0.93)	‡	0.95 (0.93-0.97)
	PIP volar joint recess in longitudinal	0.91 (0.87 – 0.94)	0.63* (0.48 – 0.74)
	PIP dorsal joint recess in longitudinal	0.94 (0.88-0.97)	0.84 (0.70 – 0.92)	‡	‡
Knee	Suprapatellar joint recess	0.93 (0.89 – 0.95)	0.88 (0.82 – 0.92)	‡	‡
	Medial parapatellar joint recess	0.92 (0.88 – 0.95)	0.90 (0.85 – 0.93)	‡	‡
	Lateral parapatellar joint recess	0.94 (0.91 – 0.96)	0.90 (0.84 – 0.93)	‡	‡
Ankle	Anterior tibiotalar joint recess	0.92 (0.88 – 0.95)	0.87 (0.75 – 0.93)	‡	‡
	Talonavicular joint recess	0.66* (0.41 – 0.82)	0.91 (0.8 – 0.96)	0.84 (0.71-0.91)	‡
	Anterior subtalar joint recess (medial aspect)	0.77 (0.63 – 0.86)	0.83 (0.59 – 0.94)	‡	‡
	Posterior subtalar joint recess (lateral aspect)	0.73* (0.57 – 0.84)	†	0.90 (0.85-0.96)	0.91 (0.83-0.95)
	Tendons—Medial, lateral and anterior	0.53* (0.26 – 0.73)	0.55* (0.1 – 0.81)	0.82 (0.73-0.88)	0.95 (0.91-0.97)
*Desired reliability of ≥ 0.75 not obtained. †Insufficient data to assess. ‡Not applicable Interrater reliability was estimated using ICC. Excellent ICC was defined to be between 0.75-1.00, good 0.60-0.74, fair 0.40 – 0.59, and poor < 0.40. [18, 19air 0.40 – 0.59, and poor < 0.40. [18, 19]

For the remaining views in which excellent reliability was not reached, a second round of education and scoring exercises was performed, with the results listed in Table 2. For this scoring exercise, a total number of 234 B-mode and PD-mode still images were scored. Desired excellent interrater reliability was obtained for the remaining B-mode and PD-mode views with the exception of those delineating the tendons of the wrists. For this area, a fair reliability was obtained.

Feedback regarding the educational sessions was obtained in real-time from participants. Positive feedback included the flexibility of the curriculum given the video format with electronic handouts, as well as the structure of educational sessions. Suggestions for improvement included incorporating more examples of image scoring for both B-mode and PD-mode views during the interactive virtual review sessions.

Discussion

This proof-of-concept study explored the ability of pediatric physicians with varying degrees of expertise to learn and demonstrate a novel pediatric musculoskeletal POCUS scoring system. From an educational standpoint, the information provided to participants built upon basic musculoskeletal anatomy, and participants were able to gain or fine tune knowledge regarding joint pathology as visualized on B- and PD-modes. Knowledge of the scoring protocol was also imparted. The format of the educational sessions, which included video lectures encompassing the related sonographic anatomy, pathology, and scoring system of each joint, allowed participants to learn independently. In addition, during the virtual, interactive review sessions the participants were able to practice scoring, concept review, and discussion of the scoring systems in real-time. Immediate, informal feedback at the end of each session allowed incorporation of suggestions into subsequent educational sessions.

Use of musculoskeletal POCUS facilitates real-time diagnosis, intervention, and monitoring by the clinician ultrasonographer [20, 21]. In addition, the evolution of technology has provided the medical community with multiple portable options, which has allowed pediatric providers increased access to this mode of imaging [21]. In terms of musculoskeletal POCUS, there are multiple routes of education including rheumatology fellowship training, mentored training that is either structured or unstructured, or self-directed and non-mentored training [20]. For pediatric musculoskeletal POCUS, basic educational opportunities include the training program offered by the Ultrasound School of North American Rheumatologists (USSONAR), workshops and courses previously offered by the American College of Rheumatology (ACR) or Childhood Arthritis Rheumatology Research Alliance (CARRA) in the pre-COVID era, as well as self-directed online resources such as Ped-MUS [6; 22-23]. Twenty pediatric rheumatology fellowship programs also offer musculoskeletal POCUS training [8].

To our knowledge, this is the first effort to investigate a virtual educational format to train pediatricians in the assessment of musculoskeletal POCUS studies as they pertain to JIA using advanced concepts. While it is possible that the size and flexibility of this participant group facilitated this investigation, it is feasible that these educational sessions can be replicated among larger participant groups in the future. The pandemic has accelerated the era of virtual meetings, which some have found to be beneficial in radiological education [3], and our demonstration of teachability and reproducibility of this scoring system via a virtual meet space is likewise encouraging. The era of COVID-19 proved to be a new obstacle in live in-person musculoskeletal POCUS training but also provided the impetus for developing virtual, remote learning options, which will likely continue to be an integral part of this training [24]. This also highlights the potential to provide virtual education for clinicians in underserved areas, including global outreach programs.

The limitations of this study included the reduced number of participants involved. A lower ICC may reflect not only the degree of interrater agreement, but also a smaller number of raters or the diverse experience of the raters. In our particular study, we had four participants, which could have contributed to this finding. Competency assessment of the proposed online curriculum was not pursued mainly given the size of the team evaluated. In addition, solicited feedback was not anonymous given the size of the group.

The one area that only attained fair interrater reliability after the second round of training in this study was B-mode and PD-mode views for the tendons of the wrists.  One possible explanation for this is that scoring involved static images, as opposed to cine clips which can better distinguish hypoechoic muscle surrounding a tendon at the myotendinous junction from fluid. In addition, the presence of the retinaculum adjacent to the dorsal wrist tendons can exhibit a hypoechoic appearance, which without dynamic study to assess for anisotropy or compressibility, could potentially affect interpretation by mimicking tenosynovitis [25]. 

Finally, as this was a proof-of-concept study, we did not perform longitudinal follow-up of participant knowledge recall and therefore cannot comment on retention. Future qualitative studies should investigate areas of optimization in education of this pediatric musculoskeletal scoring system among varying levels of expertise, including pre- and post-curriculum knowledge assessment, long-term follow-up, and where applicable, impact on clinical practice.

Conclusion

Our study showed that virtual educational exercises covering normal musculoskeletal POCUS anatomy and pathologic variations related to pediatric arthritis can provide a knowledge base to physicians from different subspecialties at varying points in their training and careers. Further qualitative studies should be performed to assess areas of optimization in education of pediatric musculoskeletal POCUS and pediatric-specific musculoskeletal POCUS scoring systems. Finally, our study demonstrated that a virtual platform for pediatric musculoskeletal POCUS curricula is a feasible option, and could be implemented for pediatric rheumatology fellowship programs that do not have ready access to trained faculty or resources.

Disclosures

Dr. Vega-Fernandez’s work was supported by the Center for Clinical & Translational Science & Training (CCTST) at the University of Cincinnati funded by the National Institutes of Health (NIH) Clinical and Translational Science Award (CTSA) program, grant 2UL1TR001425-05A1 and KL2 (2KL2TR001426-05A). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

References:

1.  Tarique U, Tang B, Singh M, Kulasegaram KM, Ailon J. Ultrasound Curricula in Undergraduate Medical Education: A Scoping Review. J Ultrasound Med. 2018 Jan;37(1):69-82. doi: 10.1002/jum.14333. Epub 2017 Jul 27. PMID: 28748549. 

2.  Zavitz J, Sarwal A, Schoeneck J, Glass C, Hays B, Shen E, Bryant C, Gupta K. Virtual multispecialty point-of-care ultrasound rotation for fourth-year medical students during COVID-19: Innovative teaching techniques improve ultrasound knowledge and image interpretation. AEM Educ Train. 2021 Aug 1;5(4):e10632. doi: 10.1002/aet2.10632. PMID: 34179677; PMCID: PMC8209882. 

3.  Larocque N, Shenoy-Bhangle A, Brook A, Eisenberg R, Chang YM, Mehta P. Resident Experiences With Virtual Radiology Learning During the COVID-19 Pandemic. Acad Radiol. 2021 May;28(5):704-710. doi: 10.1016/j.acra.2021.02.006. Epub 2021 Feb 15. PMID: 33640229; PMCID: PMC7883720. 

4.  Wright SA, Bell AL. Enhancement of undergraduate rheumatology teaching through the use of musculoskeletal ultrasound. Rheumatology (Oxford). 2008 Oct;47(10):1564-6. doi: 10.1093/rheumatology/ken324. Epub 2008 Aug 13. PMID: 18703529. 

5.  Torralba KD, Cannella AC, Kissin EY, Bolster MB, Salto LM, Higgs J, Samuels J, Nishio MJ, Kaeley GS, Evangelisto A, De Marco P, Kohler MJ. Musculoskeletal Ultrasound Instruction in Adult Rheumatology Fellowship Programs. Arthritis Care Res (Hoboken). 2020 Jun;72(6):859-870. doi: 10.1002/acr.23336. Epub 2020 May 25. PMID: 28777891.

6.  Cannella AC, Kissin EY, Torralba KD, Lin C, Higgs JB, Bolster M, et al. 2019 Rheumatologic Ultrasound (RhUS) Curriculum Supplement. American College of Rheumatology; 2019. https://assets.contentstack.io/v3/assets/bltee37abb6b278ab2c/blt457aaaf0d1d7e7a0/rheumatologic-ultrasound-curriculum-supplement.pdf. Accessed April 11, 2022.

7.  Widener BB, Cannella A, Martirossian L, Kissin EY. Modern Landscapes and Strategies for Learning Ultrasound in Rheumatology. Rheum Dis Clin North Am. 2020 Feb;46(1):61-71. doi: 10.1016/j.rdc.2019.09.002. PMID: 31757287.

8.  Lin C, Oberle E, Curran M, Roth J, Vega-Fernandez P, DeRanieri D, Ting T, Benham H, Salto L, Torralba K. Musculoskeletal Ultrasound Instruction in US and Canadian Pediatric Rheumatology Fellowship Programs. Poster presented at: ACR Convergence 2021; 2021 November 8; Virtual. https://acrabstracts.org/abstract/musculoskeletal-ultrasound-instruction-in-us-and-canadian-pediatric-rheumatology-fellowship-programs/. Accessed April 11, 2022.

9.  Roth J, Ravagnani V, Backhaus M, Balint P, Bruns A, Bruyn GA, Collado P, De la Cruz L, Guillaume-Czitrom S, Herlin T, Hernandez C, Iagnocco A, Jousse-Joulin S, Lanni S, Lilleby V, Malattia C, Magni-Manzoni S, Modesto C, Rodriguez A, Nieto JC, Ohrndorf S, Rossi-Semerano L, Selvaag AM, Swen N, Ting TV, Tzaribachev N, Vega-Fernandez P, Vojinovic J, Windschall D, D’Agostino MA, Naredo E; OMERACT Ultrasound Group. Preliminary Definitions for the Sonographic Features of Synovitis in Children. Arthritis Care Res (Hoboken). 2017 Aug;69(8):1217-1223. doi: 10.1002/acr.23130. PMID: 27748074.

10.  Roth J, Jousse-Joulin S, Magni-Manzoni S, Rodriguez A, Tzaribachev N, Iagnocco A, Naredo E, D’Agostino MA, Collado P; Outcome Measures in Rheumatology Ultrasound Group. Definitions for the sonographic features of joints in healthy children. Arthritis Care Res (Hoboken). 2015 Jan;67(1):136-42. doi: 10.1002/acr.22410. PMID: 25047864.

11.  Collado P, Windschall D, Vojinovic J, Magni-Manzoni S, Balint P, Bruyn GAW, Hernandez-Diaz C, Nieto JC, Ravagnani V, Tzaribachev N, Iagnocco A, D’Agostino MA, Naredo E; OMERACT ultrasound subtask force on pediatric. Amendment of the OMERACT ultrasound definitions of joints’ features in healthy children when using the DOPPLER technique. Pediatr Rheumatol Online J. 2018 Apr 10;16(1):23. doi: 10.1186/s12969-018-0240-2. PMID: 29631610; PMCID: PMC5892017.

12.  Sande NK, Bøyesen P, Aga AB, Hammer HB, Flatø B, Roth J, Lilleby V. Development and reliability of a novel ultrasonographic joint-specific scoring system for synovitis with reference atlas for patients with juvenile idiopathic arthritis. RMD Open. 2021 Apr;7(2):e001581. doi: 10.1136/rmdopen-2021-001581. PMID: 33883255; PMCID: PMC8061832.

13.  Ting TV, Vega-Fernandez P, Oberle EJ, De Ranieri D, Bukulmez H, Lin C, Moser D, Barrowman NJ, Zhao Y, Benham HM, Tasan L, Thatayatikom A, Roth J; Childhood Arthritis and Rheumatology Research Alliance Juvenile Idiopathic Arthritis Ultrasound Workgroup. Novel Ultrasound Image Acquisition Protocol and Scoring System for the Pediatric Knee. Arthritis Care Res (Hoboken). 2019 Jul;71(7):977-985. doi: 10.1002/acr.23746. Epub 2019 Jun 12. PMID: 30192069.

14.  Rossi-Semerano L, Breton S, Semerano L, Boubaya M, Ohanyan H, Bossert M, Boiu S, Chatelus E, Durand G, Jean S, Goumy L, Mathiot A, Mouterde G, Nugues F, Ould Hennia A, Rey B, Von Scheven A, Sparsa L, Devauchelle-Pensec V, Jousse-Joulin S. Application of the OMERACT synovitis ultrasound scoring system in juvenile idiopathic arthritis: a multicenter reliability exercise. Rheumatology (Oxford). 2021 Aug 2;60(8):3579-3587. doi: 10.1093/rheumatology/keaa804. PMID: 33374013.

15.  Vega-Fernandez P, Ting TV, Oberle EJ, McCracken C, Figueroa J, Altaye M, Cassedy A, Kaeley GS, Roth J; CARRA Musculoskeletal Ultrasound Workgroup. The MUSICAL pediatric ultrasound examination – a comprehensive, reliable, time efficient assessment of synovitis. Arthritis Care Res (Hoboken). 2021 Jul 30:10.1002/acr.24759. doi: 10.1002/acr.24759. Epub ahead of print. PMID: 34328679; PMCID: PMC8800943.

16.  Vega-Fernandez P, De Ranieri D, Oberle E, Clark M, Bukulmez H, Lin C, Shenoi S, Thatayatikom A, Woolnough L, Benham H, Brunner E, Henrickson M, Pratt LR, Proulx-Gauthier JP, Janow G, Cassedy A, Ting TV, Roth J; CARRA JIA Ultrasound Workgroup. Comprehensive and reliable sonographic assessment and scoring system for inflammatory lesions of the pediatric ankle. Rheumatology (Oxford). 2022 Oct 29:keac622. doi: 10.1093/rheumatology/keac622. Epub ahead of print. PMID: 36308429.

17.  Vega-Fernandez P, Esteban Y, Oberle E, Proulx-Gauthier JP, Clark M, Shenoi S, Thatayatikom A, Benham H, Brunner EJ, Woolnough L, Henrickson M, Pratt LR, De Ranieri D, Hoffmann S, Janow G, Bukulmez H, Altaye M, Cassedy A, Ting TV, Roth J; CARRA JIA Ultrasound Workgroup. Reliability of the Pediatric Specific Musculoskeletal Ultrasound Scoring Systems for the Elbow, Wrist, and Finger Joints. J Rheumatol. 2022 Oct 1:jrheum.220382. doi: 10.3899/jrheum.220382. Epub ahead of print. PMID: 36182106.

18.  Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment. 1994;6(4):284–90.   

19.  Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34. doi: 10.20982/tqmp.08.1.p023. PMID: 22833776; PMCID: PMC3402032. 

20.  Widener BB, Cannella A, Martirossian L, Kissin EY. Modern Landscapes and Strategies for Learning Ultrasound in Rheumatology. Rheum Dis Clin North Am. 2020 Feb;46(1):61-71. doi: 10.1016/j.rdc.2019.09.002. PMID: 31757287.

21.  Conlon TW, Nishisaki A, Singh Y, Bhombal S, De Luca D, Kessler DO, Su ER, Chen AE, Fraga MV. Moving Beyond the Stethoscope: Diagnostic Point-of-Care Ultrasound in Pediatric Practice. Pediatrics. 2019 Oct;144(4):e20191402. doi: 10.1542/peds.2019-1402. Epub 2019 Sep 3. PMID: 31481415.

22.  Childhood Arthritis and Rheumatology Research Alliance. (2019). Childhood Arthritis and Rheumatology Research Alliance 2019 Annual Meeting. CARRA – Childhood Arthritis and Rheumatology Research Alliance. Retrieved June 8, 2022, from https://carragroup.org/meetings-events/past-carra-annual-scientific-meetings/2019-annual-meeting

23.  Roth J, Malattia C, Windschall D. Abbvie – ped-mus [Internet]. Ped-MUS. [cited 2022Sep8]. Available from: https://www.ped-mus.com/

24.  Di Matteo A, Mankia K, Filippucci E, Grassi W, Rowbotham E, Wakefield RJ. Facing the challenges of running a rheumatology-based ultrasound service in the COVID-19 era. Rheumatology (Oxford). 2021 Mar 2;60(3):1013-1015. doi: 10.1093/rheumatology/keaa844. PMID: 33331890; PMCID: PMC7798732.

25.  Chiavaras MM, Jacobson JA, Yablon CM, Brigido MK, Girish G. Pitfalls in wrist and hand ultrasound. AJR Am J Roentgenol. 2014 Sep;203(3):531-40. doi: 10.2214/AJR.14.12711. PMID: 25148155.

Original version published by POCUS Journal. Indexed on PubMed Central.