J Korean Soc Geriatr Neurosurg > Volume 18(2); 2022 > Article
Na, Kim, Chung, Han, Yun, Yang, Choi, Kim, Cho, and Jeon: Classification of carotid plaque vulnerability by neurosurgical residents using ultrasonography in the clinical field



We aimed to evaluate the accuracy of the classification of carotid plaque vulnerability (unstable vs. stable plaques) by neurosurgical residents based on carotid ultrasonography (US) images.


A total of 405 subjects with 995 images were included in the study. Using a neuroradiologist’s decision as the reference value, the classification results of five reviewers were analyzed. The sensitivity, specificity, and overall accuracy were estimated. Then, a pairwise comparison of the receiver operating characteristic (ROC) curve and precision-recall curve was performed to compare the reviewers’ classification accuracy.


The mean age of the subjects was 70.5 years (range, 44-91 years) and 223 (55.1%) were female. The number of unstable and stable plaques was 236 (24.7%) and 749 (75.3%), respectively. The best-balanced classification performance of plaque vulnerability was a sensitivity of 83.7% (95% confidence interval [CI], 78.5%-88.1%), specificity of 69.0% (95% CI, 65.6%-72.3%), and overall accuracy of 72.7% (95% CI, 69.8%-75.4%). The best ROC performance was an area under the curve (AUC) of 0.583 (95% CI, 0.552-0.614). The precision-recall curve also showed low classification accuracy among the reviewers (AUC difference: 0.028; 95% bootstrap CI, 0.007-0.048).


The classification accuracy of neurosurgical residents to discriminate plaque vulnerability seen on carotid US images was low in a real-world clinical setting. Thus, it is necessary to develop systems that help to educate and automatically interpret plaque stability.


Carotid plaque refers to focal wall hypertrophy that is at least 50% greater than that of the surrounding carotid wall. The prevalence of carotid plaques is approximately 40.3% and they are found more frequently in males than in females [1]. Patients with carotid plaques are more likely to have metabolic disorders with high total cholesterol and low-density lipoprotein-cholesterol levels [1]. Because carotid plaques are formed in the late stages of atherosclerosis, they are thought to be a risk factor for cerebrovascular disease including stroke [2,3]. Accordingly, screening for carotid plaques and identifying carotid plaque morphology is conducted in the clinical field. Among various radiologic tests, carotid ultrasonography (US) is performed widely in private hospitals, as well as general and university hospitals due to its relatively low cost and ease of implementation compared to magnetic resonance imaging (MRI) or computed tomography (CT) [4-8].
The variables in interpreting carotid US images are intima-media thickness (IMT), the degree of stenosis, and the presence of a carotid plaque along with an assessment of its vulnerability to rupture. Among these variables, the accurate classification of a carotid plaque vulnerable to rupture and identifying one that is more likely to cause a stroke is difficult for general practitioners who are non-radiologists. Most neurosurgeons also have difficulties in distinguishing unstable plaques from stable carotid plaques despite the increasing number of carotid stenting procedures in recent years. In addition, neurosurgical residents are not well trained in carotid US interpretation. Few studies have addressed the classification accuracy of plaque vulnerability by non-radiologists in the domestic clinical field. Here, we investigated the classification accuracy of carotid plaque vulnerability (unstable vs. stable plaque) by neurosurgical residents to obtain objective data on the interpretation accuracy in the actual medical environment.

Material and Method

Study population

The derivation cohort was obtained from the regional stroke database between May 2021 and May 2022. We selected subjects from this database who underwent carotid US for stroke or medical evaluations. Carotid US was conducted in the B mode using a 5 to 18 MHz US probe (APLIO i800; Canon Medical Systems Corporation, Tochigi, Japan) and an 11 MHz US probe (LOGIQ E9; GE Healthcare, Wauwatosa, WI, USA). The procedure was performed by a neuroradiologist with more than 20 years of experience according to the American Society of Echocardiography guidelines [9]. Patients were positioned supine on the scan bed with their heads resting flat to facilitate high-quality and reproducible images. To relieve patients, towels or pillows were used to support knees and neck. The neck was slightly extended and rotated in a direction opposite to the probe. During the scan, the sonographer adjusted the neck to optimize the images, especially in the anterior planes. Carotid US was performed by quantifying the probe angle using an external landmark such as the Meijer arc.
Carotid plaque vulnerability was classified into two groups, stable and unstable plaques (Fig. 1). Unstable plaques referred to vulnerable plaques, which are prone to rupture or thrombosis with the following features using gray-scale imaging alone: (1) an irregular plaque surface, (2) plaque with ulceration, (3) heterogeneous echogenicity within the plaque, and (4) a non-echogenic intraplaque or central core, indicating hemorrhage or a lipid-rich necrotic core covered by a thin fibrous cap [10]. Plaque stability was determined by an experienced neuroradiologist. Four neurosurgical residents, one intern, and one neuroradiologist participated in interpreting the images. All reviewers evaluated the images independently followed by a comparison of stable and unstable carotid plaques. This study was approved by the Institutional Review Board (IRB) of the Hallym University Chuncheon Sacred Heart Hospital (IRB No. 2018-11-008-004 and 2019-06-006-008) and informed consent was waived due to the retrospective nature of the study.

Statistical analysis

Discrete variables are described as the numbers of subjects with percentages. Continuous variables are described as the mean±standard deviation. Two-by-two tables were generated to calculate sensitivity, specificity, and overall accuracy [11]. Pairwise comparison of a receiver operating characteristic (ROC) curve was generated to compare plaque vulnerability classifications among the reviewers [12]. Because there was an imbalance between the number of unstable and stable plaques, a precision-recall curve was generated, targeting the reviewers who showed the most significant difference in classification accuracy. A P-value of <0.05 was considered statistically significant. The analyses were conducted using IBM SPSS ver. 19.0 (IBM Corp., Armonk, NY, USA) and MedCalc (www.Medcalc.org).


A total of 405 patients with 995 carotid US images (axial image, n=297; and longitudinal image, n=698) were included in the analysis. The mean patient age was 70.5 years (range, 44-91 years) and 223 (55.1%) were female. The number of patients with hypertension and diabetes mellitus was 205 (50.6%) and 76 (18.8%), respectively. Cardiac disease, such as coronary artery disease or atrial fibrillation, was observed in 138 patients (34.1%). Twenty-four patients were smokers with an average of 28.25 pack-years. Among the patients who underwent carotid US, 61 (15.1%) were conducted for stroke evaluation and 344 (84.9%) were conducted for health examinations (Table 1).
The classification results of the five reviewers using a neuroradiologist’s decision as the reference value are presented in Tables 2 and 3. There were 236 (24.7%) unstable and 749 (75.3%) stable carotid plaques. Reviewer 1 showed the best-balanced interpretation with a sensitivity of 83.7% (78.5%-88.1%), specificity of 69.0% (65.6%-72.3%), and overall accuracy of 72.7% (69.8%-75.4%). Compared to the overall accuracy (ranging from 69.3% to 76.1%), the sensitivity results showed severe fluctuations (ranging from 39.0% to 83.7%) between the reviewers. Pairwise comparison of the ROC of reviewer 1 also showed the best performance in classifying plaque vulnerability with an area under the curve (AUC) of 0.583 (95% confidence interval [CI], 0.552-0.614) (Fig. 2A). Reviewer 4 had the lowest AUC of 0.533 (95% CI, 0.502-0.565). The AUC difference between reviewer 1 and reviewer 4 reached statistically significance (AUC difference, 0.049; 95% CI, 0.015-0.084; P=0.005). Except for these findings, there were no significant differences in AUC values between the remaining reviewers. If the data distribution is unbalanced, like in our case (unstable plaque: n=236, 24.7%; stable plaque: n=749, 75.3%), there is a possibility that the AUC value might be biased. To overcome this potential bias, we generated a precision-recall curve, especially for reviewer 1 and reviewer 4. The results also demonstrated a significant difference in plaque vulnerability classifications (AUC difference, 0.028; 95% bootstrap CI, 0.007-0.048) (Fig. 2B).


Carotid plaques form various shapes and occur in areas from the distal common carotid artery to proximal internal carotid artery [13]. It is more accurate to predict patient prognosis and choose treatment methods based on plaque vulnerability rather than only the degree of carotid stenosis. The major features used to describe carotid plaques in carotid US images include calcification, ulceration, hemorrhage, and a lipid-rich necrotic core, as well as IMT [10]. Calcification with ulceration is stimulated by prolonged shear stress and sudden arterial pressure changes on the carotid bifurcation [14]. This activates inflammation and metalloproteinase activity, causing erosion of the calcification to progress to ulceration of the surface [15]. Ulceration in the carotid plaque predisposes to thrombus formation. Plasminogen activator inhibitor-1 within plaques creates a thrombotic effect [16]. The expression of tissue factors, which have a major effect on the coagulation cascade, is determined by the extent and degree of inhibition of the tissue factors pathway within the plaque [17]. Tissue factor is localized to areas of lipid deposition and cholesterol crystals in carotid plaque [18]. Bos et al. [19] reported that intraplaque hemorrhage of the carotid artery significantly increased the risk of stroke (hazard ratio, 2.42; 95% CI, 1.30-4.50). Considering these facts, the accurate classification of carotid plaque vulnerability (unstable vs. stable plaque) is important for clinicians to determine treatment plans. Three radiologic tests, MRI, CT, and US are available to examine plaque vulnerability in the carotid artery. MRI provides excellent differentiation of soft tissue composition with high resolution, but its use can be limited due to the high medical costs and time-consuming procedures needed to obtain the images [4-6]. CT has the advantage of shorter imaging time than MRI but has the limitation that a contrast agent is required to distinguish plaque components [7,8]. In the domestic medical environment, MRI and CT is used in more than US in small and medium-sized general hospitals. And in most of these hospitals, radiologists are available to interpret plaque vulnerability. Compared to MRI and CT, carotid US is widely used in private hospitals, as well as for health examinations in general and university hospitals in Korea due to its relatively low cost, ease of using at the bedside, and non-invasiveness. Lukanova et al. [20] reported that carotid US showed high balanced accuracy for detecting unstable plaques with a sensitivity of 94% and specificity of 93% compared to multi-detector CT, with a sensitivity of 83% and specificity of 73%, and MRI with a sensitivity of 100% and specificity of 89%. However, the procedure and its interpretation may vary depending upon the examiner’s experience. In particular, considering the reality of private hospitals with many non-radiologists, it is necessary to properly classify plaque vulnerability. In this study, we evaluated the accuracy of plaque vulnerability classification, targeting neurosurgical residents. Although there were some differences in classification accuracy between the reviewers, the overall accuracy level was low (0.5<AUC≤0.7) [21]. Considering that stroke risk is largely dependent upon not only the degree of stenosis but also plaque vulnerability, accordingly, education on discriminating plaque vulnerability is necessary during resident training [22]. However, it is not easy to educate residents on carotid US interpretation given the reality of medical situations. In this case, a new computer-aided diagnostic system using machine learning can be an alternative. Tanaka et al. [23] demonstrated that a convolutional neural network (CNN) composed of VGG19 and ResNet152 showed accurate classifications of breast ultrasound images with a sensitivity of 90.9% (95% CI, 84.5%-97.3%), specificity of 87.0% (95% CI, 79.5%-94.5%), and AUC of 0.951 (95% CI, 0.916-0.987). Biswas et al. [24] proposed a two-stage artificial intelligence model to measure atherosclerotic wall thickness and total plaque burden. However, they did not distinguish plaque vulnerability. Lekadir et al. [25] also suggested a CNN for automatically characterizing plaque components, such as the lipid core, fibrous cap, and calcified tissue. However, they used only 56 cases. Accordingly, there is a need for artificial intelligence (AI) based on gray-scale carotid US images, which is often used in the field, and AI research that not only detects plaque composition but comprehensively informs on plaque vulnerability.
There were some limitations in this study. First, there were only five participants, and they were from the same hospital. Thus, a concern may arise as to whether the residents participating in the study actually represent non-radiologists working in private hospitals. Second, the correct answer was decided only by one neurointerventionist. Although he had more than 20 years of experience, selection bias may have occurred, and this should be taken into account when interpreting our results. Nevertheless, to the best of our knowledge, this was the first study to investigate the classification accuracy of plaque vulnerability by neurosurgical residents based on carotid US images in actual clinical settings.


The classification accuracy of neurosurgical residents to discriminate plaque vulnerability based on carotid US images was low. To address this problem, a system that helps to educate and comprehensively interpret carotid plaque morphology should be developed.


Conflicts of interest

No potential conflict of interest relevant to this article was reported.


This research was supported by the Research and Business Development Program through the Korea Institute for Advancement of Technology (KIAT) funded by the Ministry of Trade, Industry and Energy (MOTIE) (1415176646-P0018271).

Fig. 1.
Examples of carotid plaque vulnerability, including stable and unstable plaques. The red and blue circles indicate unstable and stable plaques, respectively.
Fig. 2.
(A) Pairwise comparison of receiver operating characteristics (ROC) curves for classifying plaque vulnerability. (B) The precision-recall curve between reviewer 1 and reviewer 4 who showed significant differences in the area under the curve (AUC) in the ROC comparisons. SE, standard error; CI, confidence interval.
Table 1.
Clinical characteristics of the enrolled patients
Variable Patients (n=405)
Female 223 (55.1)
Age (yr) 70.5 (44-91, ±9.1)
Previous medical history
Stroke 102 (25.2)
Hypertension 205 (50.6)
Diabetes mellitus 76 (18.8)
Cardiovascular diseases
 Coronary artery disease 75 (18.5)
  Atrial fibrillation 63 (15.6)
Hyperlipidemia 66 (16.3)
Smoking 24 (5.9)
Reason for test
 Stoke work-up 61 (15.1)
 Medical check-up 344 (84.9)

Values are presented as number (%) or mean (range, ±standard deviation).

Table 2.
Accuracy of each reviewer for the interpretation of carotid plaque characterization (stable vs. unstable plaques) with a neuroradiologist’s decision as a reference
Total images (n=995) Neuroradiologist’s decision
Unstable plaque Stable plaque
Reviewer 1
 Unstable plaque 206 232
 Stable plaque 40 517
Reviewer 2
 Unstable plaque 96 156
 Stable plaque 150 593
Reviewer 3
 Unstable plaque 131 123
 Stable plaque 115 626
Reviewer 4
 Unstable plaque 123 163
 Stable plaque 123 586
Reviewer 5
 Unstable plaque 145 202
 Stable plaque 101 547
Table 3.
Binary classification of plaque vulnerability (stable vs. unstable plaques) between the reviewers
Outcome variable Reviewer 1 Reviewer 2 Reviewer 3 Reviewer 4 Reviewer 5
Overall accuracy (95% CI) 72.7 (69.8-75.4) 69.3 (66.3-72.1) 76.1 (73.3-78.7) 71.3 (68.3-74.0) 69.6 (66.6-72.4)
Sensitivity (95% CI) 83.7 (78.5-88.1) 39.0 (32.9-45.4) 53.3 (46.8-59.6) 50.0 (43.6-56.4) 58.9 (52.5-65.2)
Specificity (95% CI) 69.0 (65.6-72.3) 79.2 (76.1-82.0) 83.6 (80.7-86.2) 78.2 (75.1-81.1) 73.0 (69.7-76.2)

CI, confidence interval.


1. Zhan C, Shi M, Yang Y, et al. Prevalence and risk factors of carotid plaque among middle-aged and elderly adults in rural Tianjin, China. Sci Rep 2016;6:23870.
crossref pmid pmc pdf
2. Falk E. Pathogenesis of atherosclerosis. J Am Coll Cardiol 2006;47(8 Suppl):C7-12.
3. Langhorne P, Stott DJ, Robertson L, et al. Medical complications after stroke: a multicenter study. Stroke 2000;31:1223-9.
crossref pmid
4. Takaya N, Yuan C, Chu B, et al. Association between carotid plaque characteristics and subsequent ischemic cerebrovascular events: a prospective assessment with MRI: initial results. Stroke 2006;37:818-23.
crossref pmid
5. Singh N, Moody AR, Gladstone DJ, et al. Moderate carotid artery stenosis: MR imaging-depicted intraplaque hemorrhage predicts risk of cerebrovascular ischemic events in asymptomatic men. Radiology 2009;252:502-8.
crossref pmid
6. Gupta A, Baradaran H, Schweitzer AD, et al. Carotid plaque MRI and stroke risk: a systematic review and meta-analysis. Stroke 2013;44:3071-7.
crossref pmid
7. U-King-Im JM, Fox AJ, Aviv RI, et al. Characterization of carotid plaque hemorrhage: a CT angiography and MR intraplaque hemorrhage study. Stroke 2010;41:1623-9.
crossref pmid
8. Walker LJ, Ismail A, McMeekin W, Lambert D, Mendelow AD, Birchall D. Computed tomography angiography for the evaluation of carotid atherosclerotic plaque: correlation with histopathology of endarterectomy specimens. Stroke 2002;33:977-81.
crossref pmid
9. Johri AM, Nambi V, Naqvi TZ, et al. Recommendations for the assessment of carotid arterial plaque by ultrasound for the characterization of atherosclerosis and evaluation of cardiovascular risk: from the American Society of Echocardiography. J Am Soc Echocardiogr 2020;33:917-33.
crossref pmid
10. Ylä-Herttuala S, Bentzon JF, Daemen M, et al. Stabilization of atherosclerotic plaques: an update. Eur Heart J 2013;34:3251-8.
crossref pmid
11. Jeon JS, Sheen SH, Hwang GJ, Kim HC, Kwon BJ. Feasibility of intravenous flat panel detector CT angiography for intracranial arterial stenosis. AJNR Am J Neuroradiol 2013;34:129-34.
crossref pmid pmc
12. Jeon JP, Kim S, Kim TY, et al. Association between copeptin and six-month neurologic outcomes in patients with moderate traumatic brain injury. Front Neurol 2022;12:749110.
crossref pmid pmc
13. Rafailidis V, Chryssogonidis I, Tegos T, Kouskouras K, Charitanti-Kouridou A. Imaging of the ulcerated carotid atherosclerotic plaque: a review of the literature. Insights Imaging 2017;8:213-25.
crossref pmid pmc pdf
14. Fuster V, Badimon L, Badimon JJ, Chesebro JH. The pathogenesis of coronary artery disease and the acute coronary syndromes. N Engl J Med 1992;326:242-50.
crossref pmid
15. Buja LM, Willerson JT. Role of inflammation in coronary plaque disruption. Circulation 1994;89:503-5.
crossref pmid
16. Raghunath PN, Tomaszewski JE, Brady ST, Caron RJ, Okada SS, Barnathan ES. Plasminogen activator system in human coronary atherosclerosis. Arterioscler Thromb Vasc Biol 1995;15:1432-43.
crossref pmid
17. Caplice NM, Mueske CS, Kleppe LS, Simari RD. Presence of tissue factor pathway inhibitor in human atherosclerotic plaques is associated with reduced tissue factor activity. Circulation 1998;98:1051-7.
crossref pmid
18. Wilcox JN, Smith KM, Schwartz SM, Gordon D. Localization of tissue factor in the normal vessel wall and in the atherosclerotic plaque. Proc Natl Acad Sci U S A 1989;86:2839-43.
crossref pmid pmc
19. Bos D, Arshi B, van den Bouwhuijsen QJA, et al. Atherosclerotic carotid plaque composition and incident stroke and coronary events. J Am Coll Cardiol 2021;77:1426-35.
crossref pmid
20. Lukanova DV, Nikolov NK, Genova KZ, Stankev MD, Georgieva EV. The accuracy of noninvasive imaging techniques in diagnosis of carotid plaque morphology. Open Access Maced J Med Sci 2015;3:224-30.
crossref pmid pmc pdf
21. Muller MP, Tomlinson G, Marrie TJ, et al. Can routine laboratory tests discriminate between severe acute respiratory syndrome and other causes of community-acquired pneumonia? Clin Infect Dis 2005;40:1079-86.
crossref pmid pmc
22. Saba L, Saam T, Jäger HR, et al. Imaging biomarkers of vulnerable carotid plaques for stroke risk prediction and their potential clinical implications. Lancet Neurol 2019;18:559-72.
crossref pmid
23. Tanaka H, Chiu SW, Watanabe T, Kaoku S, Yamaguchi T. Computer-aided diagnosis system for breast ultrasound images using deep learning. Phys Med Biol 2019;64:235013.
crossref pmid pdf
24. Biswas M, Saba L, Chakrabartty S, et al. Two-stage artificial intelligence model for jointly measurement of atherosclerotic wall thickness and plaque burden in carotid ultrasound: a screening tool for cardiovascular/stroke risk assessment. Comput Biol Med 2020;123:103847.
crossref pmid
25. Lekadir K, Galimzianova A, Betriu A, et al. A convolutional neural network for automatic characterization of plaque composition in carotid ultrasound. IEEE J Biomed Health Inform 2017;21:48-55.
crossref pmid pmc

Editorial Office
21, Namdong-daero 774beon-gil, Namdong-gu, Incheon 21565, Korea
Tel: +82-32-460-3304    Fax: +82-32-460-3899    E-mail: editor@jksgn.org                

Copyright © 2022 by Korean Society of Geriatric Neurosurgery.

Developed in M2PI

Close layer
prev next