Carotid plaque refers to focal wall hypertrophy that is at least 50% greater than that of the surrounding carotid wall. The prevalence of carotid plaques is approximately 40.3% and they are found more frequently in males than in females [1]. Patients with carotid plaques are more likely to have metabolic disorders with high total cholesterol and low-density lipoprotein-cholesterol levels [1]. Because carotid plaques are formed in the late stages of atherosclerosis, they are thought to be a risk factor for cerebrovascular disease including stroke [2,3]. Accordingly, screening for carotid plaques and identifying carotid plaque morphology is conducted in the clinical field. Among various radiologic tests, carotid ultrasonography (US) is performed widely in private hospitals, as well as general and university hospitals due to its relatively low cost and ease of implementation compared to magnetic resonance imaging (MRI) or computed tomography (CT) [4-8].

The variables in interpreting carotid US images are intima-media thickness (IMT), the degree of stenosis, and the presence of a carotid plaque along with an assessment of its vulnerability to rupture. Among these variables, the accurate classification of a carotid plaque vulnerable to rupture and identifying one that is more likely to cause a stroke is difficult for general practitioners who are non-radiologists. Most neurosurgeons also have difficulties in distinguishing unstable plaques from stable carotid plaques despite the increasing number of carotid stenting procedures in recent years. In addition, neurosurgical residents are not well trained in carotid US interpretation. Few studies have addressed the classification accuracy of plaque vulnerability by non-radiologists in the domestic clinical field. Here, we investigated the classification accuracy of carotid plaque vulnerability (unstable vs. stable plaque) by neurosurgical residents to obtain objective data on the interpretation accuracy in the actual medical environment.

A total of 405 patients with 995 carotid US images (axial image, n=297; and longitudinal image, n=698) were included in the analysis. The mean patient age was 70.5 years (range, 44-91 years) and 223 (55.1%) were female. The number of patients with hypertension and diabetes mellitus was 205 (50.6%) and 76 (18.8%), respectively. Cardiac disease, such as coronary artery disease or atrial fibrillation, was observed in 138 patients (34.1%). Twenty-four patients were smokers with an average of 28.25 pack-years. Among the patients who underwent carotid US, 61 (15.1%) were conducted for stroke evaluation and 344 (84.9%) were conducted for health examinations (Table 1).

The classification results of the five reviewers using a neuroradiologist’s decision as the reference value are presented in Tables 2 and 3. There were 236 (24.7%) unstable and 749 (75.3%) stable carotid plaques. Reviewer 1 showed the best-balanced interpretation with a sensitivity of 83.7% (78.5%-88.1%), specificity of 69.0% (65.6%-72.3%), and overall accuracy of 72.7% (69.8%-75.4%). Compared to the overall accuracy (ranging from 69.3% to 76.1%), the sensitivity results showed severe fluctuations (ranging from 39.0% to 83.7%) between the reviewers. Pairwise comparison of the ROC of reviewer 1 also showed the best performance in classifying plaque vulnerability with an area under the curve (AUC) of 0.583 (95% confidence interval [CI], 0.552-0.614) (Fig. 2A). Reviewer 4 had the lowest AUC of 0.533 (95% CI, 0.502-0.565). The AUC difference between reviewer 1 and reviewer 4 reached statistically significance (AUC difference, 0.049; 95% CI, 0.015-0.084; P=0.005). Except for these findings, there were no significant differences in AUC values between the remaining reviewers. If the data distribution is unbalanced, like in our case (unstable plaque: n=236, 24.7%; stable plaque: n=749, 75.3%), there is a possibility that the AUC value might be biased. To overcome this potential bias, we generated a precision-recall curve, especially for reviewer 1 and reviewer 4. The results also demonstrated a significant difference in plaque vulnerability classifications (AUC difference, 0.028; 95% bootstrap CI, 0.007-0.048) (Fig. 2B).

Carotid plaques form various shapes and occur in areas from the distal common carotid artery to proximal internal carotid artery [13]. It is more accurate to predict patient prognosis and choose treatment methods based on plaque vulnerability rather than only the degree of carotid stenosis. The major features used to describe carotid plaques in carotid US images include calcification, ulceration, hemorrhage, and a lipid-rich necrotic core, as well as IMT [10]. Calcification with ulceration is stimulated by prolonged shear stress and sudden arterial pressure changes on the carotid bifurcation [14]. This activates inflammation and metalloproteinase activity, causing erosion of the calcification to progress to ulceration of the surface [15]. Ulceration in the carotid plaque predisposes to thrombus formation. Plasminogen activator inhibitor-1 within plaques creates a thrombotic effect [16]. The expression of tissue factors, which have a major effect on the coagulation cascade, is determined by the extent and degree of inhibition of the tissue factors pathway within the plaque [17]. Tissue factor is localized to areas of lipid deposition and cholesterol crystals in carotid plaque [18]. Bos et al. [19] reported that intraplaque hemorrhage of the carotid artery significantly increased the risk of stroke (hazard ratio, 2.42; 95% CI, 1.30-4.50). Considering these facts, the accurate classification of carotid plaque vulnerability (unstable vs. stable plaque) is important for clinicians to determine treatment plans. Three radiologic tests, MRI, CT, and US are available to examine plaque vulnerability in the carotid artery. MRI provides excellent differentiation of soft tissue composition with high resolution, but its use can be limited due to the high medical costs and time-consuming procedures needed to obtain the images [4-6]. CT has the advantage of shorter imaging time than MRI but has the limitation that a contrast agent is required to distinguish plaque components [7,8]. In the domestic medical environment, MRI and CT is used in more than US in small and medium-sized general hospitals. And in most of these hospitals, radiologists are available to interpret plaque vulnerability. Compared to MRI and CT, carotid US is widely used in private hospitals, as well as for health examinations in general and university hospitals in Korea due to its relatively low cost, ease of using at the bedside, and non-invasiveness. Lukanova et al. [20] reported that carotid US showed high balanced accuracy for detecting unstable plaques with a sensitivity of 94% and specificity of 93% compared to multi-detector CT, with a sensitivity of 83% and specificity of 73%, and MRI with a sensitivity of 100% and specificity of 89%. However, the procedure and its interpretation may vary depending upon the examiner’s experience. In particular, considering the reality of private hospitals with many non-radiologists, it is necessary to properly classify plaque vulnerability. In this study, we evaluated the accuracy of plaque vulnerability classification, targeting neurosurgical residents. Although there were some differences in classification accuracy between the reviewers, the overall accuracy level was low (0.5<AUC≤0.7) [21]. Considering that stroke risk is largely dependent upon not only the degree of stenosis but also plaque vulnerability, accordingly, education on discriminating plaque vulnerability is necessary during resident training [22]. However, it is not easy to educate residents on carotid US interpretation given the reality of medical situations. In this case, a new computer-aided diagnostic system using machine learning can be an alternative. Tanaka et al. [23] demonstrated that a convolutional neural network (CNN) composed of VGG19 and ResNet152 showed accurate classifications of breast ultrasound images with a sensitivity of 90.9% (95% CI, 84.5%-97.3%), specificity of 87.0% (95% CI, 79.5%-94.5%), and AUC of 0.951 (95% CI, 0.916-0.987). Biswas et al. [24] proposed a two-stage artificial intelligence model to measure atherosclerotic wall thickness and total plaque burden. However, they did not distinguish plaque vulnerability. Lekadir et al. [25] also suggested a CNN for automatically characterizing plaque components, such as the lipid core, fibrous cap, and calcified tissue. However, they used only 56 cases. Accordingly, there is a need for artificial intelligence (AI) based on gray-scale carotid US images, which is often used in the field, and AI research that not only detects plaque composition but comprehensively informs on plaque vulnerability.

There were some limitations in this study. First, there were only five participants, and they were from the same hospital. Thus, a concern may arise as to whether the residents participating in the study actually represent non-radiologists working in private hospitals. Second, the correct answer was decided only by one neurointerventionist. Although he had more than 20 years of experience, selection bias may have occurred, and this should be taken into account when interpreting our results. Nevertheless, to the best of our knowledge, this was the first study to investigate the classification accuracy of plaque vulnerability by neurosurgical residents based on carotid US images in actual clinical settings.

The classification accuracy of neurosurgical residents to discriminate plaque vulnerability based on carotid US images was low. To address this problem, a system that helps to educate and comprehensively interpret carotid plaque morphology should be developed.