Validation and Refinement of Scores to Predict Stroke Risk: Prospective Cohort Study

Introduction

The Global Burden of Disease (GBD) 2021 study found that the number of new cases of stroke increased by 70.2% from 1990 to 2021 [1], highlighting a relatively insufficient emphasis on prevention, particularly in low-income countries. The number of patients with stroke in China is currently the highest in the world. Stroke is the leading cause of death and disability among adults in the country [2]. China has the highest risk of stroke globally, with an overall lifetime risk of 39.9% [3]. Consequently, China has implemented a series of measures to address the increasing burden of stroke, advocating for an integrated approach to stroke prevention, treatment, management, and rehabilitation [4].

The National Ministry of Health of China launched a significant national project called the “China National Stroke Screening Survey (CNSSS)” in 2009 to tackle the challenge posed by stroke. The China Stroke Prevention Project Committee (CSPPC) was established in April 2011. Volunteers aged 40 years and older were recruited through structured face-to-face questionnaires, and the “8+2” risk scorecard is used to screen participants and to identify high-risk groups [5]. The “8” refers to 8 risk factors (hypertension, heart disease, smoking, dyslipidemia, diabetes, physical inactivity, overweight, and family history of stroke [FHS]), and the “2” refers to transient ischemic attacks (TIAs) and previous strokes. According to the judging criteria, respondents are categorized into low-risk, medium-risk, and high-risk groups. In 2020, a total of 268,000 individuals in the high-risk group for stroke were identified across more than 240 project areas across the country [6].

Risk assessment is an effective tool for identifying prevention priorities [7]. The Framingham Stroke Risk Profile is recognized as one of the earliest and most widely used simple stroke risk assessment tools. However, validation studies in domestic populations have found that it tends to overestimate the actual stroke incidence to some extent [8]. Since its publication, the pooled cohort risk assessment equations have also been controversial, as some external validation studies suggest that this risk assessment model may overestimate the risk of atherosclerotic cardiovascular disease [9]. A study compared the performance of the Framingham cardiovascular risk equation, the pooled cohort equations, and the China-Population Attributable Risk equations in predicting the 5-year risk of atherosclerotic cardiovascular disease, including ischemic stroke. In the Uyghur and Kazakh populations, all 3 risk assessment equations consistently underestimated the risk [10]. Furthermore, although the “8+2” risk score tool has been widely used, its predictive ability remains unclear.

Therefore, we aimed to validate the “8+2” stroke risk score for predicting the 10-year risk of stroke in community cohort populations, and to optimize the scoring method to improve predictive accuracy.

Methods

Data Source

This study was a cohort study. This study followed the Transparent reporting of multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines.

The cohort was part of the China Stroke High-risk Population Screening and Intervention Program (CSHPSIP), an ongoing nationwide population-based program [5,11]. The study participants were recruited from the screening site in the Jinfeng District of Yinchuan City, Ningxia Hui Autonomous Region. A total of 22,259 community residents were enrolled in 2013 from one urban and rural monitoring site, and the outcome ascertainment was completed in 2023. The inclusion criteria for the screening participants are individuals who are aged 40 or older, permanent residents (those who have lived in the area for 6 months or more), and those who voluntarily participate by signing an informed consent form [12]. Patients younger than 40 years, those with a history of stroke or TIA, or individuals recruited after 2014 were excluded from the study cohort. The data quality control process is detailed in the Multimedia Appendix 1.

Risk Factors Measurement

Based on the “Stroke Screening and Prevention Technical Specifications” promulgated by the National Health and Family Planning Commission’s Stroke Screening and Prevention Engineering Committee, the following risk factors were assessed: hypertension, heart disease, smoking, dyslipidemia, diabetes, physical inactivity, overweight, and FHS. The detailed criteria for each risk factor are shown in Table S1 in Multimedia Appendix 1 which is also available on the China Stroke and Cardiovascular Disease website.

The criteria for classifying individuals into high-, medium-, and low-risk stroke groups were as follows: the high-risk group was defined as having 3 or more risk factors; the medium-risk group was characterized by 3 or less risk factors along with a history of chronic diseases (such as hypertension, diabetes, and heart disease); and the low-risk group was defined as having 3 or less risk factors without any history of chronic diseases (Table S1 in Multimedia Appendix 1) [13]. For high-risk individuals, follow-up visits are conducted by primary health care institutions at 6 months and 12 months after the initial assessment. For moderate-risk populations, primary health care institutions conduct a single follow-up visit at 12 months to evaluate and address their associated risk factors.

Outcome

We recorded stroke as an endpoint event by searching electronic hospitalization records in Ningxia in June 2023. Stroke was identified using the diagnostic code I60-63 from the International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD-10).

Logistic Regression Model

Logistic regression was used to determine the odds ratios (ORs) of every risk factor for incident stroke [14]. The basic equation for regression with multiple independent variables is:

${\displaystyle Y=\ln \left(\frac{P}{1-P}\right)=\alpha +{\beta }_1{X}_1++{\beta }_2{X}_2+\cdots +{\beta }_i{X}_i}$

Y is the estimated continuous outcome; α is the intercept. This is considered a constant value; β is the beta coefficients; and X_i is each risk factor.

Rothman-Keller Model

The Rothman-Keller model, initially developed by Kenneth J. Rothman and David B. Keller in the early 1970s, was designed to assess the combined impact of tobacco and alcohol consumption on the risk of oral and pharyngeal cancers [15]. This model provides a statistical framework that enables researchers to quantify both the independent and joint contributions of various risk factors to disease risk. It has been adapted and applied to a wide range of health conditions and diseases, including early-onset colorectal cancer and mild cognitive impairment [16,17]. Its flexibility in considering both additive and multiplicative effects of risk factors makes it a valuable tool for public health research and individual risk prediction.

The Rothman-Keller model uses the binomial distribution function method for risk classification. It calculates the benchmark proportion of incidence and risk scores based on the population exposure rate and OR of each risk factor. In addition, it estimates an individual’s relative risk of developing a disease by calculating their combined risk scores. The parameters of the Rothman-Keller model are calculated as follows:

1. Baseline morbidity ratio (）：$\rho =\frac{1}{{\sum }_{i=1}^{n}O{R}_i\times P_i}=1-PAR\%$P_i: the exposure rate of individuals exposed to a risk factor in the whole population; OR_i: the odds ratios of exposure to a risk factor; and PAR%: population attributable risk percentage.
2. Risk score (）：$S=\rho \times \text{O}{\text{R}}_i$
3. Total risk score (）：Pi: risk factor scores for S≥1, q_i: risk factor scores for S<1.
4. Individual risk prediction score: Individual risk of stroke=the incidence of stroke × . This expected risk of stroke is a relative value because it is measured against the overall incidence rate in the population. It can help us understand whether the individual’s likelihood of developing a stroke is higher or lower than the average level of the population.

The population exposure rate of every risk factor was derived from the literature [18]. The OR of exposure to a risk factor was sourced from the logistic regression model. Data for 100,000 participants were randomly generated by the binomial distribution functions of risk factors collected from the literature to identify nodes of high, medium, and low risk in models. The exposure rate of a risk factor in the study population was P₀. We generate 100,000 random P values of 0: 1, P<P₀ was recorded as 1 (ie, exposure), and P>P₀ was recorded as 0 (ie, nonexposure). Each risk factor simulates a column of data, summarizing the exposure of 100,000 community residents to each risk factor.

Statistical Analysis

Missing Data Interpolation

Among 11,692 participants, 1 individual did not have information on hypertension, and 2556 individuals lacked information on blood lipid levels. In accordance with previous studies [12,19,20], the incomplete data for hypertension and dyslipidemia were imputed simultaneously by multiple imputations (n=25) using the R package MICE (Stef van Buuren) [21]. Based on the Akaike information criterion value, 2 of the interpolation datasets were selected and the same analysis was performed on the selected interpolation dataset to identify results that were likely to be robust. The detailed data report of the other interpolation set is presented in the Tables S2-S5 in Multimedia Appendix 1.

Model Evaluation

First, we assessed the discrimination for the Rothman-Keller model and the logistic regression model using the area under the receiver operating characteristic curve (AUC). Our primary objective was to determine whether the predictive capability of the Rothman-Keller model surpassed that of the conventional logistic regression model, particularly in a multiclassification situation. When the increase in AUC is not statistically significant, its interpretation can become challenging [22-24]. Therefore, in addition to the AUC, we incorporated the absolute net reclassification improvement (NRI) to evaluate the relative performance of the 2 models. If the absolute NRI is greater than 0 or less than or equal to 1, it indicates a positive improvement, showing that the predictive ability of the new index has improved compared to the old index for stroke events. Conversely, if absolute NRI is less than 0 or greater than or equal to −1, it signifies a negative change, suggesting an improvement in the predictive ability of the new model for no stroke events. If the absolute NRI is equal to 0, it means that the new model shows no improvement. In our study, we analyzed the reclassification and absolute NRI for individuals who experienced a stroke event and those who did not. For individuals who have had a stroke, being reclassified into a higher-risk group was deemed an improvement in classification, whereas being reclassified into a lower-risk group was considered a failure.

A P value less than .05 was considered to indicate statistical significance. The statistical analyses were performed using R 4.2 software (R Core Team).

Ethical Considerations

The ethics review committee of The People’s Hospital of Ningxia Hui Autonomous Region approved this study (approval number: 2020-KY-053). Patients provided informed consent for using the data. Data were deidentified. No compensation was provided.

Results

Characteristics of the Study Cohort

A total of 22,259 community residents were recruited in 2013. After excluding individuals with a history of stroke (n=313), and TIA (n=384), as well as participants younger than 40 years old (n=195) and those recruited after 2014 (n=9576), 11,791 participants were included in the follow-up cohort. After a 10-year follow-up period, 99 participants were lost to follow-up. Finally, a total of 11,692 eligible participants were included in the final analysis (Figure 1). A total of 767 participants (6.6%) had a stroke by the end of the follow-up period. Based on the “8+2” stroke risk score, the 10-year stroke incidence among the 3 stroke risk groups of the community residents was as follows: low-risk group 4.47% (n=8908, 398 stroke cases); medium-risk group 14.71% (n=1074, 158 stroke cases); and high-risk group 12.34% (n=1710, 211 stroke cases) (Table 1). Kaplan-Meier survival curves are shown in Figure S1 in Multimedia Appendix 1.

‎

Model Construction

The baseline incidence ratio (ρ) of 8 factors was obtained to calculate the population attributable risk percentage (PAR%) through a previous study [18]. The OR values of these factors were assessed using a logistic regression model. The parameters of the logistic model and the Rothman-Keller model are shown in Table 2.

After ranking the stroke incidence risk based on a dataset of 100,000 random entries, 2 nodes were selected for subdividing the risk groups into low-risk, medium-risk, and high-risk categories using the logistic regression model: node a (ID=25844, risk prediction score=0.06168760) and node b (ID=77778, risk prediction score=0.16451650). In addition, for the Rothman-Keller model, 2 other nodes were chosen for the same purpose: node A (ID=25426, risk prediction score=0.0021993391) and node B (ID=64553, risk prediction score =0.0047898060) (Figure S2 in Multimedia Appendix 1).

Evaluation of the Model

The sensitivity, specificity, and AUC of the “8+2” stroke risk score, the Rothman-Keller model, and the logistic regression model are presented in Table 3. The logistic regression model and the Rothman-Keller model demonstrated significant differences in AUC values compared to the “8+2” stroke risk score (Z=2.60, P<.05; Z=3.47, P<.05, respectively). However, no difference was observed between the logistic regression model and the Rothman-Keller model (Z=0.688, P>.05). The comparison of receiver operating characteristic curve is shown in Figure S3 in Multimedia Appendix 1.

From the NRI, we found that the majority of participants remained at the same level of risk for developing a stroke as predicted by the “8+2” stroke risk score, the logistic regression model, and the Rothman-Keller model (ie, along the diagonal from the lower left to the upper right). However, some community residents were reclassified as having a different level of risk for developing stroke (Figure S4 in Multimedia Appendix 1). According to the “8+2” stroke risk score, the NRI for reclassification of stroke events by the Rothman-Keller model was 7.8%, while the NRI for nonstroke events was −2.7%. The absolute NRI was then estimated to be 0.051 (P=.01), calculated using the sum of the net estimated for individuals who developed a stroke and those who did not. The NRI for the logistic regression model was 1.7% for stroke events and −0.7% for nonstroke events, resulting in an absolute NRI of 0.010 (P=.62) .

Discussion

Principal Findings

This cohort study followed 11,692 individuals aged 40 years and older for a duration of 10 years. According to the “8+2” stroke risk score, the stroke incidence in the low-risk (n=8908), medium-risk (n=1074), and high-risk groups (n=1710) was 4.5%, 14.7%, and 12.3%, respectively. We developed a logistic regression model and a Rothman-Keller model to validate and optimize the risk score. Through a comparative analysis of the performance of the 3 models, we found that the Rothman-Keller model exhibited the best performance.

We verified the efficacy of the model using an actual database. There was no significant difference in the AUC values between the logistic regression model and the Rothman-Keller model. To evaluate model performance more accurately, we used the NRI for a more in-depth analysis. The NRI assesses the effects of low-, medium-, and high-risk reclassification for both stroke and nonstroke events, resulting in a net reclassification that provides a more accurate estimate than that obtained with other approaches [23]. Positive values of the stroke NRI indicate that the model effectively identifies patients with stroke, enabling physicians to initiate targeted detection or treatment to prevent stroke events. In contrast, a decrease in the NRI for nonstroke events suggests that community residents with a low or medium risk as determined by the “8+2” stroke risk score may actually be at a higher risk of having a stroke. Based on the overall NRI, we conclude that the Rothman-Keller model enhances the reclassification of both stroke and nonstroke events [25]. This finding aligns with the results of previous studies. Researchers used the Rothman-Keller model to predict the likelihood of mild cognitive impairment in older Chinese individuals. Upon validation with actual population data, it was found that the model had appropriate accuracy and performed well in terms of predictive efficacy [16,26]. The model can be adjusted and optimized based on new research data and epidemiological changes, thereby maintaining its predictive power in a timely manner. Its methodology can be applied to risk assessments for other populations and chronic diseases, demonstrating significant universality.

There are many model studies aimed at predicting stroke risk [27-30]. For instance, SCORE2 is a risk assessment tool developed using extensive data from a large number of European populations. It is designed to evaluate the risk of cardiovascular disease in both men and women across 4 distinct risk areas in Europe within a 10-year period [31]. While this tool is widely applied, it has limitations in terms of ethnicity and geography. Significant prediction errors may occur when applying it to populations with considerable differences [9,32]. A study estimated the 10-year risk of stroke based on a cohort analysis and found that factors such as age, systolic blood pressure, diastolic blood pressure, FHS, atrial fibrillation, diabetes, and others can significantly predict the incidence of stroke [27]. Compared with other models, this study developed a Rothman-Keller model based on questionnaire information to identify new risk nodes through simulation datasets, which provided a basis for stroke prevention. Furthermore, the model’s predictive power and accuracy were verified using real-world data. Logistic regression analysis indicated that smoking, heart disease, dyslipidemia, and physical inactivity were not related to stroke, which may be attributed to variations in demographic and stroke subtypes differences. This result was consistent with the results of previous Mendelian randomization studies [33-37].

The limitations of this study are as follows: first, the dataset used for verification only included participants from only one region. Studies in other provinces are necessary to evaluate the efficacy of our model. When research results are extrapolated to other populations with significant differences, it may be essential to consider the exposure rates of risk factors and OR or RR values for those populations in order to update and optimize the model. This process enhances the predictive accuracy and applicability of the Rothman-Keller model. Second, we were unable to perform a subgroup analysis on the various types of stroke. The primary objective of the program is to identify and intervene with high-risk populations to prevent the occurrence of stroke or reduce its risk, rather than focusing on the subtypes of ischemic stroke and hemorrhagic stroke. Furthermore, given the limitations of medical resources and the acceptability of screening, the program may prioritize the implementation of more accessible and universal preventive measures. These measures include controlling blood pressure, quitting smoking, and increasing physical activity, all of which are effective in preventing both ischemic stroke and hemorrhagic stroke [38,39]. Third, during the decade, participants may have received lifestyle interventions, pharmaceutical treatments, and early clinical treatment that influenced the incidence of stroke events. However, in the medium- and high-risk groups for stroke, these patients represent a certain proportion. Our model primarily assesses the variations in the initial screening judgments, and the outcome events remain consistent across different models, making it unlikely to influence the study results. Finally, the low sensitivity observed in this study may be attributed to the lack of several important stroke prediction factors from the risk scoring scale, thereby limiting its predictive ability. In 2021, the Guidelines for Stroke Prevention and Treatment in China recommended including homocysteine testing in routine screening and conducting carotid artery examinations for high-risk populations when conditions permit. Our research findings further support this recommendation. Although this study has some limitations, it also presents several advantages. The diagnostic criteria for risk factors in this project are based on relevant guidelines and standards established by the China Health Commission. Staff members undergo standardized training and assessment, and only those who pass the assessment are qualified to conduct screening tasks. Therefore, the identification of risk factors in this study had high accuracy and credibility. We used the binomial distribution of risk factors to construct a random dataset of 100,000 community residents, which allowed us to determine the high-, medium-, and low-risk boundary values of the models. The variables in the model were easily obtained and predicted estimates could be derived through straightforward calculations. We included a substantial number of community residences from the CSHPSIP over a decade for external validation, which exhibited good discrimination and calibration.

Conclusions

In conclusion, the Rothman-Keller model may improve the predictive efficacy of stroke screening models. In the future, verification will need to be carried out in a wider population and combined with more risk factors. The Rothman-Keller model for assessing individualized stroke risk, combined with interactive information platforms for health education, is beneficial for decreasing the incidence of stroke among high-risk groups.