Use of a Telemedicine Risk Assessment Tool to Predict the Risk of Hospitalization of 496 Outpatients With COVID-19: Retrospective Analysis

Introduction

In March 2020, the identification of SARS-CoV-2 in the United States led to the rapid closure of elective medical care at many health care institutions, with redeployment of personnel to address the rising burden of COVID-19. In the US state of Georgia, the cumulative number of cases reported by the Department of Public Health rose from 84 cases on March 15, 2020, to 4231 cases by March 31, 2020.

It was recognized from early reports that the severity of COVID-19 varies from asymptomatic to life-threatening [1,2] and that most patients have mild illness and do not require hospitalization [3]. For these patients, the recommendation is to isolate at home and monitor symptoms under the care of a medical provider [4,5]. Many US medical centers have employed telemedicine and remote monitoring programs to provide this care [6-8]. Monitoring programs require investment and staffing [7]; it may be appropriate to focus these resources on those at highest risk of hospitalization for severe COVID-19. While it is recognized that certain groups (eg, older adults and patients with diabetes) have higher rates of hospitalization [9-12], there are no validated risk assessment tools that stratify risk for outpatients undergoing home monitoring [13]. The tools in existence often require in-person criteria (eg, vital signs, labs, and imaging) that are not available by telemedicine [13,14].

In order to better target care for outpatients with COVID-19, we created a risk assessment tool to assign patients a risk tier by incorporating age, comorbidities, symptom severity and course, and the ability to isolate—criteria highlighted in the initial US Centers for Disease Control and Prevention (CDC) guidance for home monitoring of patients with COVID-19 [4]. We prospectively applied this risk tool during the telemedicine assessment of outpatients recently diagnosed with COVID-19 in a home monitoring program. Patients were followed with regular phone calls until clinical improvement or hospitalization. In this retrospective study, we analyzed patient data gathered systematically at telemedicine intake visits, including patient characteristics and assigned risk tier, and used an outcome of hospitalization related to COVID-19. We hypothesized that the multifactorial tool would predict hospitalization rates.

Methods

Ethical Approval and Consent

The study was approved by the Emory University Institutional Review Board, which granted a waiver of consent and a waiver of Health Insurance Portability and Accountability Act authorization. The study was carried out in accordance with the principles embodied in the Declaration of Helsinki.

Study Setting and Population

The study is a retrospective cohort investigation of outpatients with confirmed COVID-19 at Emory Healthcare, the largest academic health system in Georgia, serving the greater Atlanta metropolitan area. Testing was scheduled through a central COVID-19 hotline and performed at one screening clinic and one drive-through site, in addition to the emergency departments (EDs) at four hospitals. The test used was real-time reverse transcription–polymerase chain reaction (RT-PCR) detection of SARS-CoV-2 by nasopharyngeal swab. During the study period, testing for COVID-19 was available for symptomatic adults and prioritized (1) health care workers, (2) university students on campus, and (3) patients who were older or had medical comorbidities. Testing and monitoring of children (aged <18 years) was not available at Emory Healthcare. Patients with positive RT-PCR results were called by a dedicated result notification nurse team to provide self-care advice and refer for enrollment in the home monitoring program, named the Virtual Outpatient Monitoring Clinic (VOMC). Characteristics of the first 208 patients in VOMC [15] and the symptom course of VOMC patients [16] have been previously described.

The VOMC intake team included 14 physicians and 3 advanced practice providers (APPs) from two primary care clinics. VOMC follow-up call teams included 19 redeployed registered nurses (RNs) and 20 APPs. All intake providers were trained in the use of the risk assessment tool in a 1-hour webinar and conducted a median of 25 intake visits during the study period (IQR 36.5; range 5-99).

Enrollment criteria for this study included the following: (1) completion of new patient VOMC visit during the period of March 24 to May 26, 2020, and (2) documentation of positive RT-PCR result for SARS-CoV-2. Exclusion criteria included the following: (1) hospitalization prior to VOMC enrollment and (2) immediate discharge from VOMC—no follow-up calls—due to meeting CDC criteria for ending home isolation (≥14 days from symptom onset with resolution of fever and improvement in respiratory symptoms).

Exposure

VOMC intake visits comprised a 40-minute nurse intake (ie, initial data entry) followed by a 40-minute physician or APP telemedicine visit including risk assessment. The clinical care pathway for outpatients with COVID-19 in the VOMC is outlined in Multimedia Appendix 1. The risk assessment tool used by the VOMC was created based on published data about risk factors for severe COVID-19 and the natural history of disease available in March 2020. Patients were assigned to a baseline risk tier 1 to 3 by the provider upon completion of the VOMC intake visit that determined the planned frequency and duration of monitoring. Low-risk patients (Tier 1) received calls every other day for a minimum of 7 days from symptom onset. Intermediate-risk patients (Tier 2) received daily calls for a minimum of 14 days from symptom onset. High-risk patients (Tier 3) were called twice daily for a minimum of 21 days from symptom onset. There was no limit on duration of care, and calls would continue for all patients until symptom improvement or hospitalization, regardless of tier.

Details of the tier assignment by the VOMC risk assessment tool are in Multimedia Appendix 2. Tier 1 patients must meet all of the following criteria: aged <60 years; no comorbidities known to increase risk of severe COVID-19; no lower respiratory tract symptoms, except mild cough; and able to self-isolate. Tier 2 included patients aged 60 to 69 years without comorbidities and patients aged less than 60 years with moderate-risk comorbidities or with persistent symptoms (ie, no improvement) into the second week of illness. Tier 3 included patients meeting any of the following criteria: aged ≥70 years, younger age with specific high-risk comorbidity or multiple comorbidities, new or worsening lower respiratory symptoms, or uncertain ability to self-isolate. Providers were instructed to lower the risk tier by one level for patients whose intake visit occurred during the second week of illness if they reported improving symptoms, even if older age or comorbidities were present.

Outcome

Hospitalization was the primary study outcome, consistent with the stated purpose of the risk assessment tool. ED visits and observation admissions were not included as events. Hospitalization at four Emory Healthcare acute care hospitals was determined by Emory Clinical Data Warehouse (CDW) queries, last performed on July 6, 2020. External hospitalizations were identified by chart review in (1) VOMC clinical notes, (2) administrative messages, and (3) hospitalization documentation in the Emory Healthcare electronic health record per data sharing agreements with other health systems. Loss to follow-up was minimal because VOMC patients were followed until symptom improvement and for specified minimum intervals—7, 14, and 21 days for Tiers 1, 2, and 3, respectively—whichever was longer.

Covariates

Risk assessment data were obtained for all patients enrolling in the VOMC during an intake telemedicine visit utilizing synchronous two-way audio-video communication, with a telephone call as a backup option. VOMC providers completed a standard note template, including comorbidities (ie, past medical history and specific conditions with elevated COVID-19 severity risk), symptom description (ie, onset, severity, and course), social support and ability to isolate, and clinician-assigned risk tier using the risk assessment tool (Multimedia Appendix 2). These data were extracted from the completed VOMC intake notes by CDW query. Missing data were included by manual chart review by the authors (JO and GO) of provider free-text documentation in the intake note. Only data recorded at intake visits with initial risk tier were used; subsequent changes in illness severity and tier reassignments, based on worsening or improvement, were not used in the analysis. Actual age, BMI (calculated as weight in kilograms divided by height in meters squared), and race were obtained with a second CDW query of demographics and height and weight records. If a BMI of 30 or greater was recorded by the provider in the comorbidity portion the VOMC note, it was considered reported obesity. If a BMI of 30 or greater was identified by either the VOMC note or the inclusion of height and weight records, we considered this corrected obesity for analysis. As we observed underreporting of BMI values of 30 or greater in provider notes compared to height and weight records, we conducted sensitivity analyses to determine if preference for one metric or another substantially influenced results.

Statistical Methods

Survival analysis was used to determine factors associated with hospitalization to evaluate the risk tier model. Initial unadjusted hazard ratios (HRs) were calculated using a Cox proportional hazards model. A multivariable model was then constructed using a Cox proportional hazards model. Time-varying covariates were identified by individual evaluation of covariates looking at Kaplan-Meier curves and testing for a statistically significant time-variable interaction. Covariates with P values less than .05 for the time interaction term were considered time-varying.

The models developed by backward and forward selection were then manually checked by adding and removing individual variables and assessing model fit. Cases with missing data were not included in analysis during the exploratory phase. The final model did not have any missing data. To provide odds ratios (ORs) for comparison, logistic regression was performed with the same variables as the Cox regression analysis. All analyses were conducted using SPSS statistical software, version 26 (IBM Corp).

Proposed Simplified Tier Model

We considered covariates for a streamlined risk assessment model to simplify the tier-assignment process for more practical use.

Results

Participant Characteristics

We identified 551 patients completing a new VOMC visit from March 24, 2020, through May 26, 2020. We included 496 patients in the analysis after excluding 7 patients without a positive RT-PCR result, 25 patients hospitalized for COVID-19 prior to their VOMC visit, 2 patients sent to the ED and hospitalized at their first VOMC visit, 1 patient with a blank form, and 20 patients who met criteria for discharge—by duration of symptoms and improvement—and were, thus, not placed into a tier. During the study period—testing dates March 15 to May 22, 2020—the following number of nonhospitalized patients tested positive for SARS-CoV-2 by RT-PCR at Emory Healthcare: 730 in the outpatient setting and 170 in the ED. We do not have data on the patients who did not complete a VOMC intake visit.

The timing of the initial VOMC visit was similar between tiers (mean 9.3 days from symptom onset), and the mean follow-up was shorter for Tier 1 (mean 9.5 days, 95% CI 8.6-10.4) compared to the overall cohort (mean 13.1, 95% CI 12.2-13.9) (Table 1). The majority of the patients were female (330/496, 66.5%), 252 (50.8%) were Black, and 383 (77.2%) were under 60 years of age. Only 174 patients out of 496 (35.1%) reported no high-risk comorbidities, with hypertension (175/496, 35.3%) and reported BMI greater than 30 (147/496, 29.6%) as the most frequent comorbidities. Most patients (316/496, 63.7%) had mild symptoms or no symptoms at the time of the visit.

Univariate Analysis

We identified 35 VOMC patients requiring hospitalization and 461 patients who did not require hospitalization during the follow-up period. There were no deaths during VOMC care at home; 2 patients died during hospitalization and a third died shortly after hospitalization while in hospice care. Statistically significant factors for hospitalization included risk tier, age, coronary artery disease, diabetes mellitus, heart failure, reported obesity (BMI ≥30), two comorbidities, three or more comorbidities, severe symptom rating, and worsening symptom course (Table 2). Of the patients initially categorized as Tier 3, 17 out of 74 (23%) were hospitalized in the course of their care, compared with 15 out of 185 (8.1%) Tier 2 patients and 3 out of 237 (1.3%) Tier 1 patients. Among 35 hospitalized patients, the median days to admission from symptom onset was 8 in Tier 3, 11 in Tier 2, and 13 in Tier 1. Tier level had the highest unadjusted HR of all factors, with 5.29 for Tier 2 and 16.24 for Tier 3 in comparison to Tier 1.

Multivariable Analysis

The final model that predicts hospitalization among outpatients in VOMC includes (1) risk tier, (2) reported obesity, (3) aged ≥60 years, and (4) gender as strata (Table 3). This model had an overall fit that was statistically significant (P<.001). Covariates other than gender satisfied the proportional hazards assumption. Even though the risk tier rubric does take into account both age and obesity, both of these covariates remained statistically significant with HRs greater than 2 and so were retained in the final model (Multimedia Appendix 3). Gender was found to be a time-varying covariate (Multimedia Appendix 3) and was, therefore, analyzed by stratum [17]. The adjusted HRs for Tiers 2 and 3 compared to Tier 1 were 3.74 (95% CI 1.06-13.27; P=.04) and 10.87 (95% CI 3.09-38.27; P<.001), respectively. Age of 60 years or older had an adjusted HR of 2.53 (95% CI 1.27-5.02; P=.008) and reported obesity had an adjusted HR of 2.09 (95% CI 1.06-4.13; P=.03). Survival curves (Figure 1) show days from symptom onset to hospitalization by tier. Males were hospitalized earlier and more often than females. Logistic regression performed with the same variables to shadow the Cox regression analysis found similar results with adjusted ORs of 4.87 for Tier 2 and 15.38 for Tier 3 compared to Tier 1. Age of 60 years or older and reported obesity both had adjusted ORs similar to their adjusted HRs (Table 3). Gender was not statistically significant but was kept in the logistic regression model for comparison to the survival analysis.

‎View this figure

Sensitivity Analysis

We performed sensitivity analysis for obesity to see if using the actual BMI (ie, corrected obesity) would be more predictive than reported obesity from the VOMC note. The adjusted HR for corrected obesity was 3.783 (95% CI 1.761-8.126; P<.001) with only minor changes in the HR and P values for tier and for age of 60 years or older (Multimedia Appendix 4).

Proposed Simplified Tier Model

We looked at factors associated with hospitalization to propose a streamlined risk assessment model to predict which patients in the VOMC setting will not require hospitalization during COVID-19 illness. Defining a new Tier 1 as age of less than 60 years, no high-risk comorbidities, able to self-isolate, symptom severity mild or none, and symptom course stable or improving, we find a model with no hospitalizations for proposed Tier 1 patients (Table 4).

Data Sharing Statement

Deidentified data are available in a public, open access repository [18].

Discussion

Principal Findings

This study describes the outcomes of outpatients with confirmed COVID-19 who participated in a standardized telemedicine risk assessment and telephone monitoring program. We found that the risk tiers designated by a multifactorial tool predicts hospitalization rates more strongly than individual variables. We also found that age and obesity still remained significant predictors even though they were part of the risk assessment. It is likely that providers weighed initial symptom severity more in their assessment than either age or obesity, as initial symptom severity was not significant when risk tier was taken into account. Future iterations of the risk assessment tool should increase the relative weights of age and obesity. We note that many similar efforts to produce valid outpatient risk assessment tools are ongoing, but they have not yet been prospectively validated [13].

Comparison to Previous Studies

The overall hospitalization rate in this outpatient cohort was 7%, which is lower than that in other populations reported in New York (51.9%) [11] and Louisiana (39.7%) [19], likely because testing in these cohorts was concentrated in EDs, with lower numbers of patients in the outpatient setting. A more comparable cohort of outpatients monitored by text messaging in Pennsylvania reported a low rate of ED use at approximately 7% but with limited follow-up for hospitalization [20]. The individual risk factors for hospitalization in this study are similar to those identified in earlier cohort studies, particularly age, male sex, and elevated BMI [11,12], but these studies did not include any multifactorial provider risk assessment rubric.

Potential Applications

The identification of a small group of outpatients (ie, Tier 3) at the highest risk of hospitalization facilitates planning efforts for high-intensity outpatient monitoring with limited follow-up resources and may justify the expanded implementation of the risk assessment tool at the point of care. One question raised by a useful risk tier rubric is whether it can be codified into a computer-resident algorithm, an artificial intelligence (AI) application. We attempted models with tier as an output rather than input and using the objective and subjective notes and clinical observations as inputs. We were unable to develop such a model, evidently because the tier assignment includes several points where clinical judgment is required and applied. Incorporating that clinical judgment is necessary and is beyond our AI ability at this point.

The largest group identified by this risk assessment tool was Tier 1. These individuals were at low risk of hospitalization, with 3 admissions out of 237 patients. In order to rapidly identify individuals at low risk of hospitalization and, therefore, who require fewer monitoring resources, we were able to simplify criteria for a proposed new Tier 1 four-item risk score. As additional remote monitoring tools become available (eg, automated text message surveys), this population may be appropriate to assign as needed follow-up instead of proactive monitoring calls.

Strengths and Limitations

While the overall study design is retrospective, our program implemented the risk assessment tool prospectively for all new patients with COVID-19, and we were able to follow all patients until clinical improvement or hospitalization because of the availability of redeployed providers (ie, RNs and APPs), minimizing gaps in data. Furthermore, we found that the risk tiers tool predicted hospitalization risk with highly significant results in multivariate analysis and time-to-hospitalization survival analysis. This supports our hypothesis that inclusion of multiple factors in patient assessment (ie, age, risk factors, symptoms, and social factors) would most effectively identify absolute hospitalization risk and time to hospital admission.

A primary limitation of this single-center study is generalizability to other populations. We had a high proportion of working-age individuals in the first wave of the pandemic, and relatively few older adults and socially disadvantaged individuals are included in the study population. This may explain the lower hospitalization rate compared to cities with larger outbreaks. Furthermore, the time to enrollment in the VOMC (9.3 days) reflects the real-world practice at our clinic, but limits generalizability to settings (eg, urgent care) where patients may present earlier in the disease course. Another limitation is the existence of different levels of observation (ie, frequency of telephone calls, provider type for calls, and duration of follow-up calls) based on assigned tier, which may have impacted outcomes. We cannot speculate if and how more frequent calls would affect the likelihood of hospitalization. We also acknowledge the possibility of loss to follow-up: patients could end VOMC care on request and we do not have direct data for outside hospitalizations, although we reviewed all charts for documentation of such.

The risk assessment tool itself has limitations. First, it was not derived from an outpatient cohort, since none existed at the time, and was instead designed based on limited data available from reports of COVID-19 in hospitalized patients. Second, due to the differential risk posed by age ranges and specific comorbidities, the risk tool is relatively complex and required skilled medical providers to gather the necessary data. We trained a dedicated provider group in its use, but this limits external validity. Even in the optimal setting, we encountered underreporting issues (eg, reported obesity vs actual BMI).

Future Directions

In subsequent waves of the pandemic, the majority of patients in our practice remain home during acute COVID-19. Telephone monitoring continues to provide care for high-risk patients without the ability to participate in automated programs. With the introduction of technologies such as wearable monitoring devices and advanced treatments (eg, monoclonal antibodies), the identification of high-risk patients who are most likely to benefit continues to be a high priority. In this context, the refinement and validation of risk assessment rubrics across clinical sites and in diverse populations remains important to COVID-19 outpatient care. Future investigation may consider rapid tools (eg, automated identification of highest and lowest risk groups at the time of presentation for testing) as well as validation of provider assessment tools such as ours within other centralized telemedicine programs.

Conclusions

Our findings suggest that patients at low, intermediate, and high risk for hospitalization may be identified with a telemedicine risk assessment tool incorporating age, medical history, symptom severity, and social factors. The Tier 1 patients in our cohort had low hospitalization rates. We observed increasing odds of hospitalization in Tiers 2 and 3, respectively. External validation of these findings is necessary, but we also recognize that care delivery decisions need to be made immediately in the context of recently escalating cases in the COVID-19 pandemic. It is possible to use these data to create care models targeting the highest-risk patients during the highest-risk time periods, but further study of the safety and outcomes of this risk-based approach is needed. This study represents our initial experience with an outpatient telemedicine COVID-19 risk assessment tool. In the absence of clear guidelines on the risk stratification and duration of monitoring of outpatient COVID-19, these data may help guide resource allocation, planning of current care structures, and future research.