This paper is in the following e-collection/theme issue:

COVID-19 Vaccination (146) Public Health Informatics (93) Infectious Diseases (non-STD/STI) (1575)

Published on in Vol 9 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/44465, first published .
Accuracy of Self-Reported COVID-19 Vaccination Status Compared With a Public Health Vaccination Registry in Québec: Observational Diagnostic Study

Accuracy of Self-Reported COVID-19 Vaccination Status Compared With a Public Health Vaccination Registry in Québec: Observational Diagnostic Study

Accuracy of Self-Reported COVID-19 Vaccination Status Compared With a Public Health Vaccination Registry in Québec: Observational Diagnostic Study

Authors of this article:

Patrick M Archambault1, 2, 3, 4 Author Orcid Image ;   Rhonda J Rosychuk5 Author Orcid Image ;   Martyne Audet2 Author Orcid Image ;   Rajan Bola6 Author Orcid Image ;   Shabnam Vatanpour7 Author Orcid Image ;   Steven C Brooks8 Author Orcid Image ;   Raoul Daoust9 Author Orcid Image ;   Gregory Clark10 Author Orcid Image ;   Lars Grant10, 11 Author Orcid Image ;   Samuel Vaillancourt12, 13 Author Orcid Image ;   Michelle Welsford14 Author Orcid Image ;   Laurie J Morrison15, 16 Author Orcid Image ;   Corinne M Hohl6 Author Orcid Image ;   Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) investigators17 ;   Network of Canadian Emergency Researchers18 ;   Canadian Critical Care Trials Group19
Article Authors Cited by (9) Tweetations (15) Metrics

    Original Paper

    1Department of Family Medicine and Emergency Medicine, Université Laval, Québec, QC, Canada

    2Centre de recherche intégrée pour un système de santé apprenant en santé et services sociaux, Centre intégré de santé et services sociaux de Chaudière-Appalaches, Lévis, QC, Canada

    3VITAM - Centre de recherche en santé durable, Québec, QC, Canada

    4Department of Anesthesiology and Intensive Care, Université Laval, Québec, QC, Canada

    5Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, AB, Canada

    6Department of Emergency Medicine, Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada

    7Center for Health Informatics, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada

    8Department of Emergency Medicine, Queen's University, Kingston, ON, Canada

    9Department of Family Medicine and Emergency Medicine, Université de Montréal, Montréal, QC, Canada

    10Department of Emergency Medicine, Faculty of Medicine, McGill University, Montréal, QC, Canada

    11Lady Davis Institute for Medical Research, Jewish General Hospital, Montréal, QC, Canada

    12Department of Emergency Medicine, Li Ka Shing Knowledge Institute, St Michael's Hospital, Unity Health Toronto, Toronto, ON, Canada

    13Department of Medicine, University of Toronto, Toronto, ON, Canada

    14Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, ON, Canada

    15Division of Emergency Medicine, Department of Medicine, University of Toronto, Toronto, ON, Canada

    16Department of Emergency Services, Sunnybrook Health Sciences Centre, Toronto, ON, Canada

    17University of British Columbia, Vancouver, BC, Canada

    18Canadian Association of Emergency Physicians, Ottawa, ON, Canada

    19Canadian Critical Care Trials Group, Toronto, ON, Canada

    Corresponding Author:

    Patrick M Archambault, MSc, MD

    Department of Family Medicine and Emergency Medicine

    Université Laval

    Pavillon Vandry, Avenue de la Médecine

    Québec, QC, G1V 0A6

    Canada

    Phone: 1 4189552552

    Email: patrick.archambault@fmed.ulaval.ca


    Background: The accuracy of self-reported vaccination status is important to guide real-world vaccine effectiveness studies and policy making in jurisdictions where access to electronic vaccine registries is restricted.

    Objective: This study aimed to determine the accuracy of self-reported vaccination status and reliability of the self-reported number of doses, brand, and time of vaccine administration.

    Methods: This diagnostic accuracy study was completed by the Canadian COVID-19 Emergency Department Rapid Response Network. We enrolled consecutive patients presenting to 4 emergency departments (EDs) in Québec between March 24, 2020, and December 25, 2021. We included adult patients who were able to consent, could speak English or French, and had a proven COVID-19 infection. We compared the self-reported vaccination status of the patients with their vaccination status in the electronic Québec Vaccination Registry. Our primary outcome was the accuracy of the self-reported vaccination status (index test) ascertained during telephone follow-up compared with the Québec Vaccination Registry (reference standard). The accuracy was calculated by dividing all correctly self-reported vaccinated and unvaccinated participants by the sum of all correctly and incorrectly self-reported vaccinated and unvaccinated participants. We also reported interrater agreement with the reference standard as measured by unweighted Cohen κ for self-reported vaccination status at telephone follow-up and at the time of their index ED visit, number of vaccine doses, and brand.

    Results: During the study period, we included 1361 participants. At the time of the follow-up interview, 932 participants reported at least 1 dose of a COVID-19 vaccine. The accuracy of the self-reported vaccination status was 96% (95% CI 95%-97%). Cohen κ for self-reported vaccination status at phone follow-up was 0.91 (95% CI 0.89-0.93) and 0.85 (95% CI 0.77-0.92) at the time of their index ED visit. Cohen κ was 0.89 (95% CI 0.87-0.91) for the number of doses, 0.80 (95% CI 0.75-0.84) for the brand of the first dose, 0.76 (95% CI 0.70-0.83) for the brand of the second dose, and 0.59 (95% CI 0.34-0.83) for the brand of the third dose.

    Conclusions: We reported a high accuracy of self-reported vaccination status for adult patients without cognitive disorders who can express themselves in English or French. Researchers can use self-reported COVID-19 vaccination data on the number of doses received, vaccine brand name, and timing of vaccination to guide future research with patients who are capable of self-reporting their vaccination data. However, access to official electronic vaccine registries is still needed to determine the vaccination status in certain susceptible populations where self-reported vaccination data remain missing or impossible to obtain.

    Trial Registration: Clinicaltrials.gov NCT04702945; https://clinicaltrials.gov/ct2/show/NCT04702945

    JMIR Public Health Surveill 2023;9:e44465

    doi:10.2196/44465

    Keywords

    electronic vaccination registry; self-reported vaccination status; COVID-19; accuracy; diagnostic study; interrater agreement 


    Background

    Since the beginning of the COVID-19 pandemic, vaccination campaigns have been instrumental in preventing mortality and morbidity related to COVID-19 worldwide [1]. With over 13 billion doses of vaccines administered worldwide [2], public health policy makers around the world have been relying on evidence from clinical trials to guide decision-making and develop vaccination strategies to prevent the spread and consequences of this pandemic [3]. In addition to evidence from controlled clinical trials, real-world evidence regarding vaccine effectiveness is important to guide policy making [4,5] and patient decision-making [6]. In particular, real-world evidence can help understand vaccine effectiveness against different disease severities in different susceptible populations often excluded from clinical trials [7], inform policy makers about the length of vaccine protection over time [8], and document rarer side effects [9]. Real-world data can also help understand the evolving prevalence and underlying causes of vaccine hesitancy over time [10] and shape future vaccination campaigns [11,12]. Understanding the role of herd immunity and how to promote vaccination are important questions that need reliable vaccination data [13-16]. Studying COVID-19 occupational hazards for professionals working with susceptible populations also requires timely and reliable access to vaccination data [17,18]. As new variants emerge, real-world evidence informs current and future strategies to protect against COVID-19 infection [19]. To measure real-world vaccine effectiveness and inform evidence-based health promotion strategies, researchers and policy makers alike need access to reliable vaccination data.

    Unfortunately, not all researchers and policy makers have access to official electronic public health vaccine registries [20-23], while others have substantial challenges in securing access to individual-level vaccination data even when they exist [24]. Many researchers and policy makers must then rely on patients’ self-reported status to measure real-world vaccine effectiveness, and if this is inaccurate, it may bias vaccine effectiveness estimates in test-negative designs [25]. For these studies, not only are accurate data about vaccination status key, but, most importantly, reliable data about when vaccine doses were administered are essential to determine the effectiveness in preventing infections and health service use such as emergency department (ED) visits [26].

    Objective

    To inform the conduct of real-world vaccine effectiveness studies using patient self-reported vaccine information and to support the development of evidence-based public health promotion policies, this study aimed to compare the accuracy of self-reported vaccination status for COVID-19 with data from an official electronic vaccination registry in the province of Québec, Canada [27].


    Study Design and Setting

    The Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN; pronounced “SED-rin”) is a national collaboration created to harmonize data collection related to COVID-19 in over 50 EDs across 8 provinces in Canada (British Columbia, Alberta, Manitoba, Saskatchewan, Ontario, Québec, New-Brunswick, and Nova Scotia) [28]. The CCEDRRN has created a national observational database of suspected and confirmed patients with COVID-19 who presented to acute care EDs. Data from this network are used in national studies to inform decision-making, emerging treatment guidelines, resource allocation, and planning in response to the pandemic. As of April 2023, CCEDRRN had captured data on over 208,000 patients across Canada [29]. In December 2020, CCEDRRN initiated data collection on vaccination status that supports this study.

    This CCEDRRN study is a nested diagnostic accuracy study within the larger CCEDRRN cohort that was reported using the 2015 Standards for Reporting of Diagnostic Accuracy studies [30]. This study only used CCEDRRN data collected from 4 sites in the province of Québec because access to government electronic vaccine registries in the other provinces of Canada was unavailable. These 4 Québec CCEDRRN sites only collected data on patients with a positive COVID-19 test. The study enrolled consecutive consenting patients presenting to these 4 participating EDs between March 24, 2020, and December 25, 2021.

    Vaccination in Québec started on December 14, 2020, with long-term care residents and health care workers [31]. On March 1, 2021, vaccination in the general population started in sequence based on their age group. On April 14, 2021, more than 2 million Québecers had received at least 1 dose, representing a quarter of the population. On May 18, 2021, more than half of the population had received at least 1 dose, and on June 6, more than 75% of the population aged >12 years had received at least 1 dose. On July 5, 2021, Québec reduced the dosing interval to 4 weeks to maximize 2-dose coverage, such that by September 30, 2021, a total of 75% of adults were considered fully vaccinated. On September 28, 2021, a third booster dose was recommended for residents in long-term care and assisted living. Between November 16, 2021, and December 29, 2021, the booster dose was progressively authorized for specific high-risk subgroups and then in sequence for the general population based on age groups. After each dose was administered in Québec, vaccinators were required to document the vaccine brand and the time of administration directly within the provincial electronic registry [32]. The identities of the vaccinated patients were validated using at least 2 personal identification questions. Another key public health measure to consider for this study was the mandate on September 1, 2021, to implement a mandatory vaccination passport that contained information about the number of doses, vaccine brand names, and vaccination dates to access certain public services in Québec [31].

    Ethics Approval and Informed Consent

    The ethics committee at the Centre intégré de santé et de services sociaux de Chaudière-Appalaches approved this project in February 2021 (MP-23-2021-766) with a waiver of informed consent for retrospective data collected; however, informed consent was obtained during telephone follow-up for all participants. The participants did not receive any financial compensation.

    Participants

    Patients were included if they had a confirmed COVID-19 infection (≥1 nucleic acid amplification tests positive for SARS-CoV-2 from specimens collected during the index ED visit, 14 days before the ED visit in the community, or within 14 days after admission) [28]. Included patients were aged ≥18 years, were able to communicate in French or English, and consented independently to telephone follow-up and the use of their health data. All included patients sought care at any of the 4 participating Québec sites (Hôtel-Dieu de Lévis, Hôpital Sacré-Coeur de Montréal, Royal Victoria Hospital, and the Montreal General Hospital). Patients were excluded if they were unable to provide consent (eg, had died since the ED encounter), were unable to communicate in French or English, could not be reached after 5 attempts, or did not have a record in the Québec Vaccination Registry (eg, out-of-province patients).

    Data Collection and Outcome Measures

    Trained research assistants collected data for our study from different sources and at different time periods. First, starting on July 9, 2020, we used a retrospective chart review to extract demographics, the time and date of ED visits, chief complaint, laboratory tests, treatments, admission status, and comorbidities. We retrospectively collected data for the index visits of all patients and any subsequent ED visits at least 30 days after the index visit. Second, starting on March 24, 2021, we initiated data collection from the Québec Vaccination Registry and phone follow-ups. A single research assistant at each site was designated to access the Québec Vaccination Registry to determine patients’ vaccination status, vaccine brands received, the total number of doses, and dates of vaccination. Separate research assistants at each site were trained to only conduct telephone follow-ups. They were blinded to the reason for collecting self-reported vaccination data and did not access the information collected from the Québec Vaccination Registry. These research assistants contacted patients by telephone to obtain consent to use their data; to obtain additional sociodemographic data not available in their medical records (eg, income, self-identified race, and education level); and to collect their self-reported vaccination data (vaccination status, vaccine brands received, the total number of doses, and dates of vaccination). The median (IQR) time to contact the patients was 202 (108-352) days after the ED index visit. As patients were called at most 30 days after their data were consulted from the Québec Vaccination Registry, we only used self-reported vaccination data from before the date we consulted the Québec Vaccination Registry.

    In accordance with CCEDRRN’s national data management plan, after assigning a unique study identifier for each participant, deidentified data were collected and stored in REDCap (Research Electronic Data Capture; Vanderbilt University) at the University of British Columbia. After data verification, registry data were uploaded into CaraSpace, a secure private cloud for the storage and analysis of privacy-sensitive data [28]. Analysts accessed the cloud space via an encrypted virtual private network through a firewall and 2-factor authentication.

    The primary outcome of this study was the accuracy of self-reported COVID-19 vaccination status at the time of the telephone follow-up compared with their vaccination status in the Québec Vaccination Registry. We considered the self-reporting of vaccination status via telephone as the index test and the Québec Vaccination Registry as the reference standard. The index test had 2 possible results: “no vaccine dose” (self-reported as not vaccinated) or “at least one vaccine dose” (self-reported as vaccinated). We calculated the accuracy of the self-reported vaccination status as follows:

    Accuracy = (correctly self-reported as vaccinated + correctly self-reported as not vaccinated) / (correctly self-reported as vaccinated + correctly self-reported as not vaccinated + incorrectly self-reported as vaccinated + incorrectly self-reported as not vaccinated) (1)

    We also compared the clinical and demographic characteristics of patients with concordant and discordant self-reported vaccination status compared with the Québec Vaccination Registry.

    Secondary outcomes were the sensitivity, calculated as

    (correctly self-reported as vaccinated / [correctly self-reported as vaccinated + incorrectly self-reported as not vaccinated]) (2)

    and specificity, calculated as

    (correctly self-reported as not vaccinated / [correctly self-reported as not vaccinated + incorrectly self-reported as vaccinated]) (3)

    of the self-reported vaccination status and vaccination status in the Québec Vaccination Registry. Relaxing the reference standard assumption about the Québec Vaccination Registry, we also measured the interrater agreement between self-reported vaccination status and vaccination status in the Québec Vaccination Registry, interrater agreement between self-reported COVID-19 vaccine brand received and the brand recorded in the Québec Vaccination Registry, and interrater agreement between self-reported number of COVID-19 vaccine doses received compared with the number of doses recorded in the vaccine registry.

    An additional secondary outcome was the interrater agreement between vaccination status at the time of the index ED visit determined retrospectively by using the self-reported vaccination date compared with the vaccination status at the time of the index ED visit determined by the vaccination date in the Québec Vaccination Registry (Multimedia Appendix 1). Vaccination status was defined as fully, partially, and not vaccinated at the time of the index ED visit based on the number of doses and time since the last dose received to account for likely waning protection [33,34] (Multimedia Appendix 1). We defined fully vaccinated patients at the time of the index ED visit as patients who received ≥2 doses ≥14 days before the index ED visit but less than 6 months since the last dose. We defined partially vaccinated patients as having received only 1 dose before the index ED visit or having received their second or third dose <14 days before the index ED visit or having received their second or third dose >6 months before the index ED visit. Patients were considered not vaccinated if no doses had been received before their index ED visit. Patients who could not recall the date of their last vaccination were excluded from this analysis.

    Analysis

    The vaccination data in the Québec Vaccination Registry were considered the reference standard. As the interest was to compare the same information collected by telephone as in the Québec Vaccination Registry, the sample size calculations were based on a 1-sided binomial test for sensitivity [35,36]. A sample size of 613 patients was needed to achieve a 95% CI for sensitivity of half width 0.02 assuming a sensitivity of 0.98 and prevalence of at least 0.40.

    Data were summarized by using descriptive statistics (eg, mean [SD], counts, and percentage). Accuracy, sensitivity, and specificity for COVID-19 vaccination status reported by patients compared with the reference standard (ie, the Québec Vaccination Registry) were calculated with associated 95% CIs. Secondary analyses relaxed the reference standard assumption and provided interrater agreement using the unweighted Cohen κ coefficient [37]. We chose to use the unweighted Cohen κ because we had 2 raters (ie, self-report by participant and report from the Québec Vaccination Registry), and we wanted disagreements to carry the same weight. All analyses were conducted in R (R Foundation for Statistical Computing) using the rms and psych packages [38-40]. To ensure patient privacy, a cell-size restriction policy prohibited us from reporting counts of <5.


    Participant Characteristics

    During the study period, there were 5912 adult patients who presented to the Québec study sites, and 1482 met the inclusion criteria; of them, 1361 (23.02%) patients had the required data from follow-up and available in the Québec Vaccination Registry (Figure 1). Among the 4430 excluded patients, 1596 (36.03%) did not consent to participate or were not able to consent. There were also 62 (1.05%) patients who did not have any vaccination record in the Québec Vaccination Registry because they were patients from outside the province or did not have a provincial health insurance number. The baseline participant characteristics are shown in Table 1. Patients who did not or could not consent were slightly older than patients who participated (age: mean 58.6, SD 21.3 years vs mean 55.1, SD 17.4 years, respectively) and were more likely to be male individuals (nonconsenting male individuals: 871/1596, 54.57%, vs consenting male individuals: 686/1361, 50.4%). Three-quarters (1010/1361, 74.21%) of the patients presented to the ED during the second and third waves of the pandemic (August 23, 2020, to July 17, 2021).

    ‎
    Figure 1. Flow diagram of patients in the study.
    Table 1. Sociodemographic and clinical characteristics of study participants whose index visit occurred in a Québec emergency department between March 24, 2020, and December 25, 2021 (n=1361).

    		Study participants
    Age (years), mean (SD)55.1 (17.4)
    Age group (years), n (%)

    		<2445 (3.3)

    		25-39230 (16.9)

    		40-64685 (50.3)

    		65-79272 (20)

    		≥80129 (9.5)
    Sex, n (%)

    		Male686 (50.4)

    		Female675 (49.6)

    		Intersex0 (0)
    Site, n (%)

    		Hôtel-Dieu de Lévis293 (21.5)

    		Royal Victoria501 (36.8)

    		Montreal General132 (9.7)

    		Sacré-Coeur de Montréal435 (32)
    Index visit occurred during, n (%)

    		Wave 1 (March 1, 2020, to August 22, 2020)207 (15.2)

    		Wave 2 (August 23, 2020, to March 20, 2021)725 (53.3)

    		Wave 3 (March 21, 2021, to July 17, 2021)285 (20.9)

    		Wave 4 to early 5 (July 18, 2021, to December 31, 2021)144 (10.6)
    Index visit occurred before the start of vaccination campaign in Québec(December 14, 2020),n (%)

    		Prevaccination campaign490 (36)

    		Postvaccination campaign871 (64)
    Time elapsed from index visit to telephone follow-up (days), median (IQR)202 (108-352)
    Top 10 comorbid conditions, n (%)

    		Hypertension440 (32.3)

    		Dyslipidemia314 (23.1)

    		Diabetes241 (17.7)

    		Hypothyroidism160 (11.8)

    		Asthma144 (10.6)

    		Psychiatric condition or mental health diagnosis113 (8.3)

    		Coronary artery disease99 (7.3)

    		Past malignancy73 (5.4)

    		Chronic neurological conditions66 (4.8)

    		Rheumatologic disorder66 (4.8)
    EDa disposition, n (%)

    		Discharged home740 (54.4)

    		Admitted576 (42.3)

    		Transferred to other hospital38 (2.8)

    		Transfer to long-term care or rehab<5b (<0.4)

    		Other6 (0.4)

    aED: emergency department.

    bCells with fewer than 5 individuals could not be presented as requested by the ethics committee.

    Vaccination Status

    Over two-thirds (932/1361, 68.55%) of the participants reported at least 1 dose of a COVID-19 vaccine at the time of their follow-up interview (Table 2). The most administered vaccine type was Pfizer-BioNTech (689/1361, 50.6%) followed by Moderna (158/1361, 11.6%).

    Table 2. Québec Vaccination Registry information and self-reported vaccination status during the initial telephone follow-up of participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=1361).

    		Québec Vaccination Registry, n (%)Self-reported vaccination status, n (%)
    Vaccination status

    		Yes (at least 1 dose)913 (67.1)932 (68.5)

    		No (no dose given)448 (32.9)429 (31.5)

    		Unknown or not assessed0 (0)0 (0)
    First vaccine brand

    		Pfizer-BioNTech707 (51.9)689 (50.6)

    		Moderna162 (11.9)158 (11.6)

    		AstraZeneca or Covishield42 (3.1)42 (3.1)

    		Johnson & Johnson<5 (<0.4)<5a (<0.4)

    		Unknown, but vaccinated0 (0)37 (2.7)

    		Unknown or not vaccinated448 (32.9)433 (31.8)
    Number of vaccines doses

    		0448 (32.9)429 (31.5)

    		1366 (26.9)368 (27)

    		2501 (36.8)511 (37.5)

    		346 (3.4)53 (3.9)

    aCells with fewer than 5 individuals could not be presented as requested by the ethics committee.

    Self-Reported Vaccination Status Accuracy and Interrater Agreement Between Self-Reported Vaccination Data and Québec Vaccination Registry Data

    We reported high measures of accuracy (96%), sensitivity (98%), and specificity (92%) for self-reported vaccination status compared with the Québec Vaccination Registry as the reference standard (Tables 3 and 4). Moreover, Cohen κ was 0.91 (95% CI 0.89-0.93), indicating excellent agreement between patient self-report and the Québec Vaccination Registry.

    With respect to vaccine brand type, Cohen κ for dose 1 brand was 0.80 (95% CI 0.75-0.84) indicating strong agreement (Table 5). However, this estimate decreased with subsequent doses: Cohen κ was 0.76 (95% CI 0.70-0.83) and 0.59 (95% CI 0.34-0.83) for dose 2 and dose 3 brands, respectively.

    Most patients reported receiving 1 dose (368/1361, 27.04%) or 2 doses (511/1361, 37.56%) of COVID-19 vaccines at the time of the follow-up interview (Table 6). Only 3.89% (53/1361) of patients reported receiving 3 doses. One-third (429/1361, 31.52%) of the patients reported not having received the COVID-19 vaccine at the time of follow-up. Cohen κ for the number of self-reported doses compared with the Québec Vaccination Registry was very strong, at 0.89 (95% CI 0.87-0.91).

    Cohen κ for self-reported vaccination status at the time of ED index visit was 0.85 (95% CI 0.77-0.92). However, many patients (774/1361, 56.87%) could not recall the exact date of their latest vaccine dose and had to be excluded from this analysis. Of those who did recall their vaccination dates, most (538/587, 91.7%) reported not being fully vaccinated at the time of the ED visit, with only 16 (2.7%) patients being fully vaccinated (Table 7).

    The demographic, clinical, and sociocultural characteristics of patients with discordant (n=53) and concordant (n=1308) vaccination statuses are shown in Multimedia Appendix 2. Participants with discordant self-reported vaccination status compared to those with concordant self-reported vaccination status were younger (age: mean 51.1, SD 20.5 years, vs mean 55.3, SD 17.3 years, respectively), had lower self-reported income (5/53, 9% reported current income >CAD $73,701 (US $56,693) per year vs 294/1308, 22.48%), had lower education level (5/53, 9% with a university degree vs 444/1308, 33.94%), and had a lower vaccination rate according to the Québec Vaccination Registry (17/53, 32% vs 896/1308, 68.5%). The median time elapsed between the ED index visit and the telephone follow-up to determine the patient’s self-reported vaccination status was similar for discordant (median 94, IQR 53-191 days) and concordant cases (median 126, 62-196 days). Immigration status, race, and ethnicity did not influence the accuracy of our results.

    Table 3. Confusion matrix comparing self-reported vaccination status with the Québec Vaccination Registry data for all participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=1361).
    Self-reported vaccination status (index test)Vaccination status in the Québec Vaccination Registry (reference standard), n

    		YesNo
    Yes89636
    No17412
    Table 4. Sensitivity, specificity and accuracy of self-reported vaccination status and interrater agreement of self-reported vaccination status compared with the Québec Vaccination Registry for all participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=1361).

    		Measure (95% CI)
    Sensitivity0.98 (0.97-0.99)
    Specificity0.92 (0.89-0.94)
    Accuracy0.96 (0.95-0.97)
    Cohen κ0.91 (0.89-0.93)
    Table 5. Interrater agreement for self-reported brand by dose compared with the Québec Vaccination Registry for all participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021.
    Dose and self-reported vaccine brandQuébec Vaccination Registry vaccine brand, nCohen κa (95% CI)

    		PfizerModernaAstra Zeneca or CovishieldJohnson & Johnson
    10.80 (0.75-0.84)

    		Pfizer64519<50

    		Moderna1113700

    		AstraZeneca or Covishield<50380

    		Johnson & Johnson000<5

    		Unknown, but vaccinated296<50
    20.76 (0.70-0.83)

    		Pfizer38619<50

    		Moderna58400

    		AstraZeneca or Covishield0090

    		Unknown, but vaccinated16500
    30.59 (0.34-0.83)

    		Pfizer24<500

    		Moderna<5700

    		Unknown, but vaccinated<5<500

    aCohen κ calculated without participants who answered “Unknown, but vaccinated” were 0.89 (95% CI 0.86-0.93) for first dose, 0.85 (95% CI 0.80-0.91) for second dose, and 0.70 (95% CI 0.43-0.97) for third dose.

    Table 6. Interrater agreement for self-reported number of doses compared with the Québec Vaccination Registry for participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=1361).a
    Self-reported number of dosesNumber of doses in the Québec Vaccination Registry, n

    		0123
    041278<5
    120336120
    213234696
    3<501238

    aCohen κ: 0.89 (95% CI 0.87-0.91).

    Table 7. Interrater agreement for self-reported vaccination status at emergency department index visit date versus the Québec Vaccination Registry for participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=587).a
    Self-reported vaccination status determined using the date of latest vaccination dose in relation to the EDb visit datecQuébec Vaccination Registry vaccination status determined in relation to the ED index visit dated, n

    		Not vaccinatedPartially vaccinatedFully vaccinated
    Not vaccinated52756
    Partially vaccinated<5290
    Fully vaccinated0016

    aCohen κ: 0.85 (95% CI 0.77-0.92).

    bED: emergency department.

    cCategories of vaccination status at the time of ED index visit (fully, partially, and not vaccinated) for this analysis are defined in Multimedia Appendix 1.

    dA total of 774 (56.9%) participants were excluded from this analysis because they could not recall the date of their last dose.


    Principal Findings

    We found that the participants’ self-reported vaccination status was highly accurate at the time of the initial follow-up telephone call. The interrater agreement between the self-reported vaccine number of doses and the number of doses recorded in the vaccine registry was also strong. The agreement between self-reported vaccination status at the index visit and the patient’s vaccination status as determined by the registry was strong when patients could provide vaccination dates. However, many patients could not recall their vaccination dates. Although there was a strong and moderate agreement for the self-reported vaccine brand for the first and second doses, respectively, the agreement dropped to weak for the vaccine brand of the third dose. A few discordant cases concerning the accuracy of patients’ self-report of receiving at least 1 dose were found. The discordant cases involved patients of younger age groups, lower self-reported pre– and post–COVID-19 income, lower education levels, and lower vaccination rates in the Québec Vaccination Registry.

    Our results are similar to those of a study conducted in the United States by Stephenson et al [41] that showed high levels of agreement between self-reported vaccination status and official computerized vaccination registry data (unweighted κ=0.9127, 95% CI 0.9109-0.9145). Similar to our study, they also found that discordant self-reported vaccination data were associated with lower education levels and unemployment. It could be that both studies found that lower education is associated with lower accuracy of self-reported vaccination status because information provided about vaccination status tends to be complex [42]. Comprehending and remembering this information requires certain health literacy skills [43]. Rolnick et al [44] also found that education and income level could influence the accuracy of self-reported vaccination status for 8 different vaccines (pneumococcal, influenza, tetanus diphtheria, tetanus diphtheria pertussis, human papillomavirus, hepatitis A, hepatitis B, and herpes zoster).

    Contrary to our study, the study by Stephenson et al [41] found that the female sex increased the risk of discordant self-reported vaccination status. Sex and gender are important factors in understanding the real-world effectiveness of vaccines and postmarketing safety monitoring [45,46]. Sex and gender differences in vaccine hesitancy and immunization outcomes have been observed for COVID-19 and other vaccine-preventable diseases, with women typically reporting more vaccine hesitancy and higher adverse reactions than men [47-49]. Our results support the accuracy of self-reported vaccination data from both men and women.

    Ulrich et al [50] studied the accuracy of the self-reported vaccination status by university students in a 2-dose meningococcal vaccine campaign. This study showed good self-reported vaccination status accuracy at 2 months (86%) and poor self-reported vaccination status at 20 months (69%). King et al [51] also reported a decrease in the agreement of patient self-reported vaccination status compared with a registry for prior season influenza vaccination (κ=0.86) versus the current season (κ=0.95). Although our study did not observe a time-dependent increase in discordant self-reported vaccination status, we did find that patients reporting a higher number of doses were more likely to forget the vaccine brand received and did not remember the dates of their latest vaccination dose. In patients who did remember the vaccine brand for each dose, the agreement with the registry was high. This limits the use of self-reported vaccination brand names for studies comparing the effectiveness of multiple doses of different types of vaccines [52].

    The reliability of data on the timing of vaccination before being tested for a viral infection such as SARS-CoV-2 is essential for test-negative design vaccine effectiveness studies [53]. Our study was interested in determining the reliability of the self-reported vaccination data (ie, the date of their last vaccination and the total number of doses) used to retrospectively determine their vaccination status (not vaccinated, partially vaccinated, or fully vaccinated) at the time of their index ED visit. For patients who could remember their last vaccination date, our study found high levels of agreement between the self-reported vaccination status determined at the ED index visit date and the Québec Vaccination Registry. Unfortunately, more than half of our participants could not recall the date of their last dose, making this measure subject to an important selection bias when used in test-negative designs. This is an argument to support wider implementation and easy access to rigorous vaccination registries in all jurisdictions [21]. This is especially important to improve the care, research, and support of vulnerable populations presenting to the ED. In Canada, access to these registries is currently restricted by data custodians and administrative barriers that hinder timely knowledge creation and decision-making for a rapid-learning health system [54,55]. Although a study on the accuracy of self-reported influenza and pneumococcal vaccination status in a high-risk inflammatory bowel disease population showed high accuracy for self-reported influenza (97%) and pneumococcal (84%) vaccination status [56], our study did not find any difference between high-risk groups with different comorbidities. In concordance with Laurence et al [57], who found that older patients accurately self-reported their influenza (96%) and pneumococcal (87%) vaccine status, we also found that older adults adequately self-reported their COVID-19 vaccine status. Surprisingly, we found a higher proportion of discordant self-reported COVID-19 vaccination statuses among patients aged ≤34 years. This could be associated with the fact that COVID-19 presents a milder course in younger patients [57,58] and increased vaccination hesitancy in this age group [59-61].

    There are several potential explanations for the high accuracy of self-reported vaccination status at telephone follow-up and moderate to strong agreement between the self-reported number of doses and the self-reported vaccination status determined at the time of the ED index visit. First, this high accuracy and strong agreement could have been caused by the implementation of Québec’s mandatory vaccine passport, which registered dates, brands, and doses as a requirement to access the workplace and most commercial services. Although we did not ask participants to consult their passports before answering our phone questionnaire, our study personnel did not prevent participants from consulting their passports during our telephone follow-ups. Participants’ recall of their own vaccination data could also have been improved by having to consult their passport frequently, as both federal and provincial governments in Canada granted substantial benefits to fully vaccinated citizens, such as lifting travel restrictions and access to event venues, fitness centers, and restaurants via the use of a vaccination passport [62]. Second, the impact of COVID-19 on the Canadian population has been substantial, and it may have improved the recall and the accuracy of the self-reported vaccination status. Third, the COVID-19 vaccination campaign; the omnipresence of the COVID-19 pandemic in the lay press; the many frequently reported statistics portraying the evolution of this pandemic (cases, hospitalizations, deaths, and population vaccination coverage); and the vigorous public debate about vaccination public health mandates may have also raised citizens’ awareness of their own vaccination status. Fourth, this study only included patients with confirmed COVID-19 infections. This could have improved the recall of participants compared with patients who were not infected with COVID-19.

    Our results indicate that when access to official electronic vaccination registries is not available, the reliance on patients’ self-reported COVID-19 vaccination status, vaccine brand received, number of doses, and vaccination dates (when provided) can be used for research purposes. This is a substantial finding for researchers in jurisdictions that do not have access to electronic vaccine registries [20,63] and for others that have substantial challenges in securing timely access to individual-level vaccination data even when they exist [24]. Vaccine registries can also suffer from missing data for patients who have received their vaccines in different jurisdictions (as shown in our results), misclassification bias because of human error, and the timeliness of data entry into electronic vaccination registries [64]. Relying on timely and highly accurate self-reported vaccination status reduces the potential underestimation of real-world test-negative vaccine effectiveness studies introduced by vaccination-reporting bias [25]. Knowing the sensitivity and specificity of the self-reported vaccination status is also important to correct the estimation of vaccine effectiveness based on the observed risks for vaccination status misclassification [65].

    This study has limitations. First, although we assumed that the Québec Vaccination Registry was the reference standard, we cannot rule out the possibility that some vaccination data were entered incorrectly. Second, our results do not apply to patients with cognitive disorders who cannot provide consent. This represents a selection bias that we could not avoid because of ethical constraints. Patients who consented to participate in our study were also more likely to be younger and female than those who did not consent or who could not consent. Understanding the impact of the pandemic on patients who cannot consent to provide their vaccination status because of cognitive issues will still necessitate access to linked vaccination registry data. Access to a vaccination registry also avoids systematic differences in the characteristics of consenting patients versus nonconsenting patients in studies that rely on self-reporting. Third, although we had many landed immigrants constituting an ethnically diverse sample, our results are potentially not applicable to patients who do not speak English or French. Other studies have shown that the self-reported vaccination status is inaccurate in certain susceptible and hard-to-follow-up racial and ethnic groups [44,50]. Although we had to exclude patients who could not speak English or French, immigration and race and ethnicity did not influence the accuracy of our results. Fourth, our study did not verify the accuracy of vaccination status provided at the time of ED care. Future prospective studies at the point of care would be needed to validate the accuracy of self-reported data for use in guiding clinical decision-making.

    Conclusions

    Our study found that relying on self-reported COVID-19 vaccination status is accurate for adult patients without cognitive disorders who can express themselves in English or French. Researchers can use self-reported COVID-19 vaccination data on the number of doses received, vaccine brand name, and timing of vaccination to guide future research with certain groups of patients who are capable of self-reporting their vaccination data. However, access to official electronic vaccine registries is still needed to determine the vaccination status in certain susceptible populations where self-reported vaccination data remain missing or impossible to obtain.

    Acknowledgments

    The authors would like to acknowledge the assistance of Ms Amber Cragg and Ms Josie Kanu in the preparation of this manuscript. The authors would like to thank the University of British Columbia clinical coordinating center staff; legal, ethics, privacy, and contract staff; and the research staff at each of the participating institutions in the network outlined in the attached supplement (Multimedia Appendix 3). In particular, the authors would like to thank Chantal Lanthier and David Iannuzzi for their support in collecting the data for this study. The network would not exist today without the dedication of these professionals. The authors would also like to thank all the patient partners who shared their lived experiences and perspectives to ensure that the knowledge we cocreate addresses the concerns of patients and the public. Creating the largest network of collaboration across Canadian emergency departments would not have been possible without the tireless efforts of emergency department chiefs, research coordinators, and research assistants at participating sites. Finally, the authors would like to express their most humble and sincere gratitude to all their colleagues in medicine, nursing, and the allied health professions who have been on the frontlines of this pandemic from day 1, staffing ambulances, emergency departments, intensive care units, and hospitals bravely facing the risks of COVID-19 to look after fellow citizens and after one another. The authors would like to dedicate this network to them. This network was funded by the Canadian Institutes of Health Research (447679, 464947, and 466880); Ontario Ministry of Colleges and Universities (C-655-2129); Saskatchewan Health Research Foundation (5357); Genome BC (COV024 and VAC007); Fondation du CHU de Québec (Octroi No. 4007); and Sero-Surveillance and Research (COVID-19 Immunity Task Force Initiative), who provided peer-reviewed funding. The BC Academic Health Science Network and BioTalent Canada provided non–peer-reviewed funding. These organizations are not for profit and had no role in conducting the study, analysis, or manuscript preparation.

    The Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) consisted of the following members: Patrick Fok, Hana Wiemer, Samuel Campbell, Kory Arsenault, Tara Dahn, Kavish Chandra, Patrick Archambault, Joel Turner, Lars Grant, Greg Clark, Raoul Daoust, Éric Mercier, Sébastien Robert, Laurie Morrison, Steven Brooks, Ivy Cheng, Jeffrey Perry, Rob Ohle, Justin Yan, Rohit Mohindra, Megan Landes, Michelle Welsford, Tomislav Jelic, Phil Davis, Andrew McRae, Brian Rowe, Katie Lin, Stephanie VandenBerg, Jake Hayward, Jaspreet Khangura, Corinne Hohl, Daniel Ting, Maja Stachura, Frank Scheuermeyer, Balijeet Braar, John Taylor, Ian Martin, Sean Wormsbecker and Lee Graham.

    Data Availability

    Data are available on reasonable request. For investigators who wish to access Canadian COVID-19 Emergency Rapid Response Network data, proposals may be submitted to the network for review and approval by the network’s peer review publication committee, the data access and management committee, and the executive committee as per the network’s governance. Information regarding submitting proposals and accessing data may be found on the Canadian COVID-19 Emergency Department Rapid Response Network website [9].

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Definitions used to determine the interrater agreement between the self-reported vaccination status and the vaccination status determined by the Québec Vaccination Registry at the time of emergency department index visit.

    PNG File , 288 KB
    Multimedia Appendix 2

    Characteristics of all participants by concordant or discordant vaccination status.

    DOCX File , 32 KB
    Multimedia Appendix 3

    Contributors to the Canadian COVID-19 Emergency Department Rapid Response Network.

    DOCX File , 50 KB
    1. Coccia M. Optimal levels of vaccination to reduce COVID-19 infected individuals and deaths: a global analysis. Environ Res. Mar 2022;204(Pt C):112314. [FREE Full text] [CrossRef] [Medline]
    2. COVID-19 map. Johns Hopkins Coronavirus Resource Center. URL: https://coronavirus.jhu.edu/map.html [accessed 2023-04-06]
    3. Coccia M. COVID-19 vaccination is not a sufficient public policy to face crisis management of next pandemic threats. Public Organ Rev (forthcoming). Oct 17, 2022 [FREE Full text] [CrossRef]
    4. Patel MK, Bergeri I, Bresee JS, Cowling BJ, Crowcroft NS, Fahmy K, et al. Evaluation of post-introduction COVID-19 vaccine effectiveness: summary of interim guidance of the World Health Organization. Vaccine. Jul 05, 2021;39(30):4013-4024. [FREE Full text] [CrossRef] [Medline]
    5. Patel MM, Jackson ML, Ferdinands J. Postlicensure evaluation of COVID-19 vaccines. JAMA. Nov 17, 2020;324(19):1939-1940. [CrossRef] [Medline]
    6. Stacey D, Ludwig C, Archambault P, Smith M, Taljaard M, Carley M, et al. Decisions and decisional needs of Canadians from all provinces and territories during the COVID-19 pandemic: population-based cross-sectional surveys. JMIR Public Health Surveill. Mar 21, 2023;9:e43652. [FREE Full text] [CrossRef] [Medline]
    7. Shea LL, Becker A, Lee BK, Miller KK, Cooper D, Anderson K, et al. Self-reported COVID-19 vaccination acceptance and hesitancy among autistic adults. Vaccine. May 26, 2022;40(24):3288-3293. [FREE Full text] [CrossRef] [Medline]
    8. Sasanami M, Fujimoto M, Kayano T, Hayashi K, Nishiura H. Projecting the COVID-19 immune landscape in Japan in the presence of waning immunity and booster vaccination. J Theor Biol. Feb 21, 2023;559:111384. [FREE Full text] [CrossRef] [Medline]
    9. Wesselink AK, Hatch EE, Rothman KJ, Wang TR, Willis MD, Yland J, et al. A prospective cohort study of COVID-19 vaccination, SARS-CoV-2 infection, and fertility. Am J Epidemiol. Jul 23, 2022;191(8):1383-1395. [FREE Full text] [CrossRef] [Medline]
    10. Dubé E, Laberge C, Guay M, Bramadat P, Roy R, Bettinger J. Vaccine hesitancy: an overview. Hum Vaccin Immunother. Aug 2013;9(8):1763-1773. [FREE Full text] [CrossRef] [Medline]
    11. Yoon H, Choi BY, Seong WJ, Cho GJ, Na S, Jung YM, et al. COVID-19 vaccine acceptance during pregnancy and influencing factors in South Korea. J Clin Med. Sep 28, 2022;11(19):5733. [FREE Full text] [CrossRef] [Medline]
    12. Coccia M. Improving preparedness for next pandemics: max level of COVID-19 vaccinations without social impositions to design effective health policy and avoid flawed democracies. Environ Res. Oct 2022;213:113566. [FREE Full text] [CrossRef] [Medline]
    13. Etienne D, Archambault P, Aziaka D, Chipenda-Dansokho S, Dubé E, Fallon CS, et al. A personalized avatar-based web application to help people understand how social distancing can reduce the spread of COVID-19: cross-sectional, observational, pre-post study. JMIR Form Res. Apr 25, 2023;7:e38430. [FREE Full text] [CrossRef] [Medline]
    14. Hakim H, Bettinger JA, Chambers CT, Driedger SM, Dubé E, Gavaruzzi T, et al. A web application about herd immunity using personalized avatars: development study. J Med Internet Res. Oct 30, 2020;22(10):e20113. [FREE Full text] [CrossRef] [Medline]
    15. Akinbami LJ, Kruszon-Moran D, Wang CY, Storandt RJ, Clark J, Riddles MK, et al. SARS-CoV-2 serology and self-reported infection among adults - national health and nutrition examination survey, United States, August 2021-May 2022. MMWR Morb Mortal Wkly Rep. Dec 02, 2022;71(48):1522-1525. [FREE Full text] [CrossRef] [Medline]
    16. Miao Y, Zhang W, Li Y, Wu J, Xu D, Gu J, et al. Association between lifestyle and COVID-19 vaccination: a national cross-sectional study. Front Public Health. Oct 11, 2022;10:918743. [FREE Full text] [CrossRef] [Medline]
    17. Kronfli N, Dussault C, Maheu-Giroux M, Halavrezos A, Chalifoux S, Park H, et al. Importance of occupation for SARS-CoV-2 seroprevalence and COVID-19 vaccination among correctional workers in Quebec, Canada: a cross-sectional study. Front Public Health. Nov 09, 2022;10:1021871. [FREE Full text] [CrossRef] [Medline]
    18. Gormley MA, Nolan MS, Heo M, Litwin AH, Alier A, Daguise V. Differences in motivating factors for SARS-CoV-2 vaccination and perceptions of infection risk among healthcare and EMS personnel in South Carolina. South Med J. Jun 2022;115(6):381-387. [FREE Full text] [CrossRef] [Medline]
    19. Coccia M. Sources, diffusion and prediction in COVID-19 pandemic: lessons learned to face next health emergency. AIMS Public Health. Mar 02, 2023;10(1):145-168. [FREE Full text] [CrossRef] [Medline]
    20. Namageyo-Funa A, Samuel A, Bloland P, Macneil A. Considerations for the development and implementation of electronic immunization registries in Africa. Pan Afr Med J. May 29, 2018;30:81. [FREE Full text] [CrossRef] [Medline]
    21. Benati I, Coccia M. Global analysis of timely COVID-19 vaccinations: improving governance to reinforce response policies for pandemic crises. Int J Health Gov. May 31, 2022;27(3):240-253. [FREE Full text] [CrossRef]
    22. Coccia M. Pandemic prevention: lessons from COVID-19. Encycl. May 31, 2021;1(2):433-444. [FREE Full text] [CrossRef]
    23. González MJ, Rodríguez FS. Self-reported COVID-19 vaccination coverage in Venezuela: results of national sample surveys (June-December 2021). Gac Med Caracas. Jul 15, 2022;130(2S):S459-S469. [FREE Full text] [CrossRef]
    24. McRae AD, Archambault P, Fok P, Wiemer H, Morrison LJ, Herder M, et al. Canadian COVID-19 Emergency Department Rapid Response Network Investigators. Reducing barriers to accessing administrative data on SARS-CoV-2 vaccination for research. CMAJ. Jul 18, 2022;194(27):E943-E947. [FREE Full text] [CrossRef] [Medline]
    25. Jackson ML. Use of self-reported vaccination status can bias vaccine effectiveness estimates from test-negative studies. Vaccine X. Dec 29, 2018;1:100003. [FREE Full text] [CrossRef] [Medline]
    26. Feikin DR, Higdon MM, Abu-Raddad LJ, Andrews N, Araos R, Goldberg Y, et al. Duration of effectiveness of vaccines against SARS-CoV-2 infection and COVID-19 disease: results of a systematic review and meta-regression. Lancet. Mar 05, 2022;399(10328):924-944. [FREE Full text] [CrossRef] [Medline]
    27. Québec vaccination registry. Gouvernement du Québec. URL: https://www.quebec.ca/en/health/your-health-information/quebec-vaccination-registry [accessed 2023-04-06]
    28. Hohl CM, Rosychuk RJ, McRae AD, Brooks SC, Archambault P, Fok PT, et al. Canadian COVID-19 Emergency Department Rapid Response Network investigators and for the Network of Canadian Emergency Researchers and the Canadian Critical Care Trials Group. Development of the Canadian COVID-19 emergency department rapid response network population-based registry: a methodology study. CMAJ Open. Mar 17, 2021;9(1):E261-E270. [FREE Full text] [CrossRef] [Medline]
    29. Canadian COVID-19 Emergency Department Rapid Response Network. URL: https://www.ccedrrn.com/ [accessed 2023-05-28]
    30. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, et al. STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. Oct 28, 2015;351:h5527. [FREE Full text] [CrossRef] [Medline]
    31. Ligne du temps COVID-19 au Québec. Institut national de santé publique du Québec. URL: https://www.inspq.qc.ca/covid-19/donnees/ligne-du-temps [accessed 2023-04-06]
    32. Inscription obligatoire en tant que personne vaccinée. Gouvernement du Québec. Apr 20, 2018. URL: https:/​/www.​quebec.ca/​sante/​vos-informations-de-sante/​registre-de-vaccination-du-quebec/​inscription-obligatoire-en-tant-que-personne-vaccinee [accessed 2023-04-06]
    33. Britton A, Fleming-Dutra KE, Shang N, Smith ZR, Dorji T, Derado G, et al. Association of COVID-19 vaccination with symptomatic SARS-CoV-2 infection by time since vaccination and delta variant predominance. JAMA. Mar 15, 2022;327(11):1032-1041. [FREE Full text] [CrossRef] [Medline]
    34. Tartof SY, Slezak JM, Fischer H, Hong V, Ackerson BK, Ranasinghe ON, et al. Effectiveness of mRNA BNT162b2 COVID-19 vaccine up to 6 months in a large integrated health system in the USA: a retrospective cohort study. Lancet. Oct 16, 2021;398(10309):1407-1416. [FREE Full text] [CrossRef] [Medline]
    35. Hajian-Tilaki K. Sample size estimation in diagnostic test studies of biomedical informatics. J Biomed Inform. Apr 2014;48:193-204. [FREE Full text] [CrossRef] [Medline]
    36. Sample size software. NCSS: Statistical Software. 2012. URL: https://www.ncss.com/software/pass/ [accessed 2023-05-28]
    37. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-282. [FREE Full text] [Medline]
    38. R Core Team. R: a language and environment for statistical computing. R Project for Statistical Computing. URL: https://www.r-project.org/ [accessed 2023-05-28]
    39. Harrell Jr FE. Regression modeling strategies. Comprehensive R Archive Network (CRAN). May 08, 2023. URL: https://CRAN.R-project.org/package=rms [accessed 2023-05-28]
    40. Revelle W. Procedures for psychological, psychometric, and personality research. Comprehensive R Archive Network (CRAN). Mar 18, 2023. URL: https://CRAN.R-project.org/package=psych [accessed 2023-05-28]
    41. Stephenson M, Olson SM, Self WH, Ginde AA, Mohr NM, Gaglani M, et al. Ascertainment of vaccination status by self-report versus source documentation: impact on measuring COVID-19 vaccine effectiveness. Influenza Other Respir Viruses. Nov 2022;16(6):1101-1111. [FREE Full text] [CrossRef] [Medline]
    42. Lorini C, Santomauro F, Donzellini M, Capecchi L, Bechini A, Boccalini S, et al. Health literacy and vaccination: a systematic review. Hum Vaccin Immunother. Feb 01, 2018;14(2):478-488. [FREE Full text] [CrossRef] [Medline]
    43. Rowlands G. Health literacy. Hum Vaccin Immunother. 2014;10(7):2130-2135. [FREE Full text] [CrossRef] [Medline]
    44. Rolnick SJ, Parker ED, Nordin JD, Hedblom BD, Wei F, Kerby T, et al. Self-report compared to electronic medical record across eight adult vaccines: do results vary by demographic factors? Vaccine. Aug 20, 2013;31(37):3928-3935. [FREE Full text] [CrossRef] [Medline]
    45. Vassallo A, Shajahan S, Harris K, Hallam L, Hockham C, Womersley K, et al. Sex and gender in COVID-19 vaccine research: substantial evidence gaps remain. Front Glob Womens Health. Nov 01, 2021;2:761511. [FREE Full text] [CrossRef] [Medline]
    46. Heidari S, Goodman T. Critical sex and gender considerations for equitable research, development and delivery of COVID-19 vaccines. World Health Organization. Apr 18, 2021. URL: https:/​/cdn.​who.int/​media/​docs/​default-source/​immunization/​sage/​covid/​gender-covid-19-vaccines-sage-background-paper.​pdf?sfvrsn=899e8fca_15&download=true [accessed 2023-04-06]
    47. Vijayasingham L, Heidari S, Munro J, Omer S, MacDonald N. Resolving sex and gender bias in COVID-19 vaccines R&D and beyond. Hum Vaccin Immunother. Dec 31, 2022;18(1):2035142. [FREE Full text] [CrossRef] [Medline]
    48. Fischinger S, Boudreau CM, Butler AL, Streeck H, Alter G. Sex differences in vaccine-induced humoral immunity. Semin Immunopathol. Mar 2019;41(2):239-249. [FREE Full text] [CrossRef] [Medline]
    49. Zintel S, Flock C, Arbogast AL, Forster A, von Wagner C, Sieverding M. Gender differences in the intention to get vaccinated against COVID-19: a systematic review and meta-analysis. Z Gesundh Wiss. Jan 07, 2022:1-25. [FREE Full text] [CrossRef] [Medline]
    50. Ulrich AK, McKearnan SB, Lammert S, Wolfson J, Pletcher J, Halloran ME, et al. Validity of university students' self-reported vaccination status after a meningococcal B outbreak. J Am Coll Health. Apr 2022;70(3):824-829. [FREE Full text] [CrossRef] [Medline]
    51. King JP, McLean HQ, Belongia EA. Validation of self-reported influenza vaccination in the current and prior season. Influenza Other Respir Viruses. Nov 2018;12(6):808-813. [FREE Full text] [CrossRef] [Medline]
    52. Zheng C, Shao W, Chen X, Zhang B, Wang G, Zhang W. Real-world effectiveness of COVID-19 vaccines: a literature review and meta-analysis. Int J Infect Dis. Jan 2022;114:252-260. [FREE Full text] [CrossRef] [Medline]
    53. Skowronski DM, Febriani Y, Ouakki M, Setayeshgar S, El Adam S, Zou M, et al. Two-dose severe acute respiratory syndrome coronavirus 2 vaccine effectiveness with mixed schedules and extended dosing intervals: test-negative design studies from British Columbia and Quebec, Canada. Clin Infect Dis. Nov 30, 2022;75(11):1980-1992. [FREE Full text] [CrossRef] [Medline]
    54. Lamontagne F, Rowan KM, Guyatt G. Integrating research into clinical practice: challenges and solutions for Canada. CMAJ. Jan 25, 2021;193(4):E127-E131. [FREE Full text] [CrossRef] [Medline]
    55. Etheredge LM. A rapid-learning health system. Health Aff (Millwood). Mar 2007;26(2):w107-w118. [CrossRef] [Medline]
    56. Smith R, Hubers J, Farraye FA, Sampene E, Hayney MS, Caldera F. Accuracy of self-reported vaccination status in a cohort of patients with inflammatory bowel disease. Dig Dis Sci. Sep 2021;66(9):2935-2941. [FREE Full text] [CrossRef] [Medline]
    57. Laurence A, Lewis P, Gately C, Dixon A. Influenza and pneumococcal vaccination: do older people know if they have been vaccinated? Aust N Z J Public Health. Jun 2016;40(3):279-280. [FREE Full text] [CrossRef] [Medline]
    58. Lu X, Zhang L, Du H, Zhang J, Li YY, Qu J, et al. Chinese Pediatric Novel Coronavirus Study Team. SARS-CoV-2 Infection in Children. N Engl J Med. Apr 23, 2020;382(17):1663-1665. [FREE Full text] [CrossRef] [Medline]
    59. Khan MS, Nguyen TX, Lal S, Watanapongvanich S, Kadoya Y. Hesitancy towards the third dose of COVID-19 vaccine among the younger generation in Japan. Int J Environ Res Public Health. Jun 08, 2022;19(12):7041. [FREE Full text] [CrossRef] [Medline]
    60. Diesel J, Sterrett N, Dasgupta S, Kriss JL, Barry V, Vanden Esschert K, et al. COVID-19 vaccination coverage among adults - United States, December 14, 2020-May 22, 2021. MMWR Morb Mortal Wkly Rep. Jun 25, 2021;70(25):922-927. [FREE Full text] [CrossRef] [Medline]
    61. Baack BN, Abad N, Yankey D, Kahn KE, Razzaghi H, Brookmeyer K, et al. COVID-19 vaccination coverage and intent among adults aged 18-39 years - United States, March-May 2021. MMWR Morb Mortal Wkly Rep. Jun 25, 2021;70(25):928-933. [FREE Full text] [CrossRef] [Medline]
    62. COVID-19 vaccine uptake and intent: Canadian community health survey (CCHS) insight. Public Health Agency of Canada. 2021. URL: https:/​/www.​canada.ca/​en/​public-health/​services/​publications/​vaccines-immunization/​covid-19-vaccine-uptake-intent-canadian-community-health-survey.​html [accessed 2023-04-06]
    63. Sharif A, Botlero R, Hoque N, Alif SM, Nazmul Karim M, Islam SM. A pragmatic approach to COVID-19 vaccine passport. BMJ Glob Health. Oct 2021;6(10):e006956. [FREE Full text] [CrossRef] [Medline]
    64. Bosch-Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A. Accuracy and quality of immunization information systems in forty-one low income countries. Trop Med Int Health. Jan 2009;14(1):2-10. [FREE Full text] [CrossRef] [Medline]
    65. Baum U, Kulathinal S, Auranen K. Exposure misclassification bias in the estimation of vaccine effectiveness. PLoS One. May 13, 2021;16(5):e0251622. [FREE Full text] [CrossRef] [Medline]

    ‎
    CCEDRRN: Canadian COVID-19 Emergency Department Rapid Response Network
    ED: emergency department
    REDCap: Research Electronic Data Capture

    Edited by A Mavragani, T Sanchez; submitted 24.11.22; peer-reviewed by S Olson, M Coccia; comments to author 15.03.23; revised version received 11.04.23; accepted 03.05.23; published 16.06.23.

    Copyright

    ©Patrick M Archambault, Rhonda J Rosychuk, Martyne Audet, Rajan Bola, Shabnam Vatanpour, Steven C Brooks, Raoul Daoust, Gregory Clark, Lars Grant, Samuel Vaillancourt, Michelle Welsford, Laurie J Morrison, Corinne M Hohl, Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) investigators, Network of Canadian Emergency Researchers, Canadian Critical Care Trials Group. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 16.06.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.