The European Medicines Agency (EMA) released interim guidance on enhanced safety surveillance for seasonal influenza vaccines in August 2014 [1]. All Marketing Authorization Holders (MAHs) commercializing influenza vaccines in Europe must follow this guidance which sets out new standards for safety surveillance. Its goal is to rapidly detect, in near real-time early in the season, any significant increase in the frequency or severity of a defined list of adverse events of interest (AEIs). These AEIs can be local, systemic, or allergic reactions, indicating a potential or more serious risk.

For this request, the EMA defines 3 types of surveillance: (1) active surveillance, using existing methods of postauthorization surveillance; (2) enhanced passive surveillance (EPS), estimating vaccine usage rapidly and taking additional steps to facilitate passive adverse drug reactions (ADR) reporting of incidence of AEIs; and (3) electronic health record data mining (EHR-DM).

English general practice (GP) is a suitable setting to implement EPS and EHR-DM, as it has a registration-based list system with linked medical records, it has been highly computerized since 2004, and data extracted from these systems are widely used in research [2,3]. Furthermore, GPs are largely independent professional partnerships and make their own decision about which brand of influenza vaccine to purchase before the start of each influenza season [4]. Practices administer influenza vaccines to recommended groups, starting in September of each year [5].

In 2015-16, one of the major suppliers of influenza vaccines to UK health care was GSK (GlaxoSmithKline). Collaborating with the University of Surrey, GSK took the opportunity of the UK computerized infrastructure to implement EPS and EHR-DM with the aim to test the feasibility of using EPS and EHR-DM within GP to identify AEIs in subjects vaccinated with GSK’s and other influenza vaccine brands. An additional aim was to ascertain if using EPS in GP has any advantages over EHR-DM alone. The key components of this surveillance were as follows: (1) to provide a weekly estimation of vaccine coverage, by age, risk group, and vaccine brand; and (2) to estimate weekly AEI reporting rates among subjects vaccinated against seasonal influenza, by age, comorbidity, and brand from the GPs using the EHR-DM method, and those using the EPS system.

Following the request of EMA stating that MAHs must conduct enhanced seasonal influenza vaccines safety surveillance, GSK, together with the University of Surrey, launched a pilot safety surveillance study for GSK’s seasonal influenza vaccines. The University of Surrey has developed methods to extract timely surveillance data from GP. This system was developed to support the Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC) weekly surveillance reports [6] and is capable of being adapted to support the EMA-specified surveillance (ClinicalTrials.gov number: NCT02567721).

We recruited 9 GPs spread across England, from varying types of locality (North and South, and urban and rural) and using different EHR systems (Egton Medical Information Systems) and The Phoenix Partnership). We compared the demographic characteristics—age, gender, ethnicity, and deprivation scores—using the Index of Multiple Deprivation, on the basis of each patient’s Lower Super Output Area (a small local geographical location [7,8]) of the registered population, with the national average data obtained from the Office for National Statistics [9].

This feasibility study period ran from International Organization for Standardization (ISO) week 35 to week 49 of 2015 (08/24/2015 to 12/06/2015).

As this was a feasibility study, there was no attempt to determine a priori the size of the study on the basis of power calculation. However, considering the average GP size of 7034 in England and Wales, it was expected that 9 GPs would provide a study population of 63,300. Ultimately, the population registered in the recruited practices was larger (N=81,040).

GPs were recruited in summer 2015; some months after they had selected their brand of vaccine for the coming influenza season. Out of the 9 practices recruited, 3 joined the EPS group and 6 joined the EHR-DM group. As this was a feasibility study, there was no attempt to randomize the practices into groups according to the type and brand of seasonal influenza vaccine used by the practices. Each group (EPS vs EHR-DM) was based on practices’ willingness to participate in the study.

The EPS group involved every patient who was vaccinated and received a yellow card; on the basis of the standard ADR cards used by the Medicines & Healthcare Products Regulatory Agency (MHRA) [10]. The patients were asked to complete a yellow card if they experienced an AEI in the 14-day postvaccination window; they were invited to return the card to their registered practice. The reported information was coded into the patient’s EHR by their GP and extracted weekly:

All practices were given induction training and a preferred code list for the specific AEIs identified by EMA to facilitate standardized data coding. We created this list using an ontological approach [13,14]. We grouped these conditions into body system categories: respiratory, gastrointestinal, fever, sensitivity and anaphylaxis, rash, general symptoms, neurological, musculoskeletal, and local reactions (Multimedia Appendix 1). We provided GPs a full code list and a screen-side prompt list.

We requested that, when a patient presented with one of these conditions, they should use the recommended code (as per the instruction sheet). We advised GPs that GPs should code the conditions they felt a patient was reporting accurately. We stressed that the overarching purpose was not to purposively make any causal link between vaccination and a given condition, particularly for common conditions, such as cold and headache. However, they were reminded that they should report, in parallel to this study, any serious or important events on the basis of their professional judgment through the standard MHRA reporting system as per local regulation.

The primary outcome measure was to report AEI frequencies among influenza-vaccinated subjects and observe any discrepancies in these frequencies in the EPS versus EHR-DM group and GSK versus non-GSK vaccines. To identify event date, we used the event date for the actual episode, not the recorded date.

Throughout the study, we produced weekly reports of AEIs. At the end of the study, we produced an end-of-season report. The end-of-season report enabled us to identify whether the approach was appropriate to adequately capture the vaccination uptake, to enhance the collection of AEs, easily transcribe the events reported from the yellow cards to the electronic system, more comprehensively capture the AEIs experienced, and extract them on an ongoing basis in a near real-time manner.

Our established data extraction method means that only coded data (Read code version 2 and CTV3) [15] were used to collect the relevant information. Free text was not extracted as it might include patient identifiable information. We excluded people with a code that indicated that they opted out of sharing data; estimated at around 1.25% of the registered population [16]. Data were pseudonymized as close to source as possible and encrypted.

For all AEIs, we report the rate and the 95% CI, using the critical binomial function in Microsoft Excel [17].

Our practices were largely distributed across England, with most in the Midlands and East National Health Service (NHS) Region (NHS Regions: North, n=3; Midlands and East, n=5; and South, n=1). Practices were mainly from urban areas (rural, n=3; and urban, n=6). In the week 49 weekly report (data extracted in a near real-time manner), the overall population registered in the 9 GP practices was 81,040. The end-of-study cohort population was 71,407 (owing to restriction of longer registration as described earlier).

In the week 49 data extraction, the practice populations had a female to male ratio of 49.76% (40, 323/81,040) to 50.24% (40,717/81,040. The proportion of people aged 20 to 24 years and over 50 years was above the national average (Figure 1). The study population had a higher proportion of people of white ethnicity recorded 92.57% (38,154/41,218 compared with 85.42% nationally; (45,281,142/53,012,456). The study population was less deprived than the national average, with almost three-fourths 72.20% (56,545/78,322) of the population in the least deprived half (Multimedia Appendix 2).

When comparing the EHR-DM practices and the EPS practices using the end-of-season dataset, the EPS practices were slightly older, the proportion of 65 years and over was 21.64% (n=9842/45,519) in EHR-DM compared with 25.44% (6586/25,888) in EPS (Table 1). The EHR-DM practices also had a smaller proportion of at-risk patients at 40.43% (18,403/45,519) compared with 45.21% (11,703/25,888) in the EPS practices.

Most vaccinations took place between weeks 39 and 45 (Figure 2). From the weekly cumulative data extractions (weeks 35-49), 19.57% (15,863/81,040) of the registered practice population overall and 57.48% (9969/17,344) of people aged 65 years or older were vaccinated (Table 2).

This study showed that using GP data for EHR-DM was a feasible method of near real-time surveillance. This was demonstrated by the timeliness of data extraction and the validity of the data.

The use of yellow cards (EPS) to enhance surveillance was not successful as there was only a 1.97% (82/4150) return rate; and this small additional return rate did not yield useful additional information. However, there were only 3 practices in this group.

Rates of AEI (within 14 days postvaccination) in the vaccinated group were around 3% (3.02% 333/11,367 for non-GSK and 2.63% 92/3434 for GSK) with around 1% (150/15,863) of AEIs were respiratory conditions. The differences in rates were not statistically significant, and no safety concerns were raised during the weekly assessment or at the end of the study period.

The study demonstrated the feasibility of setting up a network that could, with further refinement, rapidly detect potential safety concerns, allowing prompt investigation if deemed appropriate [18], originating from a significant change in AEIs associated with influenza vaccination using EHR-DM observing, for example, trends from week to week.

This network still needs additional adaptations to be worked out in the next phase of its implementation. For example, further thought needs to be given regarding improving the return rate of yellow cards. To improve the return rate, any future ADR card should ideally have predefined categories that patients can tick if present (or record if no AEI is experienced following vaccination), to standardize coding into the EHR.

To harmonize and standardize the Enhanced Safety Surveillance in Europe, one recommendation could be to establish a network of organizations involved in EHR data mining in real time, supplemented with vaccination-customized ADR cards (ie, yellow cards in this study) reporting scheme, with criteria defining the prerequisites of data quality. However, a yellow card is not a good long-term choice of color or label. In the United Kingdom, yellow cards are developed by the MHRA and are generally completed when a GP suspects an ADR related to the vaccination. In this study, the approach was to collect as comprehensively as possible AEIs, prespecified by the EMA, regardless of the possible causal association with immunization.

Another potential way to collect AEI following vaccination could be to develop a mobile phone application to collect events in a more comprehensive manner. This approach could be an interesting approach to overcome the suboptimal return. Future research could investigate the feasibility of designing such an app to collect this information in a more reliable manner.

A near real-time enhanced brand-specific surveillance network could produce weekly reports of AEIs to meet the EMA requirements. The data extraction element of the project could readily be extended; the yellow card scheme could be refined and expanded. Introducing an ADR card as part of the process may be an optimal and sustainable way to stimulate reporting of AEIs from those with a lower propensity to consult (ie, for mild events).

Data collection over several seasons may be required to achieve a better understanding of background rates of AEIs. A particular challenge is that people immunized in the first weeks of immunization may differ from those immunized in subsequent weeks due to annual recommendations to promptly vaccinate the more vulnerable patients early in the season.

It is feasible to set up a weekly reporting, using an enhanced passive and EHR-DM surveillance system, to detect EMA-specified AEIs across specific brands; though further refinements are needed before such a system can be fully operationalized.

Data were inevitably passive surveillance data, largely derived from medically attended events for those who consult. Considering that some groups have a lower propensity to consult (eg, men [19]), this is a likely confounding factor. Future EPS may need to be designed to be more inclusive of less represented groups; for instance, using customized ADR cards to continuously enhance the reporting.

A further limitation of our design was the potential bias in the practice selection for EPS, which was conditional on the willingness of practice to participate in the study and hand out, collect, and code any data on the yellow cards. We drew conclusions about the unsuitability of the yellow card on basis of data from 3 practices, not selected at random. Return rates were low and what was written on the cards was challenging to transcribe and thus to code consistently.

In addition, in the United Kingdom, vaccines are preferentially recommended for different age and risk groups which preclude any possibility to systematically compare the findings from different vaccine brands. In addition, the sample for future studies could have improved representativeness by recruiting more inner-city practices. Our study practice populations were less ethnically mixed and less deprived than the English national average, likely due to a lack of inner-city practices.

Our remote data extraction system was reasonably reliable, although it had gaps. It is challenging to use a local extract system in a timely way, although it did fill gaps in our data. One other challenge was to properly analyze all the collected data to allow near real-time assessment. In the longer term, comparing rates with historic data may be an additional option to have more insights on baseline rates as a basis for comparison with findings in the current season. For this study, we had a lack of historical background rates of AEI in the same population or availability of an appropriate comparison group. In future studies, we should either compare rates with AEIs derived from the RCGP or from previous years of these studies, assuming that the study design and the approach remain overall unchanged.

A limitation of our design was training GPs in coding. While this should help with data within the season, it has the potential to produce biased results when comparing with historical data when no training was given. Furthermore, the preferred code list given to practices (Multimedia Appendix 1) notes that these codes should be used for adverse events postvaccination. In this respect, it does not appear to use as baseline for comparison historical data where no specific training was provided.

Overall, this study showed that using GP data was a feasible method for enhanced near-real time surveillance in terms of EHR-DM. However, the use of yellow cards (EPS) in GP did not capture a significant amount of additional data. There are many lessons learnt from this initial study, and these are reflected in the limitations of the design and study approach. Future enhanced surveillance should focus on ways to improve and standardize AEIs reporting, data collection, and extraction.