Background: Rapid diagnostic tests (RDTs) are being widely used to manage COVID-19 pandemic. However, many results remain unreported or unconfirmed, altering a correct epidemiological surveillance.
Objective: Our aim was to evaluate an artificial intelligence–based smartphone app, connected to a cloud web platform, to automatically and objectively read RDT results and assess its impact on COVID-19 pandemic management.
Methods: Overall, 252 human sera were used to inoculate a total of 1165 RDTs for training and validation purposes. We then conducted two field studies to assess the performance on real-world scenarios by testing 172 antibody RDTs at two nursing homes and 96 antigen RDTs at one hospital emergency department.
Results: Field studies demonstrated high levels of sensitivity (100%) and specificity (94.4%, CI 92.8%-96.1%) for reading IgG band of COVID-19 antibody RDTs compared to visual readings from health workers. Sensitivity of detecting IgM test bands was 100%, and specificity was 95.8% (CI 94.3%-97.3%). All COVID-19 antigen RDTs were correctly read by the app.
Conclusions: The proposed reading system is automatic, reducing variability and uncertainty associated with RDTs interpretation and can be used to read different RDT brands. The web platform serves as a real-time epidemiological tracking tool and facilitates reporting of positive RDTs to relevant health authorities.
To control COVID-19 pandemic, timely and accurate early-detection strategies of SARS-CoV-2 infections have been critical to slow down the spread of the virus. The use of rapid diagnostic tests (RDTs), both for detection of antibodies and antigens, has contributed to improve COVID-19 testing capacity, reducing costs of diagnosis, and allowing for fastest results . First, COVID-19 RDTs were intended to be used just by professional health workers who have extensive experience in the use of this tool for different infectious diseases [ , ]. Later, multiple health ministries approved home testing kits, improving the accessibility to testing and taking pressure off health institutions. Nevertheless, self-testing strategies have some limitations; the general population is not familiar with the use of RDTs; and a minimum training is needed for sampling, testing, and result interpretation. Furthermore, as it has been seen during the latest waves [ ], many results go unreported, impairing posttesting counseling and epidemiological surveillance.
Combining RDTs with digital tools, artificial intelligence (AI) and mobile health approaches can help standardize result interpretation and facilitate immediate reporting and monitoring of results . Several works have been proposed to automatically interpret photographs of RDTs using different image processing approaches, from classical methods, such as morphology-based methods, to more sophisticated machine learning or deep learning methods [ - ]. Nevertheless, these approaches are not capable of handling 2-band and 3-band RDTs indistinctly, are not connected to a cloud platform that enables the collection of mass screening results, and many require additional hardware. In this paper, we describe the development and field validation of a mobile-based tool (exhaustively tested with a variety of phone models and different lighting conditions) that could be used with any smartphone for reading and reporting multiple types of SARS-CoV-2 RDTs and is connected to a real-time epidemiological monitoring web platform ( ).
This study was divided into 2 phases: first, the training and validation of an AI algorithm for the automatic interpretation of RDTs; second, 2 field studies to assess the performance of the AI-based system for reading both COVID-19 antibody and antigen RDTs in real-world scenarios.
Ethics approval for the study was obtained from the Clinical Research Ethics Committee of the Ramón y Cajal University Hospital (127/21).
Algorithm Training and Validation Data Set
Ensuring standardized image acquisition is a key step in developing robust AI algorithms. With this purpose, all inoculated RDTs were digitized using the TiraSpot mobile app (Spotlab), which guarantees image quality and correct positioning of RDTs in the image by using a simple augmented reality system that displays a mask with the exact geometry of a given RDT in the screen of the smartphone, helping users to correctly align the RDT before making the picture (screenshot of the mobile app is presented in). Each RDT brand has its own mask that guides the user to take a standardized picture. In addition, after the picture is taken, the user is presented with the picture and asked to confirm that it is aligned and on focus. If the user rejects the picture, the user is allowed to take another one. The mobile app also allows users to record sample metadata, which together with the images and their initial visual interpretation are uploaded to the cloud platform. To gain robustness and generalizability, a total of 11 different smartphone models, ranging from low- to high-range devices, were used in this study.
An AI algorithm was developed to predict test results based on a picture of the RDT. With this purpose, the image is first preprocessed by cropping the original image to extract a region of interest that contains the relevant part of the picture (strip of the RDT). Then, image normalization and contrast enhancement (Contrast Limited Adaptive Histogram Equalization method) were applied to highlight faint bands. Finally, the processed region of interest is introduced into a convolutional neural network (MobileNet V2 ), which then predicts the test result ( ).
For generating the training image data set, 12 human sera from patients with positive SARS-CoV-2 polymerase chain reaction tests (infected between March and May 2020) and with a positive enzyme-linked immunosorbent assay (ELISA) test were used. Each serum sample was serially diluted with reference human sera (H5667; Sigma-Aldrich) until it reached a negative result when inoculated in a COVID-19 antibody test. Each dilution was tested in 3 replicates for each of the 3 brands tested (ie, 2019-nCoV IgG/IgM Rapid Test Cassette, Hangzhou AllTest Biotech Co., Ltd.; Panbio COVID-19 IgG/IgM Rapid Test Device, Abbott; and UNscience COVID-19 IgG/IgM Rapid Test, Wuhan UNscience Biotechnology Co., Ltd.), resulting in 433 RDTs inoculated (61 positive for both IgG and IgM; 166 positive for IgG and negative for IgM; 43 negative for IgG and positive for IgM; and 164 negative for both IgG and IgM). Additionally, 12 COVID-19 antigen RDTs (Panbio COVID-19 Ag Rapid Test Device, Abbott; 6 positive and 6 negative) were also included to train the algorithm to read not only 3-band tests (such as the COVID-19 antibody tests used in this study) but also 2-band RDTs, such as COVID-19 antigen tests. The entire training data set consisted of 3614 images.
For collecting the independent validation data set, 240 human sera samples independent from the ones used for training were used to inoculate 720 COVID-19 antibody RDTs (each serum was tested in triplicate using the aforementioned brands). The samples were selected ensuring all possible results are well represented along the data set (108 positive for both IgG and IgM; 321 positive for IgG and negative for IgM; 27 negative for IgG and positive for IgM; and 264 negative for both IgG and IgM).
Each RDT was visually read by multiple observers (3 to 5), and the ground truth was established as the majority result from total analyzers. All sera samples were collected between May and June 2020.
Field Validation Studies
The workflow for the field studies was as follows: a health professional digitized the RDTs by using the app and was asked for recording the visual interpretation of the test result; images were uploaded to the cloud platform and processed by the AI algorithm; and discrepancies between the interpretation made by the health professional and that obtained by the algorithm were subsequently reviewed by an external health professional through the platform.
The first field study used the system as part of a seroprevalence study conducted in two nursing homes in Madrid, Spain. A total of 172 vaccinated health care personnel were included in this study; a finger-prick blood sample was taken from them and inoculated into SARS-CoV-2 Rapid Antibody Test (Roche). A trained nurse digitized the RDTs and recorded their results using the app.
The second field validation study tested the system to read also COVID-19 antigen tests (ie, Panbio COVID-19 Ag Rapid Test Device, Abbot) composed of 2 bands (ie, control and test). This study was carried out at the emergency department of the Ramón y Cajal Hospital in Madrid, Spain, where 96 individuals’ nasal swabs were inoculated in antigen tests and digitized by experienced health professionals using the app.
All images were acquired in very diverse real-world conditions involving different users, including different environmental illuminations (eg, different lighting color temperatures and a wide range of lighting), and using different smartphone models that ranged from low- to high-range devices. This was done with the purpose of developing and validating the robustness of the algorithm that may change in real life.
AI Algorithm Training and App Validation
All images acquired with the app were uploaded to a cloud platform, where the AI algorithm processed the photographs to predict the result interpretations. As shown in(part 1), when comparing the visual interpretations (used as ground truth) against the AI algorithm, the performance was high for all brands of RDT tested, obtaining a mean sensitivity and specificity of 98% and 100%, respectively, for the detection of the IgG band; and a mean sensitivity and specificity of 80% and 89%, respectively, for the detection of the IgM band. No significant differences were found in algorithm performance between different smartphone models or across different lighting conditions, pointing out the robustness of the readout algorithm.
|Evaluation data||AUCa (95% CI)||Sensitivity (95% CI)||Specificity (95% CI)||Tests, n|
|1: RDT manufacturer (Ab) and band|
|IgG||99.5 (98.7-100)||96.4 (94.1-98.8)||100 (100-100)||94||145|
|IgM||92.5 (85.4-99.6)||80.8 (75.8-85.8)||90.7 (87.0-94.3)||184||55|
|IgG||100 (100-100)||100 (100-100)||100 (100-100)||100||140|
|IgM||89.5 (83.7-95.2)||80.0 (74.9-85.1)||88.6 (84.6-92.6)||214||26|
|IgG||99.8 (99.4-100)||97.9 (96.1-99.7)||100 (100-100)||96||144|
|IgM||90.6 (85.0-96.1)||79.6 (74.5-84.7)||86.0 (81.6-90.4)||186||54|
|IgG||99.8 (99.5-100)||98.1 (97.1-99.1)||100 (100-100)||290||429|
|IgM||90.8 (87.4-94.3)||80.0 (77.1-82.9)||89.0 (86.2-90.9)||584||135|
|2: RDT manufacturer (Ab) and band|
|IgG||100 (100-100)||100 (100-100)||94.4 (92.8-96.1)||18||154|
|IgM||99.6 (96.0-100)||100 (100-100)||95.8 (94.3-97.3)||166||6|
|3: RDT manufacturer (Ag) and band|
|Test||100 (100,100)||100 (100,100)||100 (100,100)||68||28|
aAUC: area under the curve.
Validation in Real-world Scenarios
From the 172 RDTs used in this study (5 positive for both IgG and IgM; 149 positive for IgG and negative for IgM; 1 negative for IgG and positive for IgM; and 17 negative for both IgG and IgM), we only found 9 discrepancies between test result interpretations made by health professionals and those made by the AI algorithm. From these 9 cases, 2 were incorrectly classified by the algorithm due to an incorrect image acquisition with the app. The remaining discrepant cases were further reviewed by a second professional, and the AI-based system allowed for the detection and modification of the result with respect to the initial health professional interpretation in 4 cases by confirming the result predicted by the algorithm.
The overall performance of the algorithm with respect to the ground truth is shown in(part 2). It should be noted that the performance of the system is high even when used with an RDT different from those used for training the algorithm, suggesting its potential use with any RDT on the market. The slight disparity in the performance of IgM band identification in antibody RDTs between the validation set and this field study may be explained by the presence of very faint signals that were almost invisible in the photographs.
Regarding the second field study for reading COVID-19 antigen RDTs, we found that all tests used and digitized using the TiraSpot app (ie, 58 negative and 30 positive) were correctly interpreted by the proposed system (, part 3), demonstrating that the system can also be applied for reading 2-band (ie, control and test) and 3-band (ie, IgG, IgM, and control) tests.
We described the usefulness of an app for reading and result interpretation of lateral flow RDTs for SARS-CoV-2 testing. The results are sent to a cloud platform that allows for case identification and confirmation, quality control, and real-time monitoring.
Our AI algorithm demonstrates excellent performance, especially in prospective validation of real-life scenarios and for both antibody and antigen detection tests. The algorithm performed as well in RDT brands that were not used at all for training purposes, making the solution suitable for other RDTs, including other diseases. Compared with previous studies [- ], our system is able to identify individual bands of the RDTs, allowing for complex result reading and sending them in real-time to a cloud platform. A requirement and limitation of the proposed system is the correct acquisition of the image (acquisition error in the field studies was <0.8%).
In conclusion, the use of TiraSpot () is a useful tool for reporting, real-time monitoring, and quality control, as the results can be reviewed by specialists when needed. This is especially important in contexts where massive testing is to be done and the likelihood of subjectivity and errors in the interpretation of the result is higher. It is also important in the validation of self-diagnostic tests performed by untrained users, as it avoids the loss of information in case the user does not report it, and it provides an efficient system to confirm and report data, which has been a key challenge during the latest pandemic waves.
This study has been partially funded by JYLAG Foundation (Fribourg, Switzerland). DBP was supported by grant PTQ2020-011340/AEI/10.13039/501100011033 funded by the Spanish State Investigation Agency. LL was supported by a predoctoral grant (IND2019/TIC-17167) by Comunidad de Madrid, Spain.
The raw data supporting the results of this study will be made available upon reasonable request from the corresponding author.
DBP, DMM, EA, JMN, JCG, BRH, RC, MLO, and MRD conceived and designed the study. DBP, DMM, EA, NPP, AM, ED, LL, and MRD acquired data. DBP and EA analyzed the data. EA, DC, and AV worked on the mobile app and web-based tool. DBP, DMM, and EA wrote the original draft. All authors interpreted data and revised the manuscript.
Conflicts of Interest
DBP, EA, AM, ED, LL, AV, DC, and MLO work for Spotlab. The rest of the authors declare no competing interests.
- Borges LP, Martins AF, Silva BDM, Dias BDP, Gonçalves RL, Souza DRVD, et al. Rapid diagnosis of COVID-19 in the first year of the pandemic: a systematic review. Int Immunopharmacol 2021 Dec;101(Pt A):108144 [FREE Full text] [CrossRef] [Medline]
- Weber NC, Klepser ME, Akers JM, Klepser DG, Adams AJ. Use of CLIA-waived point-of-care tests for infectious diseases in community pharmacies in the United States. Expert Rev Mol Diagn 2016 Nov 27;16(2):253-264. [CrossRef] [Medline]
- Bell SFE, Lemoire J, Debattista J, Redmond AM, Driver G, Durkin I, et al. Online HIV self-testing (HIVST) dissemination by an Australian community peer HIV organisation: a scalable way to increase access to testing, particularly for suboptimal testers. Int J Environ Res Public Health 2021 Oct 26;18(21):11252 [FREE Full text] [CrossRef] [Medline]
- Tegally H, Wilkinson E, Giovanetti M, Iranzadeh A, Fonseca V, Giandhari J, et al. Detection of a SARS-CoV-2 variant of concern in South Africa. Nature 2021 Apr 09;592(7854):438-443. [CrossRef] [Medline]
- Kadam R, White W, Banks N, Katz Z, Dittrich S, Kelly-Cirino C. Target Product Profile for a mobile app to read rapid diagnostic tests to strengthen infectious disease surveillance. PLoS One 2020 Jan 29;15(1):e0228311 [FREE Full text] [CrossRef] [Medline]
- Turbé V, Herbst C, Mngomezulu T, Meshkinfamfard S, Dlamini N, Mhlongo T, et al. Deep learning of HIV field-based rapid tests. Nat Med 2021 Jul 17;27(7):1165-1170 [FREE Full text] [CrossRef] [Medline]
- Mendels D, Dortet L, Emeraud C, Oueslati S, Girlich D, Ronat J, et al. Using artificial intelligence to improve COVID-19 rapid diagnostic test result interpretation. Proc Natl Acad Sci U S A 2021 Mar 23;118(12):e2019893118 [FREE Full text] [CrossRef] [Medline]
- Wang W, Stafford JW, Keller MD. Customization and testing of a mobile reader app for an open-access SARS-CoV-2 antigen lateral flow assay. Proceedings Volume 11950, Optics and Biophotonics in Low-Resource Settings VIII 2022 Mar 02:5009-5010. [CrossRef]
- Park C. Supporting smartphone-based image capture of rapid diagnostic tests in low-resource settings. In: Proceedings of the 2020 International Conference on Information and Communication Technologies and Development. 2020 Presented at: ICTD2020; June 17-20; New York, NY p. 1-11. [CrossRef]
- Mudanyali O, Dimitrov S, Sikora U, Padmanabhan S, Navruz I, Ozcan A. Integrated rapid-diagnostic-test reader platform on a cellphone. Lab Chip 2012 Aug 07;12(15):2678-2686 [FREE Full text] [CrossRef] [Medline]
- Xiao W, Huang C, Xu F, Yan J, Bian H, Fu Q, et al. A simple and compact smartphone-based device for the quantitative readout of colloidal gold lateral flow immunoassay strips. Sens Actuators B Chem 2018 Aug 01;266:63-70 [FREE Full text] [CrossRef] [Medline]
- Paula Vaz Cardoso L, Dias RF, Freitas AA, Hungria EM, Oliveira RM, Collovati M, et al. Development of a quantitative rapid diagnostic test for multibacillary leprosy using smart phone technology. BMC Infect Dis 2013 Oct 23;13(1):497 [FREE Full text] [CrossRef] [Medline]
- Guo W, Zhang Y, Hu X, Zhang T, Liang M, Yang X, et al. Region growing algorithm combined with fast peak detection for segmenting colloidal gold immunochromatographic strip images. IEEE Access 2019;7:169715-169723. [CrossRef]
- Miikki K, Miikki L, Wiklund J, Karakoç A. Augmented sensitivity of at-home rapid SARS-CoV-2 antigen test (RAT) kits with computer vision: a framework and proof of concept. BioMed 2022 Apr 14;2(2):199-209. [CrossRef]
- Mujtaba D, Mahapatra NR. Lateral flow test interpretation with residual networks. 2021 Presented at: International Conference on Computational Science and Computational Intelligence (CSCI); Dec 15-17; Las Vegas, NV. [CrossRef]
- Wong NCK, Meshkinfamfard S, Turbé V, Whitaker M, Moshe M, Bardanzellu A, et al. Machine learning to support visual auditing of home-based lateral flow immunoassay self-test results for SARS-CoV-2 antibodies. Commun Med (Lond) 2022 Jul 06;2(1):78 [FREE Full text] [CrossRef] [Medline]
- Wang W, Hu L, Keller MD. Effects of image capture and correction approaches on quantifying results of lateral flow assays with mobile phones. Proceedings Volume 11632, Optics and Biophotonics in Low-Resource Settings VII 2021 Mar 05:320-321. [CrossRef]
- Richardson S, Kohn MA, Bollyky J, Parsonnet J. Validity of at-home rapid antigen lateral flow assay and artificial intelligence read to detect SARS-CoV-2. Diagn Microbiol Infect Dis 2022 Nov;104(3):115763 [FREE Full text] [CrossRef] [Medline]
- Lee S, Yoo YK, Wee KW, Kim C, Lee NE, Kim KH, et al. Machine-learning-assisted lateral flow assay for COVID-19 and influenza detection. SSRN Journal 2022:1-26 [FREE Full text] [CrossRef]
- Han G, Koo HJ, Ki H, Kim M. Paper/soluble polymer hybrid-based lateral flow biosensing platform for high-performance point-of-care testing. ACS Appl Mater Interfaces 2020 Aug 05;12(31):34564-34575. [CrossRef] [Medline]
- Hoque Tania M, Lwin KT, Shabut AM, Najlah M, Chin J, Hossain M. Intelligent image-based colourimetric tests using machine learning framework for lateral flow assays. Expert Syst Appl 2020 Jan;139:112843. [CrossRef]
- Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC. MobileNetV2: inverted residuals and linear bottlenecks. 2018 Presented at: IEEE/CVF Conference on Computer Vision and Pattern Recognition; June 18-23; Salt Lake City, UT. [CrossRef]
|AI: artificial intelligence|
|RDT: rapid diagnostic test|
Edited by T Sanchez, A Mavragani; submitted 03.05.22; peer-reviewed by P Sarajlic, T Scherr; comments to author 29.08.22; revised version received 16.09.22; accepted 13.10.22; published 30.12.22Copyright
©David Bermejo-Peláez, Daniel Marcos-Mencía, Elisa Álamo, Nuria Pérez-Panizo, Adriana Mousa, Elena Dacal, Lin Lin, Alexander Vladimirov, Daniel Cuadrado, Jesús Mateos-Nozal, Juan Carlos Galán, Beatriz Romero-Hernandez, Rafael Cantón, Miguel Luengo-Oroz, Mario Rodriguez-Dominguez. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 30.12.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.