This study was a scoping review and adheres to PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-analyses Extension for Scoping Reviews) guidelines [21]. The PRISMA-ScR checklist is reported in Table S1 in Multimedia Appendix 1.

The protocol for this scoping review was prospectively registered [19] and published [20]. We chose a scoping review methodology because our objectives focus broadly on the strategies required to evaluate digital interventions rather than the efficacy of digital interventions (which is typically the case in overviews). We apply 2 aspects taken from the overview methodology: (1) we assessed the overlap among primary studies cited in the included reviews to investigate the uniqueness of existing evidence, and (2) we appraised the included systematic reviews to investigate the sources of weaknesses in existing evidence. Since the methods applied in this scoping review were already reported in detail in our published protocol [20], only a short summary is provided here. There were no changes between the published protocol [20] and the objectives, methods, and results reported in this scoping review.

The eligibility criteria ([20], Textbox 1) for this scoping review were derived from the Population, Intervention, Comparison, Outcome, and Study type (PICOS) criteria.

We used (1) international databases (MEDLINE, PsycINFO, and CINAHL) and (2) the reference sections of studies (reviews) included in our scoping review.

The electronic search strategy (Multimedia Appendix 2) was developed and performed under the supervision of an experienced librarian. The electronic search was performed from database inception to March 19, 2021, without any limits, in 3 international databases.

The electronic search returned 8272 records that were stored and processed in EndNote X9 (Clarivate); after duplicates were removed, 4912 records remained. Reviews of any type were identified using smart group settings in EndNote and assessed for eligibility by any 2 authors independently. Based on title and abstract screening, reviews that met the inclusion criteria were selected for full-text screening. The reference sections of eligible full-text reviews were also manually screened to identify additional relevant reviews. Final eligibility was decided by consensus. Final list of included and excluded studies is shown in Table S2 in Multimedia Appendix 1).

Data were coded into a self-developed spreadsheet (Excel, version 10; Microsoft Inc). The spreadsheet was pilot-tested and calibrated within the team. Data coding was performed independently by 2 authors, and consensus was reached by discussion.

Data items (Textbox 2) were coded either quantitatively into predefined categories or qualitatively using definitions or author statements from the included reviews.

The operational definitions of the 2 key concepts (digital interventions and physical activity promotion) are summarized in Textbox 3.

We performed critical appraisals using AMSTAR2 (A Measurement Tool to Assess Systematic Reviews, version 2) [23]) of all systematic reviews to identify weaknesses in existing evidence. The appraisal procedure was explained in detail in the published protocol [20]. Two authors appraised all systematic reviews independently and reached consensus by discussion. The overall confidence ratings in the results of each systematic review (high, moderate, low, or critically low) were established based on the type and the number of weaknesses in each review [23] (Multimedia Appendix 3).

Coded data were synthesized using either descriptive statistics of quantitative data (relative frequencies out of all studies) or narrative descriptions of qualitative data (by identifying common themes). The AMSTAR2 appraisal outcomes (overall confidence ratings) were synthesized for all systematic reviews using a bar graph. Evidence maps were used to visualize the results based on the objectives of this scoping review (Figure 1).

This scoping review shows that 40 reviews (rapid, scoping, or systematic) that had been published within the last 15 years mentioned the issue of evaluation of digital interventions for physical activity promotion. All reviews addressed different evaluation targets, which included user outcomes or tool performance in the context of physical activity promotion. The reviews mentioned that evaluation methods relied predominantly upon objective tool data, although data from self-reports or assessments were also used. Only approximately half of all reviews mentioned evaluation frameworks and concluded that various aspects of behavior change theory were applied to develop digital interventions but not to evaluate the user outcomes of such digital interventions.

We found that many reviews that have been published to date mentioned the issue of evaluation of digital interventions for physical activity promotion. While evaluation targets and methods were mentioned in all reviews, only some reviews addressed evaluation frameworks. Among these reviews, most suggest that evaluation frameworks seem to be considered in development of digital interventions; however, it is unclear if evaluation frameworks are used to evaluate the outcomes of digital interventions. There are several possible explanations for these findings First, most reviews aimed to synthesize the literature on the effects of digital interventions on various user outcomes, and information on evaluation frameworks may not have been coded from the primary studies by review authors. This seems unlikely because evaluation was often discussed in reviews meaning that details on evaluation frameworks were probably not reported in primary studies. Indeed, the description of the respective theoretical background of digital interventions may not be sufficiently reported in primary studies for it to be coded by reviewers [51]. Second, a focus on theoretical frameworks for development but not for the evaluation of user outcomes suggests that some digital interventions may be developed for profit, while benefits to users remain secondary or unclear. Theoretical frameworks appear to inform mechanisms of action (how digital interventions work) but are also required to define how digital interventions affect user outcomes and contribute to behavior change [25]. Third, the highly heterogeneous terminology used in the field of digital interventions means that the terms evaluation or evaluation frameworks may not have been explicitly mentioned in primary studies or reviews. Since the term evaluation was included in our search syntax, we only identified reviews that specifically referred to evaluation in titles, abstracts, or key words. Thus, more literature on theoretical frameworks in the context of digital interventions likely exists but was not located using our strategy. Indeed, the reviews included in this scoping review cited different primary studies meaning that the overlap in the primary literature among the reviews was very low despite the common topic (digital interventions for physical activity promotion). There were only 4 primary studies [60-63] that were cited in 5 to 6 systematic reviews. Interestingly, all 4 studies are reasonably old (published 2014-2017), given the rapid technological advancement and interest in digital tools to support physical activity. In general, all 4 studies [60-63] compared the physical activity outcomes of digital interventions supported by different digital tools with or without other engagement methods, such as human coaching, reminders, or feedback. The results and implications of these 4 studies can be summarized as follows: (1) physical activity outcomes were evaluated using objective tool data, (2) similar physical activity benefits were evident when using modern digital tools with feedback, such as smartphone apps or activity trackers, to those evident when using traditional tools, such as pedometers, (3) physical activity benefits were higher when digital interventions were combined with human support or feedback, and (4) preference for and acceptance of modern digital tools was high based on feedback from participants and use patterns recorded by the tools. Future research is required to determine the benefits of digital interventions relative to baseline physical activity and to evaluate the effectiveness of complex interventions incorporating digital tools and human coaching for physical activity promotion.

Our results suggest that the production of yet another scoping review of primary literature on the topic of evaluation of digital interventions for physical activity promotion (planned as part 2 of this review) may not be necessary and could contribute to research waste. Instead, based on our results of part 1 of our study (this scoping review of reviews) we propose the following topics for future research.

First, more work is needed to identify factors that could improve the effectiveness of digital interventions for physical activity promotion. According to the majority of the included reviews, the identification of such factors could help to increase adherence to digital interventions and contribute to evaluation of efficacy of digital interventions. Digital interventions are typically complex interventions that require several elements for their effectiveness, such as personalization, feedback, engagement with the tool, or human support [64]. The contribution of these elements to the success or failure of digital interventions for physical activity promotion is unclear, primarily because this information was either not coded in reviews or not reported in the primary studies. Furthermore, sociodemographic factors, including age, gender, education, income, and digital health literacy, affect the use of and interest in digital health technologies [1] and could also facilitate or hinder the efficacy of digital interventions. Further research is needed to identify health needs or barriers associated with digital health technology use in low socioeconomic settings [65] to improve the efficacy of digital interventions for physical activity promotion in such populations [66].

Second, evaluation guidelines are required for digital interventions because complex interventions, such as digital interventions, are often insufficiently reported [67]. Until these guidelines are in place, for the description of digital interventions, authors could use already established reporting guidelines, such as the TIDieR Checklist [68], which includes a description of the rationale, theory, or goal of the elements essential to the intervention in item 2. Therefore, adherence to this reporting guideline could improve the inclusion of theoretical frameworks for digital interventions in future studies.

Third, standardized reporting of digital intervention components and outcomes is needed. The key difficulty is that a standardized and universally accepted definition of digital interventions does not exist yet. Generic terms, such as eHealth, mobile health (mHealth), or telehealth typically refer to very different digital health approaches. A recent guideline from the World Health Organization [69] refers to digital interventions as digitally supported interventions delivered via the internet or digital tools with mobile apps. This definition includes the elements of eHealth and mHealth and encompasses the modern digital tools, such as smartphone apps, but also established technologies, such as the internet in general (websites and email), mobile phones able to deliver SMS reminders or wearable sensors able to quantify physical activity. There are two main differences among any of these digital interventions: (1) the level of digital health literacy required to operate or interact with the digital technologies included in the digital intervention, which can be low for noninteractive wearable sensors to high when operating a smartphone app, and (2) the level of engagement and feedback, which can range from a passive use of a website, obtaining reminders via email or SMS to continuous tracing and feedback from smartphone apps or activity trackers. Furthermore, similar to that of nondigital interventions, the development of core physical activity outcome sets is necessary to evaluate digital interventions in different populations [70,71].

Fourth, objective and homogeneous outcome measures should be evaluated for digital interventions using appropriate study designs with long-term follow-up. The potential for digital interventions to improve health has been scarcely realized, partly due to an insufficient knowledge base of guiding principles in the development and evaluation of such interventions [72]. While the gold standard for evaluating a health intervention is conducting a randomized controlled trial, these can take a long time and typically require many participants and extensive financial resources. Long delays to evaluate novel digital interventions in the rapidly evolving field might result in the digital interventions becoming obsolete or nonfunctional by the time the trial is completed [73]. Thus, the evaluation of digital interventions potentially requires new study designs and methods that take the iterative and rapidly evolving nature of such interventions and continuous data collection into account. Furthermore, digital interventions are at the intersection of various fields, such as behavioral, biomedical, and computing sciences. Thus, methods taken from multiple disciplines are required for development and outcome evaluation of digital interventions [74].

Fifth, the reviews included in this scoping review predominantly focused on mixed healthy or clinical samples and predominantly adult populations. Since digital interventions can support health promotion and disease prevention [1], future research should focus on the evaluation of digital interventions or digital tools in healthy populations to promote healthy lifestyle, including physical activity. Furthermore, children and adolescents are important target populations for digital interventions that focus on physical activity promotion because the use of digital technologies contributes to sedentary behavior, especially in early childhood [75]. Therefore, future research should consider potential benefits but also harms of digital technologies and thus, evaluate if digital interventions promote or hinder physical activity in children and adolescents [76].

Although this scoping review focused on evaluation strategies rather than outcomes of such evaluations, we performed an appraisal of the 30 systematic reviews included in our study with the AMSTAR2 tool. We found that the overall methodological quality of systematic reviews of digital interventions for physical activity promotion needs improvement, which has already been suggested in the context of other health interventions [77-79]. The overall confidence in the results of systematic reviews of digital interventions for physical activity promotion could be improved by better adherence to established reporting guidelines for systematic reviews and the prospective registration of review protocols. Due to potential financial interests in the field of digital interventions, the sources of funding for primary studies should be documented in systematic reviews. Our results are in line with those of other studies [80-82] that assessed the methodological quality of systematic reviews in telemedicine [80]; digital methods for maximizing participant engagement, participation, and retention in cohort studies [81]; or digital interventions for reducing behavioral risks of cardiovascular disease [82]. These studies [80-82] demonstrated that the majority or all of the included systematic reviews regarding digital interventions had low methodological quality, and thus, the overall confidence in the results of these systematic reviews was considered to be critically low.

There were several limitations in this scoping review. First, the search strategy for literature was conservative due to the inclusion of the term evaluation in the syntax. Thus, we did not find reviews of digital interventions for physical activity promotion that omitted the term evaluation from their titles, abstracts, or keywords. A manual search for such reviews was beyond the scope of this review. Second, study selection and data coding were difficult due to highly heterogeneous terminology for digital interventions, physical activity outcomes and evaluation. These difficulties contributed to partially superficial coding of data with little detail on specific aspects of evaluation. We also struggled to code the data item evaluation target into user outcomes or tool performance. While the evaluation of well-defined user outcomes (ie, promotion of a specific physical activity outcome) was a focus of most reviews, the focus on tool performance was less clear and sometimes included as part of user outcomes (eg, acceptance of the tool). Although any 2 authors coded the data and reached consensus by discussion, a coding manual with more detail could have improved the quality of coding.

The evaluation of digital interventions is a high priority based on the 40 reviews included in this scoping review. Evaluations of digital interventions, including mobile apps or wearables for physical activity promotion, typically target any user outcomes and rely on objective tool data. While the development of digital interventions appears to be guided by various aspects of behavior change theory, evaluation frameworks are also required to evaluate user outcomes. Behavior change theory may provide useful guidance not only for the development of digital interventions but also for the evaluation of user outcomes in the context of physical activity promotion. Evidence gaps mentioned in most reviews included a need to (1) identify factors that could improve the effectiveness of digital interventions for physical activity promotion, such as personalization, feedback, engagement with the tool, human support, and digital literacy; (2) develop evaluation guidelines; and (3) standardize the reporting of digital intervention components and outcomes. The implementation of evaluation frameworks at the development stage and to assess user outcomes is required to ensure that digital interventions are effective for physical activity promotion.