Psychiatric Annals

CME Article 

Benefits and Limitations of Implementing Mental Health Apps Among the Working Population

Nathan M. Jones, MD, MPH; Matthew Johnson, MD; Aakash V. Sathappan, MD; John Torous, MD, MBI

Abstract

Mental health conditions can have a large impact on both individual employees and organizations. Employers may be able to leverage wide spread mobile technology platforms to deliver mental health benefits to their employees. Indeed, there is a large inventory of mental health applications (apps) that have already been developed and disseminated to the public. Although mental health apps can be relatively inexpensive and easily accessible by many employees, many companies will want to consider organizational outcomes such as worker presenteeism and absenteeism when making decisions regarding the implementation of such a program. Current scientific evidence surrounding mental health apps within working populations is limited relative to the number of available programs. Practitioners should proceed with caution in recommending mental health apps for widespread use in the workplace. [Psychiatr Ann. 2021;51(2):76–83.]

Abstract

Mental health conditions can have a large impact on both individual employees and organizations. Employers may be able to leverage wide spread mobile technology platforms to deliver mental health benefits to their employees. Indeed, there is a large inventory of mental health applications (apps) that have already been developed and disseminated to the public. Although mental health apps can be relatively inexpensive and easily accessible by many employees, many companies will want to consider organizational outcomes such as worker presenteeism and absenteeism when making decisions regarding the implementation of such a program. Current scientific evidence surrounding mental health apps within working populations is limited relative to the number of available programs. Practitioners should proceed with caution in recommending mental health apps for widespread use in the workplace. [Psychiatr Ann. 2021;51(2):76–83.]

Anxiety and depression cost the global economy $1 trillion annually in decreased productivity alone.1 Working people with depression are estimated to have an of average of 4 times more lost productive working time compared to their colleagues without depression.2 The high costs incurred by mental health conditions through missed work time (absenteeism) and less productive time at work (presenteeism) compel many employers to invest in workplace mental health programs.3 One survey of large companies in the United States documented the availability of such initiatives including employee assistance programs (EAPs) (90%), mental health insurance coverage (87%), substance abuse treatment coverage (73%), mental health assessment (35%), and stress management programs (23%).4

What Are Mental Health Apps?

Mental health mobile applications (apps) offer an attractive supplement or alternative to the historical mainstays of employee mental health programs. Apps (including those targeting mental health) refer to software programs that are available for download onto smart-phones, tablets, and other digital mobile devices. Most mental health apps deliver a self-help psychological treatment/education program targeted at specific conditions and are often based on established psychotherapy principles.5 However, other mental health apps are designed to promote “mental well-being,” deliver “cognitive training,” or otherwise provide preventive mental health care even in the absence of a specific diagnosis. It is no exaggeration to state that mental health apps have exploded onto the therapeutic landscape, with over 10,000 such programs available in 2017.6

Why Use Mental Health Apps in the Workplace?

Companies have followed the general surge of interest in mental health apps due to advantages such as on-demand accessibility, low cost, wide availability, and (relative) anonymity.7 An especially significant aspect of mental health apps is the potential for implementation on a massive scale; more than 80% of adults in the US are now smartphone users,8 and 76% of smartphone users report they would use a mental health app.9 This is in contrast to utilization rates of EAPs that are as low as 1% to 5%.10 A mental health program that is available on demand and from the convenience of a mobile device is another appealing aspect of mental health apps, especially for workers who may face workplace stigmatization for disclosing a mental health condition.

Employers may also appreciate the consistent user experience of mental health apps among their workers, as opposed to the potentially variable satisfaction different workers may report with a particular EAP clinician. Employees have reported that anonymity and convenience are especially attractive aspects of mental health apps compared to traditional models of workplace-based mental health programs.11

In the current setting of the severe acute respiratory syndrome coronavirus 2 pandemic, many workplaces are significantly disrupted. Onsite operations, in addition to wellness programs, have been moved to virtual platforms. As some workplaces begin to reopen in limited capacities, employers will need to consider additional remote opportunities for employee wellness offerings, including mental health initiatives.

What Is the Current Evidence for Mental Health Apps in the Workplace?

Methods

We searched Pubmed and Google Scholar from January 2020 until April 2020. Search strategies were drafted by an academic clinician with additional research-focused training. Additional sources were found by reviewing the references of articles identified in the search. We grouped the studies based on target outcomes, quality of the evidence including study design (randomized controlled trials) and also the reliability, validity, and clinical relevance of the psychometric scales used to measure outcomes.

Eligibility Criteria

We included only experimental studies, with a single group or randomized controlled design. Studies were required to include a mental health-based intervention delivered by mobile apps to employees, managers, or other working populations. We excluded studies of apps in a non-working population or with an intervention delivery method by online- or web-based measures. To be included, we required that studies be published in peer-reviewed journals, written in English, and report measured outcomes of mental health, physical health, a psychosocial outcome (stress, resilience, wellbeing), and/or occupational outcomes (productivity, absenteeism, presenteeism) via a validated scaled measure (Table 1).

Summary of Studies Surrounding Mental Health Apps in Occupational PopulationsSummary of Studies Surrounding Mental Health Apps in Occupational PopulationsSummary of Studies Surrounding Mental Health Apps in Occupational Populations

Table 1:

Summary of Studies Surrounding Mental Health Apps in Occupational Populations

Results

Discussion

Mental health apps are marketed to employers to promote the mental wellbeing of individual employees, and thereby benefit the company overall. The supporting evidence for these claims alludes to an association between use of the app and increased productivity, relying on inference as opposed to scientific method.

For example, the resource “Head-space for Work” ( https://www.headspace.com/work) attempts to build a logical chain between reduced employee stress and improved organizational monetary outcomes. The resource cites occupational health studies, which note an association between stress and monetary losses, as well as articles that identify an association between mindfulness and a reduction in stress. Finally, Headspace for Work” highlights the importance of mindfulness and its effects on teamwork, creativity, and cognition.

Most of the app-based literature makes inferences based on numerous limitations of study design; the most glaring of which is a failure to assess human functioning. For instance, does the measured outcome have a meaningful effect on the patient's affect and behavior? Or does it simply represent a reduction on a psychological scale? One could argue that trait-based variables such as stress, well-being, and happiness are less detrimental to function compared to mental health disorders. Loss of functioning or impairment in work/play is a key criterion for the diagnosis of depression and anxiety disorders based on the Diagnostic and Statistical Manual of Mental Disorders, 5th edition.12 As such, apps that claim significant success in preventing or treating psychological disorders may have a higher likelihood of improving functional outcomes.

However, this does not mean that a reduction on a scale of anxiety or depression is necessarily clinically (let alone functionally/occupationally) relevant. It is possible, for instance, that a given change on a symptomatic scale is not large enough to have a plausible clinical effect. In other words, if a mental health app is meant to decrease depressive symptoms, does a given reduction in the Patient Health Questionnaire-9 (PHQ-9) score then result in a significant and relevant effect on occupational or enterprise outcomes?13 For example, does a lower score on the Perceived Stress Scale (PSS-10) cause a clinically meaningful reduction in stress to improve productivity in the work environment (Work Productivity and Impairment Questionnaire)?

Strength of evidence for claims that apps improve occupational outcomes depends on statistically significant direct effects demonstrated in experimental trials. These trials should use validated physical health, mental health, and/or psychosocial scales that are either clinically relevant and/or demonstrate a direct effect on occupational outcomes (via similarly well validated and clinically relevant scales). The ideal study to validate the claims of a workplace-based mental health app intervention would be a randomized controlled trial conducted among employees from businesses of varying sizes and industries. Outcome measures should include validated and clinically relevant scales of mental health (anxiety, depression), psychological wellbeing, and occupational metrics. Based on our review of the literature, this study does not yet exist.

In this article, we identified 12 studies published between 2014 and 2019 (Table 1) from the primary literature that met our inclusion criteria. Five studies were designed with a single-group and seven studies were randomized controlled trials. Certain studies directly measured changes in neuropsychiatric outcomes (ie, depression, anxiety) among the target population, which we have labeled “direct.” Other studies measured changes in related outcomes, including physical health, well-being, and productivity, which we have labeled as “indirect.” Of note, several studies included both “direct” and “indirect” outcomes measures (Table 1). We considered these studies in the context of the “direct” or “indirect” outcomes they reported.

Among the “direct” cohort of studies,14–17 target outcomes focused on reduction of depression and anxiety in employees. All four of these studies demonstrated that use of a mental health app caused a significant reduction in scores on psychometric scales for depression and/or anxiety among workers. The findings of Bostock et al.14 and Birney et al.15 are most relevant and applicable given their large sample size, randomized controlled design, and broad targeted population of employees. Of note, Birney et al.15 is the only study included in our review where the recruited employees demonstrated mild to moderate depression (per PHQ-9 rating scale) at baseline. The other studies either included healthy employees14,17 or those who self-reported mental health symptoms.16

A peculiar finding from the Birney et al.15 study was that among those workers randomized to the cognitive-behavioral therapy app, employees who were recruited through an EAP demonstrated a more robust reduction in depressive symptoms compared to the non-EAP workers. This raises the question of whether app-based interventions in the workplace are generally more effective when they are introduced and supported by face-to-face mental health programs.

Although there are changes in scores on depressive scales across these four studies, deeper evaluation is necessary to determine clinical relevance. Birney et al.15 report a mean difference between app use and no app use of 2.6 on the PHQ-9.15 Similarly, Deady et al.17 found statistical significance with a reduction in PHQ-9 scores of 2.3. Bostock et al.14 used a less common clinical scale (the Hospital Anxiety and Depressive Scale), which ranges from 0 to 21. The statistically significant change in those employees using the app was 1.69 in anxiety scores and 1.45 in depressive scores.14 Lappalainen et al.16 used the original Beck Depression Inventory and noted a reduction in depression scores by 8.46 in their intervention arm. However, the control arm also showed a reduction in scores by 4.16 Clinical relevance seems unlikely given the total score is out of 63.

The “indirect” outcome measures that were most studied were “reduction in stress”16–23 followed by improvement in psychological well-being.14,17,20,22,23 Most of these studies found that people assigned to the app-based intervention group reported significant improvements in stress and psychological well-being with respect to the controls. Other “indirect” outcomes studied include mindfulness24 and general well-being.19

Only one study included an active control group, in which patients were guided through self-observation and self-direction programming that spanned the intervention duration.20 The authors reported that the intervention group showed significant improvement in general well-being (World Health Organization-Five Well-Being Scale), work-related well-being, perceived stress (PSS), work-related stress, and general stress compared to the active control group. Another study examined a mental health app targeted specifically at middle managers versus general employees.18 Heber et al.19 was one of the few studies included in our analysis that reported long-term follow up data. At the 6-month follow-up, the intervention group continued to have significantly lower PSS-10 scores compared to the wait-list control group.19

None of the studies we identified evaluated the effects of app usage in the workplace on broad, organization-level outcomes including company productivity and sales growth. One study, however, noted an “indirect” effect of mental health apps on individual organizational outcomes in the workplace including a reduction in absenteeism and an increase in work performance.17 Another study noted statistically significant secondary outcomes in job control and job social support when employees used a mental health app for 8 weeks.14 But two other articles showed no significant improvement in worker engagement22 or in the sense of social community at work,23 respectively.

Future Work

Although the overall results of these studies seem to be promising, there are several broad, thematic limitations to be addressed by future studies before we can confidently recommend adoption of workplace mental health app interventions (Table 2).

Limitations to Adopting Workplace Mental Health App Interventions

Table 2:

Limitations to Adopting Workplace Mental Health App Interventions

Conclusions

Mental health apps represent an exciting opportunity for organizations to leverage an already widely distributed and utilized platform for the purposes of improving the mental health of their workforce. Although a major limitation of the digital mental health field in general is the lack of regulation and rigorous scientific evaluation, there are individual studies and even some meta-analyses that demonstrate benefit from such programs.

Clinicians and other experts who are asked to evaluate a particular mental health app for potential implementation into a workplace should take into consideration the general limitations in digital mental health research noted in the section above. Individual mental health apps should be evaluated based on the evidence available for the specific program in question and the selected outcome(s) of interest. In addition, the application of a formalized framework for the analysis of mental health apps may be useful for decision-making processes.25

References

  1. World Health Organization. Mental health in the workplace. Accessed January 12, 2021. https://www.who.int/news-room/commentaries/detail/mental-health-in-the-workplace
  2. Stewart WF, Ricci JA, Chee E, Morganstein D. Lost productive work time costs from health conditions in the United States: results from the American Productivity Audit. J Occup Environ Med. 2003;45(12):1234–1246. doi:10.1097/01.jom.0000099999.27348.78 [CrossRef] PMID:14665809
  3. Johnston DA, Harvey SB, Glozier N, Calvo RA, Christensen H, Deady M. The relationship between depression symptoms, absenteeism and presenteeism. J Affect Disord. 2019;256:536–540. doi:10.1016/j.jad.2019.06.041 [CrossRef] PMID:31280078
  4. International Foundation of Employee Benefit Plans. Workplace wellness trends: 2019 survey report. Accessed January 12, 2021. https://www.ifebp.org/store/Pages/work-place-wellness.aspx
  5. Christensen H, Hickie IB. Using e-health applications to deliver new mental health services. Med J Aust. 2010;192(S11):S53–S56. doi:10.5694/j.1326-5377.2010.tb03695.x [CrossRef] PMID:20528711
  6. Torous J, Roberts LW. Needed innovation in digital health and smartphone applications for mental health: transparency and trust. JAMA Psychiatry. 2017;74(5):437–438. doi:10.1001/jamapsychiatry.2017.0262 [CrossRef] PMID:28384700
  7. Firth J, Torous J, Yung AR. Ecological momentary assessment and beyond: the rising interest in e-mental health research. J Psychiatr Res. 2016;80:3–4. doi:10.1016/j.jpsychires.2016.05.002 [CrossRef] PMID:27236099
  8. Pew Research Center. Mobile fact sheet. Accessed January 12, 2021. https://www.pewre-search.org/internet/fact-sheet/mobile/
  9. Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR Ment Health. 2016;3(1):e7. doi:10.2196/mental.4984 [CrossRef] PMID:26932350
  10. Sharar D. Where do we go from here? Improving value and pricing in EAP. J Employee Assistance. 2019;49(2):10–13.
  11. Carolan S, de Visser RO. Employees' perspectives on the facilitators and barriers to engaging with digital mental health interventions in the workplace: qualitative study. JMIR Ment Health. 2018;5(1):e8. doi:10.2196/mental.9146 [CrossRef] PMID:29351900
  12. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. American Psychiatric Publishing; 2013.
  13. Howarth A, Quesada J, Silva J, Judycki S, Mills PR. The impact of digital health interventions on health-related outcomes in the workplace: a systematic review. Digit Health. 2018;4:2055207618770861. doi:10.1177/2055207618770861 [CrossRef] PMID:29942631
  14. Bostock S, Crosswell AD, Prather AA, Steptoe A. Mindfulness on-the-go: effects of a mindfulness meditation app on work stress and well-being. J Occup Health Psychol. 2019;24(1):127–138. doi:10.1037/ocp0000118 [CrossRef] PMID:29723001
  15. Birney AJ, Gunn R, Russell JK, Ary DV. MoodHacker mobile web app with email for adults to self-manage mild-to-moderate depression: randomized controlled trial. JMIR Mhealth Uhealth. 2016;4(1):e8. doi:10.2196/mhealth.4231 [CrossRef] PMID:26813737
  16. Lappalainen P, Kaipainen K, Lappalainen R, et al. Feasibility of a personal health technology-based psychological intervention for men with stress and mood problems: randomized controlled pilot trial. JMIR Res Protoc. 2013;2(1):e1. doi:10.2196/resprot.2389 [CrossRef] PMID:23611946
  17. Deady M, Johnston D, Milne D, et al. Preliminary effectiveness of a smartphone app to reduce depressive symptoms in the workplace: feasibility and acceptability study. JMIR Mhealth Uhealth. 2018;6(12):e11661. doi:10.2196/11661 [CrossRef] PMID:30514694
  18. Ly K, Asplund K, Andersson G. Stress management for middle managers via an acceptance and commitment-based smartphone application: a randomized controlled trial. Internet Interv. 2014;1(3):95–101. doi:10.1016/j.invent.2014.06.003 [CrossRef]
  19. Heber E, Lehr D, Ebert DD, Berking M, Riper H. Web-based and mobile stress management intervention for employees: a randomized controlled trial. J Med Internet Res. 2016;18(1):e21. doi:10.2196/jmir.5112 [CrossRef] PMID:26818683
  20. Coelhoso CC, Tobo PR, Lacerda SS, et al. A new mental health mobile app for well-being and stress reduction in working women: randomized controlled trial. J Med Internet Res. 2019;21(11):e14269. doi:10.2196/14269 [CrossRef] PMID:31697244
  21. Ahtinen A, Mattila E, Välkkynen P, et al. Mobile mental wellness training for stress management: feasibility and design implications based on a one-month field study. JMIR Mhealth Uhealth. 2013;1(2):e11. doi:10.2196/mhealth.2596 [CrossRef] PMID:25100683
  22. Muuraiskangas S, Harjumaa M, Kaipainen K, Ermes M. Process and effects evaluation of a digital mental health intervention targeted at improving occupational well-being: lessons from an intervention study with failed adoption. JMIR Ment Health. 2016;3(2):e13. doi:10.2196/mental.4465 [CrossRef] PMID:27170553
  23. Weber S, Lorenz C, Hemmings N. Improving stress and positive mental health at work via an app-based intervention: a large-scale multi-center randomized control trial. Front Psychol. 2019;10:2745. doi:10.3389/fpsyg.2019.02745 [CrossRef] PMID:31866915
  24. Wen L, Sweeney TE, Welton L, Trockel M, Katznelson L. Encouraging mindfulness in medical house staff via smartphone app: a pilot study. Acad Psychiatry. 2017;41(5):646–650. doi:10.1007/s40596-017-0768-3 [CrossRef] PMID:28795335
  25. Torous JB, Chan SR, Gipson SYT, et al. A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. Psychiatr Serv. 2018;69(5):498–500. doi:10.1176/appi.ps.201700423 [CrossRef] PMID:29446337
  26. Jiang Y, Jiao N, Nguyen HD, et al. Effect of a mHealth programme on coronary heart disease prevention among working population in Singapore: a single group pretest–post-test design. J Adv Nurs. 2019;75(9):1922–1932. doi:10.1111/jan.13980 [CrossRef] PMID:30786051

Summary of Studies Surrounding Mental Health Apps in Occupational Populations

Study Target Mental health app Study design Sample size/population Primary endpoint duration Primary outcome measure/psychometric scale Result highlights Effect
Bostock et al.14 Psychological well-being, depression, and anxiety Headspace RCT/intervention (n= 128) vs. control (n= 110) N = 228/office-based employees from two United Kingdom Fortune 500 companies (pharmaceutical and big-tech) 8 weeks; 16-week follow-up Psychological well-being (WEMWBS), depression/anxiety (HADS), job strain (Whitehall II Questionnaire) Intervention group (Head Space) reported improvementa in WEMBS, HADS, Whitehall II Questionnaire vs. control; sustained positive effectsa for Whitehall II Questionnaire at 16 weeks in intervention group Direct, indirect
Birney et al.15 Depression Mood Hacker (CBT) RCT/intervention (n= 150) vs. control (n= 150) N= 300/employees who had mild to moderate depression (PHQ-9) 6 weeks Patient Health Questionnaire (PHQ-9) Intervention group (MoodHacker) had lower PHQa than control Direct
Lappalainen et al.16 Stress-related psychological problems, depression P4Well (CBT and ACT) RCT/intervention (n= 11) vs. control (n= 12) N = 11/employed men 3 months; 6-month follow-up Depression (BDI), psychological symptoms (SCL-90), stress/burnout (BBI-15) Intervention group (P4Well) showed significant reductiona in depressive symptoms (BDI) compared to control group; participants maintained positive changes at 6-month follow-up Direct, indirect
Deady et al.17 Well-being, resilience, depression/anxiety, work performance HeadGear (Behavioral activation and mindfulness) 1-arm trial N= 84/industries with mostly men employees (agriculture, engineering, transport, forestry, mining, plumbing, and construction) 5 weeks Depression/anxiety (PHQ-9, GAD-2, PHQ-2), resilience (CD-RISC), well-being (WHO-5), work performance (WHO-HPQ) Significant reductionsa in absenteeism, depression, and anxiety and increasesa in worker performance compared to baseline Direct, indirect
Ly et al.18 Stress ACT-based app RCT/intervention (n= 36) vs. control (n= 37) N = 73/middle managers working for mid to large companies (>50 employees) 6 weeks General health (GHQ-12), stress (PSS) Intervention group (ACT app) had lower GHQ-12a and PSSa scores vs. control Indirect
Heber et al.19 Stress GET.ON Stress RCT/intervention (n= 132) vs. control (n= 132) N= 264/employees recruited from a large German health insurance company 7 weeks; 6-month follow-up Stress (PSS-10) Intervention group (GET.ON Stress) had lower PSSa at 7 weeks and 6-month follow-up vs. control Indirect
Coelhoso et al.20 Stress, psychological well-being Well-Being RCT/intervention (n= 250) vs. control (n= 240) N= 490/women working at a large private tertiary care hospital in Sao Paulo, Brazil 8 weeks Stress (PSS-10), well-being (WHO-5) Intervention group (Well Being App) showed increasea in (WHO-5), work-related well-being,a and reductiona in PSS, work-related stress, and general stress vs. control Indirect
Ahtinen et al.21 Stress management Oiva (ACT) 1-arm trial N= 15/staff of a technical university 4 weeks Stress (single-item stress scale), life satisfaction (SWLS), psychological flexibility (AAQ-II) Significanta pre-post improvements were obtained in stress ratings and SWLS, but not in psychological flexibility Indirect
Muuraiskangas et al.22 Well-being, stress ACT-based app 1-arm trial N= 27/human resource managers and employees of two information and communication technology companies Baseline, 2 months, 4 months; 5-month follow-up Stress (1-item stress questionnaire), work engagement (UWES-9) No significance in primary outcome measures; poor adoption (27 of 332) employees and poor engagement (mean use of 4.8 different days) Indirect
Weber et al.23 Well-being, stress, resilience, sleep, social community, work productivity, physical health impairment Kelaa Mental Resilience RCT/intervention (n= 210) vs. control (n= 332) N= 532/6 different European businesses in Germany, England, and Northern Ireland from the private and public sector 4 weeks; 2-week follow-up Stress (COPSOQ II), well-being (WEMWBS), resilience, sleeping troubles, social community at work, physical health impairment, work productivity, and activity impairment Intervention group (Kelaa Mental Resilience) experienced decreasea in general plus cognitive stress and improvementa in well-being compared to waitlist control group Indirect
Wen et al.24 Mood, mindfulness Headspace 1-arm trial N = 43/medical residents 4 weeks Affect (PANAS) for mood, mindfulness inventory (FMI) for mindfulness Intervention group (Head-space) had an increase in FMI,a a trend toward increase in PAS, no change in NAS vs. control Indirect, direct
Jiang et al.26 Heart health Care4Heart 1-arm trial N = 160/workers in Singapore 4 weeks; 6-month follow-up Heart disease knowledge (HDFQ-2), BRFSS, and stress (PSS-10) Intervention group (Care4Heart) had higher total mean scorea for HDFQ-2 at 4 weeks and 6-month follow-up vs. control. No significant differences in BRFS or PSS between groups Indirect

Limitations to Adopting Workplace Mental Health App Interventions

Characteristic Limitation explanation
Digital placebo effect Most literature fails to use an active control group as part of the randomized controlled design. This shortcoming increases the chance of participant bias via the “digital placebo effect” (placebo-like effects seen from mobile health interventions irrespective of their efficacy)
Sample size Most studies to date have demonstrated small to medium effect sizes that could be improved with recruitment of large sample sizes, thereby increasing power
Generalizability Most studies recruited participants from a specific target population (ie, financial sector workers, agricultural workers), so application of results to the broader workforce or different industries is limited
Follow-up data Durability of effects or attenuation of outcomes cannot be commented on due to a general lack of long-term follow-up data
Selection bias Most participants that volunteered for the studies represented a “convenient” sampling of self-interested employees
Engagement Several studies failed to comment on app adherence rates and daily usage activity, limiting our interpretation of results
Objective measures Most studies measured outcomes based on self-reported rating scales. Objective occupational measures (eg, employee productivity/performance scores) may be especially compelling to employers to support implementation of mental health apps
False positives Multiple hypothesis testing can lead to false-positive results
Variability Due to large variability in how people navigate digital apps and the presence of multiple different modules within individual apps, it is difficult to conclude which elements of the app contributed to the results observed and which elements were nonsignificant. Future studies should monitor adherence and engagement within specific elements of the app to account for this variation
Authors

Nathan M. Jones, MD, MPH, is an Occupational Medicine Resident Physician, Harvard T.H. Chan School of Public Health. Matthew Johnson, MD, is a Psychiatry Resident Physician, Beth Israel Deaconess Medical Center. Aakash V. Sathappan, MD, is a Psychiatry Resident Physician, Beth Israel Deaconess Medical Center. John Torous, MD, MBI, is the Director, Digital Psychiatry Division, Beth Israel Deaconess Medical Center.

Address correspondence to Nathan M. Jones, MD, MPH, Harvard T.H. Chan School of Public Health, 665 Huntington Avenue, Building 1, 14th Floor, Suite 1406, Boston, MA 02115; email: nathanmjonesmd@gmail.com.

Disclosure: The authors have no relevant financial relationships to disclose.

10.3928/00485713-20210112-01

Sign up to receive

Journal E-contents