A Survey of Notification Designs in Commercial mHealth Apps

Overview

mHealth app notifications can aid in behavioral change by increasing users’ app engagement and adherence to health objectives. To explore how mHealth apps can help users achieve their health objective via push notifications, we investigated if there was a gap between notification design recommendations in the literature and practice by surveying notifications from 50 mHealth apps. We found disconnects in notification occurrence and content. We are pleased to share that our paper “A Survey of Notification Designs in Commercial mHealth Apps” was accepted to ACM SIGCHI 2021 as a Late-Breaking Work. Here is our presentation at the CHI 2021 virtual conference.

Motivation and Project Goal

mHealth applications can aid in supporting users’ behavioral change toward healthier habits (e.g., increasing activity levels and quitting smoking) through many techniques, such as goal setting, self-monitoring, gamification, and notifications. App notifications have been shown to help users achieve their health goals through increasing app engagement and commitment to health goals in various mHealth contexts such as mental disorder treatment, weight loss maintenance, and sleep.

Previous studies have established empirically-derived notification design recommendations, such as sending notifications based on the user’s context, tailoring messages to the specific user, varying content, and allowing for user customization. However, prior work has found that few commercial mHealth apps are either grounded in health behavior theories or properly evaluated. Therefore, we wanted to examine if existing evidence-based design recommendations are being implemented in current mHealth apps.

Method

To examine how notifications are utilized in current mobile health (mHealth) applications, we surveyed 50 mHealth apps. We (1) identified 50 mHealth apps using a systematic search process, (2) iteratively generated a coding scheme of notification dimensions based on prior work, (3) downloaded and interacted with each app for five days, and qualitatively coded each notification that appeared using our coding scheme.

(1) App Identification

To select a representative sample of mHealth apps, we started with an updated version of Xu and Liu’s database of mHealth apps. The database consists of apps found in the “Health & Fitness” and “Medical” categories from both the US Apple App Store and US Google Play Store. At the time of our study, the database contained 78,734 iOS apps from the Apple App Store and 44,517 Android apps from the Google Play Store.

We excluded apps that had missing information in the database, that were not free, or that had fewer than 10 user reviews (leaving 4,855 iOS apps and 13,045 Android apps). We then applied four keywords to the app descriptions to find apps relevant to our study context: “notif” (capturing notification, notify, etc.), “remind”, “alert”, and “text message.”

To further simplify the subset, we focused on widely used apps by only including apps that had 500 or more user reviews and that were successful (i.e., apps with 3 or more stars). After filtering out the apps, we had a subset of 131 iOS apps and 460 Android apps.

From this subset, we identified 25 iOS apps and 25 Android apps to survey by randomly selecting the apps and confirming they met our final inclusion criteria: they were in English, available to download at the time of analysis, included notifications, not geared towards healthcare professionals, and focused on the goal of behavior change (i.e., tracking and setting goals).

^{_{Themes of the 50 Mobile Health Applications.}}

(2) Notification Coding Scheme

We developed a coding scheme based on prior work [1, 2, 3]. To ensure the code set was reliable, we refined the codes in an iterative process:

We randomly chose 6 apps (3 iOS and 3 Android) from the subset that were not part of the final 50 apps.
Two researchers independently interacted with all 6 apps for 5 consecutive days and coded the notifications that occurred.
After 5 days of coding, a discussion of disagreements and agreements led to the refinement of the coding scheme.

Our final coding scheme included 13 dimensions focused on a notification’s occurrence, content, and personalization, as shown in the table below.

_{^{Coding Scheme of Notification Dimensions.}}

For readers who are interested in knowing more about qualitative coding, here are some resources I find useful:

Chapters from Auerbach & Silverstein’s Qualitative Data: An Introduction to Coding and Analysis
The Long Interview by Grant McCracken
The Essential Guide to Coding Qualitative Data by the development team of the Delve software

(3) App Usage

For surveying the apps, the list of 50 apps was randomly split between the two researchers who did the initial coding. The two researchers independently used each of the apps for 5 consecutive days. To control for variation in notifications across apps, we established a set of defined information criteria to input into the apps. In order to generate as many notifications as possible, we created the values to be equal to, under, and over current health recommendations. As an example, the following screenshot shows the information criteria for food and weight data:

The 5-day usage period was broken down into: (Day 1) the values aligned with health recommendations, (Day 2) the values were under recommendations, (Day 3) the values were over recommendations, and (Days 4 and 5) no values were input to resemble a user either forgetting or becoming disinterested. For days one through three, the two researchers input values in the morning, afternoon, and night following the set information criteria. The researchers independently transcribed and qualitatively coded the notifications along the 13 dimensions in the coding scheme.

Analysis

We accumulated a total of 1,390 notifications for analysis. We found that the mHealth app notifications align with recommendations on including:

Content type (e.g., statements over questions)
Aesthetics (e.g., images, different font sizes)
Notification settings
Notification customization (e.g., changing the time, turning on/off)

We found a disconnect between current practice in mHealth app notifications and recommendations in several areas:

Frequency: Most sent daily (92.45%), but sending more lowers user retention
Content Purpose: The majority were reminders (89.6%), but reminders can hinder habit development
Interactivity: Did not include interactive elements (59.5%), but interactivity increases user engagement
Occurrence: Triggered due to time (84.1%) instead of user’s activities and environment
Sender: Did not explicitly state the sender (83.2%), but including sender increases credibility
Tailoring: Content not tailored to the user (67.4%), but tailoring increases engagement

We identified open research questions for mHealth app notifications:

Occurrence: What contextual triggers would be effective in increasing engagement and adherence in mHealth apps?
Frequency: How often should notifications be sent to keep users engaged in mHealth? Does it differ based on the app theme?
Interactivity: What interactive elements help the most with mHealth app engagement?
Notification Type: Do push or in-app notifications aid in behavioral change?
Goal Type: How do short-term or long-term goal notifications affect behavioral change?
Creating Notifications: How does using default or manual notifications affect mHealth app engagement?

Takeaways

We surveyed notifications from 50 mHealth apps and found a disconnect between current practice and notification design recommendations in several areas: Occurrence, Frequency, Content Purpose, Interactivity, Tailoring, and Sender.
We identified open research questions in the area of mHealth app notifications

Files