4.2.1 Collecting tool information and user reviews

In March 2019, we collected additional information about these tools (including number of users, average ratings, and their latest store descriptions), as well as user reviews publicly available on their store pages. For apps on the Google Play store, we used a pre-existing script (Olano 2018b) to download tool information and reviews. For apps on the Apple App store, we used a pre-existing script (Olano 2018a) to download reviews, and wrote our own script (using R packages rvest and RSelenium) to collect tool information. As the script for obtaining reviews from the Apple App store sometimes failed to retrieve all reviews visible on tools’ store page, we wrote an additional R script for collecting reviews when manual inspection found some to be missing. For extensions on the Chrome Web store, we wrote an R script to download tool information and reviews.

4.2.2 Analysing user numbers and ratings

All stores displayed average ratings as well as number of ratings. However, the stores provided different types of information in relation to user numbers: The Chrome Web store provided an exact number of users for an extension, and the Google Play store provided a ‘minimum number of installs’ (e.g. ‘100,000+’). The Apple App store provided no direct information about user numbers. For tools on the Apple App store we therefore used number of ratings as a proxy for relative number of users in analyses where tools were ranked by number of users.

4.2.3 Thematic analysis

To cover the design space, we first sampled up to 20 reviews from each of the 5 tools with the highest number of users, from each store⁶, from each main functionality type that were assigned to tools as their first category. As functionality types were not evenly distributed across stores (for example, blocking tools are rare on the Apple App store), we subsequently randomly sampled additional reviews, from tools targeting each type of functionality, for a total of 961 reviews (cf. Roffarello and De Russis 2019a).

Next, we conducted an inductive thematic analysis, following the ‘reflexive’ approach described in V. Braun and Clarke (2006) and Virginia Braun et al. (2018). First, we read through all the reviews and conducted an initial coding of recurrent patterns in the data relevant to our research aims. Subsequently, we read through all extracts associated with each code, recoded instances, and iteratively sorted codes into potential themes, and discussed emerging themes.

We used the Dedoose software for thematic analysis, and R for all quantitative analyses.

4.3.1 User numbers and ratings

For DSCTs from the Google Play store, the median ‘minimum number of installs’ was 10,000 (min = 5, max = 5,000,000), from the Chrome Web store, the median number of users was 194.5 (min = 1, max = 1,736,018)), and from the Apple App store, the median number of ratings was 0 (min = 0, max = 14,900). The median average rating given to tools (for tools with more than 30 ratings) was 4.3 (individual ratings range from 1 to 5 stars on all three stores; 5 is best).

Figure 4.1 shows top tools in terms of number of users and their types of features (on the Apple App store, ranked by number of ratings; only 13 tools from this store had received any ratings). The distributions of feature types in top tools differed by store, mirroring differences between stores overall (cf. Chapter 3) — for example, block/removal functionality is rare on the Apple App store compared to the Google Play and Chrome Web store.

Figure 4.2 displays top tools in terms of average rating (to obtain a representative sample underlying the average, this figure includes only tools with more than 30 ratings; only 5 tools from the Apple App store had received more than 30 ratings). 59% (37 out of 63) of tools shown in Figure 4.2 also appear in Figure 4.1, suggesting that tools with higher user numbers have a large overlap with tools with higher ratings.

Reward/punishment features were the least common in top tools, but some tools implementing such features ranked highly: Forest and Flora, which both gamify self-control by growing virtual trees or plants, topped the number of users on the Google Play and Apple App stores, and Forest was in the top 5 on the Chrome Web store.

Goal advancement features were more common among top tools ranked by average rating (32%, 20/63) than by number of users (25%, 16/63), and than the overall prevalence of this feature type (22% of tools, 75/334).

Across all tools, 64% implemented a single type of design pattern, whereas 32% combined two — very few combined three (3%) or all four (1%, cf. section 3.3.2.2). Tools whose features combined two or more types of design patterns had significantly higher average ratings than tools which implemented a single type (comparing tools with more than 30 ratings; median average rating with a single type = 4.2, median for tools combining two or more = 4.39, p = 0.008 in Wilcoxon’s rank-sum test, cf. Figure 4.3).

4.3.2 User reviews

The median number of reviews for a tool was 6 (min = 1, max = 4600), and the median number of words in a review was 14 (min = 1, max = 640). Figure 4.4 shows the overall frequency of terms used in reviews, excluding the terms ‘app/s,’ ‘phone’ and ‘extension.’ Apart from ‘app’ (46% of reviews), the most frequent term was ‘time,’ which was included in 17.5% of all reviews. The terms most commonly following or preceding ‘time’ included ‘usage,’ ‘limit,’ ‘screen,’ ‘spent,’ ‘set,’ ‘management,’ and ‘wasting’ (see Table 4.1), suggesting that managing time spent on digital devices was one of the most central topics.

4.4.1 The landscape of DSCTs as commentary on default designs

The landscape of DSCTs available online provides a window into user needs and struggles in the face of common designs. The high average ratings given to these tools overall, as well as the content of user reviews, suggest that many people (at least among those that choose to try them out) find them very useful. Thus, tools on this market foreshadowed generic user needs that Google and Apple recently begun to address in the form of ‘Digital Wellbeing’ (Google) and ‘Screen Time’ (Apple) tools for monitoring and limiting device usage (cf. section 2.1.4). They also suggest specific design solutions, such as browser extensions that tackle information overload by hiding one’s email inbox until one makes a deliberate choice to see it (e.g., Inbox When Ready) or hiding Facebook’s newsfeed (e.g., Newsfeed Eradicator).

Comparing popularity metrics, our results suggest some potential disconnect between tools’ usefulness and how widespread they are. I speculate that our finding that tools which include goal advancement features rank higher on average ratings than on user numbers, and that some reward/punish tools rank at the very top of user numbers despite being rare, can be explained by how attention-grabbing they are: for example, scaffolding self-control using virtual trees that die if one’s device is used inappropriately might be more ‘sexy’ and likely to be shared with others than are tools which provide simple but useful reminders of the task one tries to focus on. This would match research on online virality which has found that content evoking high-arousal emotions are more likely to be shared (Berger and Milkman 2012; Pressgrove, McKeever, and Jang 2017), as well as findings in behaviour change research where strategies’ popularity and their efficiency often diverge (Kelly and Barker 2016). An implication is that whereas store metrics of popularity may provide useful clues about the utility of various design patterns, they should be used as a rough guide for subsequent controlled studies rather than taken as ‘ground truth’ (cf. Donald A. Norman 1999).

4.4.2 Self-control struggles and the ‘just right’ level of friction

Our finding that DSCTs are used when people focus on important but effortful tasks matches findings from research using other methods with, e.g., Tran et al. (2019)‘s recent interview study finding that engagement in tedious or effortful activities is a common trigger of compulsive smartphone use (also Reinecke and Hofmann 2016). Similarly, the search for tools that provide an optimal level of friction can be conceptualised as use of ’microboundaries’ (Cecchinato, Cox, and Bird 2017) and ‘design frictions’ (Cox et al. 2016) to promote goal-oriented engagement with digital technology, and aligns with wider research in personal informatics where, e.g., people planning for exercise try to set up a ‘sweet spot’ of contextual factors to succeed (Paruthi et al. 2018).

To understand the psychological mechanisms involved, the dual systems framework presented in Chapter 3 may provide a useful lens. To recap, according to this framework, behaviour results from internal competition between potential actions. Potential actions are activated either by automatic responses to internal and external cues (‘System 1’ control, e.g., checking one’s phone out of habit) or by conscious intentions (‘System 2’ control, e.g., taking out one’s phone because one has a conscious goal to text a friend). What wins out in behaviour is the action with the highest activation value, which may or may not be in alignment with one’s longer-term goals (e.g., checking one’s smartphone in a social situation out of habit despite having a goal to not do so).

From this perspective, DSCTs are used in situations where the internal struggle between potential actions are often not resolved in line with one’s enduringly valued conscious goals (cf. section 2.2.2). Users seek tools that provide a level of friction or motivation which biases this competition such that the actions that win out are in line with one’s longer-term goals — for example by using blocking tools to prevent unwanted System 1 responses from being triggered, by using goal advancement tools to remind oneself of one’s usage goals and thereby enable System 2 control, or by using reward/punish tools to provide extra incentives for System 2 control (cf. Chapter 3).

We suggest the finding that tools whose features combine more than one type of design pattern can usefully be interpreted through this lens: tools that combine two or more types of patterns (for example, a tool which does not simply remove Facebook’s newsfeed (block/removal), but replaces it with a todo list (goal advancement)) may be more likely to bias internal action competition sufficiently to cause a change in behaviour without being overly restrictive or annoying, than are tools which implement a single type. This may be because by combining design patterns that target different psychological mechanisms, each can be implemented at a more ‘gentle’ level of friction or intrusiveness while still in aggregate provide sufficient bias to the self-regulatory system to bring about an intended behaviour change. By contrast, tools focusing on a single design pattern targeting a single psychological mechanism may need to ramp up the level of friction more highly before sufficient impetus for behaviour change is provided — and in so doing is at greater risk of being perceived as annoying or ‘too strict’ by the average user.

For example, Forest combines growing of virtual trees with a countdown timer, which influences both the expected reward for control and sensitivity to delays, which seems to be effective, given this tool’s popularity with over 10 million users on Android alone. A version which simply grew virtual trees without a countdown timer might need to correspondingly increase the magnitude of reward to remain useful, but in doing so be more likely to be perceived as too forceful or manipulative (cf. M. K. Lee, Kiesler, and Forlizzi 2011).

4.4.3 The meaning of distraction

An important user need emerging from the reviews is that users look for digital self-control tools that capture their personal definitions of ‘distraction.’ We suggest that this has implications in two domains: customisation, and cross-device integration.

In terms of customisation, many tools received a combination of excited reviews from users who felt it matched their needs well, and reviews from less satisfied users who wanted specific features to be added to capture their situation (e.g., exempting particular apps or subdomain from being blocked, limiting weekly rather than daily usage, or setting custom schedules). For example, the Forest app on iOS (Seekrtech 2018), which rewards users for not using their smartphone at all during focus sessions, addresses the concerns of a student who considers all phone use while studying a distraction, but not those of a student who wants to access math apps. A conceptual implication is that studies of digital self-control tools should focus on assessing effects on people’s sense of using their devices in ways that align with their long-term goals, rather than on overall ‘screen time’ (cf. section 2.2). A design implication is that tools should include adequate controls for users to customise them to their needs or that the ecosystems of tools should be diverse enough for users to find tools that match their needs.

There may be a trade-off between customisation and friction in a single tool: providing more fine-grained control runs the risk of increasing the amount of effort involved in using the tool in the first place, which could undermine usefulness. Apple and Google may therefore be facing distinct challenges: Because Apple do not allow mobile developers the level of permissions required to implement proper usage logging and blocking tools (Digital Wellness Warriors 2018), only their own Screen Time app is capable of providing many basic features of DSCTs. This implies that this single app needs to be carefully designed to accommodate a wide range of user needs. Google, however, provide less restrictive permissions to developers, and an accordingly diverse range of potent DSCTs are therefore available on Android. Therefore, Google may face less pressure to ‘get it right’ with their own Digital Wellbeing tools, because less common user needs could be addressed by their app ecosystem.

In terms of cross-device integration, reviews suggested that tools should be sensitive to the overall affordances provided by the devices people use in combination. A research implication is that studies of digital self-control tools should move beyond considering a single device in isolation and towards studying the ecology of devices that people use, and what ‘distraction’ means in this context (cf. section 2.2). A design implication is that tools should be able to sync between devices, to accommodate a need to curb the availability of all digital distractions. This has been explored in two previous studies (cf. section 2.3.4) with, e.g., J. Kim, Cho, and Lee (2017) finding that multi-device blocking reduces amount of mental effort required to manage self-interruptions.

4.4.4 Limitations

Many user reviews provide little depth and may consist of only generic praise or critique, an angry bug report, be very short, or intend to paint a tool in a specific light for marketing purposes. Furthermore, we may not have an equal amount of data for all areas of the design space, and lack of reviews could represent poor marketing efforts or bad luck, rather than design ideas with low utility. Therefore, care must taken when interpreting evidence from user reviews and ratings (cf. Panichella et al. 2015).

In this chapter, we sampled reviews from top tools across the design space, similar to previous research (Roffarello and De Russis 2019a). The fact that our findings from reflexive thematic analysis resonated well with findings from studies using other methods such as surveys and interviews (Ko et al. 2015; Tran et al. 2019) suggest that this approach did provide valid evidence. However, future research may wish to leverage e.g. automated ways of identifying high-informative reviews in advance of thematic analysis (N. Chen et al. 2014; Panichella et al. 2015).