At the same time, numerous commercial system providers have demonstrated the positive effects of carousel-based interfaces. The authors of the corresponding publications have proposed algorithmic improvements, e.g., to optimize how the collections are ordered among each other, how they are filled with items, and how labels are assigned (Wu et al., 2016, 2021; McInerney et al., 2018; Bendada et al., 2020; Lo et al., 2021). Singal et al. (2021) even investigated how to implement carousels independently of the underlying algorithms, requiring only standard user-item representations. Based on dimensionality reduction and clustering of the resulting item embeddings, this approach provides a generic way to create item collections, predict their usefulness, and find appropriate labels. Table 1 outlines the various contributions, but also shows that most of the findings stem from offline experiments and (more rarely) online A/B tests. Accordingly, metrics were used that relied purely on item clicks, largely ignoring richer behavioral data such as scrolling, responses to the mere presence of labels or items, and specific characteristics of the domain, user, or situation. Moreover, most comparisons were made against single lists, since authors (including academics) tried to optimize each collection individually (cf. Bendada et al., 2020; Jeunen and Goethals, 2021).

However, there are some notable exceptions, where the authors have attempted to model user behavior specifically for multiple carousels. Inspired by studies on search user interfaces, Felicioni et al. (2021) assumed that users follow a “golden triangle,” i.e., their attention decreases from the top-left corner to the bottom and the right. From this, Ferrari Dacrema et al. (2022) formally defined an extension of the well-known NDCG metric, N2DCG, where the discounted cumulative gain g is calculated as follows:

With V and H representing the number of carousels (vertical) and items (horizontal), respectively. The proposed discount function d takes into account both the above assumption and the number of scrolls required to reveal an item. Using the normalized version of (1) averaged over all users, the authors found that typical algorithms for implementing different carousel types perform differently when they are combined in a MLRI instead of being evaluated alone. Based on findings from three real-world datasets, they concluded that it is important to account for the availability of multiple collections when choosing an appropriate recommendation method. Consequently, selecting the right carousels becomes a very complex problem, which is why the authors recently proposed to use quantum computing to find a solution (Ferrari Dacrema et al., 2021). Aside from some of the aforementioned industry publications, few other authors have studied MLRI with such a holistic view. For instance, Xi et al. (2023) proposed an attentional re-ranking model that captures the user interaction with a whole page. They even went a step further by considering the special case of “F-shaped” pages, i.e., interleavings of vertical and horizontal collections, as well as the fact that users behave differently depending on the carousel type. The authors also reviewed the recent advances in page-level optimization, but these approaches are beyond the user-centered scope of this mini review.

Finally, Rahdari et al. (2022b) extended the cascade model, which describes user behavior in ranked search result lists (Craswell et al., 2008). Contrary to the above assumption, this resulted in a carousel click model that simulates user interaction under the premise that before users begin to examine the items, they explore vertically until they find a collection with a label that catches their attention. For the corresponding experiment, the authors chose labels based on genre information from the MovieLens dataset, i.e., only simple content-based explanations. The main finding was that the simulated users were more efficient than when scanning one-dimensional lists. Rahdari et al. (2021) also explored how to improve interactive control in MLRI by allowing users to fine-tune the importance of the topics represented by individual carousels. In a recent publication, they further demonstrated the successful use of carousels in a more practical domain, i.e., medical advice (Rahdari et al., 2022a). Without user studies, however, the empirical basis for the design of MLRI still remains weak, especially in light of other domains, where the user experience may be different depending on, e.g., item complexity and user familiarity.

Among the publications listed in Table 1, only a few report a user experiment. Jannach et al. (2021) conducted a large online study to investigate the impact of different design alternatives (N = 775). Their exploratory study provided initial insights into the usage and assessment of carousels in the context of similar-item recommendations: Participants were slower in their decision making when they were confronted with multiple lists, but explored longer before settling on a movie. Compared to a grid without labels, the grouped organization also increased the perceived diversity and novelty of the recommended items, remarkably even with labels that did not have a meaning. With respect to labeling, the study also showed that user- and item-based carousels were preferred over references to, e.g., movie genre, director, or release date. Finally, it is worth noting that removing duplicate items did not make a difference.

Starke et al. (2022) compared a carousel-based recipe recommendation interface with a conventional vertical list and a grid (N = 150). Although the carousels had descriptive labels, they found no positive effects on choice satisfaction or difficulty compared to the grid, where the rows had no explanations and could not be scrolled horizontally. The authors noted several reasons for this finding, but it could also have been the result of the very specific task (“find the most suitable vegetarian recipe”) combined with the fact that the dataset consisted only of vegetarian dishes and the labels were not very distinctive (e.g., “vegetarian recipes,” “salad recipes”). However, compared to the list, both the grid and the carousels were perceived as easier to use, although it was more difficult to choose an item. Other aspects related to user experience, such as carousel length or individual decision-making traits, were not taken into account.

In another study (N = 366), however, Starke et al. (2021) examined the effects of personal characteristics and explanation styles. They found that carousels had a positive effect on choice satisfaction and perceived diversity. On the other hand, but consistent with the literature (Iyengar and Lepper, 2000), participants needed more time to make a decision than with a conventional list. While cooking experience was positively correlated with comprehensibility and satisfaction, there were no interaction effects, i.e., the MLRI had no general advantage. Moreover, no differences were found when comparing carousels with and without explanations. Apparently, labels such as “similar recipes that contain fewer calories” neither made the decision easier nor led to greater satisfaction with the chosen item. Since this contradicted earlier findings on grouped interfaces (see above), the authors concluded that it still needs to be investigated whether item details, images, or descriptive texts are more critical for making decisions than carousel labels. However, it is also important to note that the study was limited to similar-item recommendations. Given the very specific domain and the interface, which was quite different from real-world systems (few recommendations, no personalization), it is therefore difficult to generalize the results.

Only recently, Starke et al. (2023) conducted another study in their series of experiments on using carousels to promote healthy food choices (N = 164). Again, they compared a single- with a multi-list format, but also varied the personalization of the labels. While the results were consistent in terms of diversity and comprehensibility, participants were less satisfied with their choices in the multi-list condition, contrary to the findings above. Moreover, the previously observed differences in choice difficulty were not present. Regarding personalization, the authors found that labels without a focus on nutrition were preferred, e.g., “these recipes [match] your low level of cooking experience.” The personalization also led to unhealthier recipe choices, possibly because participants developed negative feelings when the explanations were explicit about the relationship between personal characteristics and nutritional value. However, as acknowledged by the authors, some of the findings, including those related to the influence of health consciousness and domain knowledge, require further confirmation, especially since it was not possible to fit a structural equation model to analyze mediating effects in more depth. Besides, the relatively small recommendation sets and the fact that the crowdworkers participating in the study probably did not consume the chosen recipes may have compromised the ecological validity.