ESM-Quest consists of four main software components: the backend, the study administration frontend, an Android app, and an iOS app. The backend was developed with the open-source PHP framework Laravel and is hosted by the IT service provider of the University of Vienna. Its main task is to process, store, and provide data that is stored in a MySQL database, such as items, answers to those items, and study parameters. The backend provides a RESTAPI (Representational State Transfer Application Programming Interface) to receive data from and to send data to the other components. The study administration frontend application is based on the TypeScript open-source framework Angular and offers a user-friendly browser-based interface to manage ESM-Quest studies. The native mobile applications for data collection are written in Java (Android) and Swift (iOS).

ESM-Quest enables researchers to conduct ESM studies in which participants can use their own mobile devices. Study participants can download the app for free from the Google Play Store or the Apple App Store. The configuration of the ESM-Quest study can be done in a browser-based web interface. While researchers at the University of Vienna use their single sign-on credentials to access the study administration, researchers not affiliated with the University of Vienna will be able to register for a guest account.

The system offers a questionnaire builder tool and different types of questionnaires including various answer formats (see below). A baseline questionnaire (e.g., demographic data, trait questionnaire) can be used at the beginning, a repeated state questionnaire during, and an end questionnaire (e.g., feedback) at the end of study participation.

Each ESM-Quest study defines a unique study code. Participants of the corresponding ESM-Quest study enter this code to participate (anonymously) in the study. All of the required content (e.g., texts and items) and parameters (e.g., number of signals) will be downloaded from the backend and stored on the participant’s device.

Study administrators must choose one of the following main variants of ESM studies: random sampling or event-based random sampling. While the random sampling mode leads to random signals during the entire period of study duration, the event-based random sampling mode requires that participants activate the system in predefined situations (e.g., at the beginning of a lesson at school) to get random signals within a configurable period of data collection. Random sampling mode requires the configuration of a timetable, where start and end time of random signals must be specified for each day of the week (multiple time intervals per day are currently not supported). In this mode the mobile device calculates all random points within the timeframe of the study under consideration of the study settings. In event-based random sampling mode, the timing of the signals is calculated each time the participant activates the system. The distribution of the random signals in both modes can be affected by setting the duration of the time interval of data collection, the number of random signals within the interval, and the minimum pause between two signals. The pause parameter helps to avoid clustering random signals.

Other modes, for example a pure event-based mode, where special events (e.g., physiological parameters or pressing buttons) trigger a state questionnaire are currently not available, but planned for future development.

For each calculated time point in both modes, an auditory signal is scheduled, at which point the smartphone sends a local notification to the participant with the request to fill out a state questionnaire. A clear strength of this approach is the possibility to run studies offline without an internet connection. In this case, a connection to the server is only mandatory while entering the study code and for sending answers to the server at the end of the study. If there is an internet connection on the mobile device during participation, the app tries to send all locally stored answers after each questionnaire to prevent data loss.

The collected data can be downloaded from the backend in CSV format or as a Microsoft Excel file in XSLX format. The data is divided into two parts. The first part contains the answers to each questionnaire. Answers in the dataset are linked to the timestamp of the signal for the corresponding state questionnaire. In the second part all these signal timestamps are listed together with the status of response for each questionnaire: started (questionnaire was opened) and completed (questionnaire was completed). Participants have no time limit to open a state questionnaire. The time between signal and starting a questionnaire can be calculated from the data. If a state questionnaire has not been answered before the next signal, it is marked as missed in the dataset.

In addition, for studies in event-based random sampling mode, the status canceled means that the planned questionnaires were cancelled due to a premature deactivation of the system by the user. In this mode, the time of activation and deactivation of the ESM system is listed for each timestamp.

After entering the internet address for ESM-Quest study administration in the browser, a login screen is displayed. The next screen of study administration displays an overview of current studies of the logged-in user. The selection of a study or a click on the “New study” button leads to the first main menu point. In the “Settings” menu point the main settings of an ESM-Quest study can be made. Figure 1 shows a screenshot of the main configuration options.

Study administrators should first define the mode of the ESM-Quest study in the main settings: random sampling or event-based random sampling. The study duration field defines the days of overall data collection after entering the study code for participation on a mobile device. In this menu an optional disclaimer, the baseline questionnaire, and the end questionnaire can be activated. In the next main menu point of “Questionnaires,” a questionnaire builder tool is offered to create questionnaires. This tool supports different types of items and questionnaire content: Information texts, open-ended questions, single-choice items, multiple-choice items, Likert scales, and a slider question type. The main menu point “Texts” provides the possibility to customize most of the texts displayed in the app, for example, disclaimer text, instructions at the beginning, or acknowledgment messages at the end of the data collection. This customization also gives the possibility to create ESM-Quest studies in different languages. The last main menu point “Dashboard” is used for downloading the data and monitoring ESM studies in progress. The “Dashboard” displays an overview of the total response states overall as well as individually for each participant.

After specifying all settings described above in the ESM-Quest study administration frontend, the study can be activated for testing or data collection. From this point, participants can take part in the ESM-Quest study with their Android or iOS devices. Figure 2 shows screenshots of the mobile client.

After downloading and installing, the ESM-Quest app asks for the study code. A valid study code results in downloading all corresponding settings, texts, and questionnaires to the smartphone. The next page shows an optional configurable disclaimer text that must be accepted with a checkbox to go further. If a baseline questionnaire has been configured in the ESM-Quest study settings, it will be displayed on the next pages. After finishing the baseline questionnaire, the app displays an information text in the random sampling mode, that the app can be closed now. In event-based random sampling mode, each time the ESM-Quest app is started, a screen with a slider button is shown, which activates the system for generating and sending signals.

Scheduled signals alert the user with a specific sound and vibration, depending on the device’s current alarm settings. Participants start a state questionnaire by clicking on the local notification or by opening the app. All items of the questionnaire are displayed side by side. After completing a questionnaire, the app can be closed again. When the period of data collection is over, the app optionally displays the end questionnaire, if this has been configured in the ESM-Quest study settings. On the last page of the app, there is a reminder to send all data to the server. After all data have been sent to the backend server, a configurable “thank you for participation”-message and an optional link for incentives are displayed. Participants are informed that the app can now be uninstalled.

More information about the app can be found on the website of ESM-Quest.¹ Researchers who are interested in using the ESM-Quest App for their research projects can send a request to the email address provided on the ESM-Quest website.

The data collected with the ESM-Quest app allows for investigating the intraindividual fluctuation of constructs. Figure 3 displays such fluctuations in the three constructs of control, value, and enjoyment within three selected participants across time. Each color indicates one participant. Points indicate single assessments nested within learning situations (left panel, Study 1) or days (right panel, Study 2). The x-axis depicts the numbered learning situations and days, respectively. Figure 3 illustrates that there are pronounced intraindividual (= within-person) fluctuations in constructs over time. For example, in the left panel, we can see that the mean level of control experienced in learning situations changes considerably across time for participant 1 (displayed in yellow). Though less pronounced, we also observe changes in average experienced control, value, and enjoyment per day over the week of data collection in the adult lifespan sample (right panel). Here, participant 1 (displayed in yellow) displays little to no fluctuation in control, but considerable fluctuation in value, which illustrates that patterns of intraindividual fluctuation may depend on the construct in question. Using experience sampling allows us to study such intraindividual fluctuations and compare them across constructs or individuals. Research questions could aim to explain intraindividual fluctuations, for example by using contextual or dispositional aspects as predictors, or investigate temporary aspects, such as increases or decreases occurring naturally or linked to interventions.

The data collected with the ESM-Quest app allows us to compare intraindividual variation across individuals. Such interindividual differences in the variance of constructs within persons are shown in Figure 4. The histograms display the amount of variance in state assessments of control, value, and enjoyment on the x-axis by individual. The y-axis indicates the number of individuals who display the respective amount of variance in their state ratings of control, value, and enjoyment. Overall, variances mostly accumulate between values of 0 and 1, indicating zero to moderate variation for most individuals. However, some individuals experience much higher variation than others (i.e., their experiences of control, value, or enjoyment are much more variable over time). The right panel (adult lifespan sample) also indicates smaller amounts of intraindividual variance in control (= fluctuates less strongly) than value (which is also illustrated by Figure 4). Research questions could aim to explain why some individuals show stronger fluctuations of constructs than other individuals. For example, individual differences such as older age or lower excitability may come with smaller fluctuations (e.g., Röcke et al., 2009). Also, fluctuations themselves may serve as predictors. Higher fluctuations and variability could be both an indicator of adaptability and resilience or, in turn, maladaptation and vulnerability depending on the circumstances and construct in question (for an overview see Mac Donald and Stawski, 2014).

The data collected with the ESM-Quest app allows for the analysis of intra, as well as interindividual relations (see Table 2). Intraindividual relations refer to relations between variables on the level of state assessments, which can capture temporal fluctuations. For example, if individuals perceived higher control during learning (Study 1) or performing a mental activity (Study 2) than they normally would, they also perceived relatively higher value (r = 0.19/0.14 for Study 1/2). Interindividual relations refer to relations between variables on the level of aggregated states across assessments. Results as outlined in Table 2 show, for example, that individuals who perceive higher control on average, also report higher value on average, in both samples (r = 0.29/0.49 for Study 1/2). These examples illustrate that inter- and intraindividual relations can deviate from each other. In this case, relations between control and value were weak on the intraindividual level (= situational/state assessment; Level 1), but moderate on the interindividual level (= aggregated state values; Level 2). As the majority of research and literature focuses on interindividual relations, innovative research questions could ask whether established associations on the interindividual level can also be found on the intraindividual level (see Voelkle et al., 2014; Goetz et al., 2016b; Murayama et al., 2017). That is, whether traits and dispositions are merely associated (i.e., personal tendencies to feel in control and to attribute value to cognitive situations), or whether the relations are more interwoven in a given situation.

With the availability of state-data from multiple participants it is possible to inspect whether the variance of a construct is primarily driven by differences within individuals, between individuals or between other clusters (= levels) included in the data. A method to analyze how much variance is attributed to the different levels is variance decomposition. Figure 5 (left panel) depicts the variance decomposition for the Study 1 constructs, where we differentiated between intraindividual variance (= Level 1) and interindividual variance (= Level 2). It becomes evident that the larger proportion of overall variance is driven by variation within individuals.

Figure 5 (right panel) depicts the variance decomposition for Study 2, where we built a three-level structure of intraindividual variance on the situational level (= Level 1), the day level (= Level 2), as well as the interindividual level (= Level 3). Variance can be found on all three levels, with the lowest proportion on the day level and the highest on the situational (= intraindividual) level.

Importantly, the lowest level of variance also contains residual variance (i.e., variance because of assessment errors and unreliable and manifest assessment of constructs). Research questions can address which level determines the variation in a variable of interest – the situation, the person, or both – and whether a multilevel structure is indeed necessary to consider. For example, there could only be a negligible amount of variance either on the intraindividual level or the interindividual level.

Many studies have used aggregated state assessments as a proxy for the corresponding trait construct (see Conner and Barrett, 2012). However, the assumption that an aggregation of momentary experiences (i.e., states) reflects an individual’s disposition toward a certain experience (i.e., trait) is not a given. The experience sampling design in Study 1 allows us to further explore the question of aggregated state vs. trait, as we utilized items with parallel wording for trait and state measures of control, value, and enjoyment.

The trait versions of the items explicitly asked participants for an overall judgment of the control, value, and enjoyment they usually experience in learning situations, while state items asked for an immediate judgment of one’s current experience. These two modes of assessment allow us to not only compare trait and singular state assessments but also to compare trait assessments with the aggregated states (= individual means of state assessments, calculated by aggregating Level 1 assessments to Level 2 means).

Firstly, we can compare mean values of aggregated states with mean values of traits and examine them for correspondence. As shown in Table 3, mean values of trait control are significantly lower than aggregated state means, and mean values of trait value and trait enjoyment are significantly higher than the corresponding aggregated state means. The differences in the distributions are illustrated in Figure 6, which shows the violin and boxplots of aggregated state values in color and violin plots of trait values in gray.

Secondly, we can analyze the relation between aggregated states and corresponding trait measures and again examine them for correspondence. Prior research has shown that relations between aggregated states and traits differ depending on the construct of analysis (see, for example, Rauthmann et al., 2019). As shown in Table 4, we find correlations of moderate effect size (r = 0.34 to.42, p < 0.001) between aggregated states and their corresponding trait measures. Using experience sampling, research questions can investigate differences in mean values and correlations between aggregated states and traits of different concepts, and thereby investigate trait–state homomorphy (i.e., whether trait and state items measure the same constructs) for different psychological constructs (Rauthmann et al., 2019).

If trait constructs represent aggregated states of those constructs, one could assume that relations among trait variables are in line with relations between corresponding variables on the level of aggregated state assessments. Using items with parallel wording for trait and state measures in Study 1 allowed us to compare these relations using the different types of measurement. For variables in the aggregated state as shown in Table 5, perceived control relates positively to perceived value (r = 0.22; p = 0.03). Interestingly, this relationship is not in line with the respective trait-level constructs (r = 0.07; p = 0.48; see Table 6). There is no significant correlation between trait control and trait value. Discrepancies like these can easily be uncovered using a combination of state and trait assessments via ESM. Research questions might address whether known associations are dependent on the measurement of the construct (e.g., as a trait or aggregated state). Further, one could investigate whether differences in associations hold when controlling for concepts which are known to influence trait rather than state responses (e.g., gender; Goetz et al., 2013).

Future directions in ESM research on psychological variables might be to combine experience sampling with other types of assessments. Even self-report, as used in ESM, might generally be a good choice for the assessment of psychological variables, however, self-report variables have limitations in that they are restricted to accessible processes and bear the possibility of self-report biases. Therefore, it can be useful to complement self-report with other methods, such as physiological indicators and observation of facial expression (while considering that these methods have their own limitations in terms of reduced sensitivity and/or specificity; see, e.g., Harley et al., 2015).

A highly important area of future ESM research lies in its application within the realm of adaptive systems, which have experienced a notable surge in significance in recent years. For example, adaptive technical learning systems have become increasingly prominent as they allow for a more individualized type of learning. For instance, computerized adaptive testing (CAT; e.g., Wainer, 2000; Asseburg and Frey, 2013) can be useful in reducing situations of non-optimal challenge during tests, where individuals may be either over- or underchallenged. In CAT, items are selected individually based on the test takers’ previous responses. Therefore, if a wrong answer is given, an easier item will be presented next, and vice versa. In learning situations as well, the difficulty of the material can be adjusted based on students’ current competence level on the topic. This testing strategy is, for example, expected to significantly reduce boredom, which is a common result of being over- or underchallenged (Goetz et al., 2023). However, adaptive systems can go beyond considering just competence level and can also take psychological variables such as metacognition, motivation, and emotions into account. For example, specific content areas on a particular difficulty level within a domain can be selected based on these psychological variables (e.g., more abstract or concrete material). For example, if a system recognizes a decrease in a student’s enjoyment while working on highly abstract material, it might react by presenting more concrete materials with the aim of rekindling enjoyment. Real-time assessments are essential for the use of such adaptive technical learning systems, with the quality of adaptive systems heavily relying on accurately assessing an individual’s current state. In this regard, the ESM is an incredibly valuable tool. As demonstrated in this paper, emotions, for instance, can fluctuate significantly within students or adults engaging in mental activities. Thus, real-time reactions of adaptive systems are warranted based on ongoing assessments of these fluctuations. It is important to note that, currently, self-report as used in ESM is the only valid way to assess the affective component of emotional experiences in real-time (Pekrun et al., 2023). Likewise, other psychological variables like metacognitive and motivational constructs can only be assessed to a limited extent beyond self-report.

With respect to the outlined future directions, but also beyond those lines of research, the presented app, ESM-Quest, can be a highly valuable tool for data collection. First of all, as it allows event-based random sampling, it is possible to assess randomized data within given situations. For example, academic emotions, being highly domain-specific in nature (Goetz et al., 2007), can be assessed in one specific domain (e.g., math classes) by activating the app, for instance, at the beginning of a class, and then conducting a number of randomized assessments during the class. In other words, ESM-Quest allows a focus on specific domains within the academic context and beyond (e.g., doing sports, eating, shopping), enabling analyses regarding these domains and potential differences across them.