We developed two COVID-19 self-management apps for epidemiological investigations: the KODARI and MEDARI apps. The KODARI app is the Korean version of an application that can be used in South Korea. The MEDARI app can be used in English-speaking countries. The main functions of the two apps are the same, with the only difference being the GPS-based location-storage map method.

This study focused on users of the KODARI app. The goal of the app is to support epidemiological investigations to minimize the spread of infectious diseases and to support individual health management. The KODARI and MEDARI apps were developed for the aforementioned purpose and designed to overcome infectious diseases through their universal use. The app database was designed for research purposes and developed based on cloud services. These two apps were not registered in the App Stores. This study was conducted with actual users prior to the official launch of the app. The total number of users was 489, among them 276 participated in this study.

The crucial information to be collected during epidemiological investigations was extracted from the Epidemiological Investigation Form for COVID-19 (version 6) (for public health) (13). We developed the KODARI and MEDARI apps using crucial information. The main functions include daily movement routes (information on means of transportation, location, companions, etc.), daily symptom records, export of movement routes to epidemiological investigators, infectious disease management records (vaccines, test results, and exception certificates), overseas entry and exit records, and health-related information, such as underlying diseases. The detailed functions and interface of the KODARI app can be found on the KODARI (12) and MEDARI (14) websites (see Figure 1).

Online surveys were conducted among users of the KODARI app a COVID-19 self-management application, for epidemiological investigations. This app was released on November 1, 2022. This study targeted individuals who had used the KODARI app for more than a week (from November 2022 to the time of the survey). None of the respondents withdrew from the study.

Participants provided voluntary online consent to participate in the study prior to the online survey. The participants were compensated. The participants were given 450 points according to the survey company’s regulations on Data Spring.

This study adapted a previous assessment framework to evaluate the KODARI app (15). The seven main domains of the existing assessment framework from epidemiological and legal perspectives (15) are purpose, usability, information accuracy, organizational attributes and reputation, transparency, privacy, and user control and self-determination. Each domain has 40 criteria, ranging from a minimum of four to a maximum of seven items.

However, because the existing framework is a checklist for app developers, there are limitations to using it to conduct surveys of app users. Therefore, we deleted and supplemented the criteria to adapt the framework for use in our survey of KODARI app users, including changing some phrases when modifying the questionnaire for South Korean users and adjusting the framework to facilitate user (rather than developer) responses. In addition to the main elements of the existing evaluation framework, some items specific to this study were added, such as the intention to use and effectiveness. We added a demographic characteristic item that included basic user information, such as sex, age, and location, as well as information related to COVID-19, such as the user experience of COVID-19 testing and vaccination. The questionnaire items are included in Appendix Table A1.

The process of developing the questionnaire for this study for KODARI app users was as follows: First, a questionnaire based on an existing framework for application developers was modified for application users (15). Items regarding variables that were irrelevant to app users were deleted or supplemented. For example, an item referred to as criteria and variable was referred to as domain in a previous study. Consequently, 50 items were derived from the existing framework for seven domains: six items for purpose, five for usability, four for information accuracy, five for organizational attributes and reputation, six for transparency, seven for privacy, and seven for user control and self-determination. In addition, we added two variables based on relevant literature: intention to use and effectiveness. The total number of items, including the demographic characteristics, was 63.

The variables not bound by the factor analysis included purpose, organizational attributes, reputation, and privacy. Excluding these three variables, for the remaining six variables, our analysis used usability for two of the five items, information accuracy for two of the four items, transparency for two of the six items, self-determination for three of the seven items, effectiveness for three of the six items, and intention to use for three of the four items. To evaluate the application of the KODARI, four of the six variables were based on the existing research framework (15): usability, information accuracy, transparency, and self-determination. These six variables are defined as follows:

We conducted an online survey of KODARI users during the study. Among the existing app users, the final sample included 284 participants who volunteered to participate in the study. The survey was conducted between April 28 and May 25, 2023. Eight KODARI users who did not participate in the survey and one who responded to the survey without using the KODARI app were excluded from analysis. In total 276 valid questionnaires were completed. The survey was provided through an online link and was conducted after obtaining consent from all participants before the survey began.

To verify the reliability and validity of the questionnaire used in this study, we conducted factor and frequency analyses of six factors (intention to use, effectiveness, self-determination, information accuracy, transparency, and usability) to evaluate the overall KODARI app. We conducted two independent-sample t-tests to compare the demographic characteristics of the groups. The target variables were intention to use, effectiveness, self-determination, information accuracy, transparency, and usability, according to demographic characteristics. Finally, we investigated the factors affecting the intention to use the KODARI app using a multiple regression analysis. Multiple regression analysis was conducted using the enter method, in which the five independent variables (effectiveness, self-determination, information accuracy, transparency, and usability) were simultaneously input, and data were analyzed using the IBM® SPSS® Statistics 27.0 software package.

The study procedures were performed in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of D University (IRB number: DKU 2023–02–011-004). Participants’ data were de-identified.

Table 1 shows the demographic characteristics of the participants in this study evaluating the effectiveness of the KODARI application. The total number of participants is 276.

Among the 276 participants, 137 (49.8%) were male and 139 (50.4%) were female. The majority of the participants were 20–29 (33.0%), 30–39 (29.3%), and 40–49 (34.1%) years old. Participants aged 18–19 years were not excluded from the analysis because they were considered adults in South Korea. Regarding marital status, 134 participants (48.6%) were married and 142 participants (51.4%) were single. In terms of educational level, 201 participants (72.8%) had graduated from college or higher, followed by 47 college students (17%), and 28 high school graduates (10.1%). Professional/managerial/office jobs accounted for the largest number of occupations (141 participants, 51.1%), followed by homemakers and students (71 participants, 25.7%). A total of 142 participants (51.4%) had occupations that involved frequent interactions with others. A total of 134 participants (48.6%) had occupations involving relatively few interactions with others. Among them, 163 (59.1%) lived in the capital city. The capital area of South Korea is comprised of three districts: Seoul, Gyeonggi-do, and Incheon Metropolitan City. Non-capital areas refer to the remainder of the district.

We conducted factor analysis to evaluate the validity of the questionnaire. Principal component analysis and varimax rotation were performed. For validity verification, variables whose values were less than 0.6 or not bound by factors were excluded from the analysis. In addition, Cronbach’s alpha coefficient, which indicates the internal consistency between measurement items, was used for reliability analysis (Table 2).

Six factors were derived from factor analysis: intention to use, effectiveness, information accuracy, self-determination, transparency, and usability. The maximum and minimum values were 0.855 and 0.600, respectively, and the analysis confirmed that all values were 0.6 or more (16, 19, 20). In addition, the eigenvalue, which constituted 75.041% of the total variance, was greater than 1.0. Cronbach’s alpha was used to verify reliability, which is the internal consistency of the variable. The maximum and minimum Cronbach’s alpha values were 0.819 and 0.674, respectively. Therefore, because all values were 0.6 or more (19, 21), it was considered reliable.

Figure 2 shows the overall evaluation of the KODARI application and assessment results for the six variables. The six variables were intention to use the KODARI app, effectiveness, self-determination, information accuracy, transparency, and usability. The six variables are tied based on the factor analyses presented in Table 2. The detailed assessment results from the KODARI app are included in Appendix Table A2.

First, when participants were asked whether they intended to use the app, 125 (45.3%) answered moderately. However, 101 participants (36.6%) responded that they were willing or extremely willing to use the KODARI app.

Effectiveness refers to the efficiency of an app in epidemiological investigations. Among the total respondents, 198 (71.7%) responded that the KODARI app would be useful for epidemiological investigations and would reduce the workload of epidemiological investigators.

Additionally, 143 participants (51.8%) responded positively to the app’s self-determination features, accounting for more than a majority of participants. A total of 110 (39.9%) and 253 (91.7%) participants expressed moderate and positive opinions, respectively.

Among the participants, 218 (79.0%) responded positively to the information accuracy question. The information accuracy variable had the least negative response (six participants, 2.8%) among the six variables.

Those who positively rated transparency, including 142 participants (51.4%) who responded “agree,” accounted for 68.8% (190 participants) of all participants. Finally, 183 participants responded positively to usability and 66.3% evaluated it positively.

The results of two independent sample t-tests were used to determine the differences between each variable (intention to use, effectiveness, self-determination, information accuracy, transparency, and usability) according to the participants’ demographic characteristics. First, the participants were divided into two groups based on their demographic characteristics: sex, marital status, occupational characteristics, and location (Table 1). Age was divided into two groups based on the age of 35 years, which was the mean age (34.88) of all participants. The average age of the respondents was 34.88 years, whereas their median age was 36 years: standard deviation (8.853), variance (78.372), skewness (−0.094), and kurtosis (−1.209) values. Age was skewed to the left, and the distribution of participants in their 20s and 40s had similar characteristics. Consequently, the sample was divided into subgroups of 35 years, which approximated the average age.

Educational level and occupation were excluded from the analysis because they could not be divided into two groups or based on the averages. Second, because the purpose of the KODARI app is to support epidemiological investigations, we used variables related to epidemiological investigations for the analysis: experience with epidemiological investigations and the form of epidemiological investigations (Table 1). Therefore, the demographic characteristics included seven variables: sex, age, marital status, occupational characteristics, location, experience with epidemiological investigation, and the form of epidemiological investigation. The summary details are presented in Table 3 and the full results are presented in Appendix Table A3.

The results showed a significant difference in information accuracy and intention to use, according to sex. For male (M = 3.741, SD = 0.770) and female (M = 3.921, SD = 0.668) participants, there was a significant difference in information accuracy (t = −2.075, p = 0.039) according to sex. In addition, for male (M = 3.416, SD = 0.791) and female (M = 3.161, SD = 0.865) participants, there was a significant difference in the intention to use (t = 2.559, p = 0.011). Females had higher information accuracy, whereas males had a higher intention to use (p < 0.05).

There were significant differences in information accuracy and transparency depending on the participants’ age. The mean and standard deviation of participants aged ≤35 years (M = 4.030, SD = 0.665) and > 35 years (M = 3.647, SD = 0.731) showed a significant difference in information accuracy (t = 4.544, p = 0.000) according to age, based on the age of 35 years. In addition, participants aged ≤35 years (M = 3.707, SD = 0.776) and those aged >35 years (M = 3.510, SD = 0.716) showed a significant difference in transparency (t = 2.185, p = 0.030). Participants under 35 years old had higher information accuracy and transparency (p < 0.05, p < 0.001).

Regarding to marital status, there was a significant difference in the information accuracy. The mean and standard deviation of married participants (M = 3.724, SD = 0.665) and single participants (M = 3.944, SD = 0.765) showed a significant difference in information accuracy (t = −2.418, p = 0.016) depending on marital status. Regarding marital status, single participants had a higher information accuracy (p < 0.05).

There were significant differences in self-determination according to occupational characteristics and experience in epidemiological investigations. Occupations involving frequent interactions with others (M = 3.617, SD = 0.715) and those involving relatively fewer interactions with others (M = 3.353, SD = 0.639) showed a significant difference in self-determination (t = 3.229, p = 0.001) according to occupational characteristics.

Finally, participants with experience in epidemiological investigations (M = 3.603, SD = 0.662) and those with no epidemiological investigation experience (M = 3.419, SD = 0.700) showed a significant difference in self-determination, depending on the presence or absence of epidemiological investigation experience (t = 2.164, p = 0.031). Therefore, participants with occupations involving frequent interactions with others and experience with epidemiological investigations had a higher self-determination (p < 0.01, p < 0.001).

Table 4 summarizes the results of this analysis. The model, which included all (five) independent variables, had an F-statistical value of 41.273 and a significance value of 0.000. Differences were considered statistically significant at significance level < 0.001. The coefficient of determination (R²) of this model was 0.433, and the independent variables of effectiveness, information accuracy, self-determination, transparency, and usability explained the change in the dependent variable of intention to use by 43.3%.

Table 5 presents the results of the statistical significance analysis and contribution of each independent variable to the dependent variable. Effectiveness (t = 4.457, p = 0.000), self-determination (t = 4.504, p = 0.000), and usability (t = 4.268, p = 0.000) had significant impacts on the dependent variable and intention to use the KODARI app. Information accuracy (t = 0.399, p = 0.690) and transparency (t = 0.488, p = 0.626) were rejected as a result of the analysis. According to the standardized coefficient, which represents the relative statistical contribution of the three independent variables, effectiveness (0.266), self-determination (0.258), and usability (0.252) affected the participants’ intention to use the KODARI app.

This study has several important implications. First, in addition to the technology acceptance model (TAM) and unified theory of acceptance and use of technology model (UTAUT) (28–30), which are widely used in research on technology introduction or intention to use, we applied a new framework (15) for technologies related to COVID-19. Diverse technologies for infectious diseases such as COVID-19 have been introduced, but the situation is distinct. New attempts are required to evaluate technologies such as the KODARI app. Therefore, this study is meaningful in that it used a new evaluation tool that is different from existing studies, and confirmed the possibility of research with the variables used in this study. This study, which evaluated this app, is meaningful for developing the app and evaluating technology, considering the specificity of the infectious situation. However, further studies are required to evaluate technologies, such as the KODARI app. Therefore, this study contributes to the literature by using a new evaluation tool that is different from that used in previous studies, and confirms the possibility of further research with the variables used in this study.

Second, this study is meaningful because it directly developed an application related to infectious diseases and evaluated its effectiveness. Based on this, we found that an infectious disease management application for epidemiological investigations, such as KODARI and MEDARI apps, is helpful for epidemiological investigations and reduces the workload of epidemiological investigators. Scholars have conducted diverse studies on infectious diseases, such as COVID-19 (8, 31–35), and healthcare (36–39) apps and technologies. Yamamoto et al. reported that a personal health record (PHR)-based COVID-19 symptom-tracking app was an effective countermeasure against infectious diseases (40). However, few studies have evaluated the effectiveness of existing or directly developed infectious disease applications (41, 42). Therefore, this study is important because the researchers who participated in the app design process evaluated their effectiveness. Thus, the KODARI app provides a cornerstone for future application development.

Third, we found differences in sex, age, marital status, occupational characteristics, and experience with epidemiological investigations when evaluating the apps. This does not mean that apps should be designed differently based on specific occupations, experiences, or gender. It is desirable to use this to operate an app. For example, males had a higher intention to use the app than females. Based on these results, we will devise operational plans, such as promoting their application to female.

This study provides a basis for inventing or developing new applications in the event of future outbreaks of infectious diseases such as COVID-19 by identifying the factors affecting their introduction. The findings make an important contribution to literature on future infectious diseases.

Despite this study’s meaningful results, it had some limitations. First, the total number of participants was 276, all of whom were KODARI app users; therefore, the results of this study have limitations in terms of generalization. We were unable to target all Koreans because of limited time and cost. As of 2023, the projected population of South Korea is approximately 51 million (43), and the cumulative number of confirmed COVID-19 cases is approximately 35 million (as of August 31, 2023; Korean Standard Time [KST]) (3). Future studies should focus on larger numbers of users. Second, there was a bias against vaccination. Among participants, 95.7% were confirmed to have completed the second- or higher-dose vaccination, and only 12 participants had never been vaccinated or only completed the first dose vaccination. The second-dose vaccination (fully vaccinated) rate for COVID-19 in South Korea is 86.6% (as of September 5, 2023, KST) (44). This study revealed a higher rate of secondary vaccination. The results would have been more meaningful if vaccination rates were similar. Third, the purpose of our modified questionnaire differs from that of the original evaluation framework (15). The original framework introduced a checklist that developers can use when creating apps. In this study, the questionnaire was modified for application to our study of app users and was administered to evaluate the effectiveness of the KODARI app, although we attempted to retain the intent of the existing framework as much as possible. Fourth, some domains and criteria were excluded when adapting the questionnaire and conducting factor analysis of the evaluation framework (15). In this study, this domain was referred to as a variable. There are seven domains in the evaluation framework (15): purpose, usability, information accuracy, organizational attributes, reputation (in this study, expressed as organizational attributes), transparency, privacy, user control, and self-determination (expressed as self-determination). Several criteria are presented for each domain. However, we used only four domains (usability, information accuracy, transparency, user control, and self-determination) and excluded three domains (purpose, organizational attributes, reputation, and privacy). Among the three excluded variables, privacy can affect an app’s effectiveness and intention to use. Privacy was excluded from this study because it was not included in the factor analysis. Therefore, further research should be conducted on purpose, organizational attributes, reputation, and privacy. Finally, we distributed the KODARI app on November 1, 2022. We targeted users who had used the app for more than a week (from November 2022 to the time of the survey). However, the results may differ depending on the period or amount of app use. In future research, it would be meaningful to target users who have been using the app for a long time.