Front. Public HealthFrontiers in Public HealthFront. Public Health2296-2565Frontiers Media S.A.10.3389/fpubh.2016.00274Public HealthOriginal ResearchDevelopment and Validation of a Self-reported Questionnaire for Measuring Internet Search DependenceWangYifan1WuLingdan2ZhouHongli1XuJiaojing1DongGuangheng13*1Department of Psychology, Zhejiang Normal University, Jinhua, China2Department of Psychology, University of Konstanz, Konstanz, Germany3Institute of Psychological and Brain Sciences, Zhejiang Normal University, Jinhua, China
Edited by: Patty Kostkova, University College London, UK
Reviewed by: Pier Luigi Lopalco, University of Pisa, Italy; Ricardo Mexia, National Institute of Health Dr. Ricardo Jorge, Portugal; Michele Tizzoni, ISI Foundation, Italy; Andrew McNeill, Northumbria University, UK
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Internet search has become the most common way that people deal with issues and problems in everyday life. The wide use of Internet search has largely changed the way people search for and store information. There is a growing interest in the impact of Internet search on users’ affect, cognition, and behavior. Thus, it is essential to develop a tool to measure the changes in psychological characteristics as a result of long-term use of Internet search. The aim of this study is to develop a Questionnaire on Internet Search Dependence (QISD) and test its reliability and validity. We first proposed a preliminary structure and items of the QISD based on literature review, supplemental investigations, and interviews. And then, we assessed the psychometric properties and explored the factor structure of the initial version via exploratory factor analysis (EFA). The EFA results indicated that four dimensions of the QISD were very reliable, i.e., habitual use of Internet search, withdrawal reaction, Internet search trust, and external storage under Internet search. Finally, we tested the factor solution obtained from EFA through confirmatory factor analysis (CFA). The results of CFA confirmed that the four dimensions model fits the data well. In all, this study suggests that the 12-item QISD is of high reliability and validity and can serve as a preliminary tool to measure the features of Internet search dependence.
Internet search dependencequestionnairepsychological characteristicvalidityreliabilityIntroduction
The advent of the Internet, with powerful and diverse search engines, has made it much easier now than in the past to access information. The high-speed Internet enables individuals to instantly exchange knowledge and information all over the world. As the Internet has become an essential tool in our daily life, it is important to notice that people are susceptible to the unprecedented Internet search environment (1–3). The majority of population spends many hours per day on the Internet and become increasingly dependent on the Internet search to meet various needs (3, 4). To date, there has been a growing interest in the affective, cognitive, and behavioral changes as a result of Internet search (5, 6), although empirical studies are still lacking. Further studies are needed to investigate the impact of Internet search on users’ psychological characteristics.
Previous studies have shown that the wide use of Internet search engines has largely modified the way human beings search for and store information. Sparrow et al. (7) found that the Internet search has become human beings’ primary form of external and transactive memory. Individuals are prone to turn to the Internet and search for answers online when facing difficult questions and controversial issues (8). People have gradually shifted from remembering the answers to searching for answers on the Internet and remembering where they could effectively find the right things (9). Consistently, a recent study found that Internet access reduced people’s feeling of knowing and lowered their willingness to voluntarily generate answers (10). Once the Internet is out of reach and when they cannot immediately find out what they want, people would show withdrawal-like symptoms, such as irritability, tension, and depression (11). Those symptoms are largely overlapping with withdrawal symptoms of substance addictions and pathological Internet use (12–14).
Nowadays more and more people are likely to make decisions or solve problems according to the suggestions and advices from Internet. A symbiotic relationship has been built up over the past years between people and the Internet (15). It becomes very common that people trust the online information blindly and are incapable of discriminating between reliable information and unreliable information (16). People have formed specific searching habits owing to long-term use of search engines. The more dependent on Internet search, the more keyword spotting, browsing, and scanning behaviors, and less contemplation would be detected among Internet users (17, 18). These findings indicate that Internet users may have developed certain search strategies to make the best use of the Internet rather than independent and in-depth thinking.
In all, previous studies have shown that Internet users are associated with various features of Internet search dependence. Future studies are needed to discern those features in details. Thus, developing a valid tool, such as a questionnaire, to measure people’s psychological characteristics of Internet search dependence is of great importance. This study aimed to develop a questionnaire with proper dimensions and good validity for measuring features of Internet search dependence, which we hope will be a practical tool for future studies to further explore Internet search dependence.
Materials and MethodsStep 1: Developing the Initial Questionnaire
The initial questionnaire consists of questions of basic information (such as gender, age, and history of Internet search) and 12 items about psychological characteristics related to the Internet search dependence, based on a comprehensive literature review, supplemental investigations, and interviews.
First, we queried several databases such as PubMed, ScienceDirect, Web of Science, and Springer Link with two keywords, Internet search and Internet use. Approximately 50 papers (1, 2, 5, 18, 19), published between year 2008 and year 2016, were carefully chosen to review.
Subsequently, we compiled 16 items to represent psychological characteristics associated with Internet search dependence, based on the literature review and a follow-up interview with 50 randomly selected university students (see Supplementary Material). We adopted the six criteria for behavioral addiction formulated by Griffiths (i.e., salience, mood modification, tolerance, withdrawal, conflict, and relapse) (20) and extracted the contents that are closely linked to Internet search. All the 16 items (see Supplementary Material) were then refined for the sake of unambiguity, simplicity, and clarity, which resulted in a 12-item questionnaire of Internet Search Dependence (QISD). This questionnaire has six dimensions covering all psychological characteristics relevant to Internet search dependence, i.e., trust in Internet search, withdrawal symptoms, search impulsivity, keywords spotting, decreased contemplation, and external storage. For each item, we used a five-point Likert-type scale, ranging from 0 (“Never”) to 4 (“Always”).
In addition, we conducted a supplemental investigation with 100 randomly selected university students (see Supplementary Material), through which we identified 5 extra characteristics of Internet search dependence. Those items were assigned to the section of “basic information” considering that they were mostly about the time and frequency of the Internet search (e.g., number of years that a person uses Internet search).
This research was approved by the Human Investigations Committee of Zhejiang Normal University. The initial version of the QISD was tested among a randomly selected sample of 320 university students from Zhejiang Normal University. The returning rate of questionnaires is 90.30%. Two hundred eighty-nine subjects completed the questionnaire (75 males and 214 females, the age ranged from 18 to 27 years, M = 22 years, SD = 1.98), while 31 subjects failed to finish the questionnaire and thus were excluded from data analysis. All subjects were promised anonymity and thanked with a pen or a mini notebook for their participation. All participants in this research signed a written informed consent and no vulnerable populations were involved.
Data Analysis
Psychometric properties, reliability and feasibility, were assessed by means of consistency coefficients (Cronbach’s α) and item analyses, respectively. Cronbach’s α is a measure of reliability, or internal consistency, of a set of items (21). Item analyses, which were composed of mean, standard deviation, kurtosis, skewness, critical ratio (CR), and item correlations, measure quality of individual items.
Exploratory factor analysis was implemented to investigate the construct validity of QISD by means of principle component analysis method. The analyses of Kaiser–Meyer–Olkin (KMO) and Bartlett’s test of sphericity suggested whether the data were suitable for orthogonal factor dimension (22). In addition, the principle component analysis was performed with Varimax rotation to facilitate interpretation of the factors. All analyses were performed using SPSS, version 22.
The CFA was conducted among a newly randomly recruited sample of 324 university students to examine the dimensional structure and the validation of the revised QISD. Thirty-three incomplete questionnaires were excluded. The remaining 291 questionnaires obtained from 138 males and 153 females were used for data analysis. The mean age of the subjects was 22 years (SD = 2.14). All subjects were promised anonymity and thanked with a pen or a mini notebook for their participation.
Data Analysis
The CFA was carried out using maximum-likelihood estimation analysis in AMOS 21.0 (www.ibm.com/legal/copytrade.shtml) to verify the goodness of fit of the model that was built according to the results of EFA. Besides the χ2 and the χ2/df (cutoff ≤ 3) (23), other fit indices and incremental indices were also calculated, including the root mean square residual (RMR; cutoff ≤ 0.05), the goodness-of-fit index (GFI; cutoff ≥ 0.90) (24), the Tucker Lewis index (TLI; cutoff ≥ 0.90) (25), the comparative fit index (cutoff ≥ 0.90) (26), and the root mean square error of approximation (RMSEA; 0.05 < cutoff < 0.08) (27).
ResultsResults of Item Analyses and Cronbach’s α
Table 1 shows the distribution of the item characteristics. The average of the total scores of the QISD was 25.81 (SD = 5.93). All items were statistically significant on CR (p < 0.01), suggesting that the discrimination of each item was good. Eight of the 12 items were right skewed (g1 > 0), ranging from 0.09 to 0.29; the remaining items were left skewed (g1 < 0), ranging from −0.46 to −0.26. The highest level of kurtosis and skewness were evident on the fifth item and the sixth item, respectively.
Item characteristics compose of critical ratio (CR), mean, skewness, kurtosis, and correlation coefficients of each item.
Item
CR (p)
Mean (SD)
Skewness
Kurtosis
Correlation coefficient
1
−8.62**
2.94 (0.72)
−0.26
−0.19
0.54**
2
−10.20**
2.03 (1.09)
0.29
−0.60
0.52**
3
−6.64**
2.18 (1.01)
0.09
−0.68
0.42**
4
−8.31**
2.17 (0.97)
0.26
−0.42
0.51**
5
−8.36**
1.89 (1.15)
0.14
−0.80
0.53**
6
−10.11**
2.82 (0.82)
−0.46
0.03
0.59**
7
−8.18**
2.61 (0.82)
−0.20
−0.25
0.50**
8
−10.11**
1.75 (0.93)
0.11
−0.28
0.51**
9
−10.58**
1.58 (1.03)
0.11
−0.67
0.59**
10
−8.73**
1.80 (1.05)
0.26
−0.67
0.47**
11
−9.79**
1.69 (0.90)
0.21
−0.02
0.57**
12
−9.54**
2.35 (0.90)
−0.14
0.13
0.52**
**p<0.01.
Furthermore, the score of each item was positively related to the total score of QISD, with all the correlation coefficients greater than 0.40 (p < 0.01). The Cronbach’s α of QISD was 0.75, and the average inter-item correlation was 0.2 (ranging from 0.04 to 0.42), suggesting that the internal consistency is acceptable.
EFA Results
An EFA using principle component analysis was performed using SPSS to figure out the model that best fits the data. The value of KMO was 0.79, and Bartlett’s test met a significant level (χ2 = 561.19, p < 0.001), suggesting that the data were suitable for factor analysis. Following the recommendation of some relevant literatures, the dimensions of the model were extracted based on the percentage of variance, the scree plot, and the Kaiser–Guttman method (28). Accordingly, the factorial solution suggested a model with four dimensions that accounted for 55.92% of the total variance. The scree plot showed that the eigenvalues of the first four dimensions were greater than 1, and the eigenvalues tended to be equal after the fifth factor (Figure 1). In general, these results indicated that a large proportion e of the total variance was explained by the four factors.
The scree plot obtained from exploratory factor analysis.
The eigenvalues and corresponding percentage of variance accounting for each component are presented in Table 2, and factor loadings are presented in Table 3. The first dimension, “Habitual use of Internet search” (e.g., item 3, “try to abstract key words from a complex question”) measures the habits of Internet search. It is composed of item 1, 3, 4, 6, and 7. This dimension had an eigenvalue of 3.34 and accounted for 27.84% of the total variance. The second dimension, “withdrawal symptoms,” measures negative reactions when Internet search was not available (e.g., item 8, “I will be upset if I cannot use Internet search when encountering complex questions”). This dimension is composed of item 8 and 11 and had an eigenvalue of 1.28 and accounted for 10.70% of the total variance. The third dimension, “Internet search trust,” measures the degree of confidence to Internet search. It is composed of item 9, 10, and 12. This dimension had an eigenvalue of 1.08 and accounted for 8.99% of the total variance. The fourth dimension, “external storage under Internet search,” had an eigenvalue of 1.01 and accounted for 8.40% of the total variance. Each dimension was significantly correlated with the total score, and the correlation coefficients were greater than 0.6 (The Cronbach’s α coefficients for each dimension were 0.69, 0.67, 0.71, and 0.65).
Eigenvalues and corresponding percentage of variance explained.
Eigenvalue
% of variance
Cumulative variance
3.34
27.84
27.84
1.28
10.70
38.53
1.08
8.99
47.53
1.00
8.39
55.92
0.91
7.60
63.52
0.8
6.71
70.23
0.76
6.32
76.55
0.7
5.79
82.34
0.63
5.28
87.61
0.53
4.40
92.01
0.51
4.24
96.26
0.45
3.74
100.00
The results of exploratory factor analysis: factor loadings.
Item
Factor 1
Factor 2
Factor 3
Factor 4
3
0.71
7
0.63
6
0.56
4
0.56
0.35
1
0.50
8
0.73
11
0.69
12
0.78
10
0.42
0.56
9
0.55
2
0.83
5
0.79
According to the results of EFA, we revised the QISD by amending some sentences and words of the items for better comprehensibility. For example, the item 8, “I will be upset if I cannot use Internet search when encountering complex questions,” was modified as “I will be upset if I cannot find an answer to a complex question through Internet search.” In addition, we did not eliminate any items at this stage.
CFA Results
Based on the aforementioned results, a four-factor model was further examined using CFA. The results of CFA are as follows: χ2 = 97.49, χ2/df = 2.03, p < 0.01; RMR = 0.045; GFI = 0.95; IFI = 0.91; TLI = 0.86; GFI = 0.90; RMSEA = 0.06. Although the TLI did not reach the cutoff value, it was quite close to 0.90. The results suggest that the four-factor model is a good fit to the data.
Discussion
This study aimed to develop a psychometric tool to measure psychological characteristics of Internet search dependence and tested the validity and reliability of the questionnaire. The Cronbach’s α coefficient of reliability index was 0.75, suggesting that the questionnaire was reliable. Factor analyses revealed that the questionnaire contained four dimensions: habitual use of Internet search, withdrawal reaction, Internet search trust, and external storage under Internet search. Further analysis via CFA supported the four-factor model as most of the validity indexes were larger than the cutoff value. The results suggest that the QISD is valid and reliable.
Habitual use of Internet search implies the behavioral habits that people have cultivated during Internet search. Once people get used to Internet search, they gradually develop certain behavioral habits associated with Internet search when facing new and/or complex situations. For example, people are prone to abstract keywords from complex problems and seek for answers via Internet search when facing disputes or new questions (17).
The withdrawal symptoms refer to the response to a sudden abstinence from Internet search. Since Internet search has become a part of people’s daily routine, people may exhibit negative symptoms, such as upset and lacking of confidence, when the Internet is not available. The Internet search has made people dependent on search tools rather than their own memories.
Internet search trust refers to the people who trust the information obtained through search engines. Most people insist that they have benefited a lot from Internet search, as the Internet provides them with an effective way to find information and to gain skills that are important to their study, work as well as daily life, which in turn may facilitate the development of Internet search dependence.
External storage under Internet search describes the phenomenon that people are prone to remember where to access information instead of processing and storing the information by themselves, the so-called “Google effect” (7). The Internet, in general, has become an external storage for people to store information.
Given the ubiquitous presence of Internet, nowadays people can easily reach a huge amount of information with relatively little effort in a short period of time. With the frequent use of the Internet and the rapid advancement of technology, individuals are less willing to take notes or think independently and deeply, which in turn may result in a blind trust and firm reliance or dependence on Internet search. It has been evident that the over use of search engines leads to structural and functional brain changes (2, 8, 29). Although many researchers have noted that Internet search modifies psychological characteristics and facilitates some specific types of behaviors (e.g., seeking information), further study are required to address on those changes in more details.
In all, this study designed a questionnaire for measuring Internet search dependence and tested its reliability and validity through various analyses. The results of all analyses suggest that the QISD is an effective tool with good reliability and validity, which can serve as a practical tool for future studies to investigate the impacts of Internet search on people’s psychological characteristics. However, it should be aware that there are two limitations of this study: first, the current study provided a preliminary tool to measure the level of Internet search dependence, without providing norms or cutting lines. Thus, it cannot be used as a diagnostic tool. Second, the participants recruited for this research were all college students, and the majority was females, who may not be the representative of the general population. Future studies are needed to further test the QISD among various samples, such as adolescents.
Ethics Statement
This research was approved by the Human Investigations Committee of Zhejiang Normal University. All participants in this research signed a written informed consent. No vulnerable populations were involved.
Author Contributions
YW analyzed the data and wrote the first draft of the manuscript. HZ and YW performed the experiments and collected the data. GD conceived and designed the experiments. GD, LW, and JX revised and improved the manuscript.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Funding
This research was supported by National Science Foundation of China (30131023).
Supplementary Material
The Supplementary Material for this article can be found online at http://journal.frontiersin.org/article/10.3389/fpubh.2016.00274/full#supplementary-material.
ReferencesCarrN. Is Google making us stupid?Yearb Natl Soc Study Educ (2008) 107(2):89–94.10.1111/j.1744-7984.2008.00172.xLohKKKanaiR. How has the internet reshaped human cognition?Neuroscientist (2015) 22(5):506–20.10.1177/1073858415595005ZickuhrKMaddenM. Older adults and internet use. Pew Internet Am Life Proj (2012) 6:2–23.YangHChenYZhengLXuXCaoX. Analysis of internet use behaviors among clinical medical students in China. BMC Med Educ (2014) 14(1):67.10.1186/1472-6920-14-6724690437CarrN. The Shallows: What the Internet Is Doing to Our Brains. New York: WW Norton & Company (2011).ForsmanAKNordmyrJ. Psychosocial links between internet use and mental health in later life a systematic review of quantitative and qualitative evidence. J Appl Gerontol (2015).10.1177/0733464815595509SparrowBLiuJWegnerDM. Google effects on memory: cognitive consequences of having information at our fingertips. Science (2011) 333(6043):776–8.10.1126/science.120774521764755DongGPotenzaMN. Behavioural and brain responses related to Internet search and memory. Eur J Neurosci (2015) 42(8):2546–54.10.1111/ejn.1303926262779PeltokorpiV. Transactive memory systems. Rev Gen Psychol (2008) 12(4):378.10.1037/1089-2680.12.4.378FergusonAMMcLeanDRiskoEF. Answers at your fingertips: access to the Internet influences willingness to answer questions. Conscious Cogn (2015) 37:91–102.10.1016/j.concog.2015.08.00826340105BlockJJ. Issues for DSM-V: internet addiction. Am J Psychiatry (2008) 165:306–7.10.1176/appi.ajp.2007.07101556DavisRA. A cognitive-behavioral model of pathological Internet use. Comput Human Behav (2001) 17(2):187–95.10.1016/S0747-5632(00)00041-8DongGPotenzaMN. A cognitive-behavioral model of Internet gaming disorder: theoretical underpinnings and clinical implications. J Psychiatr Res (2014) 58:7–11.10.1016/j.jpsychires.2014.07.00525062755YoungKS. Internet addiction: symptoms, evaluation and treatment. Innov Clin Pract (1999) 17:19–31.ClarkA. Natural-Born Cyborgs: Minds, Technologies, and the Future of Human Intelligence. Oxford: Oxford University Press (2004).BartlettJMillerC. Truth, Lies and the Internet: A Report into Young People’s Digital Fluency. London: Demos (2011).NicholasDHuntingtonPJamaliHRRowlandsIFieldhouseM. Student digital information-seeking behaviour in context. J Doc (2009) 65(1):106–32.10.1108/00220410910926149NicholasDRowlandsIClarkDWilliamsP. Google generation II: web behaviour experiments with the BBC. Aslib Proceedings. (2011) 63(1):28–45.10.1108/00012531111103768RowlandsINicholasDWilliamsPHuntingtonPFieldhouseMGunterBThe Google generation: the information behaviour of the researcher of the future. Aslib Proceedings. (2008) 60(4):290–310.10.1108/00012530810887953GriffithsM. Internet addiction: fact or fiction?Psychologist (1999) 12(5):246–50.CronbachLJ. Coefficient alpha and the internal structure of tests. Psychometrika (1951) 16(3):297–334.10.1007/BF02310555KozakM. Comparative analysis of tourist motivations by nationality and destinations. Tour Manag (2002) 23(3):221–32.10.1016/S0261-5177(01)00090-5HerbertWJohnB. Goodness of fit in confirmatory factor analysis: the effects of sample size and model parsimony. Qual Quant (1994) 28(2):185–217.10.1007/BF01102761TanakaJSHubaGJ. A fit index for covariance structure models under arbitrary GLS estimation. Br J Math Statis Psychol (1985) 38(2):197–201.10.1111/j.2044-8317.1985.tb00834.xBentlerPMBonettDG. Significance tests and goodness of fit in the analysis of covariance structures. Psychol Bull (1980) 88(3):588.10.1037/0033-2909.88.3.588BentlerPM. Comparative fit indexes in structural models. Psychol Bull (1990) 107(2):238.10.1037/0033-2909.107.2.2382320703BrowneMWCudeckRBollenKALongJS. Alternative ways of assessing model fit. Sage Focus Edn (1993) 154:136–136.NunnallyJCBernsteinI. Psychological Theory. New York: McGraw-Hill (1994).GrevingHSassenbergK. Counter-regulation online: threat biases retrieval of information during Internet search. Comput Human Behav (2015) 50:291–8.10.1016/j.chb.2015.03.077AppendixThe Questionnaire on Internet Search Dependence
Hello, we are researchers in the Department of Psychology of Zhejiang Normal University. We carry this investigation to explore how you search the Internet. This result will not be open to the public; please answer the questions honestly and seriously.
Please read the following information first:
Internet search means individuals using Internet search engines (for example, Google and Baidu) to find information on the Internet.
The apparatus where people use Internet search engines include computers, IPADs, smart phones, and other terminal sets.
Basic information:
Gender: Male/Female:
Age:
Grade:
Major:
Years since your first use Internet:
Years since your first Internet search:
The time you spend on Internet search every day: minutes:
Please estimate the times you search the Internet everyday:
The percentage of the time spent on using search engine takes____% of the total time online.
Please provide your answers according to the following criteria:
0. Never 1. Seldom 2. Sometimes 3. Usually 4. Always.
When facing debatable questions, I prefer to search Google/Baidu first rather than thinking by myself.
I won’t take any notes if I know I can search for the information on the Internet.
When I am asked a complex question, I usually try to abstract the key words from it.
When someone disagrees with my points, I usually search the Internet for the answer.
I think it is not necessary to remember a thing if we can find it from Internet search.
When somebody asks me a question, I will search the Internet for answers if I cannot figure it out immediately.
I can quickly abstract keywords from a sentence for a potential Internet search.
I will be upset if I cannot find an answer to a complex question through Internet search.
I think Internet search can satisfy daily needs, including learning and living.
I usually start to search online unconsciously when I am idle.
I am not confident about the answers in my memory if I cannot double-check them through Internet search.
I think we can find reliable information through Internet search.
Thanks for your sincere participation!
The Dimensions of the Questionnaire on Internet Search Dependence
Habitual use of Internet search:
When facing debatable questions, I prefer to search Google/Baidu first rather than thinking by myself.
When I am asked a complex question, I usually try to abstract the key words from it.
When someone disagrees with my points, I usually search the Internet for the answer.
When somebody asks me a question, I will search the Internet for answers if I cannot figure it out immediately.
I can quickly abstract keywords from a sentence for a potential Internet search.
Withdrawal reaction:
I will be upset if I cannot find an answer to a complex question through Internet search.
I am not confident about the answers in my memory if I cannot double-check them through Internet search.
Internet search trust:
I think Internet search can satisfy daily needs, including learning and living.
I usually start to search online unconsciously when I am idle.
I think we can find reliable information through Internet search.
External storage under Internet search:
I won’t take any notes if I know I can search for the information on the Internet.
I think it is not necessary to remember a thing if we can find it from Internet search.