Authors:
Hannah Schmidt Department of Psychiatry and Psychotherapy, University of Lübeck, Translational Psychiatry Unit (TPU), Research Group S:TEP (Substance Use and Related Disorders: Treatment, Epidemiology, and Prevention), Lübeck, Germany
Department of Pediatric and Adolescent Medicine, University of Lübeck, Lübeck, Germany

Search for other papers by Hannah Schmidt in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0002-9393-5862
,
Dominique Brandt Department of Psychiatry and Psychotherapy, University of Lübeck, Translational Psychiatry Unit (TPU), Research Group S:TEP (Substance Use and Related Disorders: Treatment, Epidemiology, and Prevention), Lübeck, Germany

Search for other papers by Dominique Brandt in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0002-9753-4134
,
Anja Bischof Department of Psychiatry and Psychotherapy, University of Lübeck, Translational Psychiatry Unit (TPU), Research Group S:TEP (Substance Use and Related Disorders: Treatment, Epidemiology, and Prevention), Lübeck, Germany

Search for other papers by Anja Bischof in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0003-3176-3329
,
Silja Heidbrink Department of Psychiatry and Psychotherapy, University of Lübeck, Translational Psychiatry Unit (TPU), Research Group S:TEP (Substance Use and Related Disorders: Treatment, Epidemiology, and Prevention), Lübeck, Germany

Search for other papers by Silja Heidbrink in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0002-0310-1726
,
Gallus Bischof Department of Psychiatry and Psychotherapy, University of Lübeck, Translational Psychiatry Unit (TPU), Research Group S:TEP (Substance Use and Related Disorders: Treatment, Epidemiology, and Prevention), Lübeck, Germany

Search for other papers by Gallus Bischof in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0003-0432-5497
,
Stefan Borgwardt Department of Psychiatry and Psychotherapy, University of Lübeck, Translational Psychiatry Unit (TPU), Research Group S:TEP (Substance Use and Related Disorders: Treatment, Epidemiology, and Prevention), Lübeck, Germany

Search for other papers by Stefan Borgwardt in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0003-0924-8478
, and
Hans-Jürgen Rumpf Department of Psychiatry and Psychotherapy, University of Lübeck, Translational Psychiatry Unit (TPU), Research Group S:TEP (Substance Use and Related Disorders: Treatment, Epidemiology, and Prevention), Lübeck, Germany

Search for other papers by Hans-Jürgen Rumpf in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0001-6848-920X
Open access

Abstract

Background

Despite the constant publication of new screening instruments for Internet use disorders (IUD), little is known about their content validity. This study aimed to identify potential mismatches between the items' intention and young adults' interpretation of these items when answering three screening instruments that are commonly used in research and clinical settings: The Compulsive Internet Use Scale (CIUS), the 10 Item-Internet Gaming Disorder Test (IGDT-10), and the Bergen Social Media Addiction Scale (BSMAS).

Methods

In total, 30 vocational students (50% female, age = 21.3; SD = 2.1) took part in individual think-aloud interviews. All participants were asked to report their thoughts while completing the CIUS. In addition, participants who reported online games (OG) as their main Internet activity (n = 11) answered the IGDT-10. Participants who reported social networks (SN) as their main Internet activity (n = 18) answered the BSMAS. One participant used OG and SN equally and completed both screening instruments. All interviews were audio-recorded, transcribed, and content-analysed.

Results

Overall, four potential sources for errors were identified: (1) High scorings were often not congruent with the underlying diagnostic criteria. In particular, such discrepancies were found in items aimed to assess dysfunctional emotional regulation strategies and cognitive involvement. (2) Some participants were uncertain which time frame or Internet activity should be considered in their answers. (3) Long and complex items led to the building of mean values or the choice of the middle answer category. (4) Some wordings were perceived to be outdated and difficult to understand.

Discussion

These findings might help to provide recommendations on how to improve screening instruments for IUD. Most important, items should more clearly distinguish between Internet use as a “normal” leisure activity and Internet use that leads to functional impairments in daily life.

Abstract

Background

Despite the constant publication of new screening instruments for Internet use disorders (IUD), little is known about their content validity. This study aimed to identify potential mismatches between the items' intention and young adults' interpretation of these items when answering three screening instruments that are commonly used in research and clinical settings: The Compulsive Internet Use Scale (CIUS), the 10 Item-Internet Gaming Disorder Test (IGDT-10), and the Bergen Social Media Addiction Scale (BSMAS).

Methods

In total, 30 vocational students (50% female, age = 21.3; SD = 2.1) took part in individual think-aloud interviews. All participants were asked to report their thoughts while completing the CIUS. In addition, participants who reported online games (OG) as their main Internet activity (n = 11) answered the IGDT-10. Participants who reported social networks (SN) as their main Internet activity (n = 18) answered the BSMAS. One participant used OG and SN equally and completed both screening instruments. All interviews were audio-recorded, transcribed, and content-analysed.

Results

Overall, four potential sources for errors were identified: (1) High scorings were often not congruent with the underlying diagnostic criteria. In particular, such discrepancies were found in items aimed to assess dysfunctional emotional regulation strategies and cognitive involvement. (2) Some participants were uncertain which time frame or Internet activity should be considered in their answers. (3) Long and complex items led to the building of mean values or the choice of the middle answer category. (4) Some wordings were perceived to be outdated and difficult to understand.

Discussion

These findings might help to provide recommendations on how to improve screening instruments for IUD. Most important, items should more clearly distinguish between Internet use as a “normal” leisure activity and Internet use that leads to functional impairments in daily life.

Introduction

Internet use disorders (IUD) are widely discussed as a potential new diagnosis in the spectrum of behavioral addictions. To date, the conceptual framework of IUD is still part of a controversial debate. Several researchers have criticized the term “IUD” to be inadequate because it passes over important differences between heterogeneous Internet activities (Starcevic & Billieux, 2017). In the last years, research has mainly focused on the psychological mechanism underlying pathological gaming. This has led to the inclusion of the Internet gaming disorder (IGD) in the 5th edition of the “Diagnostic and Statistical Manual of Mental Disorders” (DSM-5; American Psychiatric Association [APA], 2013). Besides, Gaming Disorder (GD) was included in the 11th edition of the “International Classification of Diseases and Related Health Problems” (ICD-11, World Health Organization [WHO], 2011). Although not yet listed in the DSM-5 or ICD-11 as distinct Internet activity, there is growing evidence that the pathological use of online social networks can also be classified as addictive behavior that leads to significant impairments in daily life (D’Arienzo, Boursier, & Griffiths, 2019).

Screening instruments for IUD

Despite the constant development and publication of new screening instruments, the optimal assessment of IUD is still unclear. Most screening instruments suffer from an insufficient conceptual or methodological background (King, Chamberlain, et al., 2020). Besides, there is a tendency in research to label any excessive Internet use behavior as a potential new behavioral addiction (Billieux, Schimmenti, Khazaal, Maurage, & Heeren, 2015). This may result in overpathologizing “normal” leisure activities of young adults and high rates of false-positive results (Billieux et al., 2015). To date, existing screening instruments either assess Internet use as a general construct or focus on specific Internet applications (e.g., gaming or social networks). The Compulsive Internet Use Scale (CIUS), the 10 Item-Internet Gaming Disorder Test (IGDT-10), and the Bergen Social Media Addiction Scale (BSMAS) are examples of such screening instruments that are widely used for research purposes and in clinical settings.

Compulsive Internet Use Scale (CIUS)

The CIUS (Meerkerk, Van Den Eijnden, Vermulst, & Garretsen, 2009) measures IUD as a general construct. In total, 14 items represent the following criteria for Compulsive Internet Use (CIU): Salience, withdrawal, loss of control, conflict, and mood modification (Meerkerk et al., 2009). These criteria were adopted by the criteria for DSM-IV Dependence (APA, 1994), DSM-IV Pathological Gambling (APA, 1994), and the criteria of behavioral addictions as suggested by Griffiths (1999). All items can be answered on a 5-point Likert scale, ranging from “never” (0) to “very often” (4). Participants scoring at least 28 points in the CIUS are at increased risk for pathological Internet use (Meerkerk et al., 2009). The psychometric quality of the CIUS was tested in previous studies. Good internal reliability criteria (Cronbach's α ranging from 0.88 to 0.90) and a stable one-factor solution were found among different samples (Meerkerk et al., 2009). All items of the CIUS are listed in Table 2.

10 Item-Internet Gaming Disorder Test (IGDT-10)

The IGDT-10 is a commonly used screening instrument to assess pathological Internet gaming behavior (Király et al., 2019). Its structure is based on the nine IGD criteria as suggested in the DSM-5 (Király et al., 2019). Each criterion is operationalized by a single item, apart from the last criterion “jeopardizing or losing a significant relationship, job, or educational or career opportunity because of participation in Internet games”. This item was operationalized by two items because of its complexity. All items can be answered on a 3-point Likert scale labeled 0 (“never”), 1 (“sometimes”), and 2 (“often”). In previous studies, an unidimensional factor structure provided a good fit to the data in cross-cultural samples (Király et al., 2019). In a Chinese sample, the IGDT-10 was validated with a structured interview (Chiu, Pan, & Lin, 2018). Good internal reliability criteria (Cronbach's α = 0.85) and an adequate diagnostic efficiency (area under the curve = 0.81) were found (Chiu et al., 2018). In a Finish sample, the IGDT-10 exhibited a good internal consistency (Cronbach's α = 0.87) and acceptable standardized item loadings (Männikkö, Routsalainen, Tolvanen, & Kääriäinen, 2019). All items of the IGDT-10 are listed in Table 3.

Bergen Social Media Addiction Scale (BSMAS)

The popularity of online social networks changes quickly. Thus, screening instruments addressing specific social networks (e.g., Facebook or Instagram) lose their up-to-dateness easily. The BSMAS is an adoption of the Bergen Facebook Addiction Scale (Andreassen, Torsheim, Brunborg, & Pallesen, 2012) and considers social networks as a general construct. The BSMAS includes six items that refer to the core components of behavioral addiction as proposed by Griffiths (2005): Salience, mood modification, tolerance, withdrawal, conflict, and relapse. All items can be answered on a 5-point Likert scale ranging from “never” (0) to “very often” (4). The BSMAS has a good fit of the model to the data across different gender and age groups, thus confirming the single-factor structure of the instrument (Andreassen et al., 2012). All items of the BSMAS are listed in Table 4.

Think-aloud interviews

Despite the psychometric quality of the CIUS, the IGDT-10, and the BSMAS has been tested frequently in previous studies, little is known about the content validity of these screening instruments. When developing a new screening instrument, the wording of each item should be reviewed for its clarity and relevance to the underlying psychological construct (American Educational Research Association [AERA] et al., 2014). In the field of IUD, such reviews have often been carried out through expert ratings (e.g., King, Billieux, Carragher, & Delfabbro, 2020). However, experts may fail to recognize typical problems of lay individuals, who often interpret items in unexpected ways. Lay individuals may experience difficulties in understanding the item's content, recalling necessary information from the memory, reflecting and judging their own Internet use, and finally choose an answer (AERA et al., 2014; Presser et al., 2004; Ryan, Gannon-Slater, & Culbertson, 2012). Each step is a potential source of errors which directly impairs the screening's validity.

To investigate the validity of screening instruments in a thorough manner, it is of importance to analyse cognitive processes underlying responders' answers. In the broad spectrum of cognitive methods in psychological research, think-aloud interviews are an established approach to make such cognitive processes more explicit (Ryan et al., 2012; Wolcott & Lobczowski, 2021). Think-aloud interviews are conducted under standardized conditions and follow a strict protocol (Ryan et al., 2012). After receiving instructions and comprehensive training on how to “think-aloud”, participants are asked to spontaneously verbalize all thoughts that come to their mind while answering a screening instrument (Ryan et al., 2012). To avoid potential bias in participants' response behavior, the interviewing process must be non-intrusive (Ryan et al., 2012). The interviewer's role is to observe the interview process and remind the participant to constantly verbalize all cognitions (Ryan et al., 2012). As a result, problem-solving processes can be objectively recorded, transcribed, and content-analysed (Presser et al., 2004). Other cognitive methods, such as verbal probes, are typically more directive and intrusive by asking targeted interview questions (Ryan et al., 2012). Summing up, think-aloud interviews are an appropriate method for testing the content validity of screening instruments by lay persons. The feasibility of think-aloud interviews has already been tested in samples of adolescents in the field of substance-related disorders (e.g., Nehlin, Carlsson, & Öster, 2020). In the field of IUD, such in-depth analyses of cognitive processes underlying participants' answers have not yet been carried out.

Aim of this study

This study aimed to investigate the content validity of the CIUS, the IGDT-10, and the BSMAS by analysing young adults' cognitive processes when reporting and valuing their Internet use behavior. Based on clinical impressions, it is hypothesized that several items may have the potential of leading to misconception or overpathologizing of unproblematic Internet use. Since the CIUS, the IGDT-10, and the BSMAS have been developed many years ago and Internet applications are changing fast, it is further hypothesized that the wording is outdated which results in reduced content validity. Our findings might contribute to a better understanding when interpreting quantitative results of other studies. Besides, it may provide recommendations for possible improvements of these screening instruments.

Methods

Participants

All participants were part of the study “Intervening in Problematic Internet use“ (iPIN; Schmidt et al., submitted for publication). The iPIN study aimed to investigate the efficacy of a brief motivational intervention for adolescents and young adults with at-risk Internet use. Participants were proactively screened for problematic Internet use with the CIUS in 12 vocational schools in northern Germany. The tablet-assisted screening took place between March 2018 and March 2019. Inclusion criteria were sufficient knowledge of the German language and a minimum age of 16 years. In total, 8.230 screenings were realized and 1.475 participants with at-risk use of the Internet agreed to be contacted for telephone-based diagnostic baseline interviews. In the baseline interview, IUD was assessed using the structured, clinical interview “Internet-related disorders-Clinical Assessment Tool (I-CAT)”. I-CAT was developed by the iPIN project team and is based on the structure of the Munich Composite International Diagnostic Interview (M-CIDI; Wittchen et al., 1995) which is a fully standardized diagnostic interview to determine mental disorders following the ICD-10 Diagnostic Criteria for Research. Independent from the time spent on the Internet, I-CAT covers all nine criteria for IUD in line with the DSM-5 diagnostic criteria of IGD. Of 937 realized interviews, 497 participants fulfilling at least two criteria for IUD in the last three months were randomly assigned either to the intervention or the control group. The intervention group received up to three telephone consultations. The control group received a brochure with recommendations on how to modify problematic behavior on the Internet. Telephone-based follow-up interviews were conducted after five months (n = 301) and ten months (n = 284). After completing the second follow-up interview, participants were asked to take part in individual think-aloud interviews. In total, 197 participants agreed to take part in the think-aloud interview. Of these, 30 individuals who were stratified by gender, the main activity on the Internet, and the number of IUD criteria in the second follow-up interview were contacted. The sample size was based on reflections on the study aim, sample specificity, and analyses strategy. If eligible individuals could not be contacted or did not appear for the interview, they were invited once again. One potential participant who did not appear twice was excluded. Another participant of those who agreed to take part in the think-aloud study was recruited. The mean time between the second follow-up and the think-aloud interview was 66.6 (SD = 55.8) days with a range from 9 to 175 days. All participants received an incentive of 50 Euros for participation in the think-aloud interview.

Procedure of the think-aloud interview

The interviews were based on the think-aloud approach (Wolcott & Lobczowski, 2021) and a semi-structured interview guide. The BSMAS and IGDT-10 were translated into German. Double back-translation was performed to avoid language bias. Prior to the interview, three exercises were performed to practice “thinking-aloud”. For example, participants were asked to think aloud while they imagined going through their home and counting all windows (“Visualize the place where you live, go slowly through every room and count all windows”). To facilitate this process and to lower potential barriers of thinking aloud, all exercises were demonstrated by the interviewers in advance.

Once these exercises were complete, the interviewer asked the participants to engage in the same think-aloud process as they answer each item of the following screening instruments. First, all items of the CIUS were rated by all participants. While verbalizing all thoughts, participants were asked to fill in the items on the screening sheets they had available in front of them. Depending on their main Internet activity, participants were given paper versions either of the IGDT-10 (those with online games as main Internet activity) or the BSMAS (those with social networks as main Internet activity). Again, participants rated all items either of the BSMAS or of the IGDT-10 while verbalizing all thoughts. Participants were asked to not reflect their thoughts but to report their thoughts concurrently. Besides, they were reminded that this study aimes to understand cognitive processes rather than analysing the item rating itself.

Quality assurance

All interviews (20–30 min) were conducted by trained project members (HS, SS, AB, NM) and took place between January and February 2020. All interviewers received standard operating procedures (SOP's) to ensure the objectivity of the implementation. Prior to the main data collection, three test interviews were conducted with students of the University of Lübeck. Afterwards, minor adjustments were made in terms of the wording and length of the instruction. Weekly project meetings and supervisions with all interviewers, student assistants, and at least one of the senior researchers of the working group (DB, AB, HJR) were conducted to discuss potential difficulties during the interview process.

Analysis

All interviews were audio-recorded, verbatim transcribed, and content-analysed. The transcripts were initially coded by at least two members of the project team, independently from each other. A-priori categories were developed deductively in the project team and named as follows: Quality of statement, comprehension, susceptibility to interference, similarity to other items, and indication of Internet-related problems. The further analysis was an iterative process focusing on general problems while answering the screening instrument. Unclear cases were discussed in detail in the project team with all interviewers and senior researchers until a consensus was formed.

Ethics

The study procedure was carried out in line with the Declaration of Helsinki. It was approved by the ethics committee of the University of Lübeck on 24th September 2019. All participants were informed about the purpose of the study and all provided informed consent.

Clinical trial registration

The interviews were an additional project of the iPIN study that was registered on August 24th, 2018. The clinical trial registration of the iPIN study is available here: https://clinicaltrials.gov/ct2/show/NCT03646448?term=iPIN&draw=2&rank=1.

Results

Sample characteristics

Of 30 participants, 11 participants reported online games as their main Internet activity and filled out the IGDT-10. Besides, 18 participants mainly used social networks and filled out the BSMAS. One participant reported to use online games and social networks equally. Therefore, he filled out both screening instruments (BSMAS and IGDT-10). The sample characteristics are shown in Table 1.

Table 1.

Sample characteristics

Total1 (N = 30) BSMAS (n = 19) IGTD-10 (n = 12)
Sociodemographic variables
Age, M (SD) 21.3 (2.1) 21.1 (2.1) 21.6 (2.1)
Female gender, n (%) 15 (50.0) 15 (78.9) 0 (0.0)
Residental situation
 Alone, n (%) 5 (16.7) 2 (10.5) 3 (25.0)
 With parents/grand-parents, n (%) 19 (63.3) 12 (63.2) 8 (66.7)
 With partner, n (%) 1 (3.3) 1 (5.3) 0 (0.0)
 Shared flat, n (%) 5 (16.7) 4 (21.1) 1 (8.3)
Vocational school situation
 Still in vocational school, n (%) 18 (60.0) 12 (63.2) 6 (50.0)
 Already finished vocational school, n (%) 10 (33.3) 5 (26.3) 6 (50.0)
 Dropped out of vocational school, n (%) 2 (6.7) 2 (10.5) 0 (0.0)
Partnership, n (%) 12 (40.0) 9 (47.4) 3 (25.0)
Internet-related variables
Number of fulfilled IUD criteria
 0–2 (unproblematic Internet use), n (%) 16 (53.3) 10 (52.6) 7 (58.3)
 3–4 (problematic Internet use), n (%) 8 (26.7) 4 (21.1) 4 (33.3)
 5–9 (pathological Internet use), n (%) 6 (20.0) 5 (26.3) 1 (8.3)

Notes: 1 In total, 30 participants were interviewed. One participant reported to use online games and social media equally and completed both screening instruments (BSMAS and IGDT-10). All sociodemographic data and the number of fulfilled IUD criteria were collected in the second follow-up interview of the iPIN study.

Results of the think-aloud interviews

General difficulties when answering the screening instrument are presented separately for the CIUS, the IGDT-10, and the BSMAS. Specific difficulties of each item and suggestions for improvement are presented in Tables 24.

Table 2.

Compulsive Internet Use Scale (CIUS)

CIUS instruction: “Choose an answer that best applies to you.”
Item Criterion (Meerkerk et al., 2009) Difficulties Suggestions for improvement Statements of the participants
(1) Do you find it difficult to stop using the Internet when you are online? Loss of control The item was often associated with high scorings although participants were able to stop their Internet use in case of important other tasks. Consider to specify the context to assess the negative impact on daily life (e.g., “[…] to stop using the Internet even if you have other important things to do?”). “It's not difficult for me to stop when I know that I have to fulfill responsibilities.”
(2) Do you often continue to use the Internet despite your intention to stop? Loss of control Strong similarity to the first item. Check for redundancy; possibly delete one of these items. “I compare that item with the first question. They definitely look similar to me […].”

“That's the same […] that actually results from the first question.”
(3) Do others (e.g., friends and family) say you should use the Internet less? Conflict Participants' answers depend on their age. The Internet use of young people is often criticized by parents or grand-parents. This item has the potential to cover a generation conflict rather than characteristics of a behavioral addiction. Consider to split this item into (1) conflicts with same-aged peers or partners and (2) conflicts with family members. “Actually, no one except my grandma […]”.

“Often, especially my parents.”
(4) Do you prefer to use the Internet instead of spending time with others (e.g., friends and family)? Preoccupation To date, many real-life friendships of young adults are maintained online. Some reasons are long distances after moving to another town or sudden life-changing events such as the corona pandemic. In this context, spending time with others online is not necessarily a problematic behavior but (quite the opposite) a functional way to maintain real-life friendships. Consider to reword this item to assess problematic social withdrawal in real life (e.g., “How often do you prefer to use the Internet rather than spend time offline with others?”) “[…; My] friends don't live here. The Internet is actually the only option to communicate with them. So, actually very often.”
(5) Are you short of sleep because of the Internet? Loss of control No relevant difficulties. The item was easy to understand.
(6) Do you think about the Internet, even when you are not online? Preoccupation Participants often relate to boredom or to situations when alternative leisure activities are not available. It remains unclear if high scorings of this item were associated with a negative impact on daily life. Consider to reword and specify this item (e.g., “How often do you have a strong desire to use the Internet although you have other important things to do?”). “I guess when it's boring […]. When you're busy, you don't think about it.”
The term “when you are not online” seems outdated. Nowadays, access to the Internet is possible almost always and everywhere. See above. “Sometimes, because I'm actually always on the Internet. No matter whether I'm on the move or at home […].”
Participants think of online social interactions or important messages that are expected but not about “the Internet” per se. See above. “I'm thinking a lot about the Internet […] whether someone important has written to me […] I'm still waiting for a very important answer from a colleague.”

“Internet is a very broad topic, that's why you think often about the different dialogues you had on the Internet […].”
(7) Do you look forward to your next Internet session? Preoccupation See item 6: The term “look forward to your next Internet session” seems outdated. Internet access is possible almost always and everywhere. See item 6. Consider to delete one of these items. “I never really look forward to it, but […] I take it for granted.”
The anticipation to use the Internet refers to specific activities or social contacts maintained online. This does not reflect a strong desire to use the Internet in a way that leads to negative consequences. See above. “That happens […] often because I'm actually looking forward to talk to my colleagues again in the evening.”
The anticipation to use the Internet refers to end unpleasant work or activities and use the Internet as a “normal” leisure activity to relax. See above. “After a long day, I actually always look forward to it […] I sit down on the couch, play a little bit. I say […] often”
The term “Internet session” seems difficult to understand. See above. “What does Internet session mean? […] Is it about WhatsApp? Writing a message? Or just watching a movie or playing games? […]”
(8) Do you think you should use the Internet less often? Conflict No relevant comments. The item was easy to understand.
(9) Have you unsuccessfully tried to spend less time on the Internet? Loss of control It is unclear if the Internet use leads to negative consequences in daily life. Consider to reword and specify this item (e.g., “How often have you unsuccessfully tried to spend less time on the Internet because you have noticed negative consequences of your Internet use?”) “It happened a lot. Considering that I've been on the Internet almost daily since I was 14 years old […] it often happened that I didn't managed to spend less time on the Internet.”
Overall, participants had significant difficulties to understand the content of the item. Consider to rephrase the item. “I don't understand the question.”

“Unsuccessful, I would take rarely […] presupposing that I have [… tried to reduce] I have rarely done that. […] I think that's […] strange.”
(10) Do you rush through your homework/schoolwork to use the Internet? Conflict Participants relate to the general avoidance of unpleasant tasks but not to the Internet per se. Consider to rephrase the item (e.g., “How often do you rush through activities you actually enjoy to get on the Internet?”). “That happens a lot. […] I vacuum faster […] to write again […] with other people […]. It's not necessarily “the Internet” but rather the free time that you have again. So, I would say […] that's often the case.”
(11) Do you neglect your daily obligations (work, school, or family life) because you prefer to use the Internet? Conflict Increased risk of overpathologizing: It is not clear if the Internet use leads to significant impairment of daily life. Consider to assess the temporal context and negative consequences more clearly (e.g., “How often do you experience negative consequences or neglect responsibilities because you prefer to use the Internet?”). “I think maybe I should watch a movie with my parents […] instead of being online […]. But […] their interests are different. I would say […] sometimes.”
(12) Do you use the Internet when you are feeling sad? Coping/Mood modification Participants tend to relate to functional emotional regulation strategies (e.g., social support they receive online). Consider to rephrase the item and add a temporal reference (e.g., “How often do you use the Internet when you feel sad to forget about your problems?“) “[…] when something doesn't go the way it should […] you go online for a while […] you have your friends there and chat with them […] they usually calm you down […].”
(13) Do you use the Internet to escape from sorrows or negative feelings? Coping/Mood modification Strong similarity to Item 12. Check for redundancy; possibly delete one of these items. “That's exactly the same.“

“For me, this is similar to question 12 […] I will answer question 13 the same way I answered number 12.”
(14) How often do you feel anxious, frustrated, or irritated when you are not able to use the Internet? Withdrawal Despite high scorings of this item, participants do not refer to “withdrawal” but rather to technical problems. Consider to specify the instruction of the CIUS, e.g., “In your answers, please do not refer to technical problems.“ “When the Internet router breaks down […] it's annoying […] that's why […] I'm […] very often irritated.”
Participants had difficulties to respond to this item. Nowadays, access to the Internet is possible almost always and everywhere. Consider to specify the situation, e.g., “[…] you are not able to use the Internet because of daily obligations (e.g., school, work, or family life).” “[…] I always have access to the Internet through my mobile phone. For me, such situations don't exist.”
Increased risk of overpathologizing: Participants refer to situations when important social concerns need to be clarified via digital channels. Consider to rephrase the item or the instruction of the CIUS to make sure that participants do not refer to (constructive) problem-solving approaches in real life. “I feel restless when […] I have to clarify something important, and I can't answer my mobile phone or use the Internet.”
Overall findings in the CIUS Participants' answers strongly depend on the respective Internet activity. Consider to assess the main activity while using the Internet in the instruction (e.g., “Please name the Internet application you use most frequently at the moment: _______. When answering the items, please refer to the application you named above.” “It depends […] when I play video games, I stop. WhatsApp, Instagram […] I actually continue.”
The time frame was unclear. Consider to specify the time frame in the instruction. “Is that […] related to the current situation or […] the past?”
Table 3.

Internet Gaming Disorder Test - 10 Items (IGDT-10)

IGDT-10 instruction: “Please read the statements below regarding video gaming. The questionnaire refers to video games (both online and offline, played on any platform). Please indicate on a scale from 0 to 2 (0 = never, 1 = sometimes, 2 = often) to what extent and how often these statements applied to you over the past 12 months.”
Item Criterion (Király et al., 2019) Difficulties Suggestions for improvement Statements of the participants
(1) When you were not playing, how often have you fantasized about gaming, thought of previous gaming sessions, and/or anticipated the next game? Preoccupation Participants do not refer to the criterion “preoccupation” but rather to gaming as a leisure activity with friends or to boredom without a negative impact on daily life. Consider to define the situation more clearly (e.g., “[…] although you have other important things to do?”). “Of course, because […] you can relax, meet your friends without leaving your house […].”

“I often think back to LAN parties with friends […].”

“Actually, I always look forward to such things when I have nothing else to do […].”
The term “imagine playing games“ seems too abstract. Consider to delete this term. “Why should I imagine […] oh yeah […] a controller would be really nice in my hand […] (laughs)?”
(2) How often have you felt restless, irritable, anxious, and/or sad when you were unable to play or played less than usual? Withdrawal No relevant comments.
(3) Have you ever in the past 12 months felt the need to play more often or played for longer periods to feel that you have played enough? Tolerance Participants often refer to a temporary desire to finish a story (particularly when they had to pay for the game). Rephrase the item and define the situation more clearly (“[…] although you have other important things to do?”) “I want to finish the game and then put it aside because otherwise I feel like I've wasted the money.”

“Sometimes when a game is very captivating […] you want to reach a certain level or the end of a campaign […] just to see an end or the progress of a story […].”
(4) Have you ever in the past 12 months unsuccessfully tried to reduce the time spent on gaming? Loss of control Participants' reported Internet use behavior does not lead to negative consequences in daily life. Rephrase the item to assess more clearly if the gaming behavior leads to negative consequences (e.g., “[…] because you have noticed negative consequences?”) “When my friends are online […] I want to play with them and ask myself: ‘What else should I do?’ And I have nothing to do anyway. It doesn't have a negative effect on me anyway, I can do it, as long as everything else is going on, I can go online. […] So yes, sometimes.”
(5) Have you ever in the past 12 months played games rather than meet your friends or participate in hobbies and pastimes that you used to enjoy before? Giving up other activities Participants refer to positive social gaming experiences. The desire to play online games might be normative and socially determined. Define the situation more clearly (e.g., “[…] with friends outside of online activities”). “My friends […] are online as well […] this connects us.”
(6) Have you played a lot despite negative consequences (for instance losing sleep, not being able to do well in school or work, having arguments with your family or friends, and/or neglecting important duties)? Continuation The item was perceived to be long and complicated. Consider to shorten the examples. “The question is too long. I'm trying to read the question again.”

“Boah, this is lengthy.”
There are various aspects in one question. Participants tend to build mean values across all options instead of considering whether at least one of the mentioned options applies to them. In the instruction of the IGDT-10, point out that referring to at least one option is sufficient. “Lack of sleep actually quite often because […] I prefer to game instead of sleeping. I'm not sure about the loss of performance […]. I'm still good in sports. […] I don't really have any duties […]. So I can't say that I've lost much. I would say […] sometimes.”
(7) Have you tried to keep your family, friends, or other important people from knowing how much you were gaming or have you lied to them regarding your gaming? Deception Participants refer to financial aspects of gaming. Consider to extend the item (e.g., “[…] to hide how much time or money you spent with gaming […]?”). “ […] invested with money but that question is related to time […].”
(8) Have you played to relieve from a negative mood (for instance helplessness, guilt, or anxiety)? Escape Participants played games to temporarily distract from negative emotions. However, despite high scorings they usually don't play games to avoid problem solving or functional emotional regulation in a long term. Rephrase the item and define situation more clearly, e.g., “[…] to escape from negative mood […] in a long term?”. “[…] to forget about my negative thoughts for a while […] and deal with the problem afterwards.”
(9) Have you ever in the past 12 months risked or lost a significant relationship because of gaming? Negative consequences Participants mainly refer to romantic relationships. They did not consider a potential negative impact of problematic gaming on family members or friends. Consider to define the term “relationship” more clearly. “I don't know how to interpret the term relationship […] whether it's really a romantic relationship or […] a friendly relationship.”

“Not in the last twelve months. I was single.”

“No […]. At the moment, I'm single.”
(10) Have you ever in the past 12 months jeopardized your school or work performance because of gaming? Negative consequences No difficulties. The item was easy to understand.
Overall findings of the IGDT-10 Offering only three answer options (never, sometimes, always) was perceived to be undifferentiated. Consider to add more answer options. “I take sometimes because never is a lie. But sometimes sounds like a lot […] I'll take sometimes anyway because there is nothing in-between […].”
Table 4.

Bergen Social Media Addiction Scale (BSMAS)

BSMAS instruction: ”Below you find some questions about your relationship to social media and your use of social media (Facebook, Twitter, Instagram, …). Choose for each question the response that describes you best. How often during the last year have you…”
Item Criterion (Andreassen et al., 2012) Difficulties Suggestions for improvement Statements of the participants
(1) […] spent a lot of time thinking about social media or planned use of social media? Salience High scorings rarely correspond to the underlying criterion “salience”. Participants rather thought about specific content or news they saw online or interactions with other people on social media. Consider to explore if social media use has a negative impact on participants' daily life (e.g., “[…] although you have other important things to do?”). “When something happens and it's in the news and I see it on Facebook […] I think about it.”

“Of course […] when you have a conflict with someone or maybe you're planning a meeting […]. Then you probably think about it a lot.”
The term “planning” seems too abstract in the context of social media use. Consider to delete the complete term “planning to use social media”. “I don't plan it. […] To plan […] that's strange with social media.”
(2) […] felt an urge to use social media more and more? Tolerance Participants thought about important social interactions without significant impairments in daily life. See above (e.g., “[…] although you have other important things to do?”). “This particular person I'm writing with, online right now […] we are talking about something important. So, I would say […] it happened more often that I really felt the urge to do so.”
(3) […] used social media to forget about personal problems? Mood modification High scorings were often associated with functional emotional regulation strategies, e.g., seeking social support via social media. Reword the item to make sure that participants do not refer to functional emotional regulation strategies. “[…] only to contact friends and talk about the problems and forget about them.”
Participants had difficulties to define the term “personal problems”. Consider to clarify this term. “It depends on how you define personal problems […] if small things like boredom are part of it […] then sometimes to […] often.”
(4) […] tried to cut down on the use of social media without success? Relapse Participants had difficulties to define the term “cut down”. Consider to clarify the term (e.g., “[…] significantly cut down your social media use to feel confident?”) “What does cut down mean? […] Is it about a few minutes? Is it about hours?”
(5) […] become restless or troubled if you have been prohibited from using social media? Withdrawal Participants tend to refer to long-term technical problems. Consider to specify the instruction of the BSMAS (e.g., “In your answers, please do not refer to technical problems”). “If our Wi-Fi […] doesn't work, you get a bit anxious. Because you can't do anything […] if you don't have Internet access for two or three weeks.”

“When I was at school and I forgot my phone at home […] I was worried because I lived in the village and I couldn't get away there myself.”
Participants rarely correspond to the underlying criterion “withdrawal” but rather to important social events that happen in real-life. Consider to specify the context (e.g., “[…] when you couldn't use social media because of daily obligations?”) “The other day I confessed my love to my best buddy […]. I was a bit worried that he doesn't answer me […].”
(6) […] used social media so much that it has had a negative impact on your job/studies? Conflict No relevant comments. This item was easy to understand.
Overall findings of the BSMAS The time frame of 12 months seems too long. Participants usually refer to the past weeks. Consider to shorten the time frame. “I think a year is such an insanely long period […]. I couldn't summarize something like that.”
The instruction “How often in the last year have you” is mentioned once above all items. Many participants skipped the instruction and were confused about the time frame when they tried to answer the items. Consider to include the time frame in the beginning of each item. “Hard to answer because I don't know what time frame is intended […].”

“Oh, up there, I didn't even read this […].”

CIUS

The CIUS instruction (“Chose the answer that best applies to you”) was perceived to be unspecific and caused several comprehension problems. Participants were not sure which time frame (“Is this about the last few months or the last year?”) or Internet activity should be considered (“I'm wondering. I associate being online with WhatsApp or Facebook […] I don't know if that was meant here?”). Items aimed to assess excessive Internet use as a dysfunctional emotional regulation strategy were often rated with high scorings. However, when analysing participants' cognitions underlying their scoring, we rarely found evidence for dysfunctional emotional regulation strategies. Despite the Internet was temporarily used to distract from daily problems, participants did not generally avoid specific emotions (e.g., sadness, anger) as suggested in the underlying DSM-5 criterion (“If something went wrong, you go online for a while, meet your friends there […] and calm down”). High scorings on items aimed to assess cognitive involvement (e.g., “Do you think about the Internet, even when you are not online?” or “Do you already look forward to your next Internet session?”) were often related to a lack of leisure alternatives or boredom. Besides, participants report that they were waiting for important news and social interactions. This does not necessarily reflect problematic Internet use. Besides, several items were perceived as redundant (“I compare that to the first question, they definitely look similar to me”). In contrast, items aimed to assess impairments caused by dysfunctional Internet use (e.g., insufficient sleep) were merely well understood. A detailed overview of participants' difficulties when answering the CIUS and suggestions for improvement are presented in Table 2.

IGDT-10

The 3-point Likert scale of the IGDT-10 was perceived to be undifferentiated. In cases of uncertainty, participants often chose the middle response category (“I take sometimes because never would be a lie. But sometimes sounds like a lot […] I'll take sometimes anyway because there is nothing in-between […].”). Furthermore, several items were perceived to be long and complicated. Offering multiple options (“Did you continue playing despite negative consequences, e.g., negative impact on sleep or performance at school or work, arguments with family or friends, and/or neglect of important responsibilities”) has led to the building of mean values across all options instead of reflecting if at least one option is true. High scorings of items underlying the criterion craving were often related to unproblematic use, e.g., the anticipation of new game releases or LAN parties with friends. Besides, participants reported that the IGDT-10 tends to overpathologize (“[…the IGDT-10] is pretty focused on negative things […] not things like finding friends on the Internet, maintaining contact with friends online, such things are not included, that's […] what I do on the Internet. I find new people with whom I can talk, I […] keep in touch simply via the Internet. I've met really cool friends just through the Internet. I've never met them, but we even know what we look like, we know each other REALLY well. And that […] is really missing in this questionnaire […] positive aspects of the Internet […].”). Table 3 shows all occurring difficulties while answering the IGDT-10 and suggestions for modifications.

BSMAS

Following the original version of the BSMAS, the introduction (“How often in the last year have you…”) was mentioned only once in the introduction. Participants repeatedly skipped the information about the time frame (“So, first question […] oh […] How often in the last year […] I almost missed it.”). Furthermore, reflecting the Internet use for one year was perceived to be difficult (“One year is such a long period. It's insanely difficult to summarize.”). Participants were not sure what social media applications should be considered (“I'm wondering: Social media […] what does that term include? […] Snapchat, Instagram, WhatsApp. I'm not sure if that's part of it?”). Participants did not refer to dysfunctional emotional regulation strategies but to long-term technical problems (“When I was at school, and I forgot my phone at home, or there was not enough battery, I was worried because I lived in the village, and I couldn't get away there myself.”). This does not reflect addictive behavior. In some cases, even functional emotional regulation strategies (e.g., seeking social support online) were associated with high scorings (“[…] only to contact friends and talk about the problems […]”).

Despite most items were well understood, some specific wordings seem rather unusual or too abstract. Participants reported that their social network use was not planned but simply took place (“If I want to, I'll use social media, but I don't plan it. […] The word planning […] that's strange with social media.”). Additionally, participants thought about the content they saw online rather than the use of the social networks, e.g., real-life social events that were planned via social media. Table 4 presents details and possible modifications of the BSMAS for each item separately.

Discussion

Despite the constant development and publication of new screening instruments for IUD, to date, little is known about their content validity. In this study, we analysed cognitive processes of young adults when valuing their Internet use behavior with the three established screening instruments (CIUS, IGDT-10, and BSMAS). The aim was to identify potential mismatches between items' intention and participants' comprehension of these items. Based on clinical impressions, it was hypothesized that some items tend to overpathologize young adults' Internet use. Besides, it was hypothesized that the wording of some items is outdated which might lead to misperceptions.

Across all screening instruments, we found that high scorings were often not congruent with the underlying diagnostic criteria for problematic Internet use. These mismatches were particularly noted for items assessing Internet use as a dysfunctional emotional regulation strategy and cognitive involvement. This finding is in line with an expert rating analysing the face validity of screening instruments for problematic online gaming (King, Billieux, et al., 2020). Despite high scorings, participants relate to Internet use as a “normal” leisure activity without negative impairments in daily life. Besides, participants thought about specific content they saw online or about real-life relationships maintained online but not about “the Internet” per se. All in all, such discrepancies between items' intention and participants' comprehension of these items may lead to an increased risk of overpathologizing leisure activites of young adults and may generate false-positive results. In the light of the global Covid-19 pandemic, in which many leisure activities necessarily took place online, careful attention should be paid to ensure that items do not misjudge normal or even functional behavior of young adults (e.g., the fulfillment of social needs via online platforms). In upcoming revisions of these screenings instruments, the context and potential negative impairments caused by dysfunctional Internet use should be clarified, e.g., by adding “even if you have other important things to do” to the original item “How often do you find it difficult to stop using the Internet when you are online?”. Alternatively, the instruction text of the screening instrument could highlight functional impairments caused by excessive Internet use: “In your answers, please consider if your Internet use behavior leads to neglecting important obligations (e.g., friendships, family, work, or school)”. To further avoid false-positive results, the instruction of the screening instrument should ensure that participants do not refer to technical problems.

Besides these mismatches between items' intention and participants' comprehension of items, participants often experience difficulties with the time frame. Both, the fact that no time frame was given (CIUS) and long time frames of 12 months (IGDT-10, BSMAS) caused difficulties. In line with recommendations to improve screening instruments in other psychological fields (Meerwijk & Weiss, 2017), it should be discussed to shorten the time frame of the IGDT-10 and BSMAS. Despite both screening instruments have defined a time frame of 12 months, participants still tend to refer to situations they had previously encountered or chose a rough estimate. It is further recommended either to integrate the time frame at the beginning of each item or to highlight the time frame more clearly in the instruction with capatial letters. Apart from the unclear time frame, participants were not sure to which application they should refer to when answering the CIUS or the BSMAS. Thus, it might be helpful to ask for the main Internet application in the instruction.

Another finding was that several items were perceived outdated or incomprehensible. Rereading or stumbling are important indicators of an invalid responding in think-aloud interviews (Darker & French, 2009). Such phenomena are highly associated with choosing the middle response option in case of uncertainty (Darker & French, 2009). In this study, these phenomena occurred with very long items or those with many examples in parentheses. It is recommended to shorten items that were considered too long and complicated. Furthermore, screening instruments for IUD might need to account for cultural and social differences. In line with an other think-aloud approach in a sample of adolescents answering screening instruments for alcohol use (Nehlin et al., 2020), we found that outdated wordings that are not commonly used in this age group might lead to misconceptions (e.g., “planning to use social media” in the BSMAS). In upcoming revisions, adolescents and young adults should be included in the item development process. Redundancies should be avoided. Particularly in the CIUS, participants experienced difficulties because of strong similarities (e.g., Item 1 and Item 2).

Summing up, these findings may contribute to a better understanding about how young adults perceive and interpret self-report screening instruments for IUD that are commonly used in both research and in clinical work. These findings may further help to explain heterogeneous prevalence rates for IUD that may strongly depend on the utilized screening instrument. Self-report screening instruments are not developed for diagnosing a specific disorder. It is important to be aware that individuals may fulfill symptoms above a specific cut-off but still not meet criteria for clinically relevant distress or functional impairments caused by dysfunctional Internet use. The rationale of IUD screening instruments based on self-ratings is to identify individuals at risk for problematic or even pathological Internet use behavior to refer them to professionals for in-depth clinical interviews. To further improve the content validity, it seems necessary to evaluate the CIUS, the IGDT-10, and the BSMAS in both clinical and general populations with different ages and in addition with in-depth clinical diagnostic interviews.

Limitations and strengths

Several limitations should be addressed when interpreting the results. In general, the quality of think-aloud interviews depends on the participant's ability to articulate own thoughts while answering a screening instrument. Specific personality traits (e.g., introversion) and social desirability might impair this process. Besides, only male participants reported online games as main activity on the Internet. Therefore, the content validity of the IGDT-10 was analysed in men but not in women. Conversely, our sample allows limited conclusions about the comprehension of the BSMAS from the perspective of men. Another limitation is the small sample size (n = 30) that limits the generalizability of the results. However, this study had an exploratory intention and does not claim to provide a representative sample. The sample size was carefully considered in line with other qualitative approaches and think-aloud studies (e.g., Blair & Conrad, 2011). Results of such qualitative studies help to reword and improve screening instruments. In the next step, the revised version could be validated in the context of large samples. Despite these limitations and to the best of our knowledge, this is the first study aimed to investigate the content validity of commonly used screening instruments for IUD. The think-aloud approach provides less distraction or bias in terms of social desirability compared to “classic” question-answer interviews. The sample consists of individuals with a heterogeneous amount of fulfilled criteria for IUD which provides as much representativity as possible. Nevertheless, replications in unselected participants would be helpful to add further findings of individuals without prior screening for problematic Internet use.

Conclusion

This study provides insights into young adults' cognitive processes when responding to the three commonly used screening instruments for problematic Internet use behavior. We identified four main sources for errors: (1) High scorings were often not congruent with the underlying diagnostic criteria for problematic Internet use. This increases the risk of overpathologizing “normal” Internet use of young adults. In particular, such discrepancies were found in items assessing Internet use as a dysfunctional emotion regulation strategy and cognitive involvement. (2) The instructions of the screening instruments were unclear. Participants were not sure to what time frame or Internet application they should relate. (3) Long periods of 12 months and complex items with many examples lead to difficulties. As a result, participants tend to build mean values or chose the middle answer category. (4) Several wordings were perceived to be outdated and incomprehensible. All in all, these findings might help to generate recommendations on how to revise and clarify several items. As a main indication for upcoming revisions, items should separate more clearly between “normal” Internet use without negative consequences and Internet use that leads to functional impairments in daily life. To further improve the content validity, it might be helpful to include a heterogeneous sample of adolescents and young adults with and without IUD in the process of item development.

Funding sources

The think-aloud interviews were part of the iPIN-study that was funded by the German Federal Ministry of Health (Grant number ZMVI1-2517DSM210).

Authors’ contribution

HS: Study concept and design, gathering of screening data in vocational schools as part of the iPIN study, conduction of baseline telephone interviews and conduction of the think-aloud interviews, interpretation of the findings, preparation of the manuscript draft. DB: Study concept and design, major analysis and interpretation of the findings, revision of the manuscript. AB: Study concept and design, conduction of think-aloud interviews, analysis and interpretation of the findings, revision of the manuscript. GB, SH: Study concept and design, analysis and interpretation of the findings, revision of the manuscript. SB: Conceptual support and revision of the manuscript. HJR: PI, study concept and design, obtained the conduction of the study, analysis, and interpretation of findings, revision of the manuscript.

Conflict of interest

All authors declare that they do not have a conflict of interest.

Acknowledgment

We would like to thank all vocational students who have agreed to participate in the study. Furthermore, we thank all students and project team members who supported us with the conduction and the analysis of all think-aloud interviews: Clara Becker, Fanny Lindner, Nina Meyer, Lisa Feldhoff, Svenja Sürig, and Charlotte Miriam Tiemann.

References

  • American Educational Research Association, American Psychological Association, National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    • Search Google Scholar
    • Export Citation
  • American Psychiatric Association (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: American Psychiatric Association.

    • Search Google Scholar
    • Export Citation
  • American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association.

    • Search Google Scholar
    • Export Citation
  • Andreassen, C. S. , Torsheim, T. , Brunborg, G. S. , & Pallesen, S. (2012). Development of a Facebook addiction scale. Psychological Reports, 110(2), 501517. https://doi.org/10.2466/02.09.18.PR0.110.2.501-517.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Billieux, J. , Schimmenti, A. , Khazaal, Y. , Maurage, P. , & Heeren, A. (2015). Are we overpathologizing everyday life? A tenable blueprint for behavioral addiction research. Journal of Behavioral Addictions, 4(3), 119123. https://doi.org/10.1556/2006.4.2015.009.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Blair, J. , & Conrad, F. G. (2011). Sample size for cognitive interview pretesting. Public Opinion Quarterly, 75(4), 636658. https://doi.org/10.1093/poq/nfr035.

    • Search Google Scholar
    • Export Citation
  • Chiu, Y.-D. , Pan, Y.-C. , & Lin, Y.-H. (2018). Chinese adaptation of the Ten-Item Internet Gaming Disorder Test and prevalence estimate of Internet gaming disorder among adolescents in Taiwan. Journal of Behavioral Addiction, 7(3), 719726. https://doi.org/10.1556/2006.7.2018.92.

    • Search Google Scholar
    • Export Citation
  • D’Arienzo, M. C. , Boursier, V. , & Griffiths, M. D. (2019). Addiction to social media and attachment styles: A systematic literature & review. International Journal of Mental Health and Addiction, 17, 10941118. https://doi.org/10.1007/s11469-019-00082.

    • Search Google Scholar
    • Export Citation
  • Darker, C. D. , & French, D. P. (2009). What sense do people make of a theory of planned behaviour questionnaire? A think-aloud study. Journal of Health Psychology, 14, 861871. https://doi.org/10.1177/1359105309340983.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Griffiths, M. (1999). Internet addiction: Fact or fiction? The Psychologist, 12(5), 246250.

  • Griffiths, M. (2005). A ‘components’ model of addiction within a biopsychosocial framework. Journal of Substance Use, 10(4), 191197. https://doi.org/10.1080/14659890500114359.

    • Search Google Scholar
    • Export Citation
  • King, D. L. , Billieux, J. , Carragher, N. , & Delfabbro, P. H. (2020b). Face validity evaluation of screening tools for gaming disorder: Scope, language, and overpathologizing issues. Journal of Behavioral Addictions, 9(1), 113. https://doi.org/10.1556/2006.2020.00001.

    • Search Google Scholar
    • Export Citation
  • King, D. L. , Chamberlain, S. R. , Carragher, N. , Billieux, J. , Stein, D. , Mueller, K. , … Delfabbro, P. H. (2020a). Screening and assessment tools for gaming disorder: A comprehensive systematic review. Clinical Psychology Review, 77, 101831. https://doi.org/10.1016/j.cpr.2020.101831.

    • Search Google Scholar
    • Export Citation
  • Király, O. , Bothe, B. , Ramos-Diaz, J. , Rahimi-Movaghar, A. , Lukavska, K. , Hrabec, O. , & Demetrovics, Z. (2019). Ten-item Internet Gaming Disorder Test (IGDT-10): Measurement invariance and cross cultural validation across seven language-based samples. Psychology of Addictive Behaviors, 33(1), 91103. https://doi.org/10.1037/adb0000433.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Männikkö, N. , Routsalainen, H. , Tolvanen, A. , & Kääriäinen, M. (2019). Psychometric properties of the Internet Gaming Disorder Test (IGDT-10) and problematic gaming behavior among Finnish vocational school students. Scandinavian Journal of Psychology, 60(3), 252260. https://doi.org/10.1111/sjop.12533.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Meerkerk, G. J. , Van Den Eijnden, R. , Vermulst, A. A. , & Garretsen, H. F. L. (2009). The compulsive Internet use scale (CIUS): Some psychometric properties. Cyberpsychology & Behavior, 12(1), 16. https://doi.org/10.1089/cpb.2008.0181.

    • Search Google Scholar
    • Export Citation
  • Meerwijk, E. L. , & Weiss, S. J. (2017). Utility of a time frame in assessing psychological pain and suicide ideation. PeerJ, 5, e3491. https://doi.org/10.7717/peerj.3491.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Nehlin, C. , Carlsson, K. , & Öster, C. (2020). How adolescents think when responding to alcohol-related questionnaires: A think-aloud study. Youth & Society, 116. https://doi.org/10.1177/0044118X20959239.

    • Search Google Scholar
    • Export Citation
  • Presser, S. , Couper, M. P. , Lessler, J. T. , Martin, E. , Martin, J. , Rothgeb, J. M. , & Singer, E. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68(1), 109130. https://doi.org/10.1093/poq/nfh008.

    • Search Google Scholar
    • Export Citation
  • Ryan, K. , Gannon-Slater, N. , & Culbertson, M. J. (2012). Improving survey methods with cognitive interviews in small-and medium-scale evaluations. American Journal of Evaluation, 33(3), 414430. https://doi.org/10.1177/1098214012441499.

    • Search Google Scholar
    • Export Citation
  • Schmidt, H. , Brandt, D. , Meyer, C. , Bischof, A. , Bischof, G. , Trachte, A. , … Rumpf, H.-J. (2022). Motivational brief interventions for adolescents and young adults with Internet use disorders: A randomized-controlled trial. Manuscript submitted for publication.

    • Search Google Scholar
    • Export Citation
  • Starcevic, V. , & Billieux, J. (2017). Does the construct of Internet addiction reflect a single entity or a spectrum of disorders? Clinical Neuropsychiatry: Journal of Treatment Evaluation, 14(1), 510.

    • Search Google Scholar
    • Export Citation
  • Wittchen, H.-U. , Beloch, E. , Garczynski, E. , Holly, A. , Lachner, G. , & Perkonigg, A. , et al. (1995). Munich Composite International Diagnostic Interview (M-CIDI), version 2.2. Munich: Max Planck Institute of Psychiatry, Clinical Psychology and Epidemiology.

    • Search Google Scholar
    • Export Citation
  • Wolcott, M. D. , & Lobczowski, N. G. (2021). Using cognitive interviews and think-aloud protocols to understand thought processes. Current in Pharmacy Teaching and Learning, 13(2), 181188. https://doi.org/10.1016/j.cptl.2020.09.005.

    • Search Google Scholar
    • Export Citation
  • World Health Organization (2011). ICD-11 for mortality and morbidity statistics. Retrieved online: https://icd.who.int/browse11/l-m/en#/http://id.who.int/icd/entity/1448597234.

    • Search Google Scholar
    • Export Citation
  • American Educational Research Association, American Psychological Association, National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    • Search Google Scholar
    • Export Citation
  • American Psychiatric Association (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: American Psychiatric Association.

    • Search Google Scholar
    • Export Citation
  • American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders (DSM-5®). Washington, DC: American Psychiatric Association.

    • Search Google Scholar
    • Export Citation
  • Andreassen, C. S. , Torsheim, T. , Brunborg, G. S. , & Pallesen, S. (2012). Development of a Facebook addiction scale. Psychological Reports, 110(2), 501517. https://doi.org/10.2466/02.09.18.PR0.110.2.501-517.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Billieux, J. , Schimmenti, A. , Khazaal, Y. , Maurage, P. , & Heeren, A. (2015). Are we overpathologizing everyday life? A tenable blueprint for behavioral addiction research. Journal of Behavioral Addictions, 4(3), 119123. https://doi.org/10.1556/2006.4.2015.009.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Blair, J. , & Conrad, F. G. (2011). Sample size for cognitive interview pretesting. Public Opinion Quarterly, 75(4), 636658. https://doi.org/10.1093/poq/nfr035.

    • Search Google Scholar
    • Export Citation
  • Chiu, Y.-D. , Pan, Y.-C. , & Lin, Y.-H. (2018). Chinese adaptation of the Ten-Item Internet Gaming Disorder Test and prevalence estimate of Internet gaming disorder among adolescents in Taiwan. Journal of Behavioral Addiction, 7(3), 719726. https://doi.org/10.1556/2006.7.2018.92.

    • Search Google Scholar
    • Export Citation
  • D’Arienzo, M. C. , Boursier, V. , & Griffiths, M. D. (2019). Addiction to social media and attachment styles: A systematic literature & review. International Journal of Mental Health and Addiction, 17, 10941118. https://doi.org/10.1007/s11469-019-00082.

    • Search Google Scholar
    • Export Citation
  • Darker, C. D. , & French, D. P. (2009). What sense do people make of a theory of planned behaviour questionnaire? A think-aloud study. Journal of Health Psychology, 14, 861871. https://doi.org/10.1177/1359105309340983.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Griffiths, M. (1999). Internet addiction: Fact or fiction? The Psychologist, 12(5), 246250.

  • Griffiths, M. (2005). A ‘components’ model of addiction within a biopsychosocial framework. Journal of Substance Use, 10(4), 191197. https://doi.org/10.1080/14659890500114359.

    • Search Google Scholar
    • Export Citation
  • King, D. L. , Billieux, J. , Carragher, N. , & Delfabbro, P. H. (2020b). Face validity evaluation of screening tools for gaming disorder: Scope, language, and overpathologizing issues. Journal of Behavioral Addictions, 9(1), 113. https://doi.org/10.1556/2006.2020.00001.

    • Search Google Scholar
    • Export Citation
  • King, D. L. , Chamberlain, S. R. , Carragher, N. , Billieux, J. , Stein, D. , Mueller, K. , … Delfabbro, P. H. (2020a). Screening and assessment tools for gaming disorder: A comprehensive systematic review. Clinical Psychology Review, 77, 101831. https://doi.org/10.1016/j.cpr.2020.101831.

    • Search Google Scholar
    • Export Citation
  • Király, O. , Bothe, B. , Ramos-Diaz, J. , Rahimi-Movaghar, A. , Lukavska, K. , Hrabec, O. , & Demetrovics, Z. (2019). Ten-item Internet Gaming Disorder Test (IGDT-10): Measurement invariance and cross cultural validation across seven language-based samples. Psychology of Addictive Behaviors, 33(1), 91103. https://doi.org/10.1037/adb0000433.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Männikkö, N. , Routsalainen, H. , Tolvanen, A. , & Kääriäinen, M. (2019). Psychometric properties of the Internet Gaming Disorder Test (IGDT-10) and problematic gaming behavior among Finnish vocational school students. Scandinavian Journal of Psychology, 60(3), 252260. https://doi.org/10.1111/sjop.12533.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Meerkerk, G. J. , Van Den Eijnden, R. , Vermulst, A. A. , & Garretsen, H. F. L. (2009). The compulsive Internet use scale (CIUS): Some psychometric properties. Cyberpsychology & Behavior, 12(1), 16. https://doi.org/10.1089/cpb.2008.0181.

    • Search Google Scholar
    • Export Citation
  • Meerwijk, E. L. , & Weiss, S. J. (2017). Utility of a time frame in assessing psychological pain and suicide ideation. PeerJ, 5, e3491. https://doi.org/10.7717/peerj.3491.

    • PubMed
    • Search Google Scholar
    • Export Citation
  • Nehlin, C. , Carlsson, K. , & Öster, C. (2020). How adolescents think when responding to alcohol-related questionnaires: A think-aloud study. Youth & Society, 116. https://doi.org/10.1177/0044118X20959239.

    • Search Google Scholar
    • Export Citation
  • Presser, S. , Couper, M. P. , Lessler, J. T. , Martin, E. , Martin, J. , Rothgeb, J. M. , & Singer, E. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68(1), 109130. https://doi.org/10.1093/poq/nfh008.

    • Search Google Scholar
    • Export Citation
  • Ryan, K. , Gannon-Slater, N. , & Culbertson, M. J. (2012). Improving survey methods with cognitive interviews in small-and medium-scale evaluations. American Journal of Evaluation, 33(3), 414430. https://doi.org/10.1177/1098214012441499.

    • Search Google Scholar
    • Export Citation
  • Schmidt, H. , Brandt, D. , Meyer, C. , Bischof, A. , Bischof, G. , Trachte, A. , … Rumpf, H.-J. (2022). Motivational brief interventions for adolescents and young adults with Internet use disorders: A randomized-controlled trial. Manuscript submitted for publication.

    • Search Google Scholar
    • Export Citation
  • Starcevic, V. , & Billieux, J. (2017). Does the construct of Internet addiction reflect a single entity or a spectrum of disorders? Clinical Neuropsychiatry: Journal of Treatment Evaluation, 14(1), 510.

    • Search Google Scholar
    • Export Citation
  • Wittchen, H.-U. , Beloch, E. , Garczynski, E. , Holly, A. , Lachner, G. , & Perkonigg, A. , et al. (1995). Munich Composite International Diagnostic Interview (M-CIDI), version 2.2. Munich: Max Planck Institute of Psychiatry, Clinical Psychology and Epidemiology.

    • Search Google Scholar
    • Export Citation
  • Wolcott, M. D. , & Lobczowski, N. G. (2021). Using cognitive interviews and think-aloud protocols to understand thought processes. Current in Pharmacy Teaching and Learning, 13(2), 181188. https://doi.org/10.1016/j.cptl.2020.09.005.

    • Search Google Scholar
    • Export Citation
  • World Health Organization (2011). ICD-11 for mortality and morbidity statistics. Retrieved online: https://icd.who.int/browse11/l-m/en#/http://id.who.int/icd/entity/1448597234.

    • Search Google Scholar
    • Export Citation
  • Collapse
  • Expand
The author instruction is available in PDF.
Please, download the file from HERE

Dr. Zsolt Demetrovics
Institute of Psychology, ELTE Eötvös Loránd University
Address: Izabella u. 46. H-1064 Budapest, Hungary
Phone: +36-1-461-2681
E-mail: jba@ppk.elte.hu

Indexing and Abstracting Services:

  • Web of Science [Science Citation Index Expanded (also known as SciSearch®)
  • Journal Citation Reports/Science Edition
  • Social Sciences Citation Index®
  • Journal Citation Reports/ Social Sciences Edition
  • Current Contents®/Social and Behavioral Sciences
  • EBSCO
  • GoogleScholar
  • PsycINFO
  • PubMed Central
  • SCOPUS
  • Medline
  • CABI

2021  
Web of Science  
Total Cites
WoS
5223
Journal Impact Factor 7,772
Rank by Impact Factor Psychiatry SCIE 26/155
Psychiatry SSCI 19/142
Impact Factor
without
Journal Self Cites
7,130
5 Year
Impact Factor
9,026
Journal Citation Indicator 1,39
Rank by Journal Citation Indicator

Psychiatry 34/257

Scimago  
Scimago
H-index
56
Scimago
Journal Rank
1,951
Scimago Quartile Score Clinical Psychology (Q1)
Medicine (miscellaneous) (Q1)
Psychiatry and Mental Health (Q1)
Scopus  
Scopus
Cite Score
11,5
Scopus
CIte Score Rank
Clinical Psychology 5/292 (D1)
Psychiatry and Mental Health 20/529 (D1)
Medicine (miscellaneous) 17/276 (D1)
Scopus
SNIP
2,184

2020  
Total Cites 4024
WoS
Journal
Impact Factor
6,756
Rank by Psychiatry (SSCI) 12/143 (Q1)
Impact Factor Psychiatry 19/156 (Q1)
Impact Factor 6,052
without
Journal Self Cites
5 Year 8,735
Impact Factor
Journal  1,48
Citation Indicator  
Rank by Journal  Psychiatry 24/250 (Q1)
Citation Indicator   
Citable 86
Items
Total 74
Articles
Total 12
Reviews
Scimago 47
H-index
Scimago 2,265
Journal Rank
Scimago Clinical Psychology Q1
Quartile Score Psychiatry and Mental Health Q1
  Medicine (miscellaneous) Q1
Scopus 3593/367=9,8
Scite Score  
Scopus Clinical Psychology 7/283 (Q1)
Scite Score Rank Psychiatry and Mental Health 22/502 (Q1)
Scopus 2,026
SNIP  
Days from  38
submission  
to 1st decision  
Days from  37
acceptance  
to publication  
Acceptance 31%
Rate  

2019  
Total Cites
WoS
2 184
Impact Factor 5,143
Impact Factor
without
Journal Self Cites
4,346
5 Year
Impact Factor
5,758
Immediacy
Index
0,587
Citable
Items
75
Total
Articles
67
Total
Reviews
8
Cited
Half-Life
3,3
Citing
Half-Life
6,8
Eigenfactor
Score
0,00597
Article Influence
Score
1,447
% Articles
in
Citable Items
89,33
Normalized
Eigenfactor
0,7294
Average
IF
Percentile
87,923
Scimago
H-index
37
Scimago
Journal Rank
1,767
Scopus
Scite Score
2540/376=6,8
Scopus
Scite Score Rank
Cllinical Psychology 16/275 (Q1)
Medicine (miscellenous) 31/219 (Q1)
Psychiatry and Mental Health 47/506 (Q1)
Scopus
SNIP
1,441
Acceptance
Rate
32%

 

Journal of Behavioral Addictions
Publication Model Gold Open Access
Submission Fee none
Article Processing Charge 850 EUR/article
Printed Color Illustrations 40 EUR (or 10 000 HUF) + VAT / piece
Regional discounts on country of the funding agency World Bank Lower-middle-income economies: 50%
World Bank Low-income economies: 100%
Further Discounts Editorial Board / Advisory Board members: 50%
Corresponding authors, affiliated to an EISZ member institution subscribing to the journal package of Akadémiai Kiadó: 100%
Subscription Information Gold Open Access

Journal of Behavioral Addictions
Language English
Size A4
Year of
Foundation
2011
Volumes
per Year
1
Issues
per Year
4
Founder Eötvös Loránd Tudományegyetem
Founder's
Address
H-1053 Budapest, Hungary Egyetem tér 1-3.
Publisher Akadémiai Kiadó
Publisher's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
Responsible
Publisher
Chief Executive Officer, Akadémiai Kiadó
ISSN 2062-5871 (Print)
ISSN 2063-5303 (Online)

Senior editors

Editor(s)-in-Chief: Zsolt DEMETROVICS

Assistant Editor(s): Csilla ÁGOSTON

Associate Editors

  • Joel BILLIEUX (University of Lausanne, Switzerland)
  • Beáta BŐTHE (University of Montreal, Canada)
  • Matthias BRAND (University of Duisburg-Essen, Germany)
  • Luke CLARK (University of British Columbia, Canada)
  • Daniel KING (Flinders University, Australia)
  • Ludwig KRAUS (IFT Institute for Therapy Research, Germany)
  • H. N. Alexander LOGEMANN (ELTE Eötvös Loránd University, Hungary)
  • Astrid MÜLLER (Hannover Medical School, Germany)
  • Marc N. POTENZA (Yale University, USA)
  • Hans-Jurgen RUMPF (University of Lübeck, Germany)
  • Attila SZABÓ (ELTE Eötvös Loránd University, Hungary)
  • Róbert URBÁN (ELTE Eötvös Loránd University, Hungary)
  • Aviv M. WEINSTEIN (Ariel University, Israel)

Editorial Board

  • Max W. ABBOTT (Auckland University of Technology, New Zealand)
  • Elias N. ABOUJAOUDE (Stanford University School of Medicine, USA)
  • Hojjat ADELI (Ohio State University, USA)
  • Alex BALDACCHINO (University of Dundee, United Kingdom)
  • Alex BLASZCZYNSKI (University of Sidney, Australia)
  • Judit BALÁZS (ELTE Eötvös Loránd University, Hungary)
  • Kenneth BLUM (University of Florida, USA)
  • Henrietta BOWDEN-JONES (Imperial College, United Kingdom)
  • Wim VAN DEN BRINK (University of Amsterdam, The Netherlands)
  • Gerhard BÜHRINGER (Technische Universität Dresden, Germany)
  • Sam-Wook CHOI (Eulji University, Republic of Korea)
  • Damiaan DENYS (University of Amsterdam, The Netherlands)
  • Jeffrey L. DEREVENSKY (McGill University, Canada)
  • Naomi FINEBERG (University of Hertfordshire, United Kingdom)
  • Marie GRALL-BRONNEC (University Hospital of Nantes, France)
  • Jon E. GRANT (University of Minnesota, USA)
  • Mark GRIFFITHS (Nottingham Trent University, United Kingdom)
  • Anneke GOUDRIAAN (University of Amsterdam, The Netherlands)
  • Heather HAUSENBLAS (Jacksonville University, USA)
  • Tobias HAYER (University of Bremen, Germany)
  • Susumu HIGUCHI (National Hospital Organization Kurihama Medical and Addiction Center, Japan)
  • David HODGINS (University of Calgary, Canada)
  • Eric HOLLANDER (Albert Einstein College of Medicine, USA)
  • Jaeseung JEONG (Korea Advanced Institute of Science and Technology, Republic of Korea)
  • Yasser KHAZAAL (Geneva University Hospital, Switzerland)
  • Orsolya KIRÁLY (Eötvös Loránd University, Hungary)
  • Emmanuel KUNTSCHE (La Trobe University, Australia)
  • Hae Kook LEE (The Catholic University of Korea, Republic of Korea)
  • Michel LEJOXEUX (Paris University, France)
  • Anikó MARÁZ (Humboldt-Universität zu Berlin, Germany)
  • Giovanni MARTINOTTI (‘Gabriele d’Annunzio’ University of Chieti-Pescara, Italy)
  • Frederick GERARD MOELLER (University of Texas, USA)
  • Daniel Thor OLASON (University of Iceland, Iceland)
  • Nancy PETRY (University of Connecticut, USA)
  • Bettina PIKÓ (University of Szeged, Hungary)
  • Afarin RAHIMI-MOVAGHAR (Teheran University of Medical Sciences, Iran)
  • József RÁCZ (Hungarian Academy of Sciences, Hungary)
  • Rory C. REID (University of California Los Angeles, USA)
  • Marcantanio M. SPADA (London South Bank University, United Kingdom)
  • Daniel SPRITZER (Study Group on Technological Addictions, Brazil)
  • Dan J. STEIN (University of Cape Town, South Africa)
  • Sherry H. STEWART (Dalhousie University, Canada)
  • Attila SZABÓ (Eötvös Loránd University, Hungary)
  • Ferenc TÚRY (Semmelweis University, Hungary)
  • Alfred UHL (Austrian Federal Health Institute, Austria)
  • Johan VANDERLINDEN (University Psychiatric Center K.U.Leuven, Belgium)
  • Alexander E. VOISKOUNSKY (Moscow State University, Russia)
  • Kimberly YOUNG (Center for Internet Addiction, USA)

 

Monthly Content Usage

Abstract Views Full Text Views PDF Downloads
Jun 2022 0 111 82
Jul 2022 0 103 65
Aug 2022 0 143 80
Sep 2022 0 57 59
Oct 2022 0 64 51
Nov 2022 0 49 54
Dec 2022 0 0 0