Abstract
In a recent pilot study, we examined the potential benefits and opportunities that ChatGPT can bring to higher education, particularly from the perspective of business students and educators. The study included 41 participants and aimed to explore their opinions on using ChatGPT in business language classes. Twelve students did not use ChatGPT during the course (control group), while 29 students used it actively (experimental group). Examining their experiences and comparing the two groups, it is evident that students see the benefits and disadvantages of ChatGPT and use it for tasks they find helpful. However, the practice and hands-on experience helped the experimental group members gain much more diverse and nuanced opinions about ChatGPT. These results underline that universities and their boards must embrace the technology and find reasonable areas to use ChatGPT. These may not only be connected to assignment forms and plagiarism, but can embrace more general topics, like equal eligibility to these new technologies or strengthening the students' social and emotional intelligence and skills to help their future lives.
1 Introduction
The emergence of generative artificial intelligence (AI), such as ChatGPT, poses unprecedented challenges for higher education. On 30 November 2022, Open AI released its sophisticated large language model, ChatGPT, and millions of users started to test its free and pro versions within just a few days (Mitrović et al. 2023). Students and teachers reacted to the news with excitement and trepidation, respectively. Students sensed an opportunity to enhance their learning, while teachers saw a potential threat to academic integrity. ChatGPT has questioned the current practices in education and the appropriate ways of accountability from one day to another. Moreover, increasing potential for the direct substitution of human labor has emerged in many areas of the business sector, which has anticipated the possibility of a fast transformation in the labor market, and consequently in higher education, that supplies the labor market with a well-trained workforce holding up-to-date knowledge. In other words, besides “how”, the question of “what” will be taught in the future's higher education institutions has also arisen:
To get to know the students' first reactions in more detail and understand their perspectives on these changes, new potentials, and threats, research was carried out in the spring of 2023, shortly after the launch of ChatGPT, to evaluate its potential as a tool for productive and fair use in the teaching and learning process. This posttest-only nonequivalent control group quasi-experimental research aims to unveil how the students perceive the benefits of ChatGPT, what their expectations in terms of changes in the learning mode and the teaching style are, what their attitudes are towards regulation, how they interpret the role of ChatGPT specifically, how they reinterpret the role of the universities, and finally, how these perceptions may be influenced by the actual practical experience they acquire using the tools. Accordingly, in the following, we will provide a theoretical background and essential information about the conceptual framework, then discuss the applied methodology and the experiment's findings. After discussing the results and lamenting their consequences, we close our paper with a short conclusion addressing our investigation's limitations and further research directions.
2 Theoretical background and conceptual framework
AI can be briefly defined as the ability of machines, including computers and related technologies, to exhibit adaptive, problem-solving, and intelligent behavior similar to humans (Coppin 2004). This concept involves the integration of computers, machines, and information communication technologies, enabling them to perform tasks with human-like professionalism. The introduction of AI in the education sector is extensive, in line with the broader integration of emerging technologies (Chen et al. 2020).
To discuss the changes in the content and the teaching methods of higher education in its complexity, Gibson et al. (2023), for example, propose a model that can capture the effects of AI at three levels: in supporting (1) the individual (micro level), in which the role of AI is to support individual learners, develop personalized learning pathways and strategies, and enhance individual knowledge and skills; (2) the community (meso level), when the focus is on learning processes, collaboration, knowledge sharing and community knowledge building of teams, peer groups and knowledge communities; and (3) the cultural learning processes (macro), when we look at the role of AI from a socio-cultural development perspective.
Our research aims to provide micro-level empirics but also aspires to have a more holistic perspective and interpret the results on the macro level, considering the embeddedness of the micro and meso levels.
2.1 Future skills and digital literacy
Increasing students' ability to acquire key competencies and develop new skills that may qualify them for the current and future demands of the labor market is undoubtedly one of the crucial tasks of higher education institutions (HEIs) (Becker et al. 2017; Garcia-Esteban – Jahnke 2020) that affects HEIs' market orientation strategies (Chandler et al. 2021). Many studies focus on the new powerful drivers that change the job market (e.g. Medland 2016; Ehlers – Kellermann 2019; Davies – van Seventer 2020; Maxwell – Gallagher 2020) and on identifying these new desirable skills and skill profiles explicitly (e.g. González – Wagenaar 2003; EC Eurobarometer 2014; Weng 2015; Bakhshi et al. 2017; CEDEFOP 2018; Garcia-Esteban – Jahnke 2019).
Trends like the spread of smart machines that can replace or augment humans in offices or factories (Davies – van Seventer 2020) or the emergence of technologies that amplify human performance (Bakhshi et al. 2017) raise the inconvenient question of what unique properties the human brain has, or with other words, what comparative advantage it can provide in a job market where tasks are computable, programmable and designable based on enormous quantities of data (Davies – van Seventer 2020) and the application of AI. The answer to these questions might fundamentally impact the desired skills and students' expectations from higher education.
The empirical literature unveils that before the rise of Language Learning Models (LLMs) in 2022, the discourse about future skills that higher education develops was already about the possible automation of tasks or whole occupations (see MacCrory et al. 2014; Hughes 2017) and the role of the human brain in the economy.
Weng (2015) argues that although technological products can replace human work in many domains, higher-level thinking skills, like perceiving the meanings of expression to help solve problems or make decisions, remain essential in the long run. She considers the increasingly technological world and multiculturalism two of the significant changes in job markets and lists computational thinking (understanding data, systems, programs, and software designs), new media literacy (ability to engage and communicate utilizing user-generated media) and cross-cultural competence (ability to function effectively in multi-cultural or different cultural contexts) as the most important future skills from these two perspectives. When we focus on the human brain, however, skills like sense-making (reacting cognitively and emotionally towards different objects, perceiving surroundings, and comprehending the complex world to perform better in various situations), social, emotional, and cultural intelligence; design mindset (seeking for workable best solutions), novel and adaptive thinking (going beyond the rule-based patterns), and management of cognitive load (capacity for filtering information for significance and maximizing cognitive functioning) play crucial roles (Weng 2015).
Instead of digitalization or globalization, the concept of a risk society can also be a point for departure in our argumentation. Ulf Ehlers and Sarah Kellermann (2019) argue that the most significant challenge students need to be prepared for through higher education is greater connection to the constant need for adaptation through learning in a constantly changing future work environment and dealing with uncertainty. These are thus the main foundations of the most important future skills that the authors divide into three groups: (1) self-development-related skills (e.g., autonomy, agility, self-management); (2) object-related skills (e.g., ability to reflect, creativity, digital literacy); and (3) organization-related skills (communication skill, sense-making).
However, the pressures on higher education exist beyond the changing expectations related to skill development. Employing digital technologies for teaching and learning, student engagement, or curriculum development is an opportunity and a challenge. “Digital literacy” has emerged as a key concept of integrating technologies into higher education and primarily captures the skills and dispositions required for effective use of digital media (Pangrazio et al. 2020). Integrating AI as a digital technology into higher education has many possible domains, like providing feedback (Wang et al. 2024), enhancing writing and language skills, assisting students with disabilities, and fostering critical thinking and media literacy abilities. However, it can also be challenging due to possible biases, misinformation, and an over-dependence on AI tools (Ciampa et al. 2023).
These expectations, trends, and challenges related to the “demands” toward higher education (Király – Géring 2021; Fűzi et al. 2022) affect what and how higher education “supplies,” that is, the teaching and learning in HEIs.
2.2 Changes in teaching and learning – related, but not limited to ChatGPT
The following focuses on the teaching and learning process and its challenges and possibilities related to LLMs and related new technologies, especially the most prevalent, ChatGPT.
ChatGPT is the best-known advanced LLM by the company OpenAI (OpenAI 2023). It has been available since November 2022 and can perform various tasks. It can create authentic and inventive text formats, translate languages, and provide informative answers to queries. These features make it a precious resource for researchers studying human-computer communication and educators exploring its potential.
While ChatGPT is a relatively new tool – and the publication timeline is typically around one year – very few analyses can be found about its practical use and effects in higher education based on actual projects. Nonetheless, in Table 1, we have collected some of the advantages and challenges mentioned in the academic discourse up to early 2024.
Advantages and challenges of ChatGPT in higher education
Level | Advantages | Challenges/disadvantages |
Micro-level: individual learner and teacher |
|
|
Meso-level: higher education sector |
|
|
Macro-level: society |
|
|
Source: authors.
The appearance of ChatGPT and its fast spread at first resulted in a genuine fear related to the teachers' role, namely, if ChatGPT "knows the answer for everything," then there is no need for teachers anymore. Furthermore, based on the experiences related to ChatGPT, it seems that it could be an appropriate tool for personalized learning because it could help students progress at their own pace, using this tool in stages and at problems of their learning where some difficulty arises (for example, a missing theoretical issue or definition, or a missing skill to describe data scientifically). Additionally, its online feature always makes it available in any space if somebody has the necessary mobile device. This tool allows easy and convenient access, and these are the main characteristics of flexible learning, which is a desirable direction in higher education for students and the labor market (see Géring – Király 2020). ChatGPT can also be used for various tasks that require different tools. For example, it can summarize texts, translate dialogues, collect ideas or arguments, analyze data, etc. Therefore, it seems to be an “all-in-one” tool instead of simultaneously opening numerous applications and tabs (see Kohnke et al. 2023).
However, we should be aware that ChatGPT is not a complex digital teaching and learning device. Therefore, it is unsuitable to evaluate students' contribution or to help them develop further in line with their level of understanding. So, we should remember that ChatGPT-type digital LLMs are not online learning tools; they are not Intelligent Tutoring Systems (ITS) or Dialog-based Tutoring (DBT) applications that use cooperative dialogue meant to stimulate critical thinking and deeper comprehension (Afzal et al. 2019). They are “only” content-providing tools for information collection, and specific instructions are necessary. In that sense, even if they could compete with HEIs in providing content (which is not the case at this moment in the early 2020s), they do not cover the whole teaching and learning process, including skill development, critical thinking, system thinking, and future orientation (Selwyn et al. 2020).
As emphasized in Table 1. both on the students' and teachers' sides, additional skills and competencies are necessary to implement and use ChatGPT successfully. As Benmamoun (2023) calls it, “Knowledge Readiness” reflects on the problem of whether students have the necessary skills and knowledge to fully understand and critically assess what ChatGPT produces, while “Course Readiness” draws attention to the role of the teachers in evaluating whether ChatGPT is an appropriate tool for given courses or not (Benmamoun 2023). Furthermore, Kohnke and colleagues (2023: 546) collected ten digital competencies necessary to properly use ChatGPT, including technological proficiency, pedagogical compatibility, and social awareness.
Furthermore, we should not forget that higher education is not only about providing knowledge and skills for the students but also introducing them to the ethical framework of the given profession and social norms, especially a place to socialize and build relationships (Fűzi et al. 2022).
Another aspect of teaching and learning is the assessment question, which also seems threatened by ChatGPT and similar LLMs. Using ChatGPT, students may outsource their work (e.g. Mhlanga 2023; Sallam 2023) and will produce no genuine knowledge (Nye 2023). For example, ChatGPT could pass four separate examinations at the University of Minnesota Law School without human assistance. It underperformed the human students, but despite its struggles with writing essays, its performance was sufficient to earn a degree from a highly selective law school (Choi et al. 2023). In an era where remote examinations are available, ChatGPT also raises questions about teachers' routines of testing students' factual knowledge that reflects their abilities or readiness to practice the given profession. Again, this requires reconsidering the ways and content of assessment, moving toward new ways of teaching and learning focused on skill development, such as problem-solving in project- and group-work settings, where the formal assessment techniques can be replaced (Király – Géring 2020).
Concerning this changing “nature” of teachers' role in higher education, the focus is rapidly moving from knowledge transfer to skill development and collaborative knowledge- and content production. However, this change requires a rather facilitative mindset, where the lecturer and the students are partners and work together. Educators play an essential role not only in ensuring high-quality education but also in fostering positive learning experiences. In these environments, the teachers are mentors and coaches for the students, helping their intellectual development with teaching methods like problem-based learning, flipped classrooms, action learning, etc. They rely heavily on digital forms and solutions for essential knowledge transfer and information-collection tasks.
These tools and solutions are essential if we consider the human aspect of teaching and learning (Géring – Király 2020). Several sources (e.g. Jeon – Lee 2023; Karakose et al. 2023; Loos et al. 2023; Pardos-Bhandari 2023) emphasize that humans will always be essential in learning and knowledge creation, since the most important, long-lasting, and formative educational experiences differ from simple content memorization. The main focus should be on how to get students involved in the content intellectually. However, intellectual engagement requires critical reading, reflection on one's understanding, and engagement in academic discussions about the content. These activities presuppose close connections not only among the students but also between the students and the skilled tutor, i.e., the teacher.
There are two additional important aspects we should consider related to digitalization in the higher education context, namely, academic integrity (e.g. Bin-Nashwan et al. 2023; Chen et al. 2020; Eke, 2023) and the inequalities connected to the digital environment (e.g. Khan – Paliwal 2023; Santiago – Ruiz, 2023). These are essential aspects. As for academic integrity, the question of authorship and authenticity and the problems with detecting plagiarism are the main challenges in the academic field (Lee 2024). Regarding the inequalities related to the digital environment, the difficulties are twofold. On the one hand, based on access, there could be differences between students and/or between universities, too. That is, the necessary device and appropriate internet connection require a sound financial background, which is not equally available to all students (Géring – Király 2020). Just as at the organizational level, not all HEIs have a similar level of digital infrastructure. On the other hand, the devices and the level of digital literacy necessary to effectively use them could be a source of inequalities, both at the individual level (students, teachers) and the organizational level (Benmamoun 2023). The different access to free and paid services could enhance these possible fields of inequalities.
Nonetheless, these opportunities and challenges clearly show that ChatGPT and the general phenomenon of generative AI are essential issues that need to be addressed in higher education. To do so, this pilot study focuses on the students and their first experiences with ChatGPT in the context of business higher education.
3 Research process and methodology
To capture the students' first impressions of ChatGPT, we conducted pilot research between March and June in the spring semester of 2023 at the largest business university in Hungary, which involved BA-level students. The public release of ChatGPT was on 30 November 2022. Therefore, this tool was so new that the pilot research focused on the students' first application and experiences with it in different learning processes. To get this information, students of German Business classes (N = 41) had to solve exercises that mirrored the language exam structure and the typical tasks of a digital working environment. Slightly more than half of our students were female (53.7%), and two-thirds (68.7%) had never used ChatGPT before the pilot project.
In the first exercise, the students needed to briefly explain a grammar topic and provide short practices. In the second exercise, they had to answer a straightforward question related to their compulsory language colloquium topics. In the third exercise, they had to analyze a chart. As for their fourth exercise, they had to write a short business letter, and in the last one, they had to prepare a business dialogue on a common topic in business communication, like ordering or complaining about a product or service. They had to perform three rounds of these five exercises during the spring semester, so every student had fifteen overall. The students received regular feedback from the lecturer for their exercises, just like in any standard classes, and regardless of the use of ChatGPT, there were no differences in grading between the control and experimental groups. This pilot study focused only on students' views, so there was no interview with the lecturers.
Before starting work, we randomly put the students into research and control groups. The students in the research groups had received a message from their teacher that they should use ChatGPT to solve the five different types of exercises. In contrast, the students in the control group did not receive any message or instruction. We also asked the students in the research group to keep their ability to use ChatGPT a secret, and not mention it to their peers. Nevertheless, the responses suggest that some control group members also used the tool, as 70% of the students in the questionnaire said they had used ChatGPT in some form for the tasks. Therefore, in the following chapters, we will use the answers of the active users of ChatGPT as the experimental group (29 students) and the non-users (12 students) as the control group.
Accordingly, the research project was exploratory. However, it utilized experimental research elements, dividing the students into control and experimental groups to capture their first impressions and experiences with ChatGPT.1
At the end of the semester, every student filled out a questionnaire related to their experience, views, and opinions about the exercises and ChatGPT. The results of this questionnaire are displayed in the next section.
4 Results
Regarding the results of this pilot study, we focus on three main topics: (1) the students' opinion about ChatGPT related to different types of tasks and exercises connected to future skills; (2) based on their own experiences, the students' opinion about possibilities of utilizing for ChatGPT in teaching and learning, and (3) the challenges they envision. However, before we delve into these details, it is essential to note that 90% of the students are against banning ChatGPT, and half are against any regulation. Results of the comparison between the experimental group who used ChatGPT (n = 29, average: 1.42) and the control group (n = 12, average: 1.5) show that the experimental group is more against the ban of ChatGPT than the control group; however, the difference is not significant (P = 0.636), so the use of tool did not influence students' opinions significantly (Fig. 1).
Opinions about the statement that universities should ban ChatGPT (number of students, N = 41)
Source: authors.
Citation: Society and Economy 2025; 10.1556/204.2024.00007
However, one-third of the students think universities should regulate ChatGPT. According to the averages, the control group (average = 2.75) favors regulation slightly more than the experimental group (average: 2.59), but again, the difference is not significant (P = 0.179). Interestingly, the experimental group is more divided than the control group; they have more polarized views on regulating ChatGPT (Fig. 2).
Opinions about the statement that universities should regulate ChatGPT (number of students, N = 41)
Source: authors.
Citation: Society and Economy 2025; 10.1556/204.2024.00007
According to these results, it is recommended that universities and their boards consider embracing modern technology, such as ChatGPT, and explore suitable areas where it can be utilized effectively.
4.1 Future skills, digital literacy
As Fig. 3 illustrates, the students used ChatGPT most frequently to answer straightforward questions related to professional topics of their compulsory language colloquium, while ChatGPT was hardly used to analyze charts. As one student responded, it is not possible to upload a specific diagram and do the analysis based on a picture. It clearly shows the limitations and text-driven focus of ChatGPT (at least then). We can find much better and more sophisticated AI tools that deal effectively with a graph.
Tasks solved by Chat GPT (number of students, N = 41)
Source: authors.
Citation: Society and Economy 2025; 10.1556/204.2024.00007
The students realized and mentioned the benefits of ChatGPT and used it actively for tasks they found helpful (Fig. 4).
Usefulness of Chat GPT for different tasks (number of students, N = 41)
Source: authors.
Citation: Society and Economy 2025; 10.1556/204.2024.00007
In addition to the German tasks, students also used ChatGPT for other purposes (Fig. 5); the most popular was collecting information and asking questions. However, they also used it for different subjects, like English essays and preparing for various exams. Some liked the quick summarizing and the quick and more practical search function compared to other search engines like Google.
Other tasks or purposes supported by ChatGPT (number of students, N = 41)
Source: authors.
Citation: Society and Economy 2025; 10.1556/204.2024.00007
After utilizing ChatGPT for the different exercises, which mirror typical tasks in a digital working environment, we focus on the more general fields students mentioned as possible future directions in this area.
4.2 Changes in the learning-teaching process
Students see tremendous potential in implementing ChatGPT at university. However, the focus should be on ensuring that the views and interests of all stakeholders in higher education are equally considered in the implementation process. Figure 6 shows the most frequent areas and tasks mentioned by the students where ChatGPT has potential in HE. The obvious choice is to use it for essays, but some students think it would be unfair. Thus, essays or papers no longer make sense as assignments in their current form.
What could universities use ChatGPT for? (number of students, N = 41)
Source: authors.
Citation: Society and Economy 2025; 10.1556/204.2024.00007
The control group discovered more benefits for essays and data collection, but their responses were rather general. The students in the experimental group gave much more varied fields of use for ChatGPT, which is likely related to practical experience. Interestingly, only the experimental group saw the benefits of ChatGPT for learning languages.
In addition to the above, some students see ChatGPT as helpful for learning in general, motivating teachers' creativity, providing explanations, for administration, as a tool in AI classes, for text summaries, brainstorming, coding, business models, exploring teaching and learning styles, improving education, and for gamification.
One student had an interesting opinion that universities should focus more on developing emotional intelligence instead of intensively using ChatGPT.
The students think ChatGPT is a game changer for teaching and learning. However, 73% of the respondents at least partly agree that ChatGPT will affect the learning mode of the students, while only 44% expect this to be related to the lecturers' teaching styles. In other words, students think teachers will continue their usual teaching practice and need help adapting to the changes caused by ChatGPT.
If we compare the answers of the control and experimental groups, we see no significant differences (P = 0.365) in the averages (4.25 for the control and 3.93 for the experimental group). The students in the experimental group are slightly more pessimistic or, according to their experience with ChatGPT, more realistic (as seen in Fig. 7). However, everyone rules out the possibility that ChatGPT does not affect learning.
Opinions about the learning mode of the students (number of students, N = 41)
Source: authors.
Citation: Society and Economy 2025; 10.1556/204.2024.00007
Let us compare the answers of the control and experimental groups relating to the teachers' teaching style. Again, we see no significant differences (P = 0.596) in the averages (3.25 for the control and 3.31 for the experimental group). Still, we see a different pattern. The more intensive the use of ChatGPT, the more polarized students' opinions become about teachers' teaching styles (as seen in Fig. 8).
Opinions about the teachers' teaching style (number of students, N = 41)
Source: authors.
Citation: Society and Economy 2025; 10.1556/204.2024.00007
The more students use ChatGPT, the less romantic and monotonous their image of it becomes. The more intensively they use it, the more they learn about its advantages and disadvantages, and the more polarized their opinions become.
5 Discussion
As discussed earlier, ChatGPT as a learning and teaching tool has potential for HEIs, but disadvantages and limitations co-exist. In this research, our focus was on the micro-level, especially on students' perspectives and their first experiences and opinions. According to the students, the results show that it is easy to integrate ChatGPT into the learning process when the learners write texts of different lengths and purposes. It could not only help the learner to collect information, but it can even carry out the task of drafting entirely. Despite this easy integration, ChatGPT is not the ultimate learning tool that increases students' knowledge faster, or that can replace textbooks or teachers. Solving text-related exercises quicker or easier using ChatGPT does not necessarily mean more efficient learning. As the results indicated, ChatGPT is only fit for some tasks; therefore, its usability has limitations.
Nonetheless, the students mainly see ChatGPT as a help or aid, so it could be reasonable to implement it as a digital assistant in the classes and define and describe the concrete amount of its participation and work. This opinion is in line with the first experiences and suggestions in the academic discourse, where authors suggest incorporating ChatGPT into higher education (see, for example, Pang 2024; Benmamoun 2023; Sinatra – Wang 2023). However, it is essential to provide additional skills to the students and the teachers (Benmamoun 2023; Kohnke et al. 2023).
However, in our research project, the ChatGPT-user experimental group was similarly, or even slightly less convinced that the learning mode will change in the future due to the application of ChatGPT compared to the non-user control group. Thus, experiencing the benefits of ChatGPT did not lead immediately to a more optimistic view in this regard. This can indicate that although there are several evident application areas of ChatGPT for students during their studies, its role in the learning process is not apparent. That is why finding the correct place to use LLMs in higher education is so challenging, and this is what raises so many questions about its regulation (Lee 2024; Nye 2023).
Another important issue at the meso-level was the topic of academic integrity and the ethical considerations related to ChatGPT (see above). Our respondents disagreed on the need for regulation, and the ChatGPT-user experimental group was particularly polarized on this issue. We argue that the issue is not even limited to regulation. To find reasonable areas to use ChatGPT (with or without rules), HEIs need to be conscious and proactive and apply a pedagogical methodological perspective. This way, ChatGPT will not mean a shortcut to a master's degree but a way to reach a higher level of knowledge during the given training program. This might lead to rethinking educational practices in general and the roles of teachers and students in particular (Liu 2024).
Although our research focuses specifically on the students' perspectives, we can phrase some implications related to the perceived changing role of the university teachers in a setup where ChatGPT is also involved in the learning process. Our respondents were quite polarized in their expectations related to the changes in teaching styles, and the experience in using ChatGPT stayed the same and changed into an even more polarized direction. In other words, it is not clear to students that ChatGPT might change teachers' role and that this might imply a change in their teaching methods and attitudes (Géring – Király 2020). Again, this can be connected with the fact that ChatGPT's position in the learning and teaching process is not clear to the students, and it has an unclear role within HE in general. Finding ChatGPT's correct and universally accepted place within HE and defining its working frameworks are essential in turning such software applications from threats to opportunities, managing risks, and maximizing their potential systematically (Kostka – Toncelli 2023). One of the main risks, for example, is related to the difficulties in the assessment tasks of HE. This will require innovation in teaching methods and formative and summative assessment methods and may lead to the proliferation of previously unknown solutions.
For example, universities should revise the thesis as a primary assignment form and final output because LLMs make it a frivolous task. From now on, every student can effortlessly produce “convincing” written products. In our research, even some students reacted that using these tools was cheating and did not reflect fundamental knowledge.
The first option would be to replace written tasks with more in-class or oral exams, quick questions, and challenges. The second option would be to demand unconventional, more creative, and experiential genres, which need more specific details and which LLMs cannot currently provide. The third option would require more outputs as a final assessment, like in portfolios. With this diversification, we could decrease the temptation to use LLMs. If we keep the thesis, an exciting option would be demanding a second paper in which the students narrate the whole writing process and how they got to their results. This second narrative paper would be an essential part of the final evaluation. An interesting idea would be to change the evaluation process, and instead of evaluating the final product, we should value the many steps and phases along the way.
The whole issue can increase the quality of education and teaching practice because universities and teachers must ask why they are doing and demanding what they are doing and demanding. They must also address the pedagogical purpose of educational activities, processes, and outputs. Technological tools for innovative competency assessment in education have received little scholarly attention. Therefore, future research should explore the potential of such tools to advance innovation in evaluation and promote new educational methods.
However, these challenges should be addressed at the meso-, that is, at the university level, not only at the teacher- or class-level. Higher education institutions should address these issues by focusing on ChatGPT, the whole field of LLMs, and related generative AI applications. These new technologies and applications will be part of our life in the future; therefore, it is essential to include them in the general frameworks of education. Furthermore, it is also a important to teach students how to use these tools responsibly, which will be paramount in the working environment they are prepared at the HEIs. Some consider prompting an essential future skill (e.g. Clavié et al. 2023; Walter 2024).
Nonetheless, we should be aware that there are other aspects of these new digital changes and challenges besides the changing teaching and learning modes. Namely, we should also consider the macro-level consequences, such as the socialization role and responsibility of HEIs. Previous research shows that personal contact and socialization are still essential during learning (Füzi et al. 2022). Additionally, the social and professional contacts and networks collected during university could be crucial later in the labor market. Regarding future skills, the importance of social skills and emotional intelligence seem to increase because they will/can be an added value to this advanced technology (e.g. Bakhshi et al. 2017; Ehlers – Kellermann 2019). Therefore, the responsibility of HEIs is twofold: they should not only provide eligibility to these new technologies for everybody equally (which is not necessarily the case if we think about the economic aspects of digital technology), but at the same time, HEIs should strengthen the social and emotional intelligence and skills of their students to help their future life (Kohnke et al. 2023).
Finally, we should address the data concerns as well. This is not connected to ChatGPT only because AI models will change, some will disappear, and new ones will emerge. It is already happening. Besides ChatGPT, all big tech companies have already released their systems, which will become increasingly complex and sophisticated. Therefore, future models and additions will ensure personal data privacy and allow for the building of one's prompt memory. These more advanced models would be a massive benefit because the AI could be a fully personalized and safe teaching assistant tool. Universities should prefer to provide AI models fitting these categories. However, data protection is a huge task and responsibility during this endeavor, which means immediate challenges for HEIs.
Overall, there are promising opportunities in the appearance and use of LLM-type tools and generative AI in higher education. However, HEIs are facing severe challenges, from the problem of fair and equal availability through ethical concerns related to genuineness to the changing mode of teaching and learning.
6 Conclusion
In November 2022, the appearance of ChatGPT provided a rather unexpected shock to higher education. Alternatively, as New York University Abu Dhabi vice-chancellor Mariët Westermann said: “It has jolted universities even more profoundly than Covid-19 did when it forced teaching online.” (Westermann 2023). The main concerns were related to the originality of students' work and the “outsourced” learning process when it seemed that ChatGPT could answer many questions correctly and could even provide short essays and arguments.
Reflecting on the challenges and trying to understand the students' opinions as well, right in the spring semester of 2023, one of the authors conducted a pilot project during a German business language course, dividing the students into an experiment group (which was encouraged to use ChatGPT at solving the different tasks of the course) and a control group (which was asked not to use it). We presented the results of these first experiences in this paper, and they focus our attention on the complexity of the situation and the opportunities and challenges we face in higher education due to these new generative AI technologies.
However, there are some limitations to our research project because it was a pilot study; therefore, it was a very fast first attempt to capture the students' first experiences. For example, the experimental design was not strictly applied; thus, the initial 50%–50% ratio of the control and experimental groups became a 30%–70% division of the students. Furthermore, the results are based on one class's experiences at a business language course. Therefore, they should not be generalized automatically to other groups and/or other topics. Nonetheless, this pilot study provided exciting insights into the thinking and opinions of higher education students at their first endeavor with ChatGPT.
The results indicated that it is crucial for higher education institutions to carefully consider the impact of regulations on the learning and assessment processes. This may also require some adjustments from the teachers' side to ensure that students receive the best possible education. However, they do not necessarily replace the teachers. Furthermore, we should remember that this issue is related to more general education questions, like equality, fairness, and personal data protection.
Therefore, further research should aim at more detailed and extended analysis among students and embedding teachers' opinions. Just as with the appearance of new LLMs, they can be compared to the future experiences with ChatGPT. Additionally, further research projects could move from the micro-level experiences to the sector- or even social level, analyzing the effect of these new LLMs beyond the walls (or digital barriers) of universities. The rapid spread of generative AI programs in every area of our lives (from transportation to customer relations) requires national or even international attention, as can be seen, for example, in the European Union's new AI Act.2
Bibliography
Afzal, S. – Dhamecha, T. – Mukhi, N. – Sindhgatta, R. – Marvaniya, S. – Ventura, M. – Yarbro, J. (2019): Development and Deployment of a Large-Scale Dialog-Based Intelligent Tutoring System. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Volume 2 (Industry Papers), pp. 114–121.
Bakhshi, H. – Downing, J. – Osborne, M. – Schneider, P. (2017): The Future of Skills: Employment in 2030. London: Pearson.
Becker, S. A. – Cummins, M. – Davis, A. – Freeman, A. – Hall, C. G. – Ananthanarayanan, V. (2017): NMC Horizon Report: 2017 Higher Education Edition. Austin: The New Media Consortium.
Benmamoun, M. (2023): Reinventing International Business Education: Integrating the Power of Generative AI. AIB Insights 24(1). https://doi.org/10.46697/001c.90397.
Bin-Nashwan, S. A. – Sadallah, M. – Bouteraa, M. (2023): Use of ChatGPT in Academia: Academic Integrity Hangs in the Balance. Technology in Society 75: 102370. https://doi.org/10.1016/j.techsoc.2023.102370.
CEDEFOP (2018): Skills Forecast: Trends and Challenges to 2030 .Luxembourg: Publications Office.
Chandler, N. – Heidrich, B. – Szászvári, K., – Kása, R. (2021): Reframing Market-Orientation: A Comparative Study of the Market Orientation Concept in the Subcultures of University Employees. Society and Economy 43(3): 270–288. https://doi.org/10.1556/204.2021.00011.
Chen, L. – Chen, P. – Lin, Z. (2020): Artificial Intelligence in Education: A Review. Ieee Access 8: 75264–75278.
Choi, J. H. – Hickman, K. E. – Monahan, A. – Schwarcz, D. (2023): ChatGPT Goes to Law School. Journal of Legal Education 71(3): 387–400.
Ciampa, K. – Wolfe, Z. M., – Bronstein, B. (2023): ChatGPT in Education: Transforming Digital Literacy Practices. Journal of Adolescent & Adult Literacy 67(3): 186–195. https://doi.org/10.1002/jaal.1310.
Clavié, B. – Ciceu, A. – Naylor, F. – Soulié, G. – Brightwell, T. (2023). Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification. International Conference on Applications of Natural Language to Information Systems, pp. 3–17.
Coppin, B. (2004): Artificial Intelligence Illuminated. Jones & Bartlett Learning.
Davies, R. H. – van Seventer, D. (2020): Labour Market Polarization in South Africa: A Decomposition Analysis. WIDER Working Paper 2020/17.
EC Eurobarometer (2014): Public Opinion in the European Union .Brussels: European Commission.
Ehlers, U. D. – Kellermann, S. A. (2019): Future Skills: The Future of Learning and Higher Education. Karlsruhe.
Eke, D. O. (2023): ChatGPT and the Rise of Generative AI: Threat to Academic Integrity? Journal of Responsible Technology 13: 100060. https://doi.org/10.1016/j.jrt.2023.100060.
Fűzi, B. – Géring, Z. – Szendrei-Pál, E. (2022): Changing Expectations Related to Digitalisation and Socialisation in Higher Education. Horizon Scanning of Pre- and Post-COVID-19 Discourses. Educational Review 74(3): 484–516. https://doi.org/10.1080/00131911.2021.2023101.
Garcia-Esteban, S. – Jahnke, S. (2020): Skills in European Higher Education Mobility Programmes: Outlining a Conceptual Framework. Higher Education, Skills and Work-Based Learning 10(3): 519–539. https://doi.org/10.1108/HESWBL-09-2019-0111.
Géring, Z. – Király, G. (2020): Changes in Teaching and Learning – the Challenges of Flexible Learning. Horizon Scanning Report Series, Volume II. Budapest: Future of Higher Education Research Centre, Budapest Business School.
Gibson, D. – Kovanovic, V. – Ifenthaler, D. – Dexter, S. – Feng, S. (2023): Learning Theories for Artificial Intelligence Promoting Learning Processes. British Journal of Educational Technology 54: 1125–1146. https://doi.org/10.1111/bjet.13341.
González, J. – Wagenaar, R. (2003): Quality and European Programme Design in Higher Education. European Journal of Education 38(3): 241–251.
Hughes, J. J. (2017): What Is the Job Creation Potential of New Technologies? In: LaGrandeur, K. – Hughes, J. J. (eds): Surviving the Machine Age: Intelligent Technology and the Transformation of Human Work .London: Palgrave Macmillan.
Jeon, J. – Lee, S. (2023): Large Language Models in Education: A Focus on the Complementary Relationship Between Human Teachers and ChatGPT. Education and Information Technologies 28: 15873–15892. https://doi.org/10.1007/s10639-023-11834-1.
Karakose, T. – Demirkol, M. – Aslan, N. – Köse, H. – Yirci, R. (2023): A Conversation with ChatGPT about the Impact of the COVID-19 Pandemic on Education: Comparative Review Based on Human–AI Collaboration. Educational Process: International Journal 12(3): 7–25.
Khan, I. A. – Paliwal, N. W. (2023): ChatGPT and Digital Inequality: A Rising Concern. Scholars Journal of Applied Medical Sciences 11(9): 1646–1647. https://doi.org/10.36347/sjams.2023.v11i09.010.
Király, G. – Géring, Z. (2020): Changes in Teaching and Learning – The Transformation of Knowledge Transfer. Horizon Scanning Report Series, Volume I. Budapest: Future of Higher Education Research Centre, Budapest Business School.
Király, G. – Géring, Z. (2021): Having Nothing but Questions? The Social Discourse on Higher Education Institutions' Legitimation Crisis. Journal of Future Studies 25(4): 57–70. https://doi.org/10.6531/JFS.202106_25(4).0005.
Kohnke, L. – Moorhouse, B. L. – Zou, D. (2023): É ChatGPT for Language Teaching and Learning. RELC Journal 54(2) 537–550. https://doi.org/10.1177/00336882231162868.
Kostka, I. – Toncelli, R. (2023): Exploring Applications of ChatGPT to English Language Teaching: Opportunities, Challenges, and Recommendations. TESL-EJ 27(3). https://doi.org/10.55593/ej.27107int.
Lee, S. (2024): Utilizing a ChatGPT Workshop to Foster Ethical Awareness and Enhance L2 English Writing Revision Processes in University Academic Settings: ChatGPT Workshop for Effective and Ethical L2 English Writing. In: Pan, F. (ed.): AI in Language Teaching, Learning, and Assessment. IGI Global, pp. 269–299.
Liu, W. (2024): The Integration of ChatGPT in Language Education: Exploring Contexts, Assessing Impacts, and Unpacking Implications. In Pan, F. (ed.): AI in Language Teaching, Learning, and Assessment. IGI Global, pp. 1–21.
Loos, E. – Gröpler, J. – Goudeau, M. L. S. (2023): Using ChatGPT in Education: Human Reflection on ChatGPT's Self-Reflection. Societies 13(8): 196. https://doi.org/10.3390/soc13080196.
MacCrory, F. – Westerman, G. – Alhammadi, Y. – Brynjolfsson, E. (2014): Racing With and Against the Machine: Changes in Occupational Skill Composition in an Era of Rapid Technological Advance. In: Myers, M. D. – Straub, D. W. (eds): Proceedings of the International Conference on Information Systems - Building a Better World through Information Systems .Auckland: Association for International Systems.
Maxwell, N. – Gallagher, S. (2020): Drivers for Change in Higher Education. New Directions for Community Colleges 2020(189): 9–22. https://doi.org/10.1002/cc.20394.
Medland, E. (2016): Assessment in Higher Education: Drivers, Barriers and Directions for Change in the UK. Assessment & Evaluation in Higher Education 41(1): 81–96.
Mhlanga, D. (2023): Open AI in Education, the Responsible and Ethical Use of ChatGPT Towards Lifelong Learning. In: FinTech and Artificial Intelligence for Sustainable Development. Sustainable Development Goals Series. London: Palgrave Macmillan.
Mitrović, S. – Andreoletti, D. – Ayoub, O. (2023): ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text. ArXiv:2301.13852. https://doi.org/10.48550/arXiv.2301.13852.
Nye, B. (2023): Authoring by Editing and Revising: Considering Generative AI Tools. In: Critical Thinking and Ethics in the Age of Generative AI in Education. A Critical Look into the Future of Learning .USC Center for Generative AI and Society, pp. 34–41.
OpenAI (2023): ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt, accessed 08/03/2024.
Pan, F., ed. (2024): AI in Language Teaching, Learning, and Assessment. IGI Global.
Pangrazio, L. – Godhe, A. L. – Ledesma, A. G. L. (2020): What is Digital Literacy? A Comparative Review of Publications Across Three Language Contexts. E-learning and Digital Media 17(6): 442–459. https://doi.org/10.1177/2042753020946291.
Pardos, Z. A. – Bhandari, S. (2023): Learning Gain Differences Between ChatGPT and Human Tutor Generated Algebra Hints. arXiv:2302.06871 .https://doi.org/10.48550/arXiv.2302.06871.
Sallam, M. (2023): The Utility of ChatGPT as an Example of Large Language Models in Healthcare Education, Research and Practice: Systematic Review on the Future Perspectives and Potential Limitations. Healthcare 11(6): 887.
Santiago-Ruiz, E. (2023): Writing with ChatGPT in a Context of Educational Inequality and Digital Divide. International Journal of Education & Development using Information & Communication Technology 19(3).
Selwyn, N. – Hillman, T. – Eynon, R. – Ferreira, G. – Knox, J. – Macgilchrist, F. – Sancho-Gil, J. M. (2020): What's Next for Ed-Tech? Critical Hopes and Concerns for the 2020s. Learning, Media and Technology 45(1): 1–6. https://doi.org/10.1080/17439884.2020.1694945.
Sinatra, G. – Wang, C. (2023): The Promise and Peril of Using AI in College Classrooms. In: Critical Thinking and Ethics in the Age of Generative AI in Education. A Critical Look into the Future of Learning. USC Center for Generative AI and Society, pp. 10–13.
Singh, A. (2021): Quasi Experimental Design in Scientific Psychology. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3793568.
Strzelecki, A. (2023): Students' Acceptance of ChatGPT in Higher Education: An Extended Unified Theory of Acceptance and Use of Technology. Innovative Higher Education, 1–23. https://doi.org/10.1007/s10755-023-09686-1.
Wang, L. – Chen, X. – Wang, C. – Xu, L. – Shadiev, R. – Li, Y. (2024): ChatGPT's Capabilities in Providing Feedback on Undergraduate Students' Argumentation: A Case Study. Thinking Skills and Creativity 51: 101440. https://doi.org/10.1016/j.tsc.2023.101440.
Walter, Y. (2024): Embracing the Future of Artificial Intelligence in the Classroom: The Relevance of AI Literacy, Prompt Engineering, and Critical Thinking in Modern Education. International Journal of Educational Technology in Higher Education 21(1): 15. https://doi.org/10.1186/s41239-024-00448-3.
Weng, W. (2015): Eight Skills in Future Work. Education 135(4): 419–422.
Westermann, M. (2023): After a Year of ChatGPT, is Academia Getting to Grips With Generative AI? Times Higher Education 23 November.
Appendix
Questionnaire
Dear Students!
The following questionnaire will take about 15 min to complete.
Please read the statements carefully and then answer the questions. Thank you in advance for your answers!
Name:
Age:
Gender:
Your course:
Did you use ChatGPT before the German course? (Yes/No)
If you used it, what did you use it for?
Did you use ChatGPT for the German exercises? (Yes/No)
Tick all the tasks for which you used ChatGPT: Select all valid ones.
Letter (letter writing)
Diagram (chart analysis)
Question (answering a question)
Dialogue (writing a dialogue)
Grammar (describing a grammatical topic)
Other:
If there was a task for which you did not use it, explain why you did not use it.
What else did you use ChatGPT for besides the German tasks?
Did you talk to your classmates about your experience with ChatGPT during the semester? (Yes/No)
If you have talked about it, please recall what you said and what you talked about.
Please indicate in the table below how useful you found ChatGPT for the following tasks Select all that apply. (1 (not at all useful) - 5 (very useful)/not used)
Letter (letter writing)
Diagram (chart analysis)
Question (answering a question)
Dialogue (writing a dialogue)
Grammar (describing a grammatical topic)
If you have any comments or additions to the above evaluation or the usefulness of ChatGPT, please comment here.
How much do you agree with the following statements? Please tick only one answer per line. (1 (strongly disagree) - 5 (strongly agree)
ChatGPT is very useful for language learning.
ChatGPT has helped me improve my language skills this semester.
ChatGPT, in its current form, is not suitable for language learning.
The use of ChatGPT should be banned on campus.
The use of ChatGPT should be regulated at the university.
Thanks to ChatGPT, teachers will teach differently.
Thanks to ChatGPT, students will learn differently.
Have you noticed any changes while using ChatGPT during the semester?
Evaluate the following statements. Select all that apply. (1 (strongly disagree) - 5 (strongly agree)
ChatGPT has improved during the semester.
ChatGPT has become more reliable over the semester.
The quality of ChatGPT's responses to questions and instructions in different languages varied.
What do you think ChatGPT could be used for on campus?
What do you think ChatGPT should be used for at university?
Do you have any other comments about ChatGPT and its use?
Instructions regarding the tasks for the students
Grammar topic
Work out a grammatical topic from nine topics (see below) in a Word file. The structure of the file should be as follows:
- •The rules
- •The use
- •Exercises (at least 20 sentences/task) with solutions
- •Your extra on the topic
The topics:
Adjective declination
Tenses of verbs
Participles
Passive voice
Rection of the verbs
Conjunctions
Infinitive clauses
Relative clause
Conditional
Exam questions
Answer one of the following official exam questions in writing, then upload it in a Word file.
Chart analysis
Analyze a current German-language diagram and upload the diagram and the analysis in a Word file.
Business letter
Write a short business letter (120–140 words) and upload it in a Word file. The information and details for the letter are below.
Business Dialogue
Prepare a dialogue in writing (Word file) and upload it.
The situations can be found on the Moodle LMS in the section “Practical.”
This exploratory study has followed a quasi-experimental design, more precisely, a non-equivalent group, posttest-only design, widely used in natural settings where randomization cannot be conducted, e.g., for practical reasons. A significant limitation of this design is that the experimental and control groups may not be the same before and after the intervention and may differ in ways that influence the outcome (Singh 2021).