The central element of any research is the selection of strategies, methods, or techniques that are applied to conduct the study (van Thiel, 2014). As part of our research, we decided to conduct an experiment in which we analyse students’ satisfaction with the feedback received from both the teacher and ChatGPT 3.5 within the context of the learning course.

The experiment involves 36 students from Taltech Virumaa College, with an average age of 30 years. The syllabus for the RAM0800 Digital Logic and Digital Systems study course (6 credits) in the Telematics bachelor’s degree program was used for the experiment. The theoretical and practical materials of the course are studied during weeks 1 and 14 of the 17-week course. The experiment was conducted during the last 4 weeks of the course (weeks 14–17).

Students’ tasks consist of four sets of assessment questions. The revised Bloom’s taxonomy by Anderson et al. (2001) was used for the formulation and design of sets of assessment questions.

Each set of questions for each learning outcome is designed to include four questions: an A, a B, a C, and a D question. The A question covers the “Remember” level of Bloom’s Taxonomy, the B question the “Understand” level, and the C question the “Apply” level. The D question covers the three higher levels (analysis, synthesis, and evaluation) of Bloom’s taxonomy. The A questions are threshold questions necessary for passing the study course. Students achieving correct answers to questions A, B, and C receive an excellent grade. D questions are designed to indicate to students that there is still much to be learned. The number of attempts for solving each question is unlimited. The time for each attempt is limited depending on the complexity of the question.

As a mechanism for automated personalized feedback on these questions, we utilize the capabilities of ChatGPT 3.5. This innovative AI platform offers significant opportunities for enhancing personalized learning experiences and shaping the future of education (Albdrani and Al-Shargabi, 2023). ChatGPT 3.5 analyses students’ responses and determines whether the correct answer has been provided, assigns grades, and offers feedback recommendations to the student.

While this feedback will be automated in the domain model we are developing, in the experiment, it was provided by the teacher, simulating the future automated feedback algorithm. To maintain a manageable workload for the teacher and simultaneously create high-quality research design, feedback using ChatGPT 3.5 was provided to only half of the students, while the other half received feedback in the usual manner. These groups alternated weekly to ensure fairness. Overall, the experiment consisted of four consecutive stages.

Stage 1: The students were divided into two groups (Group A and Group B, with 18 students each) using simple random sampling facilitated by a random number generator in Microsoft Excel. Due to the small sample size, stratified randomization was not employed. To ensure the equivalence of the groups at baseline, we conducted a comparative analysis of participants’ key characteristics (age, knowledge level, academic performance) and confirmed that no statistically significant differences were present. This allowed us to consider the groups comparable prior to the start of the experiment.

Stage 2: All students were informed in advance that they would be participating in this experiment. Each student had the option to decline participation, in which case they could complete the course in the usual manner.

During the 14th and 15th weeks of the semester, the teacher provided the first and second sets of control questions (four questions in each set). The task for all students was to complete these control assignments. Upon completion, the task had to be uploaded as a text file to the Moodle platform. The response could be in the form of a text-based essay or specific program code in the Falstad simulator.¹ Once a student uploaded their work to Moodle, the teacher received a notification.

Group A served as the focus group, receiving additional feedback. Additional feedback meant that students received a response from the teacher immediately after submitting their assignment. To generate this feedback, the teacher used ChatGPT 3.5. Group B acted as the control group and received feedback directly from the teacher within the standard timeframe (no later than 48 h after submission) during the same period.

The final deadline for completing the assignments was set at the end of the 15th week.

Stage 3: During the 16th and 17th weeks of the semester, the teacher provided the third and fourth sets of control questions (four questions in each set). At the beginning of week 16, Group B became the focus group, and Group A became the control group. The strict deadline for completing the tasks was set for the end of week 17.

Stage 4: At the end of the 15th and 17th weeks, a survey was conducted among the students. This survey gathered the students’ opinions on the automated personalized feedback provided during the experiment.

After the qualitative data analysis, we intended to assess the quality of feedback. The analysis of students’ responses to the questions evaluates the clarity, completeness, comprehensibility, and relevance of the feedback provided by the supervisor and ChatGPT 3.5 in the student survey conducted at the end of the experiment. First, we assessed the percentage of respondents for each question in the proposed survey. Second, we evaluated the average weight values of each question.

As a quantitative analysis, we conducted a comparison of the mean scores assigned by the supervisor and ChatGPT 3.5 for each assignment. We also compared the number of attempts made by students when feedback was provided by the supervisor and ChatGPT 3.5 for each assignment.

Students were asked to evaluate the feedback by indicating their level of agreement with six statements. Each response in the conducted survey has its own weight: 1—strongly disagree, 2—disagree, 3—neither disagree nor agree, 4—agree, 5—strongly agree, 6—not able to judge. In the first survey (after the 15th week), 56% of the total number of students participated. In the second survey (after the 17th week), 64% of the students who took part in the experiment participated.

The results obtained after surveying Group A are presented in Figures 1, 2. Figures 3, 4 illustrate how students assessed the feedback in Group B after weeks 15 and 17, respectively. Group B was more satisfied on all aspects than Group A. This was the case regardless of whether they were rating the feedback received from the supervisor or from ChatGPT 3.5.

We compared the average satisfaction with feedback received from the supervisor and from ChatGPT 3.5. The result of this comparison is presented in Figure 5. Again, the differences between Group A and Group B are striking, with Group A (receiving ChatGPT 3.5 feedback in week 15 and supervisor feedback in week 17) being less satisfied than Group B. This was confirmed by a linear mixed models analysis with condition, week and group as fixed effects and student ID as random effect. While there was a trend toward a difference between groups A and B, F(1, 24.19) = 2.95, p = 0.095, no effect of week, F(1, 20.87) = 0.086, p = 0.773 or condition, F(1, 20.87) = 0.42, p = 0.523, could be discerned. Entering group as random factor instead of as fixed effect did not change the results for either week or condition. In other words, students seemed equally satisfied with both types of feedback.

We conducted a comparative analysis of the average scores obtained by students for the completion of control assignments as part of the experiment. Initially, all assignments were graded using a 100-point scale, which allowed for more precise differentiation of student performance. Subsequently, to ensure consistency and facilitate interpretation of the results, the scores were converted into the traditional five-point grading scale commonly used in educational practice.

We compared the average marks awarded to students depending on whether the assessment and feedback were provided by the human supervisor or ChatGPT 3.5. This approach enabled us to analyse the impact of different types of feedback on students’ academic outcomes.

The comparison of average scores given by the supervisor and ChatGPT 3.5 is presented in Figure 6. Again, linear mixed-model analyses of differences between grades found a trend for group, F(1, 28.20) = 3.02; p = 0.093, this time with an advantage for group A. However, no effect was found for either week, F(1, 27.63) = 0.07, p = 0.79, or for condition, F(1, 27.63) = 0.02, p = 0.83. Again, the same results for week and condition held when group was entered as a random instead of fixed effect. There was thus no evidence of any change in grades as a function of the kind of feedback.

We compared the number of student response attempts made to achieve the highest result. Given that each attempt rejected by the teacher was necessarily accompanied by a comment with personalized feedback for the student, this means that the number of attempts is directly related to the number of feedback comments.

The comparison of the average number of attempts made by students to complete tasks, when working with a supervisor and with ChatGPT 3.5, is presented in Figure 7. No difference between the two conditions was found, F(1, 27.55) = 1.23, p = 0.277, nor of week, F(1, 27.55) = 1.23, p = 0.277, nor of group, F(1, 28.10) = 0.001, p = 0.97.