LLMs, such as GPT-4, are increasingly present in higher education as tools to assist with language production, research synthesis, and academic writing (Lu et al., 2024). Their growing use among university students has sparked institutional interest in understanding how these tools influence learning outcomes. However, emerging evidence suggests that the pedagogical value of LLMs depends less on their availability than on the instructional design that accompanies their use (Robleto et al., 2024).

A useful framework for analyzing the educational integration of technology is the Technological Pedagogical Content Knowledge (TPACK) model developed by (Mishra and Koehler 2006). TPACK posits that effective technology-enhanced instruction requires the intersection of three types of knowledge: disciplinary content, pedagogical strategies, and technological tools. In this model, the mere introduction of digital resources does not guarantee meaningful learning. Rather, it is the thoughtful alignment of those tools with pedagogical goals and disciplinary content that fosters deep understanding and transferable skills.

This framework is particularly relevant to the use of LLMs. In a guided implementation, students receive explicit instructions on how to use the model to support key aspects of academic writing—such as developing argument structure, paraphrasing source material, or revising according to disciplinary conventions (Yan et al., 2024). This structured use of the tool reflects the TPACK ideal: technology embedded within a coherent pedagogical plan.

By contrast, unguided use of LLMs lacks this intentional alignment. Although students may independently explore the tool's capabilities, they do so without pedagogical framing, which may result in inconsistent outcomes. Unguided users might underuse the tool, rely on it uncritically, or fail to recognize its limitations (Wang, 2022). Finally, students in the control group, with no access to LLMs, must rely entirely on their prior writing skills and internal resources. While this condition mirrors traditional academic expectations, it may pose additional cognitive and emotional challenges for students with lower confidence or weaker academic preparation (Ayeni et al., 2024). Building on this theoretical foundation, the present study investigates how different instructional approaches to LLM use—guided, unguided, or absent—affect academic writing, mental well-being, and engagement. The TPACK framework supports the hypothesis that pedagogically framed LLM use will yield superior outcomes across all domains.

In this experimental study, we report how the sample size was determined and all inclusion criteria, manipulations, and outcome measures. All anonymized data are available via the Open Science Framework (https://osf.io/htejm/?view_only=54624dbd9f11467ea26242bae037e713). The data were analyzed using SPSS, version 29. No data were collected after the data analysis began. This study was not preregistered.

An a priori power analysis was conducted using G^*Power (Faul et al., 2009) to determine the required sample size for a one-way ANCOVA with three groups and one covariate. Setting the alpha level at 0.05, power at 0.80, and anticipating a small to medium effect size (f = _0.20), the estimated minimum sample size was N = 246. The final sample consisted of two hundred and eighty eight undergraduate students enrolled in humanities and arts programs at an international university in China. Instructors from four elective courses were initially contacted via internal mailing lists distributed by the College of Humanities and Arts and were invited to authorize data collection during one of their scheduled sessions. Once instructor consent was obtained, students were approached in person during class and invited to participate. Participants were recruited using a convenience sampling strategy, and participation was strictly voluntary. Students were informed that they could decline or withdraw at any point without penalty. No academic credit, compensation, or incentive was offered. Of the approximately three hundred and twenty five students approached across the four courses, two hundred and eighty eight undergraduate (88.6%) agreed to participate and completed all study components. The final sample included one hundred and sixty eight male students (58.3%) and one hundred and twenty female students (41.7%), ranging in age from 18 to 22 years (M = 19.88, SD = 1.50). All participants completed the writing task and self-report measures under supervised classroom conditions.

This study was reviewed and approved by the Institutional Review Board (IRB) of the Xi'an International University following the Declaration of Helsinki. Ethical approval was granted on [January 15th, 2024], under the reference number [approval ID, IRB/24/072-HUMARTS]. Before participation, all students received an information sheet outlining the purpose of the study, the nature of the tasks, the voluntary nature of their participation, and their right to withdraw at any point without penalty. Written informed consent was obtained from all participants. The consent form emphasized that participation was anonymous, data would be kept confidential, and results would be used solely for academic research. Participants were also informed that using the LLM was part of an experimental educational intervention and that their course grades would not be affected by their responses or participation.

Instructors from four elective undergraduate courses in the humanities and arts were first contacted via internal mailing lists distributed by the College of Humanities and Arts. After receiving their consent to conduct the study during scheduled class time, students were invited in person to participate. The study was introduced at the beginning of the session, and all students were informed that participation was entirely voluntary and that they could withdraw at any time without consequences. Students who agreed to participate provided informed consent and completed the study during a supervised class session. Participants were randomly assigned to one of three experimental conditions: guided LLM use, unguided LLM use, or control. Random assignment was conducted at the individual level within each classroom using a pre-generated randomization list.

All participants were asked to complete the same academic writing task: a critical essay on the topic “The impact of globalization on contemporary culture”, to be written in 45 m using a computer. Participants in the two experimental conditions used OpenAI's GPT-4, accessed via a monitored institutional interface. No alternative platforms or personal devices were permitted. All students interacted with the same LLM under identical interface conditions. On average, participants in the experimental conditions spent between 25 and 35 m actively interacting with the LLM during the task. In the control condition, students received the following prompt: “Write a critical essay on the impact of globalization, using the provided readings. Structure your argument and support it with specific examples.” No access to LLMs or external writing tools was provided. In the unguided LLM use condition, students were given access to GPT-4 and instructed: “You may use the language model (LLM) in any way you find useful to complete your essay.” No additional instructions, training, or support were provided.

In the guided LLM use condition, participants received the following prompt: “Write a critical essay on the impact of globalization. Use the language model (LLM) to help you generate ideas, organize your arguments, and improve clarity. You may use it to explore different perspectives, revise paragraphs, or paraphrase content. Ensure that your essay reflects critical thinking, coherence, and academic style.”

Before beginning the writing task, this group received a brief 10-m in-class orientation delivered by the course instructor, based on a script prepared by the research team. The orientation covered three key elements: how to formulate effective prompts, how to evaluate AI-generated outputs critically, and how to use the tool ethically in academic contexts. After completing the writing task, participants responded to standardized self-report measures assessing perceived mental health, academic engagement, and a short demographic questionnaire. All responses were submitted digitally and anonymized prior to analysis. No pilot study was conducted prior to the implementation of the experiment.

All instructions, writing prompts, the manipulation check, and the self-report measures—except for one—were administered in English, in accordance with the instructional language of the international university where the study took place. The only exception was the Utrecht Work Engagement Scale – Student version (UWES-S), which was administered in its validated Chinese version due to its demonstrated psychometric reliability in Chinese undergraduate populations.

A manipulation check was administered immediately after the writing task to verify participants' adherence to their assigned intervention condition. Participants responded to two closed-ended questions: (1) “Did you use the language model (LLM) while completing the essay?” (Yes/No), and (2) “Were you instructed on how to use the LLM?” (Yes/No). These items allowed the researchers to determine whether participants in the experimental groups used the LLM as intended, and whether participants in the control group refrained from doing so.

Only participants whose responses were fully consistent with their assigned condition were retained for the main analyses. Specifically, inclusion criteria required that participants in the guided condition reported using the LLM with instructions, those in the unguided condition reported using the LLM without instructions, and those in the control condition reported not using the LLM. Participants who did not meet these criteria were excluded from the final dataset. As a result, the final sample included two hundred and forty six participants who successfully passed the manipulation check and were eligible for analysis.

The quality of participants' academic writing was assessed using the Grammarly for Education platform (Grammarly, Inc., 2024). After completing the essay, the experimenter uploaded each text under standardized conditions. Grammarly automatically generated a Performance Score, ranging from 0 to 100, which served as the primary indicator of overall text quality. This composite score reflects the extent to which the writing adheres to grammatical norms, clarity, and effective communication, and it can be improved by addressing the platform's suggested revisions. In addition to the performance score, Grammarly provides detailed linguistic metrics, including word count, average word and sentence length, readability score (based on the Flesch scale) (Flesch, 1948), and vocabulary diversity (e.g., proportion of unique and rare words). These secondary indicators were reviewed to contextualize writing complexity and stylistic variation, though only the Performance Score was used in the statistical analyses. This approach provided a replicable, standardized, and objective method for evaluating the quality of written academic work across all participants, minimizing potential biases associated with human ratings.

Perceived psychological well-being was assessed using the Ryff Psychological Well-Being Scale (PWBS) (Ryff, 1989). The scale consists of multiple subdimensions (e.g., autonomy, environmental mastery, personal growth, purpose in life), with responses given on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Higher scores indicate greater psychological well-being. The PWBS has been widely validated and used across cross-cultural educational contexts (Ryff and Keyes, 1995). Li (2014) tested a shorter version in the Chinese language, and it was used in the present study. In the current sample, internal consistency was acceptable (α = 0.78).

Academic engagement was measured using the Utrecht Work Engagement Scale—Student Version (UWES-S) (Schaufeli et al., 2002). This 17-item scale captures three core dimensions of engagement—vigor, dedication, and absorption. Participants rated each item on a 7-point scale from 0 (never) to 6 (always). Total scores were calculated by averaging across all items, with higher scores reflecting greater engagement. The UWES-S has demonstrated strong internal consistency and cross-cultural validity (Schaufeli and Bakker, 2004). The Chinese version developed by Fang et al. (2008) was used. In the present study, the scale demonstrated good internal consistency (α = 0.89).

All main analyses included participants' prior academic performance in a literature-related subject as a covariate. Academic records provided a numerical score on a 100-point scale, reflecting performance in the most recent literature course completed before the intervention. This variable was used to control for potential baseline differences in academic ability related to writing, critical reading, and content familiarity. The scores ranged from 18 to 78, with a mean of 44.37 (SD = 13.46), indicating moderate variability across the sample. Controlling for this variable allowed for a more accurate estimation of the intervention effects on the outcome measures.

Before conducting the main analyses, descriptive statistics and bivariate correlations were calculated for all continuous variables: prior academic performance, text quality, perceived mental health, and academic engagement. Table 1 presents the means and standard deviations for each variable. Pearson's correlation coefficients are presented in Table 2. All correlations were statistically significant at the p < 0.01 level. Academic performance was positively correlated with text quality (r = 0.258, p < 0.001), mental health (r = 0.329, p < 0.001), and engagement (r = 0.272, p < 0.001). Text quality also showed moderate positive correlations with mental health (r =0.406, p < 0.001) and engagement (r = 0.280, p < 0.001). The strongest association was observed between mental health and academic engagement (r = 0.568, p < 0.001), suggesting a meaningful link between students' psychological wellbeing and their engagement with academic tasks.

A series of Univariate Analyses of Covariance (ANCOVA) were conducted to examine the effects of intervention condition on three outcome variables: academic writing quality, perceived mental health, and academic engagement. The independent variable was the type of LLM integration (guided use, unguided use, and control), and prior academic performance was included as a covariate in all models.

The ANCOVA revealed a significant effect of intervention condition on writing quality, F _{(2, 253)} = 789.53, p < 0.001, with a very large effect size (R² adj = 0.863). This indicates that the intervention condition explained approximately 86% of the variance in writing performance, reflecting a strong and practically meaningful impact of structured LLM integration.

Estimated marginal means showed that students in the guided LLM use condition produced significantly higher quality texts (M = 151.35, SE = 0.775) than those in the unguided use (M = 129.61, SE = 0.770) and control groups (M = 109.61, SE = 0.700), as Table 2 shows.

Effect sizes computed with pooled standard deviations showed a very large difference between the guided LLM use and control groups (Cohen's d = 5.16), a large difference between the guided and unguided groups (d = 2.53), and a large difference between the unguided and control groups (d = 3.83). These values highlight the strong impact of guided LLM use on writing performance, and confirm that even unguided use resulted in substantially better outcomes compared to no use. All pairwise comparisons were statistically significant after Bonferroni correction (p < 0.001), as Table 3 shows.

The analysis also showed a significant effect of intervention condition on perceived mental health, F _{(2, 253)} = 5.78, p = 0.004, with a small to moderate effect size (R² adj = 0.097). This suggests that nearly 10% of the variability in self-reported mental wellbeing was attributable to the different LLM conditions, indicating a modest yet meaningful contribution of guided use to students' perceived psychological health. Students in the guided use condition reported significantly higher mental health scores (M = 4.15, SE = 0.076) than the control group (M = 3.81, SE = 0.069, p = 0.003), as Table 4 shows.

As Table 5 shows, the difference between the guided and unguided groups (M = 3.91, SE = 0.076) approached statistical significance (p = 0.070), whereas no significant difference was observed between the unguided and control conditions (p = 0.984). Effect size estimates indicated a moderate difference between the guided LLM use and control conditions (Cohen's d = 0.53), a small to moderate effect between guided and unguided use (d = 0.31), and a negligible effect between unguided use and control (d = 0.14). These findings suggest that only structured use of the LLM led to meaningful psychological benefits.

Lastly, the ANCOVA for academic engagement indicated a significant effect of intervention condition, F _{(2, 253)} = 6.70, p = 0.001, with a modest effect size (R² adj = 0.101). This means that around 10% of the variance in engagement was explained by the intervention condition, pointing to a small but practically relevant effect of structured LLM integration on students' involvement in academic activities. Students in the guided LLM use group reported the highest engagement scores (M = 14.67, SE = 0.304), significantly higher than the control group (M = 13.17, SE = 0.275, p < 0.001), as Table 6 shows.

Although the unguided use group (M = 13.74, SE = 0.302) scored higher than the control group, this difference was not statistically significant (p = 0.485), and the difference between the guided and unguided groups was marginal (p = 0.093), as Table 7 shows. For engagement, the contrast between guided use and control yielded a moderate effect size (Cohen's d = 0.55), while the effect between guided and unguided use was small to moderate (d = 0.32), and the difference between unguided use and control was small (d = 0.21). These results indicate that guided integration produced a noticeable improvement in students' involvement, whereas unguided use led to minimal gains.

These results suggest that the guided integration of LLMs can significantly enhance students' academic writing and engagement, and may also support improvements in perceived mental health, compared to both unguided use and no use of LLMs.

Hypothesis 4: Intermediate Outcomes in the Unguided LLM Use Condition

Hypothesis 4 proposed that students in the unguided LLM use condition would demonstrate intermediate levels of academic writing quality, perceived mental health, and academic engagement, higher than those in the control group but lower than those in the guided use group.

The results provided partial support for this hypothesis. In terms of academic writing quality, the unguided group (M = 129.61, SE = 0.77) scored significantly higher than the control group (M = 109.61, SE = 0.70, p < 0.001), but significantly lower than the guided group (M = 151.35, SE = 0.78, p < 0.001). These findings confirm the predicted ordinal pattern in this domain.

However, for perceived mental health, the unguided group (M = 3.91, SE = 0.076) did not differ significantly from the control group (M = 3.81, SE = 0.069, p = 0.984), although it trended lower than the guided group (M = 4.15, SE = 0.076), with this comparison approaching statistical significance (p = 0.070).

Similarly, regarding academic engagement, the unguided group (M = 13.74, SE = 0.30) did not significantly differ from the control group (M = 13.17, SE = 0.28, p = 0.485). The difference between the unguided and guided conditions (M = 14.67, SE = 0.30) was marginal (p = 0.093).

These results indicate that while the unguided LLM condition yielded intermediate outcomes for academic writing quality consistent with Hypothesis 4, the same pattern was not statistically supported in perceived mental health and academic engagement.

The results strongly support Hypothesis 1, demonstrating that students who received structured guidance using LLMs achieved significantly higher academic writing quality than those in both the unguided and control groups. The magnitude of the effect was exceptionally large, underscoring the substantial educational potential of guided LLM integration. This finding aligns with the growing body of evidence suggesting that effective integration of LLMs in academic contexts improves the quality of student output and encourages critical engagement with both the writing process and the technology itself (Cash et al., 2025). The superiority of the guided condition can be interpreted through several converging mechanisms identified in recent research. First, structured frameworks like the Writing Path, which utilize explicit outlines, have been shown to significantly improve text generation quality by aligning outputs with the user's intentions and task-specific goals (Lee et al., 2024). This alignment is particularly important in academic settings, where coherence, argument structure, and adherence to conventions are critical.

Moreover, the results reflect broader findings in human-AI collaborative writing research. Studies on tasks such as headline generation show that users achieve better outcomes when they can guide and selectively refine LLM outputs. This process enhances quality without compromising user agency or perceived authorship (Ding et al., 2023). This suggests that guided LLM use in educational settings may strike a productive balance between automation and student ownership. At a cognitive level, guided use of LLMs appears to support key phases in the writing process, particularly translation and revision. Chakrabarty et al. (2024) found that professional writers benefited most from LLM support during these stages. This insight resonates with our results and further substantiates the utility of guided approaches in educational contexts.

Finally, it is worth noting that the enhanced writing performance observed may not stem solely from the tool's linguistic capabilities, but also from reduced uncertainty and cognitive load due to structured task framing. When students know exactly how to proceed and what is expected of them in using a complex tool like an LLM, their cognitive resources may be more efficiently allocated to higher-order writing concerns, thus improving final output quality.

The findings provide empirical support for Hypothesis 2, indicating that students in the guided LLM use condition reported significantly higher levels of perceived mental health compared to those in the control group. Although the effect size was modest, the statistical significance of the difference underscores the potential of guided LLM integration as a psychologically beneficial educational tool. Notably, the comparison between the guided and unguided groups approached significance, suggesting that guidance in LLM use may play a decisive role in how such tools influence users' well-being. These results are consistent with growing evidence that conversational AI systems can enhance users' subjective mental health experiences when implemented with structured guidance. One contributing factor may be the enhanced user experience associated with anthropomorphically designed systems. For instance, Wu et al. (2024) showed that agents like Sunnie increased users' perceptions of usability and engagement. Such design strategies may foster a more human-like, empathetic interaction, which resonates with students in high-stress academic contexts.

Beyond surface-level interaction quality, systems like VITA have demonstrated that adaptive, behavior-sensitive guidance improves not just perception but also outcomes in mental well-being (Spitale et al., 2025). These systems personalize responses to individual user profiles and evolving needs, features that align well with the nature of guided LLM use in this study. When students receive structured prompts, reflective exercises, or scaffolded interactions from LLMs, the result is improved engagement and potentially heightened psychological support.

The results also echo the broader literature on LLM-based agents such as Replika, which offer on-demand, judgment-free interactions. Ma et al. (2023) highlighted how such interactions help individuals engage in self-reflection and develop confidence. In the present context, the structured engagement with LLMs may serve a similar purpose, providing students with an emotionally neutral space to articulate their thoughts and manage academic stress more effectively. Furthermore, Kumar et al. (2024) noted that AI-guided systems can integrate multiple data sources to detect subtle shifts in mental states and deliver personalized micro-interventions. Although this study did not leverage multimodal inputs, the positive outcome observed in the guided condition suggests that even text-based interventions, when strategically framed, can produce a meaningful uplift in well-being.

At the same time, the non-significant difference between the unguided and control groups raises important questions about the boundary conditions under which LLMs can support mental health. One plausible explanation is that unguided access may generate uncertainty, confusion, or even decision fatigue when students are left to navigate the system without structure. Prior research suggests that the absence of guidance can lead to overwhelming interactions or passive use of the tool, which may fail to produce affective benefits (Zhang et al., 2024; Zhu, 2024). It is possible that psychological support through AI requires not only access but also a sense of clarity, safety, and intentionality—conditions more likely to be fostered in structured interventions. In sum, the significant improvement in perceived mental health among students in the guided condition supports the hypothesis that structured, intentional interaction with LLMs can enhance psychological experiences in academic settings. These findings reinforce the view that LLMs—when deployed thoughtfully—can act as supportive companions in educational environments (Youn and Jin, 2021), particularly when combined with design principles and adaptive features that foster trust, personalization, and emotional safety (Liu et al., 2023). However, the lack of improvement in the unguided condition highlights the importance of pedagogical framing as a necessary condition for translating technological affordances into emotional gains.

The results support Hypothesis 3, indicating that students in the guided LLM use condition exhibited significantly greater academic engagement than those in the control group. The ANCOVA revealed a statistically significant effect of condition on engagement scores, with a modest effect size. Students in the guided condition reported the highest levels of engagement, reinforcing the view that structured interaction with LLMs can foster a more involved and focused learning experience. These findings align with a growing body of research highlighting the importance of guidance in shaping students' cognitive and emotional engagement in AI-supported learning environments. In particular, structured guidance during LLM use has been shown to reduce off-task behavior, such as random or superficial queries and the indiscriminate use of AI for answer retrieval (Kumar et al., 2024, 2023). By promoting intentional and reflective engagement, guided LLM interventions encourage students to assume more active roles in their learning processes.

Moreover, research on immersive and AI-integrated educational formats—such as Alternate Reality Games (ARGs) augmented with LLM guidance—has demonstrated that such designs can enhance behavioral engagement, emotional connection, and control beliefs (Cheng et al., 2022; Neary and Schueller, 2018). These findings suggest that when LLMs are embedded in pedagogically grounded frameworks, they can catalyze sustained academic motivation and participation. At the cognitive level, integrating LLMs into virtual teaching assistant roles, such as the Jill Watson system, further supports the idea that guided AI interactions can promote higher-order thinking and intellectual curiosity (Maiti and Goel, 2024). Students in such systems are not merely passive recipients of information but are actively encouraged to formulate and refine complex inquiries, fostering deeper engagement with content.

In contrast, using unguided LLMs favors quick information retrieval over sustained learning. Although such interactions may yield short-term performance gains, they do not appear to generate the same level of student investment or trust in the learning process (Kumar et al., 2025). This may help explain why the guided condition in the present study outperformed both the unguided and control groups regarding engagement. Indeed, the absence of clear instructional framing in the unguided condition may have led to uncertainty about how to use the tool productively, diluting its potential benefits for emotional or behavioral engagement. When students are unsure whether they are using a tool “correctly,” this ambiguity can undermine their sense of efficacy and reduce motivation to persist. Thus, although the results confirmed that guided LLM use fosters greater academic engagement, they also suggest that without supportive structure, LLMs may not reliably elicit active academic involvement. By combining technological capabilities with pedagogical intentionality, these systems offer an interactive and supportive environment that encourages students to actively participate, reflect, and persist in their academic work.

Hypothesis 4 posited that students in the unguided LLM use condition would exhibit intermediate levels of academic writing quality, perceived mental health, and engagement, greater than those in the control group but lower than those in the guided use condition. The results partially supported this hypothesis: while this expected pattern was observed and statistically confirmed in academic writing quality, it was not replicated in perceived mental health or academic engagement.

The writing results suggest that access to LLMs can meaningfully enhance students' output even without structured guidance. The unguided group significantly outperformed the control group, indicating that basic interaction with the tool—through prompts, content generation, or surface-level feedback—was sufficient to raise writing quality. This aligns with prior findings that when used independently, LLMs can offer valuable assistance in planning, drafting, and refining text (Jungherr, 2023; Meyer et al., 2024). Nevertheless, the superior performance of the guided group supports the idea that structured scaffolding, metacognitive prompts, and explicit instruction in tool usage amplify these benefits (Cash et al., 2025; Salimi and Hajinia, 2025).

In contrast, the data did not support the predicted intermediate pattern for perceived mental health. Although the unguided group reported slightly higher scores than the control group, this difference was not statistically significant. One possible explanation is that unguided use, while offering on-demand support, lacks the emotional structure and stress regulation strategies typically embedded in guided implementations. Without clear boundaries or reassurance about responsible usage, students may experience friction, uncertainty, or even anxiety about whether they are “using the tool correctly,” which may counteract any potential gains in psychological well-being (Zhang et al., 2024). In this context, guidance may not only clarify functionality but also serve a regulatory role—normalizing AI integration, reducing confusion, and promoting a sense of support and competence (Zhu, 2024).

A similar pattern emerged with academic engagement. Although the unguided group showed numerically higher engagement than the control group, this difference was not statistically significant. This suggests that, while unguided LLM access may spark curiosity and enable autonomous exploration, it does not consistently produce sustained or deep engagement. One likely reason is that students without instructional scaffolding may remain uncertain about how to engage productively with the tool, leading to hesitant or fragmented interaction. Prior research indicates that without pedagogical framing, students may use LLMs for surface-level information retrieval or task avoidance, limiting the depth of their involvement (Chen and Leitch, 2024). By contrast, guided use has been associated with stronger emotional and cognitive engagement, as students are trained to leverage LLMs in a reflective and goal-oriented manner (Beurer-Kellner et al., 2024; Uchendu et al., 2023).

Altogether, the findings highlight the nuanced role of guidance in realizing the potential of LLMs. While unguided use may yield modest benefits in writing quality, its impact on mental health and engagement appears more limited. The lack of significant differences between the unguided and control groups in two of the three outcome domains suggests that students may underutilize these tools or even encounter friction in their use without structured scaffolding. These results point to the importance of not only providing access to AI tools but also offering appropriate pedagogical frameworks to ensure their effective and psychologically supportive implementation.

Despite this experimental design's strengths, several limitations should be acknowledged. First, although random assignment was used to allocate participants across conditions, the study relied on self-report manipulation checks to confirm adherence to the assigned use of LLMs. While this approach ensured consistency between reported and intended use, it may introduce response bias or fail to capture subtle variations in how students interpreted and used the tool within each condition (Roshanaei, 2024). Although technically feasible alternatives such as behavioral logging (e.g., tracking LLM interactions) might offer more objective verification, such data were not collected due to ethical constraints and institutional limitations in access control. Future research could explore ways to integrate such measures in a transparent and privacy-respecting manner.

Second, the study was conducted within a single institutional context and with a relatively homogeneous sample—undergraduate students enrolled in humanities and arts programs at an international university in China. This limits the generalizability of the findings to other educational settings, disciplines, and cultural contexts (Salimi and Hajinia, 2025). In particular, students in STEM fields might interact with LLMs differently not only due to varying levels of digital literacy, but also because of the nature of the writing tasks they face, the disciplinary conventions they follow, and the specific modes of information retrieval their fields require. These differences may influence how beneficial, usable, or trustworthy LLM tools appear in practice.

Third, the primary measure of writing quality relied on the automated Performance Score generated by the Grammarly for Education platform. While this tool offers objectivity and replicability, it prioritizes surface-level features such as grammar, clarity, and lexical variety. As a result, it may not fully capture deeper dimensions of academic writing—such as argumentation structure, critical analysis, originality, synthesis of sources, or adherence to disciplinary conventions—which are essential in evaluating high-level academic work. Human-rated assessments or rubric-based evaluations could complement automated scoring in future studies to provide a more holistic picture of writing quality (Salimi and Hajinia, 2025).

Fourth, although the study included a validated measure of prior academic performance as a covariate, this measure was limited to students' most recent literature course. While this represents a meaningful control, the category of “literature-related subject” remains broad and may encompass varying levels of complexity and assessment standards. Moreover, other potentially relevant factors—such as writing experience in other languages, previous exposure to AI tools, or individual motivation—were not controlled and could have influenced the outcomes (Xu et al., 2025). Fifth, the study used perceived mental health and academic engagement as outcome variables measured through self-report scales. While these instruments are widely validated, self-reported data are subject to social desirability and may not accurately reflect behavioral engagement or psychological functioning (Meyer and Elsweiler, 2025). Future research could enhance robustness by incorporating behavioral (e.g., time-on-task) or physiological (e.g., stress monitoring) indicators to triangulate self-perceptions with observable evidence (Youn and Jin, 2021).

To sum up, the findings should be interpreted with an awareness of existing limitations. For example, while guided LLM use markedly improves performance on structured academic tasks, its efficacy may vary across genres or in more creative domains. Gómez-Rodríguez and Williams (2023) noted that human writers still maintain an advantage in areas such as humor and originality, which are difficult for LLMs to replicate reliably. Therefore, while our results highlight the transformative potential of guided LLM use, they also reinforce the need for human oversight and creative judgment in academic writing.

Integrating LLMs into academic writing instruction presents promising opportunities and pressing challenges. The present findings reinforce the importance of structured, guided use of LLMs, particularly in enhancing students' writing quality, academic engagement, and, to a certain extent, their perceived mental well-being. These results carry several practical implications for educators, instructional designers, and policymakers in higher education. Importantly, the implications presented here are directly informed by the limitations discussed above, and their placement after the limitations section reflects a deliberate decision to ensure that recommendations are realistic, context-aware, and attuned to the boundaries of the current design.

The study underscores the pedagogical value of structured engagement with LLMs. Educators should prioritize guided frameworks when introducing LLMs into learning environments (Alsobeh and Woodward, 2023). This includes providing students with clear instructions on using these tools effectively, offering structured prompts, and integrating LLM interactions into existing learning goals (Chiang and Lee, 2023). As demonstrated by approaches such as the Writing Path framework (Lee et al., 2024), guided strategies help align LLM-generated content with academic standards and user intentions, improving writing quality (Gómez-Rodríguez and Williams, 2023). Guidance also plays a crucial role in shaping student behavior during AI interaction. Research shows that structured guidance can reduce off-task or random queries and foster a more focused, problem-solving approach to writing (Kulaksiz, 2024). When embedded in pedagogy, these strategies enhance performance and increase students' trust and sense of ownership in the learning process (Bekker, 2024).

LLM integration necessitates rediscovering the educator's role—from transmitter of knowledge to AI literacy and ethical engagement facilitator (Lazebnik and Rosenfeld, 2024). Teachers should take an active role in helping students understand the limitations of LLMs, differentiate between responsible use and misuse, and navigate the ethical considerations associated with AI-generated content (Lee, 2023). Training students to critically evaluate and revise LLM outputs contributes to deeper learning and helps prevent overreliance (Liao et al., 2023). Instructors can also promote transparency by encouraging students to document their use of LLMs in the writing process, thus reinforcing principles of academic integrity and accountability (Mahmoud and Sørensen, 2024). This approach fosters a culture of AI-augmented authorship, where students learn to integrate feedback rather than delegate writing tasks to an automated agent. This pedagogical vision also aligns with global policy agendas—such as the UNESCO Futures of Education framework and the Sustainable Development Goal 4 (SDG4)—by promoting inclusive, equitable, and future-ready higher education that incorporates responsible AI use.

While the study found that even unguided use of LLMs can yield some benefits, particularly in writing quality, such benefits are significantly more limited without instructional scaffolding. Unguided use has several risks, including superficial engagement, skill stagnation, and academic integrity concerns. Without explicit instruction, students may bypass the cognitive and metacognitive processes essential to writing, relying instead on the fluency of LLMs to complete tasks (Lopes et al., 2024). Moreover, the indistinguishability of AI-generated text from human-authored work poses significant challenges for evaluation (De Villiers et al., 2024; Reinhart et al., 2024). This complicates the role of assessment and highlights the urgent need for institutional policies that address transparency, disclosure practices, and acceptable uses of generative AI in coursework. As noted in the limitations, behavioral metrics and clearer definitions of disciplinary norms could inform these policies, especially when automated scoring tools like Grammarly are involved.

Finally, educators must balance leveraging the benefits of LLMs and fostering independent writing skills (Patac and Patac Jr, 2025). While guided use can accelerate learning and reduce barriers, overdependence on AI tools may inhibit students' ability to think critically and write autonomously (Ouwehand et al., 2025). Integrating LLMs should not replace traditional instruction but complement it through strategy-based interventions, peer review, and scaffolded writing tasks (Li et al., 2023; Lin et al., 2024). Future research might also explore how these strategies can be adapted to STEM disciplines or interdisciplinary programs, as differences in writing genres and task complexity may shape how students engage with LLMs. In line with SD4′s commitment to inclusive and contextually sensitive education, such differentiated approaches are essential to ensuring that AI-enhanced instruction serves diverse learners effectively.