Valid measurement instruments are a cornerstone of research in the social sciences. Traditionally, psychometric development has relied on teamwork, iterative testing, expert judgment, and statistical analysis grounded in frameworks such as classical test theory. These methods, while backed by decades of research, take time and are limited by the resource capacity of a research project. Despite progress in reusing validated scales, researchers still encounter challenges in building and adapting measures to suit project needs (Heggestad et al., 2019). In recent years, artificial intelligence (AI) technologies—particularly large language models (LLMs) and machine learning (ML) algorithms—have begun to augment item, scale, and instrument development (Beghetto et al., 2025; Zhao et al., 2025; Tan et al., 2024).

In brief, typical AI technologies comprise a huge array of interconnected mathematical structures known as artificial neurons whose coefficients can be successively adjusted by subjecting them to large amounts of training data. Research applying these structures to the production of natural language demonstrated success in 2017 with the release of the first “generative pre-trained transformer” (GPT) models (Wang Z. et al., 2025). LLMs, trained on vast corpora of text, generate convincing linguistic outputs when prompted with the definition of a psychological construct provided by a researcher (Kim and Lee, 2024; Lee et al., 2025; Tan et al., 2024; Terry et al., 2025). ML techniques can predict factor structures, estimate reliability prior to collection of response data, and identify patterns in item responses that may elude traditional statistics (Kim and Lee, 2024; Lee et al., 2025; Stanton et al., 2024). Such innovations suggest a future where LLMs and ML might make psychometric tasks easier and quicker.

Unsurprisingly there is no free lunch. AI also creates pitfalls and unintended consequences. Generative AI provides little purchase for a human who wants to understand how the technology works. As a result, assumptions embedded in AI models often cannot be directly evaluated. For example, when AI selects or discards items from a scale, it is difficult to assess whether underlying assumptions are fulfilled (Hernandez and Nie, 2023; Qu et al., 2025). These interpretability barriers raise questions about validity, replicability, and defensibility of instruments developed with AI (e.g., Circi et al., 2023). Ethical considerations also arise in the use of AI-based methods. The immense amounts of data harvested to create AI training data may contain copyrighted material or inadvertently include inappropriate data such as personal information (Jiao et al., 2025). Training data may also reflect the inaccuracies, biases, and prejudices present in everyday speech (Hagerty and Rubinov, 2019; Sang and Stanton, 2020; Veale and Binns, 2017).

This review briefly synthesizes recent literature to examine how AI serves various stages of psychometrics, including item generation, validation, and analysis. The review highlights benefits, such as increased efficiency, as well as pitfalls, including lack of transparency, ethical concerns, and the risk of obscuring psychometric assumptions. Drawing from articles in psychology and education, as well as technical papers from fields like natural language processing, this review evaluates AI's emerging role in supporting social science measurement. The review is organized into themes addressing content generation, psychometric analytics, data quality, and ethical considerations, and concludes with a discussion.

One promising application of AI in social science measurement lies in the use of large language models (LLMs) for early-stage psychometric tasks such as item generation. LLMs can produce a large set of candidate survey items in the few moments it takes a researcher to write a suitable prompt. Studies have demonstrated that LLMs can produce syntactically and grammatically correct texts that semantically align with target constructs. Tan et al. (2024) listed 60 distinct research efforts to generate educational assessment items. Bhandari et al. (2024) conducted psychometric evaluations suggesting that AI-generated items matched the performance of human-written items.

One pitfall of using LLMs for this purpose is semantic drift, where generated items may deviate from the intended construct due to the probabilistic nature of language modeling. In typical configurations, most LLMs “sample” from a set of probable responses. AI-generated items thus reflect stochastic pattern matching rather than an epistemological connection to a defined theoretical construct. This disconnect arises in part from the absence of world knowledge in AI systems: they lack an understanding of psychological theory, cultural nuance, and the context of test administration (Ferdaus et al., 2026; Owan et al., 2023). Semantic drift may lead to AI-generated items that exhibit face validity without meeting statistical quality criteria (Qu et al., 2025; Speer et al., 2025). Replicability problems can be corrected in some AI models with appropriate configuration changes, but with the next release of the LLM it will once again become difficult to reproduce earlier outputs.

In addition to new items, AI can also creation of content for alternate test forms. Educational testing researchers have experimented with automatic item generation even prior to the advent of LLMs (Gierl and Lai, 2012; Gorgun and Bulut, 2024). Using appropriate prompt engineering, LLMs can generate alternative items that attempt to assess an existing construct, skill, or knowledge area (Tan et al., 2024). For example, Wang L. et al. (2025) developed a system that incorporated contextual knowledge from teachers into an item generation prompt to enhance the veracity of LLM-generated items. Burke (2025) generated alternative test forms through an iterative process of prompt development including refinements like this: “Exams should test the following learning objectives: procedural fluency, emergent statistical thinking, and explanatory literacy…” Kim and Lee (2024) and Speer et al. (2025) cautioned that empirical verification of the equivalence of alternate forms generated with the help of AI is a crucial analytic step.

The development of translated items applies a “cross language” LLM to creating items for a foreign language version of a scale. Such systems specialize on language-pairs, enabling translations that try to preserve the definition of the construct reflected in the original items using the target language (Grobelny et al., 2025; Wang and Duan, 2024). Some tools create workflows that combine machine-generated drafts with human editing, a hybrid approach that may improve translation quality (Herbig et al., 2019). Semantic shifts induced by automated translation may affect item functioning across cultures, potentially compromising validity (Xu et al., 2024).

An emerging application of LLMs creates data sets of artificial item responses using “silicon respondents” (Gurdil et al., 2024; Liu et al., 2025; Sun et al., 2024). For example, Hämäläinen et al. (2023) conducted a series of three experiments to evaluate the usefulness of participant data generated by an LLM and concluded that such data was of sufficiently high quality to include in pilot testing and experiment design efforts. Suh et al. (2025) reached a similar conclusion in an experiment on public opinion data. Lutz et al. (2025) examined prompting strategies and found that LLMs had difficulty generating data representing responses of minority groups. Neumann et al. (2025) evaluated nine LLMs' ability to generate opinion data and found substantial variation in data quality by model.

In a parallel vein to LLMs, researchers have explored AI-based image generation for psychological assessments, such as facial expression scales and projective tests (Koweszko et al., 2025; Lin et al., 2025; Villegas-Ch et al., 2025). AI-generated inkblot-like stimuli have been examined as alternatives to traditional projective materials (Alsabouni and Ihara, 2023; Li et al., 2025). Researchers have also tested AI tools that assist with the evaluation of projective test responses created by participants (Beltoft et al., 2026). These innovations raise similar concerns to LLM-based item: Stimuli for projective tests require variety, whereas training for generative AI systems ingest graphical data in ways that may create homogeneous outputs (AlDahoul et al., 2025; Messingschlager and Appel, 2025).

Traditionally, scale validation procedures have relied on step-by-step application of statistical procedures such as reliability estimation and factor analysis. These methods are usually grounded in psychometric frameworks like item response theory. Recently, researchers have applied methods from a domain of AI known as machine learning (ML) to predict factor structures, assess internal consistency, and identify item clusters (e.g., Goretzko and Bühner, 2020; Stanton et al., 2024). Unlike traditional statistical methods, ML generally does not embed data assumptions such as normality or linearity (Giudici et al., 2025). ML can handle atypical data types, such as graphical and linguistic data. This flexibility has advantages, for example when working with unruly distributions, but also raises transparency and interpretability questions. For example, when an ML predicts the reliability of a scale, the criteria or features it used to make that judgment ignore the theoretical underpinnings of scale development and may hinder replication efforts (Zhuang et al., 2025). Basic assumptions, such as unidimensionality or local independence, often remain unexamined and may be absent from the tested data.

The application of ML algorithms to individual problems of prediction and classification has existed for decades (e.g., Cortes and Vapnik, 1995) but these algorithms represent just a subset of what is now termed as AI. Taking a broader perspective, AI-assisted workflows are increasingly being tested to automate chains of tasks that would more usually be conducted in separate steps by a human analyst. These workflows integrate machine learning models, statistical code generation, and natural language interfaces to streamline procedures such factor analysis. For example, Zhuang et al. (2025) described a framework that applied adaptive testing principles to AI model evaluation. Almheiri et al. (2024) described an application where the LLM recommended a sequence of spreadsheet-based analyses. Similarly, AI has been used to streamline statistical modeling (Sharpnack et al., 2024; Yancey et al., 2024; Zhan et al., 2025) mainly through code generation for R and/or Python.

Automated workflows raise concerns about rigor/replicability (Duque et al., 2023). When human analysts conduct a factor analysis, for example, opportunities arise for evaluating assumptions and crosschecking. AI analytics, in their present development stage, tend to obscure the rationales behind decisions, predictions, and estimates (Zhan et al., 2025). The likelihood that a naïve user could generate nonsensical results may be unacceptably high until the automation also offers built-in crosschecking and tools to flag violated assumptions (Zhuang et al., 2025).

LLMs and supervised ML classification algorithms also offer new avenues for approaching content validation. Traditionally, content validation involves expert judgment of whether items represent the construct being measured (Haynes et al., 1995). AI augments content analysis by analyzing semantic coherence, construct coverage, and linguistic appropriateness of item pools. For example, researchers have assessed how LLMs evaluate alignment of items with construct definitions (Grobelny et al., 2025; Larsen et al., 2024). Additionally, natural language processing techniques have been used to identify redundancies, gaps, or ambiguities in item sets (Reimers and Gurevych, 2019). ML algorithms can also predict human categorization of items, offering probabilistic assessments that complement human judgment (Ahmad et al., 2020). While these approaches enhance efficiency, they may also reinforce biases present in training data (Barocas et al., 2019, p. 7).

In addition to psychometrics, AI can also assist with test administration. Adaptive testing, which dynamically adjusts item difficulty based on a respondent's performance on previously administered items, can reduce test administration time and increase measurement precision. AI has been tested for real-time item selection and scoring. For example, Mujtaba and Mahapatra (2020) described AI-driven item sequencing to maximize precision while minimizing test length. Other research has used AI to tailor assessments both to ability levels and individual item response patterns (El Msayer et al., 2024; Gan et al., 2019). These systems raise concerns about transparency and fairness, because the algorithms typically make item selections without documentation of the psychometric rationale.

As a final topic in this section, researchers have also applied AI directly to common, time-consuming measurement tasks. For example, open-ended survey responses can offer rich qualitative insights into participants' beliefs, attitudes, personality, and reasoning processes, but their analysis has traditionally been labor-intensive (Fan et al., 2023; Sun, 2021). AI can automate scoring and classification of open-ended responses by recognizing patterns in textual data and assigning scores or categories (Gweon and Schonlau, 2022). Similarly, LLMs can automatically assign grades and generate feedback for short answers and essays. These models have achieved low error rates in predicting human-assigned grades and have generated written feedback judged to be similar in quality to that of human experts (Katuka et al., 2024).

Given contemporary methods used to train AI models, developers may inadvertently create inappropriate item content when using LLMs and other generative AI tools. Researchers have long recognized that LLMs exhibit biases present in training data (Hovy and Prabhumoye, 2021; Messingschlager and Appel, 2025; Salinas et al., 2023). These biases can lead to output that participants may consider offensive (Huang and Huang, 2025). In the same vein, LLMs are known to “hallucinate” in ways that create counterfactual outputs (Chen et al., 2023). Developers have tested safety controls to limit the generation of inappropriate output, but research shows that these can be readily circumvented (Hundt et al., 2025).

Current LLM training processes require vast amounts of data (Villalobos et al., 2024). Companies harvest these data from the internet, but at the same time increasing amounts of internet content are themselves AI-generated. Thus, another training-related concern is “AI-slop:” degradation of training data and model quality caused by recursive ingestion of AI-generated material (Gross and Colson, 2025). This problem can amplify biases and cause stylistic homogenization (Ferdaus et al., 2026; Han, 2025), undermining the richness needed for full representation of a construct domain (Smith and Stanton, 1998). Poor data quality may also complicate validation workflows: When decision-making models are trained or tested on data that includes synthetic responses or AI-generated items, generalizability of the results may be compromised. This is particularly troubling in adaptive testing and IRT modeling, where item-level precision is essential. If an item pool is polluted with low-quality or derivative content, resulting trait estimates may be misleading (Bachmann et al., 2024; Kingston and Kramer, 2013).

As AI becomes integrated into psychometric tasks, ethical concerns have emerged. The convenience of using automation in research processes has the potential to reduce researcher oversight (Xu and Peng, 2025). One pressing issue is algorithmic bias, which can arise when AI systems make decisions based on training data that reflect historical inequities. As AI systems take on tasks traditionally performed by humans—such as item writing—there is a risk that researchers may defer too readily to automated outputs. This could lead to a diminished role for critical thinking in scale development/deployment such that AI testing reinforces stereotypes or excludes a breadth of cultural perspectives.

Ethical concerns also arise regarding privacy and autonomy, especially when AI systems infer psychological traits (Kim and Lee, 2024; Menard and Bott, 2025). In cases where AI infers personality traits or predicts future behavior, there is a risk of violating participant autonomy by analyzing personal data in the absence of informed consent by affected individuals (Kim and Lee, 2024). Researchers must also ensure that data handling—for training data, input prompts, and output data—complies with ethical standards and legal regulations, such as GDPR and HIPAA, and that participants affected by AI-based measures are informed about the role of AI in the research process and in testing services.

Moreover, because the internal mechanisms by which LLMs generate items are opaque, researchers may be unable to explain why certain outputs were produced, raising concerns about rigor and legal defensibility of measures. Unlike traditional methods, which allow researchers to assess/adjust for bias through statistical procedures, AI systems make it difficult to detect or correct adverse patterns (Qu et al., 2025; Zhuang et al., 2025). These issues pose challenges for peer review, replication, ethical oversight, and informed consent, especially when AI is used in processes like educational placement and employee selection.

Integration of AI into development and validation of measurement tools heralds the prospect of rapid methodological evolution for social science research. As this review suggests, AI tools show potential to enhance item generation, automate analysis, and support adaptive testing, among other tasks. These innovations enhance what a research team can accomplish with limited resources (Kim and Lee, 2024; Wang et al., 2024) but also introduce concerns that must be addressed to ensure the integrity and ethical use of AI in psychometrics.

A recurring theme throughout this review is that, while current AI systems can assist with many psychometric tasks, they do not do so transparently. Current systems may replicate traditional psychometric procedures in some contexts, but they typically cannot reveal the assumptions or decision rules underlying their outputs. This opacity raises questions about validity, replicability, and accountability—core principles of psychometrics (Bringsjord, 2011; Veldkamp, 2023). The risk is particularly acute in tasks such as data analysis, where untested or invisible assumptions may distort results.

Another critical issue is the epistemological gap between AI systems and human researchers. AI models, despite linguistic fluency, lack world knowledge and contextual understanding. This deficit affects the ability of an AI model to generate items that are not simply grammatically correct but also societally and culturally appropriate. Consistent alignment between AI model outputs, theoretical frameworks, and ethical standards for the conduct of research remains as an important future goal.

Ethical concerns compound current technical challenges. Bias in training data, lack of cultural sensitivity, and erosion of researcher oversight all threaten to undermine the fairness and defensibility of instruments created and/or administered with the assistance of AI (Kassir et al., 2023; Mujtaba and Mahapatra, 2020). Moreover, AI-slop may pose and eventual risk to the diversity and quality of psychometric tools (Hagendorff, 2021). Without careful curation of training data, AI-generated items may become increasingly homogenized, atheoretical, and ineffective in capturing psychological constructs.

Three mitigation strategies may help us achieve benefits of using AI in psychometrics: human-in-the-loop, rigorous training data curation, and transparency engineering. First, experiments with AI-enabled systems suggest that interface designs can provide checkpoints for human validation of outputs and pending decisions (Burke, 2025; Lee et al., 2025; Wu et al., 2022; Rothschild et al., 2025). This design strategy, commonly known as human-in-the-loop, casts the AI as a subordinate team member in an interactive effort to achieve a project goal.

The second strategy, rigorous data and tool curation, might require cooperation by professional societies, publishers, and research funders. Heretofore, curation of training and evaluation data for LLMs and other AI technologies has often been left to commercial firms that create AI models. There is a process called “fine-tuning” where additional training data (often from a specialized domain) is applied to a basic trained model to allow it to perform tasks that a general-purpose model cannot (Hommel and Arslan, 2025). Shared benchmarking data and tools are needed to support fine tuning and evaluate the performance of the resulting system. To support contemporary scientific standards, a concerted effort is needed by the social science community to curate high quality AI training data and evaluation tools.

Finally, emerging research suggests that transparency challenges exhibited by current AI models can eventually be rectified by new capabilities (Bacon and Menon, 2025; Liao and Vaughan, 2023; Pigac, 2025). With appropriate engineering, an AI model can communicate to users the internal assumptions used in a computation as well as the uncertainty surrounding estimates/outputs. Such features could eventually include assumption-checking protocols, post hoc simulation of distributional properties, and comparison/validation of results by multiple AI models. As an example of the latter, Liu et al. (2025) published research suggesting that an ensemble of LLMs produces better results than a single LLM working alone (see also, Lee et al., 2025). The concern for transparency extends to reports by human researchers using AI: we need scientific reporting standards that document model versions, configurations, training procedures, and known limitations.

Despite challenges outlined above, AI-assisted psychometrics promise to enhance the efficiency and scope of social science measurement tasks. When combined with traditional methods and subject to human oversight, AI tools can support efficient and ethical approaches to measurement. The future of AI in psychometrics must be guided by a hybrid approach—one that leverages the strengths of AI while preserving the judgment, expertise, and ethical accountability of human researchers. Transparent documentation, assumption-checking protocols, and interdisciplinary collaboration will be essential to ensure that AI tools enhance and do not compromise the rigor of item, scale, and instrument development in the social sciences.