Actor-network theory, rooted in science and technology studies (STS), offers a distinctive lens for analysing AI in governance by refusing to separate technical systems from political structures (Latour, 2007; Latour, 1996; Papilloud, 2018). Rather than viewing GenAI as merely a tool deployed by human actors, ANT recognizes technological systems as actants with political significance that actively reshape governance networks. This approach reveals how GenAI systems like LLMs do not simply execute predetermined functions but can actively mediate political relations, reshaping how decisions are made, policies implemented, and public accountability maintained.

Three key concepts from ANT are particularly valuable for understanding AI’s implications in governance:

First, recognizing both human and non-human actors and actants illuminates how LLMs can reshape democratic accountability by mediating between citizens, officials, and policy frameworks. When an LLM generates policy recommendations or drafts administrative decisions, it exercises a form of agency that complicates traditional governance accountability chains. The NSW Government’s integration and promotion of GenAI as a co-actant (Digital.NSW, 2024c; Digital.NSW, 2024b) within administrative processes reconfigures who—or what—exercises authority in governance networks.

Second, the concept of translation reveals how AI systems transform policy intentions through their implementation. Translation in ANT refers to the process by which actors are brought into networks, with their interests realigned to fit (Latour, 2007). As LLMs or other algorithmic platforms interpret regulations, draft documents, or analyze citizen inputs, they do not simply transmit information but actively reshape it according to their logic. This translation process raises critical questions about democratic legitimacy when non-elected technical systems mediate policy intentions established through democratic processes.

Third, blackboxing—the process by which a technology’s internal workings become opaque to users—presents fundamental challenges to democratic transparency. In government AI projects, blackboxing can obscure the logic behind algorithmic decisions, complicating efforts to ensure accountability to citizens (Gutiérrez, 2023).

Recent scholarship has advanced the conversation on how AI integration reshapes political governance processes, offering both transformative potential and presenting challenges to democratic oversight (Cheong, 2024; Taeihagh, 2021). LLMs have become a focal point within this context due to their ability to generate text resembling human language, enabling automation of traditionally human political functions like policy analysis, regulatory drafting, and citizen communication.

Studies reveal that GenAI in public administration can assist in processing substantial data volumes, identifying trends, and generating insights and future scenarios that traditional methods might miss (Reid et al., 2023). However, some studies highlight the importance of tackling ethical and accountability concerns associated with GenAI’s expanding role, addressing issues around transparency, potential bias, and the need for human oversight to prevent harm (Mergel et al., 2024).

As GenAI becomes more integral to government, policy and political processes, there is increasing attention on concerns like algorithmic transparency, bias, and accountability (Janssen and Kuk, 2016; Singhal et al., 2024; Taeihagh, 2021). Its deployment raises fundamental questions about delegating authority to GenAI, as these systems have the potential to shape or even determine policy outcomes (Veale and Brass, 2019) and are also embedded with a particular ideological view (Rozado, 2024).

The ideas of breakdown and repair (Star, 1999; Jackson, 2014) provide valuable analytical tools for understanding democratic resilience in AI-enhanced governance systems. These concepts become especially important in implementing GenAI in the government sphere, where technological systems often face setbacks and require adjustments that have political implications.

‘Breakdowns’ (Star, 1999) describe moments when systems, infrastructures, or technologies fall short of expectations, exposing the underlying connections within a network that typically remain hidden during smooth operation. This concept draws on Latour (1992) work, which explores breakdowns through the lens of everyday artefacts and their role in guiding social order and behaviors. In political and policy contexts, breakdowns occur when systems fail to align with democratic values, expose accountability gaps, or produce outcomes that undermine public trust. When an AI system produces biased outcomes or operates in ways citizens cannot understand or contest, it can create democratic deficits that require intervention.

‘Repair,’ on the other hand, follows breakdown and encompasses the technological fixes, policy updates, and ethical recalibrations needed to realign these systems with public sector values. Jackson (2014) sees repair as not a simple return to the previous state but a productive process that can lead to transformation and improvement. The repair might mean reworking the relationships and roles within the actor network when integrating GenAI into public administration, creating opportunities for ethical reflection, policy updates, and systemic enhancement. In the NSW government, breakdowns could surface as technical setbacks or ethical issues that expose weaknesses in the socio-technical infrastructure. Repair becomes essential in addressing these issues, helping build public trust by openly tackling AI-related challenges.

By combining Actor-Network Theory with concepts of breakdown and repair, this paper develops a framework for critically examining how GenAI reshapes political accountability and democratic legitimacy in NSW governance. The following section outlines the methodological approach to mapping these socio-technical dynamics in NSW’s emerging AI governance landscape.

To systematically identify and analyze relationships among actors in NSW’s GenAI ecosystem, a structured Actor-Network Mapping protocol was employed, aligned with established ANT methodological approaches (Justesen, 2020; Nimmo, 2011). The protocol consisted of four key phases:

This mapping revealed key actors and their roles in NSW’s GenAI network (Table 1), highlighting the centrality of the AI Assurance Framework as a mediating structure between technical systems and governance mechanisms.

The document analysis followed a structured protocol, selecting materials based on their relevance to GenAI integration in NSW governance (Digital.NSW, 2024c; Digital.NSW, 2024b; Digital.NSW, 2024a; Digital.NSW, 2024d; Parliament NSW, 2024). A thematic coding approach was employed, with initial codes derived from the research questions and ANT framework, supplemented by inductive codes that emerged during analysis. Qualitative analysis software organized themes, ensuring systematic tracking of coding decisions. The analysis was enriched by examining evidence from relevant cases, such as the NSW Ombudsman’s findings on AI-related failures and insights from the ‘Robodebt’ inquiry (Clarke et al., 2024; Podger, 2023; Rinta-Kahila et al., 2024). These insights will be expanded in a subsequent section of the paper.

Given that GenAI implementation in NSW is at an early stage with no significant operational failures yet documented, this study employed scenario analysis to complement document analysis and actor-network mapping. This approach enables the exploration of potential vulnerabilities without requiring direct observation of failures.

Three hypothetical scenarios were developed based on the socio-technical dynamics identified through ANT mapping. Each scenario examined how actors might realign or conflict in response to the breakdown, drawing on concepts of breakdown and repair to anticipate potential adaptations in AI governance.

Three key considerations drove the selection of the methodological approaches:

The combination of ANT mapping with scenario analysis represents a methodological innovation that addresses the challenges of studying emerging technologies whose impacts are very recent due to their early stages of implementation. It offers a framework other jurisdictions can adapt to ideating their GenAI integration initiatives.

Some limitations in these methods are worth noting, such as potential bias by the author in document selection, the subjective nature of thematic coding, and the challenge of fully capturing actor relationships in a static map. Additionally, reliance on public documents may not fully reflect the internal, current, or informal dynamics within the NSW Government, which, due to the early implementation and inception of these systems, are very difficult to obtain. The black box of governance is sometimes very closed, but this methodology aims to first articulate the topic at large. The following sections analyze the socio-technical dynamics of GenAI integration in NSW, exploring related ethical challenges and breakdown-repair concepts. The discussion will synthesize findings, examining how GenAI influences NSW Government operations and the associated ethical and political implications.

The NSW Government has introduced a governance framework for integrating AI within its public sector operations to establish democratic legitimacy while capturing efficiency benefits. The framework establishes a political authority structure that positions elected officials and public servants as the ultimate decision-makers while incorporating GenAI systems as supporting actors in governance processes. This architecture consists of several interconnected components that together form what might be termed a socio-technical governance ecosystem:

A foundational element is the AI Ethics Policy (Digital.NSW, 2024a), which establishes normative principles guiding AI’s use across government operations. The policy emphasizes transparency, accountability, fairness, and privacy, aligning AI tools with democratic values and societal expectations. By establishing these principles, the policy creates a governance mechanism to maintain democratic legitimacy while incorporating increasingly autonomous technical systems. This approach positions ethical frameworks as political instruments that mediate between technical capabilities and democratic requirements.

Complementing this normative framework, the AI Assurance Framework provides a structured review process for AI initiatives, mandating evaluations of AI projects to confirm their adherence to established ethical principles (Digital.NSW, 2024d). This framework represents an attempt to operationalize abstract ethical principles into concrete governance mechanisms. The review involves considering questions about public benefit, privacy, and security and establishing a procedural approach to maintaining human authority over algorithmic systems. Notably, this framework covers all AI projects except those involving commercial AI tools without customization, creating a potential governance gap in the oversight architecture.

These frameworks establish a hierarchical governance structure where human officials retain formal authority while AI systems are positioned as tools subject to human oversight. However, AI systems’ increasing autonomy and authority in day-to-day administrative processes may challenge this formal structure in practice. The training initiatives explored below suggest a more complex agency distribution than the formal governance architecture might imply.

The Chatbot Prompt Essentials learning module (Digital.NSW, 2024c) represents a key component of NSW’s preparation of its workforce for AI integration, revealing insights into how political authority may be redistributed in practice by GenAI. This module aligns with the infrastructure philosophy (Große, 2023) shaping the government’s adoption of GenAI and AI more broadly (Parliament NSW, 2024).

The module is designed to upskill government employees in crafting effective prompts for GenAI systems, teaching public servants to create prompts that encourage a natural conversational flow with LLMs. By introducing prompt engineering (Bozkurt, 2024; Sahoo et al., 2024; Wang et al., 2024), the module establishes a new form of human-AI interaction where public servants learn -and are expected- to guide AI systems rather than simply oversee them. This approach positions public servants as prompt engineers who shape AI outputs rather than sovereign decision-makers who merely review AI recommendations, subtly shifting the distribution of agency and authority in practice.

This training initiative reveals tension within NSW’s AI governance approach. While the formal governance architecture positions humans as the final authority over AI systems, the practical implementation suggests a more collaborative relationship where humans must learn to communicate with AI systems to achieve desired outcomes effectively. Moreover, they are expected to. This tension reflects broader challenges in establishing democratic governance of increasingly autonomous technical systems actively shaping political and administrative processes (see Figure 1).

The NSW Government’s approach to AI governance is distinctive in its breadth and proactive nature, setting a high benchmark for public sector AI frameworks globally. Compared to other jurisdictions, NSW’s governance architecture reveals political priorities and assumptions about the relationship between human and artificial agents in democratic governance.

While other regions are developing their AI governance frameworks, NSW’s strong focus on integrating ethical frameworks with workforce preparation and technical implementation distinguishes it from approaches in jurisdictions where AI governance may be more fragmented. For instance, the European Union’s AI Act offers a comprehensive regulatory framework that categorizes AI risks (Wagner et al., 2023), while the United States issued an Executive Order on AI to guide federal agencies in creating assurance structures (Saheb and Saheb, 2024). Canada’s Algorithmic Impact Assessment tool and Singapore’s AI Governance Framework also emphasize responsible AI implementation but with different emphases on risk assessment and industry collaboration (Saheb and Saheb, 2024).

These global efforts underscore the increasing focus on establishing legitimate governance structures for AI systems. Still, the NSW Government’s model is one of the most thorough in the public sector worldwide, particularly in its attention to practical workforce integration alongside formal governance frameworks. This comprehensive approach suggests an attempt to create a governance architecture that maintains democratic legitimacy while adapting to the redistribution of agency that AI systems inevitably introduce into governance processes.

The initiatives described above establish the formal governance architecture for GenAI in NSW public administration. The following sections will examine how these frameworks operate in practice, analysing their effectiveness in maintaining democratic legitimacy, accountability, and human oversight as AI systems become increasingly integrated into governance processes. Particular attention will be paid to how these formal structures address—or fail to address—the potential for breakdowns in AI-enabled governance systems and the mechanisms available for repair when such breakdowns occur.

Algorithmic models, including GenAI, raise essential considerations regarding how training data biases may influence public service outputs and potentially undermine democratic values of fairness and equality (Anthis et al., 2024; Brown, 2024; Gutiérrez, 2023). To counteract these democratic risks, the NSW Government has adopted AI Ethics principles and policies that stress fairness, inclusivity, and accessibility (Digital.NSW, 2024a). These principles require GenAI systems utilized by the government to incorporate input from diverse stakeholders and comply with anti-discrimination laws. However, in many senses, these principles represent normative aspirations rather than enforceable governance mechanisms, raising questions about their effectiveness in preventing outcomes that might be inherent to LLMs and GenAI systems (Anthis et al., 2024). The challenge is not merely ethical but fundamentally political: How can algorithmic systems respect democratic values and remain accountable to citizens rather than simply efficient at administrative tasks?

GenAI’s reliance on extensive datasets that may include personal information also raises serious privacy concerns implicating citizen rights in democratic societies. The data-driven nature of AI can amplify privacy risks, mainly if sensitive data is processed without proper safeguards, potentially undermining citizens’ privacy rights and autonomy. The Office of the Australian Information Commissioner (OAIC) has underscored privacy as a key concern in AI use. A 2023 OAIC survey revealed that many Australians feel uneasy about government agencies processing personal data with AI, highlighting the need for stringent privacy protections that respect democratic rights to informational self-determination (Pane, 2023).

While NSW’s AI Ethics Policy and Assurance Framework represent significant steps toward responsible AI governance, their adequacy for maintaining democratic legitimacy in increasingly autonomous AI systems requires systematic assessment. The NSW AI Ethics Policy establishes eight core principles to guide AI development and use: transparency, fairness, accountability, privacy, security, reliability, contestability, and public benefit (Digital.NSW, 2024a). However, when explicitly evaluated for their capacity to ensure democratic governance of GenAI systems, several strengths but also limitations emerge:

The NSW AI Assurance Approach (Digital.NSW, 2024d) is designed to operationalize the ethics policy through a structured assessment process for AI projects. However, its capacity to ensure democratic oversight and political accountability varies significantly across different dimensions, as shown in Table 2.

This assessment reveals that while the NSW AI Assurance Framework provides a reasonable foundation for technical governance, significant gaps remain in its capacity to ensure democratic legitimacy and political accountability. Most notably, the framework lacks explicit provisions for citizen participation in AI governance, has limited mechanisms for independent oversight beyond departmental self-assessment, and has insufficient attention to how AI-influenced decisions remain politically accountable in democratic terms.

When compared to emerging democratic AI governance models internationally, NSW’s approach demonstrates both strengths and significant areas for improvement. The European Union’s AI Act, for example, provides more robust democratic safeguards through a detailed risk classification system, stronger transparency requirements, and more precise political accountability mechanisms for high-risk AI systems (Wagner et al., 2023; Union, 2024). Similarly, the UK Government’s Pro-Innovation Approach to AI Regulation addresses generative AI more explicitly, with specific provisions for maintaining democratic oversight and ensuring political accountability (GOV.UK, 2023).

Critics like Acemoglu have argued that effective democratic AI governance requires not just ethical principles but robust institutional mechanisms to ensure that AI development serves broader social welfare rather than narrow technical or bureaucratic interests (Acemoglu, 2021a; Acemoglu, 2021b). Against these critiques, NSW’s framework does demonstrate a commitment to human oversight but lacks specific democratic institutions to ensure political accountability and citizen participation in AI governance.

Similarly, Marcus has emphasized the need for governance frameworks to specifically address the limitations of LLMs, including their propensity to generate plausible but false information that could undermine informed democratic deliberation (Marcus, 2023; Marcus, 2024). The NSW framework acknowledges these concerns but provides limited practical mechanisms for ensuring the accuracy of AI-generated content used in governance processes.

For NSW to maintain democratic legitimacy as GenAI becomes more deeply integrated into governance processes, its framework will need to evolve beyond technical and ethical considerations to address the political dimensions of AI governance more directly. This includes establishing clear lines of political accountability for AI-influenced decisions, creating meaningful opportunities for citizen participation in AI governance, and ensuring that democratic values of transparency, deliberation, and contestability are not sacrificed for algorithmic efficiency.

To effectively uphold democratic principles amid the growing influence of GenAI, governments must design systems that align with ethical standards and remain responsive to the political consequences of their deployment. As the following section illustrates, the moments when systems falter—whether through bias, opacity, or procedural disruption—often expose the fragility of existing governance frameworks. These breakdowns serve as critical entry points for understanding how socio-technical failures can undermine political legitimacy and where targeted repairs might reinforce democratic accountability (see Figure 2).

The Revenue NSW automated debt recovery system, operational from 2016 to 2019, represents a significant case of digital algorithm-enabled administrative failure within NSW. The system was designed to automate the recovery of unpaid fines and debts through mechanisms including bank account garnishment and property seizures.

The system employed relatively simple automation rather than sophisticated AI, yet its implementation illuminates critical vulnerabilities relevant to more advanced GenAI applications in democratic governance:

The Revenue NSW case demonstrates how algorithmic systems can undermine democratic governance when technical implementation fails to incorporate procedural fairness, transparency, and accountability. Although the current NSW AI Ethics Policy explicitly addresses many of these concerns, this example serves as a cautionary tale about the gap that can emerge between policy principles and technical implementation—a gap that could widen with more complex GenAI systems.

The federal ‘Robodebt’ scheme (officially the Online Compliance Intervention) represents one of Australia’s most significant examples of algorithmic governance failure with profound political implications. The scheme operated from 2016 to 2019 and used an automated system to identify and recover alleged welfare overpayments by matching income data from the Australian Taxation Office with welfare payment records from Centrelink (Services Australia).

The Robodebt case illustrates how algorithmic systems can undermine fundamental democratic principles when implemented without appropriate political oversight:

The Royal Commission’s findings highlighted the scheme’s technical and political failures. The algorithm’s flawed assumptions interacted with organizational culture, political priorities, and power imbalances to create a system that was resistant to democratic oversight and accountability (Podger, 2023).

While Revenue NSW and Robodebt employed relatively simple algorithms compared to contemporary language models, their failures provide crucial insights for GenAI governance in democratic systems. Table 3 maps specific failure modes from these cases to potential GenAI risks and corresponding governance safeguards necessary to maintain democratic legitimacy.

This comparative analysis table reveals that while NSW’s current AI governance framework addresses some of these risks in principle, significant gaps remain in practical implementation mechanisms. In particular, the framework provides limited guidance on:

After considering previous breakdowns and gaps in the frameworks, the concept of ‘articulation work’—ongoing human efforts to adapt systems to real-world situations and contexts (Star, 1999)—provides a valuable framework for addressing the gaps identified earlier. In the context of GenAI governance, articulation work involves several critical dimensions that maintain democratic legitimacy in technically mediated governance systems:

Drawing on Jackson (2014) framing of repair as a creative and transformative process, several potential repair mechanisms emerge as particularly relevant for maintaining democratic legitimacy in NSW’s GenAI implementation:

NSW’s current AI governance framework addresses some of these elements, particularly through its emphasis on monitoring and evaluation. However, the specific mechanisms for capturing, analyzing, and learning from GenAI failures remain underdeveloped. As the government advances its GenAI implementation, strengthening these repair mechanisms will be crucial for building resilient and trustworthy AI systems that maintain democratic legitimacy.

The lessons from Revenue NSW, Robodebt, and international cases underscore that technical systems cannot be separated from their social, organizational, and political contexts. Effective GenAI governance requires attention to technical specifications and the socio-technical networks in which these systems operate. As NSW continues to integrate GenAI into governance processes, maintaining this socio-technical perspective will be essential for anticipating vulnerabilities and developing effective repair mechanisms when breakdowns occur in ways that maintain democratic accountability, procedural fairness, and political legitimacy. The next section of this paper will engage with discussions and policy recommendations based on this broad analysis.

Given that GenAI implementation in NSW is at an early stage with limited operational history, these scenarios provide a structured approach to anticipating how breakdowns might manifest and how governance networks might respond. The scenarios were developed based on the socio-technical dynamics identified through ANT mapping, drawing on historical patterns of algorithmic failure while accounting for the unique characteristics of GenAI systems.

Each scenario explores a different type of potential breakdown, examining how actors within NSW’s governance network might realign or conflict in response and identifying potential repair mechanisms that could maintain democratic legitimacy. These scenarios are not predictions but analytical tools that illuminate potential vulnerabilities and governance challenges that might otherwise remain hidden until real-world breakdowns occur.

Drawing from the three scenarios of potential GenAI breakdown in NSW governance, a regulatory sandbox (Allen, 2019; Zetzsche et al., 2017) approach emerges as a promising framework for balancing innovation with democratic safeguards. This approach builds directly on the insights gained from analyzing how LLM-generated errors in policy development, public backlash to AI-driven services, and governance conflicts over AI-generated administrative decisions might manifest in practice.

The regulatory sandbox creates controlled testing environments where innovative GenAI applications undergo time-limited implementation with modified regulatory requirements while maintaining core democratic protections. In response to the scenario of LLM-generated errors in policy analysis, the sandbox implements assessment criteria that specifically evaluate how GenAI systems interpret and aggregate citizen input, with verification protocols that compare AI-generated summaries against original submissions. This addresses the epistemic challenges and representation risks identified when AI mediates between citizen voices and policy decisions.

Transparency mechanisms within the sandbox framework directly respond to public backlash by establishing visibility into GenAI systems’ operations. The framework requires documentation of training data sources and limitations, mandates clear disclosure when citizens interact with AI systems, and emphasizes accessible explanations of AI-influenced decisions. These measures create the conditions for early identification of potential bias patterns before they trigger the cascading trust failures depicted in the scenario analysis.

Human oversight protocols address the vulnerabilities in the administrative decision-making scenario, particularly the risk of automation bias under efficiency pressures. The sandbox establishes minimum standards for human review of AI-generated content, creates monitoring systems that track review time and modification rates, and implements periodic audits that test the quality of oversight through deliberate error introduction. These measures maintain the balance between efficiency gains and procedural fairness requirements.

Democratic accountability structures within the sandbox respond to the diffusion of responsibility identified across all three scenarios. An independent oversight body reviews implementations within the sandbox environment, regular public reporting creates transparency into the experimentation process, and statutory review rights ensure citizens can contest AI-influenced decisions. These mechanisms prevent the accountability gaps that emerged when responsibility became distributed across technical systems and human operators.

The sandbox approach fundamentally depends on the participatory mechanisms that were notably absent in the scenario breakdowns. It incorporates citizen consultation on significant AI deployments, equity impact assessments to evaluate potential disproportionate effects on marginalized communities, and feedback channels for reporting concerns. These participation structures ensure that repair mechanisms are built into the sandbox design rather than improvised after breakdowns occur.

NSW can create controlled conditions for testing GenAI applications that anticipate and mitigate the specific vulnerabilities identified in the scenario analysis through this regulatory sandbox framework. This approach enables the validation of governance mechanisms before permanent implementation, generating valuable evidence about effective oversight approaches while maintaining democratic legitimacy during the innovation process.