When Digital Systems Begin to Think for Us

AI, Epistemic Agency and the Sovereignty of the Self

Athirah Nor Kamarudin

Abstract

The first digital divide concerned access to technology. A second concerned the skills required to use it. Artificial intelligence introduces a more difficult divide because contemporary AI systems no longer simply wait for instructions and execute clearly bounded tasks; they increasingly participate in interpretation, recommendation, writing, classification, analysis and decision-making. The policy question therefore changes. It is no longer sufficient to ask whether people can operate digital systems. We must also ask whether they retain the capacity to understand, interrogate and ultimately remain responsible for the judgments produced with them.

This essay develops the idea of sovereignty of the self as a way of thinking about that problem. I use the term not in a legal or territorial sense but to describe an individual’s capacity to remain an active author of judgment: to form reasons, recognise uncertainty, question an answer, preserve knowledge, revise a position and decide when a machine’s recommendation should be rejected. This is closely related to what emerging AI scholarship describes as epistemic agency, while UNESCO’s AI competency framework similarly places human agency and critical judgment at the centre of responsible AI use.

Drawing on my experience in digital transformation, adult training and organisational systems, I argue that AI literacy should be understood as more than prompt-writing competence or familiarity with generative tools. A person may become highly efficient at obtaining answers from AI while simultaneously becoming less capable of evaluating those answers independently. The deeper policy challenge is therefore to develop people and institutions that can work productively with artificial intelligence without surrendering the intellectual capacities upon which meaningful human agency depends.

The invisible handover

Most digital tools ask the user to perform an action. A spreadsheet calculates after someone supplies a formula, an accounting system records what a user enters, and a conventional search engine retrieves material that the person must still interpret. Generative artificial intelligence alters this relationship because it increasingly occupies the space between question and judgment. It can formulate an argument, recommend a course of action, summarise evidence, classify a problem, rewrite language, anticipate objections and present the result in a tone of apparent coherence.

This makes AI extraordinarily useful, but it also changes what is being delegated. When I ask conventional software to calculate a total, I delegate computation while retaining most of the interpretive work. When I ask a generative AI system to explain why revenue has fallen, compare competing strategies, summarise a policy document or determine the strongest argument, I may be delegating part of the process through which a conclusion itself is formed.

The handover is easy to miss because the interaction still feels active. The user types the prompt, asks follow-up questions and chooses whether to accept the response. Yet much of the conceptual labour may already have occurred inside the system before the answer reaches the screen. The structure of the problem, the evidence selected as salient, the categories through which the issue is organised and the language in which the conclusion appears may all have been generated elsewhere.

For an experienced professional, this can be an extraordinary augmentation of capability. AI can expose a neglected angle, organise unfamiliar material or accelerate analysis that the user remains capable of checking independently. For a less experienced user, however, the same system may produce something quite different: an answer that is persuasive precisely because the knowledge needed to challenge it has not yet been developed.

The same interface can therefore augment one person’s agency while quietly substituting for another person’s judgment.

This distinction is fundamental to the future of digital inclusion.

From digital literacy to AI literacy

Malaysia has already travelled a considerable distance through the earlier stages of digital inclusion. Household internet access reached 97.1 per cent in 2025, while rural household access reached 90.7 per cent. Connectivity remains uneven in important ways, but the policy problem can no longer be understood primarily as a question of whether people are online.

The previous generation of digital literacy policy concentrated understandably on access and functional competence: how to use devices, navigate platforms, retrieve information, transact safely and participate in increasingly digital institutions. Artificial intelligence inherits all of these concerns but introduces another level of literacy because the technology does not merely mediate access to information. It increasingly mediates the construction of meaning.

The OECD and European Commission’s 2026 AI literacy framework defines AI literacy through the knowledge, skills and attitudes needed to understand how AI works, evaluate its outputs critically and use it responsibly. UNESCO similarly places a human-centred mindset, ethics, AI foundations and critical judgment within its competency framework, while explicitly asking learners to examine the changing boundary between human agency and machine agency.

This distinction matters because a person can be functionally excellent at using AI while remaining epistemically vulnerable to it. She may know how to obtain a polished report, refine a prompt, automate repetitive work and produce professional language, yet still lack the subject knowledge required to identify when the answer is incomplete, misleading or confidently wrong.

In conventional digital literacy, competence often means knowing how to make the tool do what one intends. In AI literacy, competence must also include knowing when not to accept what the tool has done.

The required skill is therefore not simply operation but calibration.

A capable user must have some basis for deciding when the machine deserves trust, when its answer requires verification, when the framing of the problem itself should be challenged and when judgment should remain substantially human.

The problem of the confident answer

One reason this becomes difficult is that generative AI often speaks in the same confident register whether the underlying answer is strong or weak. Unlike a hesitant colleague who signals uncertainty through tone, an AI-generated response may present a well-supported inference and a fragile speculation with almost identical rhetorical assurance.

Recent research on organisational decision-making identifies this asymmetry as an important source of risk. A 2026 systematic review covering fifteen studies and more than fifteen thousand participants found that generative AI could improve some forms of decision work while impairing others, with the authors highlighting the difficulty users face in calibrating reliance when AI outputs provide weak signals about their own reliability.

This problem is related to automation bias, the tendency to place excessive trust in automated recommendations. Experimental research has shown that users can continue to rely on incorrect AI assistance even when doing so undermines their own reasoning, while OECD work on AI in government similarly warns that apparent machine neutrality can encourage people to accept automated outputs without adequate scrutiny.

The danger is therefore not merely that AI can be wrong. Humans have always worked with imperfect information and fallible advisers. The more interesting problem is that AI can alter the user’s willingness to perform the mental work through which wrongness would ordinarily be discovered.

The user may stop searching because an answer has already arrived. She may stop constructing an argument because one has been generated. She may stop remembering because the information can be retrieved later, and she may stop developing expertise because the system appears capable of reproducing the surface characteristics of expertise on demand.

Each individual act of delegation may be rational.

The cumulative effect deserves much more scrutiny.

Sovereignty of the self

I use sovereignty of the self to describe the ability to remain the author of one’s own judgment while participating in systems that increasingly offer to perform parts of cognition on one’s behalf.

The phrase requires care because sovereignty can imply complete independence, while human knowledge has never been purely individual. We think through language inherited from others, learn through teachers and communities, rely upon institutions, use tools, consult experts and distribute cognition across books, records, technologies and relationships. The goal cannot therefore be an imaginary state in which the individual thinks entirely alone.

The relevant distinction is between assistance and abdication.

A sovereign self can use external intelligence while retaining the ability and responsibility to interrogate it. She may allow a system to extend memory without treating the system as the final authority on what should be remembered. She may use AI to generate possibilities while retaining ownership of the criteria by which those possibilities are judged. She may allow a model to challenge her reasoning without surrendering the responsibility to determine whether the challenge is justified.

Sovereignty, in this sense, means preserving enough internal capability to remain in meaningful relationship with the tool.

Emerging scholarship on AI and learning makes a closely related argument through the concept of epistemic agency. A 2026 paper proposing a theory of epistemic co-agency argues that meaningful human-AI collaboration requires users to reason not only with AI but also through and against it, challenging assumptions and retaining responsibility for knowledge construction.

This is a useful distinction because the future of human intelligence is unlikely to involve rejecting artificial intelligence. The more realistic challenge is learning how to remain epistemically active while surrounded by increasingly capable systems.

The question is not whether we use AI.

The question is what remains ours when we do.

A new form of dependence

Digital dependence has traditionally referred to access: a person who lacked a device, connectivity or the ability to use a system depended on someone else to complete a digital task. AI introduces a more subtle form of dependence because it can arise even among highly competent users.

A person may become dependent not because she cannot operate the tool but because she no longer feels capable of performing the underlying cognitive task without it.

The difference is significant. If I cannot submit a digital form, my dependence is visible. I know that I require assistance, and the person helping me can see the boundary between my capability and theirs. If I routinely ask AI to formulate my arguments, interpret difficult documents, generate my options and evaluate my choices, the boundary becomes less obvious because I remain the person apparently performing the work.

The output still appears under my name.

The cognitive infrastructure beneath it may increasingly belong elsewhere.

This does not mean that using AI weakens cognition automatically. Calculators did not abolish mathematical thought, search engines did not eliminate research, and writing tools have long shaped the way people compose language. Human capability frequently grows through well-designed tools.

The issue is whether the tool creates scaffolding or substitution.

Scaffolding enables a person to perform at a higher level while gradually developing greater independent understanding. Substitution enables the task to be completed while the underlying capability remains stagnant or deteriorates.

The distinction is familiar from education, but AI makes it relevant to professional and civic life as well.

If a junior employee relies on AI to draft every analytical argument, does the system accelerate the development of analytical judgment or allow the appearance of competence to precede the acquisition of it? If a business owner receives an AI-generated explanation of financial performance, does the tool deepen understanding or replace the need to understand? If public servants increasingly rely on automated recommendations, does institutional capacity improve or does expertise become concentrated inside systems that frontline users cannot independently evaluate?

These are not arguments against AI adoption.

They are questions about the kind of capability AI adoption produces.

The paradox of convenience

The more useful AI becomes, the more difficult this problem becomes because good technology reduces friction. Tasks that once demanded sustained effort can increasingly be completed through conversation. A complicated document can be summarised in seconds, a draft can be produced almost immediately, and unfamiliar concepts can be explained without the user navigating several sources independently.

Much of this represents genuine progress. Time previously spent on routine cognitive labour can be redirected toward higher-value activity, while people who once lacked access to specialised assistance can now obtain forms of support that would previously have required significant resources.

Yet friction has never been purely waste.

Some forms of effort are part of how capability develops.

Struggling with a difficult argument can reveal what one does not understand. Writing a first draft forces the writer to decide what she actually thinks. Comparing conflicting sources requires the reader to develop criteria of credibility. Remembering a concept strengthens connections that later allow new information to be evaluated.

When AI removes the effort, it may remove both the inefficiency and the learning process embedded within it.

The policy challenge is therefore more complex than maximising efficiency. A society that optimises every cognitive task for immediate completion may discover that it has weakened the longer-term processes through which independent judgment is formed.

This matters particularly for younger people, early-career workers and people acquiring new forms of expertise because they are precisely the users least equipped to distinguish between productive delegation and premature substitution.

An expert can outsource a task because she knows what competent performance looks like.

A novice may outsource the very experience through which that standard would have been learned.

Knowledge without ownership

My interest in this question comes partly from working at the point where people encounter unfamiliar systems.

In SME training and organisational implementation, I have repeatedly seen that completing a task does not necessarily mean understanding the logic behind it. A participant can learn which sequence of buttons produces the desired result while remaining unable to explain why the process works or how to respond when circumstances change.

Artificial intelligence intensifies this distinction because it can now provide the explanation as well as the action.

A person who does not understand a concept can ask AI to explain it. If the explanation is clear, this can be an extraordinary learning tool. However, the same person can also ask AI to apply the concept, produce the resulting document and justify the conclusion, allowing the entire chain between unfamiliarity and output to be traversed without substantial understanding ever forming.

The result is a peculiar condition: knowledge can appear in the work without becoming knowledge possessed by the worker.

This matters because institutions ultimately depend on embodied human capability. Organisations need people who can recognise when a situation falls outside the pattern, reconstruct reasoning when a system fails, defend a decision, teach another person and adapt knowledge to circumstances that have not yet been encoded into the tool.

A document may survive without its author understanding it.

An institution cannot remain resilient indefinitely if understanding itself becomes externalised.

Reflexivity: I am not outside this problem

Any argument about cognitive dependence on AI becomes suspect if written from a position that pretends to stand outside contemporary AI use.

I do not.

Artificial intelligence is already part of my own working environment. I use computational systems to explore ideas, organise information, question assumptions and accelerate forms of work that would otherwise take substantially longer. I am therefore not observing a technological change from a distance; I am participating in it while trying to understand what that participation is doing to my own habits of thought.

This produces a methodological difficulty because AI frequently makes my work feel better. It can make language clearer, expose a connection I had not initially noticed and allow a complex question to be examined from several directions rapidly. The immediate evidence therefore encourages greater use.

The harder question concerns what cannot be observed as easily: which intellectual muscles are being exercised less often because the system is always available?

I have found this question increasingly important because convenience can conceal dependency. The moment at which a tool becomes difficult to work without may occur gradually, and the user may interpret the resulting dependence as evidence of the tool’s value rather than evidence of a change in her own capability.

Reflexivity therefore requires me to ask not only whether AI improves my output but whether I remain capable of explaining, defending and reconstructing the reasoning behind that output independently.

This is where sovereignty of the self stops being an abstract philosophical concern and becomes an everyday practice.

The institution has sovereignty too

The problem extends beyond individuals because organisations can lose epistemic sovereignty as well.

An organisation possesses more than documents and databases. It contains accumulated judgment about customers, operations, risks, relationships and history. Some of this knowledge is explicit, while much of it remains tacitly distributed among people who know why a particular process exists, when a formal rule should be interpreted cautiously and what previous failures taught the institution.

Generative AI creates powerful opportunities to make this knowledge searchable and usable. An internal AI system can help staff navigate procedures, locate institutional history and interpret large bodies of documentation that would otherwise remain inaccessible.

Yet the same architecture can create a new dependency if the organisation gradually loses the capacity to understand how its own knowledge is represented.

If employees increasingly ask the AI rather than consult source material, if institutional memory becomes accessible primarily through generated summaries, or if important decisions are shaped by models whose retrieval and reasoning processes are poorly understood, the organisation can become informationally sophisticated while losing direct custody of its own reasoning.

The question of AI sovereignty therefore intersects with data sovereignty but cannot be reduced to it.

Owning the server does not guarantee ownership of judgment.

A locally hosted system can still produce cognitive dependence if users treat its outputs as epistemically authoritative.

Conversely, an externally provided system does not automatically destroy agency if users retain strong expertise, meaningful oversight and the ability to challenge its recommendations.

The deeper question is whether the organisation can continue to know what it knows, explain why it knows it and act independently when the automated layer is unavailable or wrong.

Human agency as infrastructure

This suggests that human agency should be treated as part of AI infrastructure rather than as an ethical consideration added after deployment.

Governments and organisations commonly evaluate AI readiness through computing infrastructure, data availability, governance frameworks, technical skills and regulatory capacity. These dimensions are important, but AI readiness should also ask whether the humans interacting with the technology possess sufficient domain knowledge and critical capability to remain meaningfully responsible for what the systems produce.

UNESCO’s competency framework moves in this direction by placing a human-centred mindset and human agency alongside technical knowledge. It explicitly encourages learners to understand the dynamic relationship between machine and human agency rather than imagining that responsible AI use is simply a matter of learning the technology.

The policy implication is significant.

AI training that teaches people only how to generate better outputs may increase productivity while leaving the more fundamental capability problem untouched.

A worker should learn not only how to prompt a system but how to challenge its framing, trace a claim, recognise uncertainty, compare alternative explanations and know when the task requires human expertise that cannot responsibly be delegated.

AI literacy therefore needs an epistemological dimension.

People need to understand not merely what AI can do but what kind of knowledge an AI-generated answer represents.

From human in the loop to human capable of judgment

AI governance frequently invokes the principle of keeping a “human in the loop.” The phrase is reassuring because it suggests that consequential decisions remain under human supervision.

Yet the presence of a human does not guarantee meaningful oversight.

If the person lacks the expertise, confidence, authority or time required to challenge an automated recommendation, the human may function primarily as a ceremonial approver.

A checkbox is not agency.

The important question is whether the human inside the loop retains enough independent capability to disagree.

Research on automation bias demonstrates why this matters. Users often give disproportionate weight to automated recommendations, particularly when systems appear objective or authoritative. OECD analysis warns that excessive reliance on AI in government can diminish human oversight and allow incorrect outputs to propagate through institutional decision-making.

Meaningful human oversight therefore requires more than procedural inclusion.

It requires epistemic independence.

The human must possess an alternative basis for judgment, access to underlying evidence, sufficient understanding of system limitations and institutional permission to reject the machine’s recommendation.

Without these conditions, “human in the loop” can become a phrase describing responsibility without corresponding power.

AI literacy as the defence of authorship

A useful way to understand AI literacy is therefore as the capacity to preserve authorship.

Authorship here means more than writing one’s own sentences. It means remaining responsible for the intellectual act that a piece of work represents.

If I submit an argument, I should be capable of explaining why I believe it.

If an organisation makes a decision, someone should be capable of reconstructing the reasons behind it.

If a student presents an analysis, the analysis should correspond to capabilities the student is actually developing rather than merely to capabilities available through the system.

AI can participate extensively in all of these processes without necessarily undermining authorship.

The boundary is crossed when the person can no longer distinguish between a conclusion she has examined and one she has merely received.

This is why sovereignty of the self cannot be measured by whether AI was used.

The relevant question is whether the user remained epistemically present.

What policy should protect

A serious AI-literacy agenda should therefore protect several capacities simultaneously. People need enough foundational knowledge to evaluate the outputs they receive, enough metacognitive awareness to recognise when they do not know something, enough confidence to challenge an automated answer, and enough opportunity to develop expertise without having every difficult cognitive task prematurely automated.

Educational institutions will have to decide which forms of effort remain developmentally necessary even when AI can perform them more efficiently. Employers will need to distinguish between using AI to amplify expertise and using it to conceal the absence of expertise. Public institutions will need to ensure that citizens interacting with AI-mediated services can understand consequential outcomes and meaningfully contest them.

AI systems themselves should be designed to support rather than suppress these capabilities.

A system intended to augment human judgment should make uncertainty visible, expose relevant evidence, allow competing interpretations to be examined and encourage users to retain responsibility for consequential decisions.

The goal should not be to make AI less capable.

It should be to ensure that increasing machine capability does not require decreasing human capability as its hidden price.

The right to remain intellectually present

The great promise of artificial intelligence is that it can extend human capability. It can make expertise more accessible, reduce unnecessary cognitive burden, help people cross disciplinary boundaries and allow organisations to extract value from knowledge that previously remained inaccessible.

There is no reason to romanticise the inefficiencies that AI can eliminate.

Yet efficiency is not the only value at stake.

A person who can obtain an answer instantly but cannot evaluate it possesses a fragile form of power. An organisation that can generate knowledge rapidly but cannot reconstruct its reasoning possesses a fragile form of intelligence. A society that becomes increasingly dependent on machine-generated interpretation without strengthening human judgment may become technologically capable while epistemically vulnerable.

The next generation of digital policy should therefore move beyond the question of access and even beyond the question of literacy.

It should ask whether people remain capable of participating in the production of their own knowledge.

Sovereignty of the self does not require rejecting artificial intelligence, nor does it require preserving every cognitive task simply because humans once performed it manually. It requires something more demanding: the ability to decide consciously what we delegate, what we retain, and what capacities we refuse to allow convenience to quietly erode.

The most important AI skill may ultimately be neither prompting nor programming.

It may be knowing when to stop asking the machine and think.

Methodological Note

This essay draws upon my professional experience in digital transformation, organisational systems, adult training and technology implementation, together with reflexive observation of my own increasing use of AI-assisted tools. These observations are used to identify analytical questions rather than as representative empirical evidence of Malaysian users or organisations.

My position within this subject creates an important tension because I am both interested in the risks of cognitive dependence and an active beneficiary of AI augmentation. I therefore do not approach technological use and human agency as opposing categories. The central concern of the essay is the quality of the relationship between them: whether AI expands the user’s ability to reason and act or gradually substitutes for capacities the user no longer develops independently.

Future empirical research could examine this relationship through longitudinal observation of students, workers, entrepreneurs and public servants using generative AI in authentic tasks, with particular attention to how repeated AI use changes confidence, domain knowledge, verification behaviour, memory, independent problem-solving and willingness to challenge automated recommendations.

References

OECD & European Commission. Empowering Learners for the Age of AI: An AI Literacy Framework for Primary and Secondary Education. Paris: OECD Publishing, 2026.

OECD. Governing with Artificial Intelligence. Paris: OECD Publishing, 2025.

UNESCO. AI Competency Framework for Students. Paris: UNESCO, 2024.

“Human-AI Agency in the Age of Generative AI.” Information and Organization, 2025.

“Learning with Machines: Toward a Theory of Epistemic Co-Agency.” Computers and Education: Artificial Intelligence, 2026.

“Mitigating Automation Bias in Generative AI Through Nudges: A Cognitive Reflection Test Study.” Procedia Computer Science, 2025.

“Generative AI and Organizational Decision-Making: A Systematic Review of Performance Effects.” Procedia Computer Science, 2026.