AI and the Production of Subjects

**Manav Pamnani

Introduction

Artificial intelligence (“AI”) today has gained immense prominence in almost every sphere of daily life. It is no longer a mere speculative technology operating at the margins of governance but has become an increasingly integrated routine instrument through which human beings are constantly evaluated, classified, and sorted into multiple categories. For example, border enforcement agencies in the United States have implemented large-scale facial recognition tools which enable the recognition of identified “persons of interest” through biometric matching procedures across retained image databases. Further, predictive policing systems refer to geographic risk maps to identify areas which require intensive surveillance and continued patrolling. In fact, a similar practice was followed under British colonial administration itself wherein the Criminal Tribes Act, 1871, was passed which automatically designated certain communities, tribes, and castes, as “crime-prone”, consequently leading to stigmatisation. A similar practice of classification based on identified parameters has continued to exist today with the only difference being a change in the form from humans to AI algorithms. 

Even in the financial sector, these practices exist. For example, machine-learning credit models analyse behavioural and transactional data to classify individuals into categories such as creditworthy, risky, and defaulter, which ultimately determine access to loans and insurance schemes.  Similarly, automated hiring platforms filter job applicants through algorithmic scoring mechanisms that translate curricula vitae, behavioural metrics, and psychometric indicators into employability profiles. The problem with these algorithms is not technological calculation but AI being used to produce administrative identities upon the classification of human subjects. 

Such practices exemplify a shift in the modality of power that existed earlier. Today, decisions that shape important life chances are increasingly mediated through algorithmic classification. Individuals are no longer seen only as juridical subjects endowed with rights but as data composites that have been assigned probabilistic value. In this backdrop, the present article argues that this transformation is best understood through Michael Foucault’s account of subjectification and governmentality. It uses this theoretical lens to show how algorithmic governance does not simply automate existing decisions but reorganises the relationship between knowledge, power, and population. The paper then identifies certain possible adverse consequences of such AI-integration and ultimately conceptualises a few model solutions that should be implemented to foster better and more meaningful AI governance. 

Algorithmic Governmentality and the Production of Subjects: A Foucauldian Reading of AI as a Technology of Rule

Foucault’s analysis of modern power departs from the strict top-down classical image of sovereignty as the right to command and punish. His main argument is that power has become decentralised and focused on the regulation of life rather than its blanket negation. It therefore operates through capillary, dispersed, and normalisation-oriented techniques. His central concern is the historical process through which human beings are made into subjects, an analysis which delves deep into how forms of knowledge and institutional practice produce identities that individuals both inhabit and are constrained by. Subjectification in practice operates through a double-layered mechanism. Individuals are objectified within fields of knowledge (as the mad, the criminal, the delinquent, the healthy, and the normal) and are simultaneously compelled to recognise themselves within those categories. This reveals that power in this sense is productive rather than repressive as it was earlier. It creates the kinds of persons that it governs. 

Foucault’s theory of governmentality extends these insights from the individual to the population level. He argues that modern states govern not primarily through law or discipline but through the management of collective processes including health, crime, circulation, labour, and risk, with security becoming the central legitimising principle. Instead of outrightly prohibiting certain acts or phenomena, governmental power calculates acceptable ranges, distributions, and probabilities. Foucault’s coined terminology “technologies of security” refers to the forms of statistical knowledge, demographic analysis, and administrative techniques which aim not to eliminate uncertainty but to regulate it. The theory of governmentality therefore denotes a complex ensemble or web of institutions, procedures, and forms of knowledge, through which the larger population is rendered intelligible and governable. 

Within this framework, technologies are not neutral instruments but are embedded in particular rationalities of rule because they are designed, calibrated, and deployed in accordance with governmental objectives. They do not merely record pre-existing differences among individuals but actively organise social reality by classifying and defining parameters like “normal”, “risky”, “productive”, or “deviant”. The techniques of classification, measurement and comparison produce the very realities they appear to describe. This is because once institutionalised in administrative and decision-making processes, they shape access to resources, opportunities, and rights, consequently bringing into existence the very distinctions they claim to identify. They shape social reality and practice because people often understand themselves in accordance with these algorithmically determined identities. Foucault’s conception therefore shifts the analytical focus by making the decisive question not who or what counts as a legal subject but how regimes of knowledge produce subjects who can be administered and governed. 

Foucault’s works were written in the late 1970s and early 1980s where complicated AI algorithms were absent. Therefore, his perspective does not deal with machines as are in existence today, but the interlinkage between techniques and forms of knowledge which he termed as “technologies of government”. These include statistics, administrative record-keeping, and surveillance which make populations governable. When read in the broader sense, generative AI and large language models can be understood not as neutral computational tools but as extensions of these governmental technologies. Their capacity to aggregate data, effectuate the modelling of behaviour probabilistically, and produce classificatory outputs situates them within a specific rationality of rule because they render individuals comparable and rankable in relation to institutional objectives such as security, efficiency, or risk management. The apparent technical autonomy of such systems obscures the fact that they are embedded in pre-existing regimes of knowledge and power. This is because they are trained on historically produced datasets that have been calibrated according to policy priorities and are deployed in contexts where their outputs acquire administrative force. This is in addition to the programmers’ bias which embeds certain preconceived normative assumptions about what classifications must be made depending on the type of subjects. Such bias is not a technical error but a reflection of the social, institutional, and political priorities within which the system is produced. This further reinforces Foucault’s perspective about technologies of rule being inseparable from the regimes of knowledge that animate them. Therefore, AI algorithms can be conceptualised in Foucauldian terms because they are used as techniques for structuring the field of possible action, shaping how institutions perceive populations and how they are constituted as subjects of governance. Technology today, thus, participates within the overarching government structure that defines what must be known and optimised, and who is to be acted upon. 

Algorithmic Classification and the Production of Governable Subjects

The above discussion indicates that contemporary AI systems appear as a continuation and in fact, intensification of the governmental logic that Foucault had described. Predictive policing models do not simply forecast crime, but they transform heterogeneous social life into data points capable of being distributed across a risk spectrum. For example, when a neighbourhood is deemed a “hotspot” by the AI system, its residents become potential offenders by virtue of their special position within the statistical model rather than because of any specific behaviour or evidence. Consequently, police patrolling is increased, surveillance is intensified, and residents might be stopped or questioned more frequently. They therefore get classified into the “high-risk” category irrespective of any specific proof to that effect. Similarly, credit scoring algorithms translate recorded patterns of consumption, location, and digital behaviour into a numerical index of trustworthiness, completely relying on an automated analysis of past practices instead of specific and relevant present circumstances. Additionally, even in the context of hiring systems, complex human trajectories and character traits are condensed into employability scores and classifications such as “fit” and “unfit” which might not accurately reflect the actual capabilities of the applicants. In all these cases, the stage of classification precedes the decision. The subject is first recognised as a calculable entity and only then subsequently acted upon.

This process entails a double operation of subjectification. On the one hand, individuals are objectified as data that is divisible into attributes, comparable across populations and ranked according to predictive value. On the other hand, the institutional consequences of these classifications shape the social identity of the individuals subjected. For example, being repeatedly flagged as “high-risk” leads to intensified surveillance, reduced opportunities, and, eventually, the internalisation of that status. The algorithmic label that is generated by the programmed system ultimately becomes a lived reality. What initially appears as a technical assessment is, therefore, actually a mechanism of power that distributes visibility, mobility, and credibility unequally. 

Although this paper acknowledges this power dynamic, it does not deny the benefits of algorithmic systems. They are useful in processing vast quantities of information quickly, reduce manual arbitrariness and bias to a great extent, and reveal patterns that often escape human perception and analysis. However, these merits are inseparable from its inherent drawbacks. In addition to the general downsides like the possibility of cyber intrusions, job displacements, and data privacy concerns, there are evident disadvantages that become clear in the context of the above discussion. First, the opacity of machine-learning models makes it difficult to contest the criteria of classification. This in practice means that even if an individual, for example, is classified as “not creditworthy”, that individual has potentially no recourse to challenge the rationale behind the AI system reaching that decision. Second, the blanket reliance on historical data embeds past inequalities within future decisions. This is particularly exacerbated by the programmer’s bias phenomenon which might deviate from the objectivity standard that AI systems aim to achieve, consequently subjecting individuals to automated classifications that might fail to accurately depict the true position or situation of the subjected individual. Third, the probabilistic nature of these systems encourages governance through pre-emption, where individuals are treated according to projected behaviour rather than actual conduct. This results in a security regime wherein the management of risk according to probability and prediction displaces the adjudication of responsibility regime that prevailed earlier wherein responsibility was established only after investigation and examination of evidence. 

Therefore, it is clear that algorithmic governance realises, in a digital form, the governmental rationality that Foucault identified. Power no longer operates only by direct coercion but is present by structuring the field of possible action through AI algorithms which resort to automated classification based on identified parameters. Therefore, today, individuals are governed through their position within a distribution of probabilities instead of being judged solely on the basis of concrete actions or established legal responsibility. 

Re-Politicising Algorithmic Classification: Shift Towards a More Accountable AI Governance Framework

Based on the identified discrepancies above, this section conceptualises a model framework positing a set of solutions which should be implemented to bring about a more transparent and accountable AI governance regime. The problem lies in the production of subjects through an opaque classificatory process and so, going forward, the reform must focus on the conditions under which such classification occurs. It is important to recognise at the outset itself that the potential solutions this section proposes might face implementational difficulties in practice, extending to legitimate administrative concerns such as increased costs, lack of developed resources, technical complexity faced by developers in making the AI systems transparent, and so on. However, despite these hurdles, these solutions mark a crucial starting point and a significant step towards ensuring effective AI governance in the long run.

First, algorithmic systems used in consequential decision-making should be designed around contestability. Individuals adversely affected by the decision of the AI algorithms should have the right to access the data and logic that produced their classification and the right to challenge such a decision before a forum presided by a human. This would ensure that individuals are not forced to accept the output of an algorithmic system but get the opportunity to contest the decision, if sufficient and justifiable grounds are provided. A counter-argument could be that such a mandate could open a floodgate of claims, increasing the already-existing burden of the judiciary. However, the response to this argument is that the admission of these claims would be subject to adequate reasons being provided by the complainant, clearly depicting how the AI classification is misinformed and incorrect. False, malicious, or insufficient claims would be outrightly rejected at the filing stage itself. 

Further, this response is related to the second proposal which provides for meaningful human oversight without accepting the output of the AI system unquestioningly at face value. This would mean a double-review mechanism wherein the AI-generated output is verified by the designated human officials. Such human review should not function merely as a symbolic formality but a substantive reconsideration of the decision, with the power to override algorithmic outputs. This safeguard would avoid the perils of AI being trained on historical data and the programmer’s bias problem identified earlier, giving more fair and just outcomes. A possible counter-argument to this proposal is that a double-review mechanism as conceptualised would neutralise the beneficial effects of the integration of AI in the first place, those being time efficiency, considerable objectivity, and accuracy. However, this can be responded to by clarifying that the second review would not be a completely fresh process or an inquiry which evaluates all relevant parameters from scratch. It would instead supplement the AI output and be an additional superficial examination to merely ensure and verify that the generated output and classification is correct. This would in effect amalgamate the benefits of integrating an AI system while upholding human verification, which is important considering the problems with completely relying upon automated classification, as discussed earlier. 

Third, transparency obligations should extend to the variables and proxies used in the construction of models. This would enable public scrutiny of the norms embedded in technical systems. There are bound to be certain inconsistencies with respect to this proposal, particularly that the general public does not have enough technical competence to understand complex inputs. This issue can be resolved by mandating public disclosure of technical inputs in an easily understandable format. The logic should clearly be explained without delving too much into the inherent technicalities including details of the source code. These transparency obligations would operate in a two-pronged manner. Firstly, AI providers or developers would bear the primary obligations because they understand the system’s architecture and training process the best. They should be mandated to make disclosures through a brief description of the system alongside the logic, inputs, and method of creation. This is similar to the stipulation under the European Union AI Act (“EU AI Act”), wherein providers of high-risk AI systems are mandated to create detailed, up-to-date technical documentation including a general description, development processes, risk management strategies, system evaluation, and training data summaries. Secondly, the users of these AI systems will have transparency obligations. These would include informing the targeted individuals that AI is being used to make decisions about them and that they have the right to contest the automated decisions. Implementing these suggestions would foster a balanced, transparent AI governance framework, ameliorating the major problem of opacity in decision-making. 

Fourth, the deployment of such AI algorithms systems such as biometric and predictive tools, particularly in sensitive areas like policing and border control should be strictly limited to contexts where necessity and proportionality can be sufficiently demonstrated. The necessity test would require that the use of the AI system must pursue a legitimate objective and that no less intrusive alternative means are reasonably available to achieve the same purpose. The proportionality test would further require that the benefits of deploying the technology must outweigh the harm it may cause to individual rights, ensuring that the interference with privacy, autonomy, or civil liberties is not excessive in relation to the intended security objective. The rationale behind introducing the differentiation between sensitive and non-sensitive areas stems from the fact that the decisions or classifications made in sensitive areas affect individuals to a much larger extent, possibly extending to rigorous punishments and even death in exceptional circumstances, whereas non-sensitive areas such as shortlisting of job applicants using AI algorithms do not have as high an impact. This is not to imply that there should be no monitoring in non-sensitive areas which also represent decisions that have the potential to adversely affect an individual’s personal life. 

A monitoring agency with an adequate number of technically trained personnel should be constituted to oversee the functioning of such algorithmic systems in both sensitive and non-sensitive applications. This agency should operate as an independent regulatory body composed of data scientists, legal experts, ethicists, and public policy specialists, consequently ensuring that both technical and normative aspects of AI governance are adequately addressed. Its primary mandate should include conducting periodic audits of deployed algorithmic systems, examination of the datasets used for training, reviewing the variables and proxies relied upon for classification, and assessing whether these systems produce discriminatory or unjustified outcomes. The monitoring agency should also conduct quarterly reviews of major algorithmic decision-making systems and publish detailed public reports on its findings. These reports should include information regarding the categories of decisions taken by the system, the number of individuals affected by automated decisions, the number of challenges filed against such decisions, the proportion of successful appeals, and the corrective measures implemented in response to identified flaws. In addition, the agency should possess the authority to require modifications to the system, suspend the use of algorithms that fail to meet accountability standards, and recommend regulatory action wherever necessary. Alongside this institutional oversight, the government should also be tasked with publicly disclosing the existence and purpose of algorithmic systems used in governance, conducting algorithmic impact assessments prior to deployment, and publishing at least an annual oversight report evaluating the broader societal consequences of algorithmic decision-making.

Lastly, institutional responsibility should remain with and be clearly assigned to human actors only, as is the position in Singapore. The attribution of agency to machines should never serve as a means of diffusing accountability. In practice, this requires the establishment of a clear chain of liability across the lifecycle of an AI system. Developers should be responsible for flaws arising from negligent design, biased datasets, or inadequate testing, while organisations deploying AI systems should remain accountable for flawed decisions made solely on the basis of algorithmic outputs, without the second layer of manual human review, mentioned earlier. Individuals affected by such decisions should have adequate access to remedies, including the right to explanation, the ability to contest automated decisions, and the right to seek damages or compensation from the entity responsible for the loss.

These measures are not mere technical adjustments but represent an attempt to re-politicise the classificatory practices that algorithmic governance tends to naturalise. This conceptualised model aims to make the production of subjects through AI systems visible and contestable, which will ultimately foster a governance regime which protects autonomy instead of merely optimising and reducing the overall risk through blanket automated classification. 

Conclusion

The model conceptualised above furnishes valuable guiding principles which should be incorporated both by the Legislature while drafting any AI-related legislations or amendments in the future as well as policymakers and AI developers and implementers to foster a more effective and balanced AI governance regime. The expansion of algorithmic classification today suggests that, going forward, the governance of populations will increasingly occur through systems that translate human behaviour into predictive datasets. International regulatory developments already indicate that states are beginning to recognise the political implications of such systems. Alongside the EU AI Act provisions discussed earlier, jurisdictions such as Canada have also begun implementing algorithmic impact assessment frameworks to evaluate the societal consequences of automated decision-making before deployment. These developments reflect a growing global consensus that algorithmic governance today cannot remain a purely technical matter. It must instead be embedded within institutional safeguards. 

Considering this evolving landscape, Foucault’s analytical framework is likely to become increasingly relevant rather than obsolete in the long run. Although advancements in technology and AI might result in increased accuracy on part of these classificatory systems, the underlying political logic through which populations are rendered intelligible and administrable is expected to stay unchanged. In fact, the proliferation of algorithmic infrastructure may intensify this dynamic today, despite Foucault’s theory being identified decades earlier. This is because technological advancements will potentially increase the usage of algorithmic systems to make classifications across sectors, reproducing and upholding the underlying political logic. The challenge for the future therefore lies not merely in regulating technology, but in ensuring that the knowledge produced by these systems does not silently redefine the very boundaries of autonomy, responsibility, and citizenship.

**Manav Pamnani is a final-year B.A. LL.B. (Hons.) student at the NALSAR University of Law, Hyderabad.

**Disclaimer: The views expressed in this blog do not necessarily align with the views of the Vidhi Centre for Legal Policy.