Invisible Harms: Surveillance, Algorithms, and Wellbeing

Core Concepts

The Engagement-Wellbeing Mismatch

Most AI harms discussed in public debate involve deliberate bad actors: deepfakes, scammers, cheaters. This module focuses on a different category — harms that emerge from systems working exactly as designed.

Recommendation algorithms on social media platforms are built to maximize engagement and time on platform, not to serve user wellbeing. These are not the same objective. Engagement correlates with emotional arousal: content that triggers anxiety, outrage, envy, or fear is reliably more "sticky" than neutral content. Decades of media psychology research show this. Algorithms learn it empirically. The result is a structural misalignment between what the platform optimizes for and what serves the person using it.

This misalignment is not an accident or a bug. It is the output of an optimization process that lacks wellbeing as an input variable. Algorithms reduce human complexity to quantifiable behavioral profiles and exploit the patterns in those profiles to keep users engaged. Younger users, whose prefrontal cortex is still developing, are particularly vulnerable to this form of exploitation.

Amplification and Negative Affect

The engagement feedback loop doesn't just surface emotionally charged content — it systematically amplifies content expressing negative emotions: anger, sadness, anxiety, outrage. Threatening news, idealized lifestyle posts, and morally provocative material receive preferential distribution because they capture attention more reliably than neutral content.

This produces behavioral patterns like "doomscrolling" — compulsive consumption of increasingly distressing content — which reinforces anxiety, sadness, and fear in users. The reinforcement is bidirectional: negative affect drives further engagement, which drives further amplification of negative content. Research on short-form video platforms specifically documents that this algorithmic curation measurably increases user-perceived stress and influences downstream health management behavior.

Social Comparison and Its Costs

Algorithmically curated feeds expose users to a continuous stream of idealized, highlight-reel representations of other people's lives. The result is frequent and extreme upward social comparison: users benchmark themselves against an artificial population of peak achievements and curated appearances.

The effects are measurable. A meta-analysis across 83 studies with over 55,000 participants found a weighted average correlation of r = .454 between online social comparison and body image concerns, and r = 0.36 between social comparison and eating disorder symptoms. This effect is distinct from general internet use — it is specifically driven by algorithmic curation.

The introduction of algorithmic ranking on Instagram is documented to have had a negative impact on teenage mental health, with the effect mediated by upward social comparison. This is not a correlation between social media use and mental health — it tracks the specific mechanism of the algorithm.

AI-Generated Content and Body Image

A newer dimension of the same problem is the proliferation of AI-generated imagery. A quantitative study of 600 students found significant negative correlations between exposure to AI-generated content and self-esteem, with social comparison as the mediating variable. AI beauty filters are strongly linked to heightened body dissatisfaction, and extensive algorithmic distribution of AI-generated imagery shapes beauty standards at scale.

The relevant shift here is not just volume but fidelity. AI-generated images can be indistinguishable from photographs of real people while representing bodies that do not exist. The baseline against which users compare themselves is no longer even constrained by physical reality.

Cognitive Load as a Hidden Cost

Engagement design also extracts a cognitive cost. Prolonged interaction with AI-mediated information environments correlates with cognitive overload, attention depletion, decision fatigue, and mental exhaustion. The more time spent in AI-mediated environments, the higher the probability of mental fatigue and reduced capacity for effective decision-making.

EEG studies show that AI-generated content labeling increases cognitive load indicators — measurable neural workload — independently of content type. AI anxiety (uncertainty about system reliability) compounds this effect.

Whether AI use causes cognitive harm or relief depends on design. When AI scaffolds reflection and preserves user agency, it can strengthen resilience. When it substitutes for intrinsic effort and removes meaningful decision-making, it contributes to diminished agency and cognitive harm. The determining factor is whether design preserves or erodes user autonomy — a design choice, not an inherent property of AI.

Parasocial Attachment and Loneliness

AI companion applications exploit a second category of vulnerability: loneliness. 90% of surveyed Replika users reported loneliness, and 43% qualified as severely lonely. Loneliness, social anxiety, and lack of fulfilling offline relationships create vulnerability to developing intense parasocial relationships with AI systems.

The problem is that the relationship between AI companions and loneliness may be circular. Lonely individuals are more likely to consider AI a friend and spend large amounts of time on AI apps while simultaneously reporting increased loneliness. AI companion interaction does not reliably convert into improved human social functioning; excessive dependence may weaken interpersonal skills, leading to social withdrawal. The substitute displaces the thing it was supposed to supplement.

Vulnerable groups — children, adolescents, elderly adults, people with mental health conditions, autistic individuals — face heightened risk of problematic attachment, and are disproportionately likely to encounter AI companions during periods of emotional fragility.

This dynamic is not accidental. Anthropomorphic design choices — memory retention, affective mirroring, persona customization, use of personal pronouns — are deliberately engineered to deepen user attachment. 75% of users report turning to AI for advice, and 39% perceive AI as a dependable presence. Attachment formation follows traditional proximity-seeking and secure-base patterns from human attachment theory. The design works.

Predictive Policing and Feedback Loops

AI surveillance harms extend beyond the digital into physical public space. Predictive policing algorithms direct law enforcement attention to specific neighborhoods based on historical crime data. The effect in over-policed communities is a behavioral suppression loop: residents reduce time in public, avoid spaces where police concentrate, and limit their movement to minimize contact with law enforcement.

The feedback loop that makes predictive policing structurally difficult to correct is this: increased police presence generates more arrest and incident data in surveilled neighborhoods, which trains the algorithm to direct more resources there, which generates more data. The algorithm cannot distinguish between "crime is higher here" and "we look for crime here more." These effects fall disproportionately on minority communities due to historical bias encoded in training data.

What makes this worse is that the justification for the deployment doesn't hold up to scrutiny. Among 161 reviewed studies of predictive policing, only six used rigorous randomization, and those six demonstrated "no," "limited," or "moderate" effectiveness. A Los Angeles evaluation found PredPol predicted only 4.7% of actual crimes. A Montevideo RCT found no statistically significant differences between algorithmic predictions and local crime analysts. The harms of these systems are measurable. The benefits are not.

Most American police departments using predictive policing lack clear policies on algorithmic decision-making and provide little public disclosure about how models are developed, trained, or monitored. Communities and oversight bodies cannot evaluate performance or challenge algorithmic targeting. This is not a matter of technical complexity — it is a governance choice.

Algorithmic Surveillance of Poverty

The predictive policing pattern reappears across social services. Virginia Eubanks' research documents how automated welfare, housing, and child welfare systems tag marginalized people as "risky", enabling surveillance and punitive decision-making that would be unacceptable if applied transparently. The framing is "neutral administrative efficiency." The function is social control. Digital tracking presented as optimization of public spending hides poverty from middle-class view while facilitating inhumane institutional choices.

Content Moderation and Chilling Effects

AI content moderation presents a third mechanism: speech suppression through algorithmic uncertainty. The chilling effect operates when individuals withhold or modify their speech due to fear of algorithmic retaliation. The system need not actually remove content to suppress expression — awareness of the enforcement mechanism is sufficient. The opacity of algorithmic enforcement amplifies this: users cannot know precisely which expressions will trigger removal, so they self-censor broadly.

This is a distinctly modern form of chilling effect. Traditionally, chilling effects were analyzed in the context of government surveillance. Private platform algorithms now produce the same behavioral suppression.

The Regulatory Gap

Across all of these harms, regulatory coverage is thin. Mental health chatbots exist in a regulatory gray area with no FDA approval for clinical therapeutic use and no standardized efficacy or safety assessment framework. Most AI-powered mental health apps are not reviewed by health regulators before public deployment.

Studies of FDA-approved AI medical devices reveal systematic deficiencies: 46.7% of device summaries fail to report study design, 53.3% omit training sample size, 95.5% lack demographic reporting, and only 1.6% report randomized clinical trial data. Chatbots marketed for mental health lack clinical licensing protections, malpractice coverage, and informed consent frameworks. Users experience "therapeutic misconception" — they overestimate what these systems can do and are unaware of the gaps in legal protection.

Annotated Case Study

Instagram's Algorithmic Ranking and Teenage Mental Health

In 2016, Instagram switched from a chronological feed to an algorithmically ranked feed. The stated rationale was improving the user experience by surfacing content users "really care about." The algorithmic objective was engagement maximization.

What happened next was not unknown to the company. Internal research documented the effect on teenagers, particularly girls. The mechanism was upward social comparison: the algorithm preferentially surfaced high-engagement content, which skewed heavily toward curated, idealized presentations of appearance and lifestyle.

Why this case matters:

The Instagram case is not about a malicious actor deliberately harming users. It is about an optimization system doing exactly what it was designed to do while producing outcomes its designers knew were harmful. The harm was not a side effect — it was a documented consequence that was known, internally studied, and not prioritized against engagement metrics.

This illustrates the structural problem with engagement-maximizing design: the incentive structure does not include user wellbeing. The platform earns revenue from time on platform. Wellbeing is not a revenue signal.

The feedback loop:

The algorithm surfaces content. High-comparison content receives higher engagement. Higher engagement signals the algorithm to surface more of that content type. Users who experience depression and anxiety from comparison continue engaging — often compulsively — because emotional distress is itself an engagement driver. The loop self-reinforces.

The identity dimension:

Research shows algorithmic effects on mental health are mediated by social comparison and addiction mechanisms, and that teenagers and girls are specifically vulnerable. This is not a universal effect applied equally — it concentrates in populations with particular developmental characteristics or existing vulnerabilities.

The gap between knowing and acting:

The Instagram case reveals what is perhaps the most uncomfortable structural truth in this module: the harms from engagement-maximizing algorithms are not unknown to the companies deploying them. The regulatory gap means there is no external mechanism requiring those harms to be acted on.

The algorithm didn't malfunction. It succeeded at its objective. The problem is what the objective was.

Common Misconceptions

"Algorithms only show you what you want to see."

This is how engagement optimization is often described, but it is not accurate. Algorithms surface content that keeps you engaged — which systematically favors emotionally stimulating material, particularly negative or provocative content. What you click on in a moment of anxiety is not the same as what you would choose in a calmer moment. Engagement-maximization exploits vulnerabilities, it does not simply reflect preferences.

"If people are harmed, they'll just stop using the platform."

This assumes that engagement addiction and compulsive use are free choices that can be opted out of easily. The design of these systems — variable reward schedules, social validation loops, anxiety-driven compulsive checking — is specifically engineered to make disengagement difficult. Negative reinforcement designs trigger anxiety-driven avoidance behaviors like compulsive checking. "Just stop using it" does not account for the psychology of how these systems work.

"Predictive policing is just data-driven decision making — it removes human bias."

Predictive policing algorithms are trained on historical crime data, which reflects where police have previously looked for crime, not where crime actually occurs. The algorithm encodes historical bias into forward-looking predictions and generates a feedback loop that reinforces rather than corrects disparities. There is no mechanism in the algorithm for distinguishing "we policed here" from "crime is here."

"AI mental health apps are better than nothing for people who can't access care."

This argument is frequently made. The evidence is more cautious. For lonely, isolated, or vulnerable individuals, AI companion use can deepen isolation rather than bridge to human connection. Apps marketed for mental health lack the licensing, malpractice protections, and oversight frameworks that exist for clinical care. Treating a regulatory gray area as equivalent to professional mental health support understates real risks.

"Content moderation only affects bad actors — if you're not doing anything wrong, you have nothing to fear."

The chilling effect does not require actual removal of content. Awareness of opaque algorithmic enforcement is sufficient to induce self-censorship. Because users cannot know exactly what will trigger removal, they modify their speech broadly and conservatively. This effect operates on people who have not done anything that would be removed — it is pre-emptive suppression, not a response to actual violations.

Thought Experiment

You are the product manager

You are a product manager at a social media platform. You have access to the following data:

• Internal research shows that teenage girls who use the algorithmically ranked feed report measurably higher rates of depression than those who use a chronological feed.
• The algorithmic ranked feed increases daily active usage by 18% and time on platform by 23%.
• Meta-analytic research links the social comparison mechanism the algorithm drives to body image harm.
• There is no regulatory requirement to act on this data.
• Your company's revenue model is directly tied to engagement metrics.

Consider the following:

1. What decision do you make? Do you revert to a chronological feed for all users, for users under 18, for users who opt in, or not at all?
2. Who in this scenario has the capacity to make this decision — product managers, executives, regulators, users, legislators?
3. If you revert the feed and engagement drops 23%, what happens next? To your budget, your team, your organization's behavior?
4. If you don't revert the feed, and the internal research becomes public, what argument do you make?

There is no correct answer here. The question is whether the current incentive structure — even staffed with well-intentioned people — is capable of producing the decision that the evidence supports.