AI for Social Science Research

Overview: Introduces the foundational idea of using Large Language Models (LLMs) to simulate human populations, replacing or augmenting traditional human survey respondents.

• Source that introduces it: "Out of One, Many: Using Language Models to Simulate Human Samples" establishes the premise of "silicon sampling" and the concept of "algorithmic fidelity"—the degree to which a model accurately reflects relationships between ideas, demographics, and behaviors within real subpopulations.
• Source that deepens it: "HUMANLM: Simulating Users with State Alignment Beats Response Imitation" deepens this by moving beyond surface-level text imitation. It introduces a reinforcement learning framework that aligns LLMs with underlying psychological states.
• Source that challenges it: "Digital Twins as Funhouse Mirrors: Five Key Distortions" and "Not Yet: Large Language Models Cannot Replace Human Respondents for Psychometric Research" challenge this practice by demonstrating that LLMs often distort human behavior through stereotyping, hyper-rationality, and representation bias.

Overview: Explores how well LLMs can adopt specific personas and whether they accurately reflect diverse demographic identities without reducing them to stereotypes.

• Source that introduces it: "Position: LLM Social Simulations Are a Promising Research Method" outlines the baseline framework while identifying five challenges: diversity, bias, sycophancy, alienness, and generalization.
• Source that deepens it: "Evaluating Silicon Sampling: LLM Accuracy in Simulating Public Opinion on Facial Recognition Technology" applies persona prompting across models like GPT-4o, Claude, and DeepSeek to match multinational surveys.
• Source that challenges it: Detailed analysis points out that LLMs suffer from misportrayal, group flattening (erasing subgroup heterogeneity), and identity essentialization.

Overview: Focuses on the practical and statistical guidelines required to integrate LLMs into robust empirical research, replacing tasks like qualitative hand-coding and econometric estimation.

• Source that introduces it: "Qualitative Coding with Generative Large Language Models" illustrates emulating traditional qualitative alternating deductive/inductive analysis.
• Source that deepens it: "Large Language Models: An Applied Econometric Framework" divides LLM tasks into "prediction" (no training leakage validation) and "estimation" (requiring human-validated samples to correct errors).
• Source that challenges it: "The threat of analytic flexibility in using large language models to simulate human data" warns that minor defensible choices in prompting drastic alters synthetic datasets, creating a threat to validity.

Overview: Examines whether LLMs genuinely possess a "world model" of human psychology and physics, or if they are simply executing task-specific pattern matching.

• Source that introduces it: "A foundation model to predict and capture human cognition" introduces "Centaur," fined-tuned on 160 psychological experiments to predict human behavior.
• Source that deepens it: "Playing repeated games with large language models" tests LLMs in scenarios like the Prisoner's Dilemma, showing rational self-maximization distinct from human "unforgivingness".
• Source that challenges it: "What Has a Foundation Model Found?" and "Potemkin Understanding in Large Language Models" demonstrate that LLMs memorize task-specific heuristics, exhibiting internal incoherence ("Potemkin understanding").

Overview: Investigates the frontier of using LLMs not just as data sources or classifiers, but as active participants in generating novel scientific theories and hypotheses.

• Source that introduces it: "AI Co-Scientist: Augmenting Scientific Discovery and Innovation" introduces a multi-agent system utilizing "generate, debate, and evolve" to propose novel hypotheses.
• Source that deepens it: "Theorizing with Large Language Models" uses a Generative Agent-Based Engine (GABE) to run complex interactions in silico.
• Source that challenges it: "Emergent LLM behaviors are observationally equivalent to data leakage" critiques the premise by arguing AI insights are simply regurgitated from training corpora.

Begin the unit by introducing students to the basic mechanics of how LLMs are currently being used to generate data. Start with Concept 1 to explain "silicon sampling", "algorithmic fidelity", and "state-aligned simulators" like HUMANLM. Transition to Concept 2 to explore prompting specific demographics and identifying structural biases like misportrayal and the "funhouse mirror" effect.

Teach methodological responsibility. Show how LLMs replace qualitative human coders. Introduce the Applied Econometric Framework to teach the necessity of guarding against training data leakage and utilizing human-validated anchor samples (Rectification). Utilize "analytic flexibility" to emphasize pre-registration and rigid standards.

Shift to psychology and computer science. Have students analyze how models simulate game theory, and then shatter the illusion with "Potemkin Understanding." Teach students that accurate prediction does not ensure a coherent internal "world model".

Conclude the unit by looking at AI as a peer (AI Co-Scientist and GABE). Facilitate a debate on epistemological limits—are LLMs capable of true scientific discovery, or are they stochastic search engines rearranging human history?

• Learning Objective: Students will define "silicon sampling," analyze how LLMs act as simulated socio-economic agents, and differentiate text imitation from state alignment.
• Key Concepts: Homo Silicus, Algorithmic Fidelity, Retrodiction, State Alignment (HUMANLM).
• Real-World Analogy: The Virtual Focus Group: Feeding 1,000 backstories to an LLM to predict specific demographic polling responses.
• Discussion Question: If an LLM can accurately "retrodict" public opinion from 1972 to 2021, does that mean the model genuinely understands human opinions?

• Learning Objective: Students will critically assess LLM personas, identifying how models distort, essentialize, or flatten identities.
• Key Concepts: WEIRD Bias, Misportrayal, Group Flattening, Five Key Distortions of Digital Twins.
• Real-World Analogy: The Funhouse Mirror: Resembling the shape of reality but grotesquely exaggerating features while erasing individual nuance.
• Discussion Question: Given pervasive misportrayal, is it ever justifiable to use LLMs as stand-ins for marginalized populations?

• Learning Objective: Apply econometric frameworks to utilize LLMs in empirical research, distinguish prediction vs. estimation, and implement debiasing.
• Key Concepts: Prediction vs. Estimation, Validation Samples and Rectification, Analytic Flexibility.
• Real-World Analogy: The Automated Apple Sorter: Validating an automated sorter output by manually checking 100 apples to establish statistical correction metrics.
• Discussion Question: Should we enforce standardized LLM configurations across science, or does that stifle innovation?

• Learning Objective: Evaluate whether LLMs possess genuine cognitive world models using game theory and psychometrics.
• Key Concepts: Foundation Models of Cognition (Centaur), Repeated Games behavior, Potemkin Understanding.
• Real-World Analogy: The Movie Set Facade: From the street (benchmarks), the facade looks real, but open the door and there is no underlying structure.
• Discussion Question: If a model predicts human behavior perfectly without an internal world model, does its "Potemkin illusion" matter empirically?

• Learning Objective: Synthesize the unit by exploring multi-agent scientific systems and investigating the emergence vs. leakage boundary.
• Key Concepts: Generative AI-Based Experimentation (GABE), AI Co-Scientist System, Data Leakage vs. Emergence.
• Real-World Analogy: The Scientific Debate Tournament: Autonomous agents pitching, critiquing, debating, and evolving novel hypotheses like an accelerated academic conference.
• Discussion Question: How can we definitively prove whether an AI has engaged in pure scientific discovery compared to uncredited data leakage from its training?