© Copyright 2026 Mastersinlibraryscience.org.
A research-methods guide to when synthetic participants work, where they fail, and why
Simulated users are becoming the shortcut library teams take when testing AI search and chat tools, and that shortcut has limits. Real patron studies require recruitment, scheduling, consent, and often institutional review board approval before one query is logged. An LLM can produce synthetic queries in hours.
The University of South Carolina's College of Information and Communications has put that trade-off under study: a 2026 news item describes professor Souvick Ghosh comparing human and simulated users in AI research.
Simulation answers some evaluation questions well and others poorly, so the checklist you validate against matters more than the simulator you choose. A synthetic patron rarely gives up and walks away, which is the very behavior libraries most need to observe.
Three labels now circulate for AI stand-ins in user research, and published studies often neglect to say which one they used.
Nielsen Norman Group has published guidance on each, and the boundaries between them separate a hypothesis from a measurement.
A synthetic user is an AI-generated profile that mimics a user group and produces artificial findings without any real people being studied. NN/g is blunt that this is not user research. The legitimate uses are narrow: generating hypotheses, checking the direction of broad patterns, and drafting proto personas and early user-flow maps that get revisited once real-user research exists.
A digital twin models a specific individual rather than a group.1 Interview-based twins, built from rich interview transcripts, exceeded 80 percent accuracy on survey tasks in NN/g's reported comparisons, including the General Social Survey and Big Five measures, and outperformed persona-based and demographic-based models. NN/g calls that result impressive but does not extend the figure to every task, population, or setting. Bias persists: one study predicted white participants better than other racial groups, with accuracy varying by socioeconomic class and political views.
The middle case is built from a brief paragraph a participant wrote after an interview. It represents an individual or type through a short profile, with no extensive interview material, observed history, or complete responses behind it.
How far you can trust a simulation tracks how much real human data sits behind it. NN/g found that construction method drives accuracy, and that the simplest approach tested, augmenting a prompt with rich interview data, outperformed more elaborate alternatives.
Synthetic users and digital twins differ in what they are built from: a synthetic user represents a group from a description, while a digital twin represents one person from data about that person.
At the University of South Carolina, Souvick Ghosh collected the same conversational search data twice: once from real people, once from LLM-generated stand-ins. The College of Information and Communications news item poses the question any library search evaluator eventually faces: when can a simulated participant legitimately replace a human one?
Ghosh, a faculty member in the College of Information and Communications, compared conversational-search data from real users and from simulated users. The practical difference was speed and cost. Recruiting and scheduling real participants was slower and more cumbersome, while simulated users produced thousands of conversations quickly. The analytic difference mattered more: the simulated users diverged from real ones in several ways, including a higher reading level and a wider vocabulary.
That divergence raises a question about who gets left out. Ghosh's reported concern is that systems tuned on simulated-user data may be optimized for a narrower population, potentially disadvantaging people with lower reading proficiency or people searching in a second language.
The 2026 news coverage reports these comparative findings but cites no peer-reviewed publication from the project, and it gives no sample sizes, participant demographics, statistical tests, or exact model names. It is best cited as a research-methods signal, not a settled result.
Continuing work is planned as a pilot on Bridges-2, a high-performance supercomputer at the Pittsburgh Supercomputing Center, backed by a National Science Foundation ACCESS Explore allocation of 200,000 credits for GPU time and storage. The project is titled "Measuring the Validity of LLM-Simulated Users in Conversational Information Retrieval Evaluation."
South Carolina's School of Information Science houses an MLIS program, so this is a methodological argument being made from inside library and information science, not imported from computer science alone.
Why would a library team test a search interface with made-up users instead of real ones? The practical answer is usually time. Recruiting patrons for repeated evaluation rounds is slow and hard to sustain, especially for small library teams without a dedicated research staff. You need people, a schedule, a consent process, and enough incentive to bring them back for round two. When a vendor ships an update mid-semester, that timeline collapses.
Simulation buys speed, cost, and scale at once. A synthetic set of personas can run hundreds of query variations overnight, which is useful for conversational search, catalog interfaces, and reference tools that would otherwise get a dozen human testers at most.
Reasonable uses tend to share a shape:
Simulated users are decent at ranking systems against each other and risky at predicting how any single real person will behave. Use them to decide which version deserves a human study, not to conclude what patrons need. Keep the questions broad, treat the outputs as hypotheses, and let real users settle anything that turns on emotion, confusion, or context. That keeps simulation in the role it performs best: a filter before contact, never a substitute for it.
Researchers call the distance between how an AI-simulated user behaves and how a real person behaves the "Sim2Real gap": a tool can look strong when tested against simulated users and then stumble with actual people. None of the studies below were run in libraries. They come from general AI agent and conversational system research, so treat them as warnings for catalog search and reference chat testing, not as measurements of library patrons. On the question of how accurate AI simulated users are, the answer depends on which behavior you measure. Task outcomes can look reasonable while conversational behavior drifts far from what real people do, and for library settings the most consequential gap is miscalibrated success estimates, because simulated success rates in these studies landed well above or well below the human result depending on the simulator and prompting used.
| Behavior gap | What simulated users tend to do | What real users tend to do | Why it matters for library testing |
|---|---|---|---|
| Cooperation (Sim2Real gap study) | Were excessively cooperative, creating an "easy mode" that inflated agent success rates above the human baseline | Were less uniformly cooperative than the LLM simulators | With 451 real participants and 165 tasks, the study suggests evaluations built only on cooperative simulators may overestimate real-world performance |
| Frustration, ambiguity, and style (Sim2Real gap study) | Were stylistically uniform and lacked realistic frustration or ambiguity | Showed more realistic frustration and ambiguity | Patrons who are unsure or frustrated may be underrepresented when a search or reference tool is tested only on simulators |
| Conversational initiative (Lost in Simulation) | Asked questions much more frequently and drove conversations forward more proactively | Showed different conversational initiative; real users assigned blame to the agent at a rate of 24.5% | Question-heavy simulated behavior can make interactive systems look more effective than they are with real patrons |
| Grounding in real behavior (RealUserSim framework) | Cooperative simulators overstated performance | The framework drew on behavioral data from 7,273 real users, represented through 48 binary linguistic traits and eight behavioral archetypes | Behavior-grounded simulation surfaced failure mechanisms tied to a mean task-success degradation of 3.2% to 3.5%, which cooperative simulators missed |
| Overall conversational behavior (PRISM-X) | Diverged significantly from real humans across all six measured dimensions of conversational behavior | Used their own framing, analyzed through short topic words ("atoms") without merging synonyms or standardizing vocabulary | A simulator that differs on every measured dimension may produce unreliable conclusions about patron behavior even when task outcomes look acceptable |
| Success-rate calibration (CUEing User Simulators evaluation) | Reported success varied widely by method: 0.18 for RealUserSim, and 0.70, then 0.67, then 0.89 across the reported S2R-N sequence | Human success was 0.61 | Simulator choice and prompting can substantially shift the estimated performance of the system being tested |
Simulation can rank systems well and still misjudge the patron who is actually stuck, so simulated users narrow the question rather than replace the person at the desk.
Is a study that uses simulated patrons automatically free of the ethics questions that come with human participants? No. Several concerns apply, and on some of them the field has no settled standard yet.
A simulated user is only as stable as the model and prompt behind it. Swap the model, update a version, or change a temperature setting, and the "patron" may behave differently. Report the model name and version, the full prompt or persona template, the settings, and the dates of the runs. Without those details, another team cannot check your results, and a claim that simulation matched human behavior may not hold a few months later.
Default model outputs tend to flatten difference. A simulated patron may write fluent standard English, state needs tidily, and ignore the ways real people search when they use another language, rely on assistive technology, or struggle to put a question into words. Multilingual and accessibility needs deserve explicit testing against real people, not an assumption that a persona prompt covers them.
The risk rises when personas are built from real transcripts, search logs, or interviews. The American Library Association's Code of Ethics and its privacy interpretation of the Library Bill of Rights commit libraries to protecting what users seek and receive.1 ALA privacy guidance points toward anonymous or aggregated data for analysis, minimum necessary retention, and explicit permission before sharing with third parties.1 Its AI guidance keeps patron data under library control and prohibits entering personally identifiable information into unapproved AI tools.2 Earlier ALA commentary on big data also urged avoiding linked transaction-level data unless the purpose is limited, and called for consent when data are linked.3
Removing names is not enough. A persona drawn from linkable search or reference activity can still carry sensitive detail, even if it is labeled "synthetic." Consent given for ordinary library service does not automatically cover later persona building, so disclosure should name the secondary purpose, the source data, retention, and vendor access.1
No categorical IRB exemption for synthetic participants has been established.2 Review outcomes appear to turn on whether the source data are identifiable, whether the study counts as human-subjects research, and whether living people are involved.1 Wholly simulated personas with no real-person data raise different questions from those built on patron records. Check with your own IRB and data-governance office early, and document your reasoning.
Before simulated users shape a decision about a catalog, discovery layer, or reference chat tool, run through these checks. Each one tests whether the simulation actually resembles the patrons you serve.
Building AI evaluation skills inside an MLIS program comes down to one habit: asking what a system answered, for whom, and how you would prove it. The coursework that supports that habit clusters into five areas: information retrieval, user research and information behavior, evaluation metrics, research ethics, and enough Python to inspect your own data.
Collect 15 to 20 reference questions your library actually receives. Run each one through a chatbot and record the answer. Then have a librarian review the same answers and log every correction. Compare results before and after review, and sort the errors into a few plain categories: wrong fact, missing source, missed intent. That small design teaches metric definition, coding judgment, and where synthetic evaluation quietly breaks down.
If you have never opened a notebook, mastersinlibraryscience.org's Python for Librarians guide is a practical first step before any of the above.
A vendor's simulated-user study is a starting point for questions, not an answer you can hand to your community. When a library considers AI search or chat tools, the procurement conversation should include a direct question: how was this system validated with real users, and what mix of simulated and human testing produced the results in the demo?
Even with a simulated pilot in hand, plan a small usability test with real patrons before launch and repeat it a few months after. Recruit deliberately for the groups simulation is most likely to miss: novices who do not know library vocabulary, multilingual patrons, and patrons using assistive technology. These are exactly the users a synthetic persona tends to smooth over.
The earlier evidence suggests simulation can rank systems reasonably well while misjudging the patron who is actually stuck. Keep a human channel open alongside any AI tool, staff the reference desk for the questions automation handles badly, and treat local testing as ongoing rather than a one-time gate.
For a wider view of how academic libraries are adopting generative AI, see our related article on academic libraries and generative AI usage.
