‘Silicon samples’, AI-generated synthetic data based on demographic profiles, have been proposed to help plan research. Here, I argue that Monte Carlo simulation already does this job transparently, and that generative AI is better used to help write such simulations.
Two researchers are planning the same experiment. Neither has begun data collection, and both want to adhere to best practices in study planning: specifying in advance the required sample size to achieve sufficient power, informed by a plausible effect size and expected spread of responses. The first writes a Monte Carlo simulation: for each combination of effect size, variance, and sample size, they generate 10,000 complete datasets and apply the planned analysis. They then estimate power for this analysis in each case and choose a sample size that offers at least 80% power across the scenarios they consider plausible. The second prompts a large language model (LLM) to respond to the scales of interest 10,000 times, providing a distinct demographic persona each time. They run their planned analysis on this LLM-generated data, treat the observed effect size as an estimate of the human effect, and use it to motivate a subsequent power analysis. They then adopt the sample size this analysis produces. Which approach provides the more defensible basis for planning?
This question has come to prominence in the social and behavioral sciences, as a growing body of work advocates that “silicon samples” could be useful in research planning1,2. A recent Nature paper demonstrated that LLM-derived predictions correlated strongly with treatment effects across 70 nationally representative survey experiments (although systematically overestimating their magnitudes), with pilot testing as one of the central applications explored3. An accompanying commentary drew the natural conclusion: such models could serve as low-cost tools to assist experimental design4. Even sceptics of silicon samples might grant that they could cheaply help with pretesting or planning of study designs. The question is what status this method has relative to Monte Carlo simulation when determining the design of a study.
Monte Carlo simulation-based study planning5 is routine in the social and behavioral sciences. A researcher first specifies a data-generating process (DGP) they consider plausible for the phenomenon of interest. Relevant quantities might include the assumed effect size, an error term such as ({varepsilon }_{i}, sim {N}(0,,{sigma }^{2})), and dependencies among observations. The researcher then defines plausible levels of the DGP and design parameters and repeatedly simulates complete experiments under each condition. For power planning, the proportion of simulated experiments in which the intended statistical decision is reached estimates power. Comparing conditions identifies the designs that achieve acceptable power across the range of assumptions considered plausible.
Source: Communications Psychology — Read the full article on the original site ↗