AI systems are getting good at sub-contracting to make themselves better:…AI can now productively spend money to improve itself…Researchers have developed FrontierML, a benchmark “that measures how well agents turn capital into model performance on challenging, end-to-end machine learning tasks.” It shows that today’s frontier systems are already able to, when given some money, turn this into improving their own performance on a basket of 21 distinct tasks.21 long-horizon tasks “inspired by real work at frontier labs, neolabs, and academic institutions”. These tasks span six category areas, including LLM training, efficiency, vision & audio, computer use, applied NLP, and math & games. “Each task provides the agent with the same setup: a 12-hour CPU sandbox, a wallet with $10, $50, or $200, a few validation samples, and a task description.” Example task: Agents try to solve the tasks in pretty different ways. E.g, for one task about generating screenshots, the top-scoring agent fine-tuned a diffusion model using LoRA, while other high-scoring approaches involved an LLM writing a drawing program that is then rendered.Key features:
Scaling spending: “Give agents $10, $50, and $200 per task to spend on training compute and data, and evaluate whether agents can use the additional capital productively,” they write. “In our experiments, we find that stronger agents are able to reliably improve their submissions with greater access to capital.”
Open-ended & end-to-end: “In each task, agents can choose architectures, source data, implement training, provision compute, evaluate checkpoints, and optimize inference,” they write. One notable observation is that agents do quite a lot of novel stuff; “in 28% of trajectories, agents add newly trained parts to pretrained models or train a new model from scratch.”
Decent verification: “Agents submit an inference server that serves their model on an air-gapped H100 machine,” they write. “To disincentivize overfitting, we provide limited validation samples and evaluate on a large out-of-distribution held-out test set, not shown to the agent”.
Source: Import AI — Read the full article on the original site ↗
