Karpathy's autoresearch repository is deliberately constrained: an agent edits a small training surface, runs a timed experiment on target hardware, compares a metric, and keeps or discards the change. That is a powerful loop. It is not proof that any business problem with a number can be safely optimized.
To adapt the pattern, I require a bounded editable surface, an objective metric, a holdout, a cost and risk ceiling, reversibility, and human approval around consequential actions.
The loop
- Freeze a baseline.
- Change one controlled input.
- Run the same evaluation.
- Compare the primary metric and guardrail metrics.
- Keep or discard the change.
- Log the hypothesis, result, cost, and limitations.
Never optimize and evaluate on the same examples if the result will guide a real decision.
Ten experiments
1. Inventory reorder points. Use historical sales to simulate whether changing a product's reorder threshold would have reduced stockouts without creating unacceptable overstock. Preserve a holdout period and include carrying cost, spoilage, and supplier lead time as guardrails. The experiment recommends a threshold; an inventory owner approves it before live use.
2. Job scheduling and routing. Replay historical home-service jobs with different groupings—such as zip code or time window—and compare drive minutes without changing the real schedule. Include appointment promises, technician skills, breaks, traffic assumptions, and employee constraints. A dispatcher reviews any live route.
3. Invoice follow-up timing. Compare reminder timing on approved invoice cohorts and measure days to payment, response rate, and customer complaints. Do not let the agent send arbitrary collection messages. Finance reviews the copy, cohort design, legal requirements, and every automated send policy.
4. Support response templates. Test whether a revised template reduces repeat questions on a privacy-safe historical set or a controlled live sample. Measure resolution accuracy and customer satisfaction, not only fewer replies. The agent drafts templates; support owners approve them and retain a human escalation path.
5. Membership pricing. Simulate pricing hypotheses or run a properly governed experiment that measures revenue, conversion, retention, and customer harm together. Pricing is consequential: legal, finance, and business owners define eligibility and approve the live test. An agent never changes production prices autonomously.
6. Employee shift scheduling. Replay staffing arrangements against historical demand and compare labor cost, coverage, service levels, breaks, skills, availability, and fairness. Do not optimize people into a single cost metric. Managers review schedules, employment rules, and human constraints before any change.
7. Vendor selection. Compare cost, on-time delivery, quality, minimums, and contract terms across historical purchase orders. A live vendor switch affects money and relationships, so the agent can prepare a recommendation but cannot place or reroute an order without procurement approval.
8. Healthcare appointment reminders. Test reminder timing and channel only inside a compliant, consented program. Protect patient data, honor communication preferences, and measure no-shows alongside opt-outs, complaints, and accessibility. Clinical/privacy owners and applicable law govern the live workflow; an agent does not decide patient outreach alone.
9. Warehouse pick paths. Simulate layouts or product placement against historical orders and measure pick time, congestion, safety, replenishment cost, and worker ergonomics. Test in a reversible pilot before moving inventory. Operations and safety owners approve physical changes.
10. Insurance quote response time. Map where time is spent, test draft automation on synthetic or authorized historical cases, and measure accuracy as well as speed. Never remove approval rules or auto-fill regulated customer data merely to improve a latency metric. Licensed and compliance owners approve the process and every live eligibility boundary.
What not to automate blindly
Pricing changes, healthcare reminders, customer sends, vendor orders, insurance workflows, account access, and employment schedules are not harmless editable surfaces. Even if they have a metric, they carry legal, ethical, safety, and relationship risk.
For those domains, use the loop to evaluate drafts or simulations. Keep the real action outside the autonomous experiment.
Throughput is not the goal
The original system's experiment rate depends on hardware and workload. Business experiments have different costs, delays, and feedback quality. I do not promise dozens of useful variations overnight.
The metric I care about is trustworthy learning per unit of risk. A slower experiment that preserves a clean holdout and produces an auditable decision beats a fast loop optimizing noise.
Start with one reversible internal artifact. Prove the baseline. Let the agent suggest and test changes. Make it earn the next permission.