#devchallenge #hf26challenge Slow & Steady

LazyDecider

How an open-weight agent fine-tuned a decision model for me — proposing configs, pruning the losers, keeping only winners — while I slept.

🦥 Gemma 4 E4B proposes → ✂️ ASHA prunes → ✅ keep / ↩︎ revert → 🔬 Tinker confirms

Local · open weights · sealed test · ~$0 on a laptop

The ten-minute job I refused to do again

Hyperparameter babysitting:

change the learning rate → kick off a run → stare at the loss curve
change the LoRA rank → repeat → fall asleep at the desk

So I automated it. The search runs overnight, for free, on my own laptop. I wake up and read a report.

The task — typed decisions

A model reads a case and makes a calibrated accounts-payable style call across 4 domains, scored against a teacher consensus (gold).

Domain	Cases	What the model decides
invoice_processing	100	approve / review an invoice vs its PO
customer_service	100	route / escalate a ticket
security_incidents	100	triage severity & response
agent_trace_observability	100	flag anomalous agent traces

Metric that matters: KL from gold (distribution match), not just top-1 accuracy.

The model we fine-tune

Base: Qwen3.5-0.8B (open weights)
+ Clef joint-decision head — emits a probability over all decisions
Trained with Unsloth, LoRA on an 18 GB Mac (MLX)

Two knobs the agent plays with:
head-only vs unfreeze backbone (LoRA)
LoRA rank/alpha, LR, label smoothing, weight decay, dropout, epochs

The overnight loop

🦥
Gemma 4 E4B
reads the log, proposes
1 config + 1-line reason

→

✂️
ASHA pruner
trains it, kills it early
if it's losing

→

⚖️
keep / revert
kept only if it beats
best on frozen split

→

🔬
Tinker
bigger model, same
sealed test = referee

Gemma only ever sees fit / select / calib — never the sealed test.

Why ASHA — don't waste the night on losers

Asynchronous Successive Halving: evaluate at rungs, prune the bottom performers early, let the promising runs train longer.

Rung (step)	What happens
20	first checkpoint — obvious failures killed here
60	bottom percentile pruned
140	survivors continue
240	full run — only the best reach here

A bad config costs minutes, not the whole night.

What Gemma actually tried — verbatim from the log

#	backbone	LR	LoRA r/α	smooth	wd	ep	select loss	select acc
1	frozen	1e-4	16/32	0.00	0.01	2	1.041	0.523	✅ keep
2	unfrozen	2e-4	16/32	0.00	0.01	2	0.840	0.782	✅ keep
3	unfrozen	1.5e-4	16/32	0.06	0.05	2	0.842	0.787	↩︎ revert
4	unfrozen	1e-4	16/32	0.08	0.05	1	0.884	0.767	↩︎ revert
5	unfrozen	8e-5	8/16	0.03	0.08	2	0.857	0.767	↩︎ revert

These are select-split scores measured during the search (what Gemma optimizes). Run 3 had marginally higher accuracy but higher select loss than run 2 → not kept. The headline 0.770 on later slides is the winner’s accuracy on the sealed test — a different, unseen split.

Run 2 — the night's one big move

“Enable LoRA fine-tuning on the backbone to see if adapted encoder representations improve beyond the head-only baseline.”— Gemma, run 2 (kept)

1.04 → 0.84

select loss drop

+0.26

accuracy jump (0.52 → 0.78)

Unfreezing the backbone with LoRA was the single biggest gain — and the agent found it on its own, then spent runs 3–5 chasing calibration and reverting each one.

ASHA select-loss, per config

ASHA select-loss curves per Gemma-proposed config

Run 2 (unfrozen LoRA) separates from the pack by rung 140 and holds the lead.

The “can’t cheat” part

Early on, the loop found a “winner” suspiciously fast — a too-small validation subsample let a lucky run look great. The agent wasn’t cheating; it was optimizing the number I gave it.

🔒 Sealed test — only its SHA-256 is committed; the worker can’t read the cases
📏 Scorer copies the dataset’s grading arithmetic verbatim
💸 Worker strips TINKER_* from env — the free local loop can’t spend a cent
🧊 Every finalist’s config is committed to git before its one-shot sealed read

Sealed-test results

Model	Role	Acc ↑	KL ↓	Brier ↓	ECE ↓
Prior / Uniform	reference	0.470	—	—	—
Zero-shot base	no training	0.483	—	—	—
Tinker 4B	cloud referee	0.601	0.642	0.172	0.094
LazyDecider 0.8B	the winner	0.770	0.094	0.051	0.145

A 0.8B model, found autonomously, lands KL 0.094 — the metric this benchmark optimizes — for ~$0 on an 18 GB Mac.

Per-domain — strongest on invoices

Domain	Acc ↑	KL ↓	Brier ↓	ECE ↓
invoice_processing	0.840	0.093	0.045	0.114
customer_service	0.796	0.115	0.059	0.162
security_incidents	0.738	0.074	0.043	0.163
agent_trace_observability	0.706	0.094	0.056	0.156
Overall (400)	0.770	0.094	0.051	0.145

Best accuracy on invoices (0.840); best-calibrated on security incidents (KL 0.074).

Tinker — the independent referee

A different model (Qwen3.5-4B LoRA), different hardware (cloud), same sealed test — does the recipe transfer?

0.601

Tinker 4B accuracy

0.483

zero-shot baseline

$2 / $10

hard-capped spend

A clear lift over zero-shot on the same ruler = the pipeline isn’t fooling itself.

Why open innovation matters here

🦥 Agent + model are both open weights → hundreds of reasoning calls a night cost $0
🔓 The model is yours → run it offline, push the adapter to HF, fork the recipe
🔍 Guardrails are auditable → “can’t cheat” is only credible when anyone can read the sealing logic

A closed API couldn’t give me all three: free overnight reasoning, full ownership, and verifiable integrity.

The lazy payoff

I wake up, open REPORT.md, read what Gemma tried and why, run one command to retrain the winner — and Tinker confirms it. I never babysit a sweep again.

🌐 lazydecider-demo.onrender.com ⌁ github.com/rahuls321/autoresearch-finetune-lab

Open agent · open model · sealed test · built on a laptop, while asleep.