Experiment board
| run | data | step | loss | progress | status |
|---|---|---|---|---|---|
| v2 · MagicBrush | MagicBrush 1,195 | 0 / 200 | 0.0000 | not started | |
| v3 · MagicBrush | MagicBrush 3,306 | 0 / 400 | 0.0000 | not started | |
| v3-cont · MagicBrush | MagicBrush 3,306 | 1200 / 1200 | 0.4502 | done | |
| v4 · MagicBrush | MagicBrush 5,082 | 940 / 1600 | 0.4380 | stopped | |
| mixed v2 · high-ratio | 4,109 (ext 21%) | 0 / 800 | 0.0000 | not started | |
| v5 · mixed-all | 5,962 (ext 15%) | 769 / 2000 | 0.5239 | stopped | |
| v6 · low-ratio | 5,522 (ext 8%) | 23 / 2000 | 0.5308 | stopped | |
| rft · best-of-N | RFT 5,250 | 53 / 1600 | 0.5532 | running | |
| v9 · RFT + target-informed | RFT + TI ~5.8k | 0 / 1600 | 0.0000 | not started | |
| v10 · all-data mixed | 7,619 (MB+UE+Sci) | 0 / 2000 | 0.0000 | not started | |
| v11 · all-data + TI | 8,793 (MB+UE+Sci+TI) | 0 / 2000 | 0.0000 | not started | |
| v12 · 10k all-data | 9,447 (RFT+UE+Sci+TI) | 56 / 2400 | 0.5898 | running | |
| v13 · all-data + TI v6 | 9,890 (RFT+UE+Sci+TI1.3k) | 1042 / 2400 | 0.3962 | running | |
| v13-warm · from best ckpt | 9,890 (warm start) | 954 / 1500 | 0.3905 | running | |
| v13-r32 · capacity | 9,890 (LoRA r32) | 933 / 2400 | 0.4402 | running | |
| v15 · TI 2x dose | 11,209 (TI 23.5%) | 651 / 2400 | 0.5007 | running | |
| ultraedit-only | UltraEdit 700 | 0 / 800 | 0.0000 | not started |
| model | benchmark | tasks | executed | judge pass | pass rate |
|---|---|---|---|---|---|
| zero-shot | MagicBrush dev32 partial | 32 | 26 | 3 | 9.4% |
| zero-shot | UltraEdit dev partial | 82 | 56 | 5 | 6.1% |
| zero-shot | SciForma dev | 55 | 52 | 21 | 38.2% |
| v3-cont-1200 | MagicBrush dev32 | 32 | 21 | 7 | 21.9% |
| v3-cont-1200 | UltraEdit dev partial | 36 | 21 | 6 | 16.7% |
| v3-cont-1200 | SciForma dev | 55 | 43 | 19 | 34.5% |
| v4-400 | MagicBrush dev32 | 8 | 7 | 2 | 25.0% |
| v4-400_part0 | MagicBrush dev32 | 8 | 7 | 2 | 25.0% |
| teacher deepseek | MagicBrush dev32 | 32 | 31 | 13 | 40.6% |
| teacher deepseek | UltraEdit dev | 94 | 87 | 48 | 51.1% |
| teacher deepseek | SciForma dev | 55 | 54 | 33 | 60.0% |
| teacher gpt55 | MagicBrush dev32 | 32 | 28 | 8 | 25.0% |
| teacher gpt55 | UltraEdit dev | 94 | 84 | 45 | 47.9% |
| teacher gpt55 | SciForma dev | 55 | 55 | 43 | 78.2% |
| teacher gpt56sol | MagicBrush dev32 | 32 | 30 | 18 | 56.2% |
| teacher gpt56sol | UltraEdit dev | 94 | 91 | 74 | 78.7% |
| teacher gpt56sol | SciForma dev | 55 | 50 | 47 | 85.5% |
| teacher gpt61 | MagicBrush dev32 | 32 | 29 | 21 | 65.6% |
| teacher gpt61 | UltraEdit dev | 94 | 90 | 77 | 81.9% |
| teacher gpt61 | SciForma dev | 55 | 53 | 51 | 92.7% |
| teacher luna | MagicBrush dev32 | 32 | 26 | 12 | 37.5% |
| teacher luna | UltraEdit dev | 94 | 81 | 40 | 42.6% |
| teacher luna | SciForma dev | 55 | 14 | 11 | 20.0% |
| teacher seed21pro | MagicBrush dev32 | 32 | 27 | 11 | 34.4% |
| teacher seed21pro | UltraEdit dev | 94 | 72 | 37 | 39.4% |
| teacher seed21pro | SciForma dev | 55 | 51 | 29 | 52.7% |
| teacher seed21turbo | MagicBrush dev32 | 32 | 29 | 8 | 25.0% |
| teacher seed21turbo | UltraEdit dev | 94 | 73 | 17 | 18.1% |
| teacher seed21turbo | SciForma dev | 55 | 53 | 16 | 29.1% |
| teacher terra | UltraEdit dev | 94 | 27 | 15 | 16.0% |
| teacher terra | SciForma dev | 55 | 43 | 38 | 69.1% |
Data quality · RFT & target-informed distillation
| experiment | n | judge pass | CLIP gain vs target | L1 gain vs target | notes |
|---|---|---|---|---|---|
| source-only accepted (base) | 5,826 | — | -0.0013 | — | all accepted trajectories |
| RFT best-of-N selected | 5,826 | — | -0.0000 | — | 1,691 multi-candidate; 3,012 positive |
| STaR target-informed (gpt55) | 100 | 34 | +0.0035 | -0.0011 | flip-reuse 73/100; 0 hardcoded (refined filter) |
| source-only matched base | 100 | — | +0.0011 | -0.0039 | accepted trajectories only |
Official benchmark metrics (vs target)
| model | benchmark | n | L1 ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | CLIP-I ↑ | CLIP gain ↑ | DINO ↑ | DINO gain ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| sft_magicbrush_27b_oneshot_v3_cont-1200 | MagicBrush dev32 | 21 | 0.0787 | 18.90 | 0.818 | 0.213 | 0.906 | -0.0234 | 0.000 | +0.0000 |
| v3cont1200 sciforma | SciForma dev | 43 | 0.0212 | 26.01 | 0.954 | 0.066 | 0.984 | -0.0069 | 0.000 | +0.0000 |
| zero sciforma | SciForma dev | 52 | 0.0108 | 25.13 | 0.959 | 0.067 | 0.987 | -0.0030 | 0.000 | +0.0000 |
| teacher gpt55 | MagicBrush dev32 | 28 | 0.0619 | 18.57 | 0.833 | 0.190 | 0.928 | -0.0027 | 0.000 | +0.0000 |
| teacher gpt61 | MagicBrush dev32 | 29 | 0.0628 | 19.09 | 0.824 | 0.192 | 0.921 | -0.0060 | 0.000 | +0.0000 |
SFT · Qwen3.8-27B one-shot
0 / 400training step
0.0000last loss
0.0 GBpeak memory
1,195train rows
3/32 = 9.4%zero-shot baseline (dev32)
| checkpoint | dev tasks | parsed | executed | judge pass | pass rate |
|---|---|---|---|---|---|
| waiting for first checkpoint | |||||
SFT · Qwen3.8-27B one-shot (code-only)
0 / 400training step
0.0000last loss
0.0 GBpeak memory
1,719train rows
3/32 = 9.4%zero-shot baseline (dev32)
| checkpoint | dev tasks | parsed | executed | judge pass | pass rate |
|---|---|---|---|---|---|
| waiting for checkpoint-200 | |||||
Per-teacher contribution
| teacher | attempts | exec ok | judge pass | accepted | accept rate | exec fail | teacher error |
|---|---|---|---|---|---|---|---|
| gpt61 | 6186 | 5987 | 4501 | 4537 | 73.3% | 138 | 61 |
| terra | 1685 | 1572 | 335 | 337 | 20.0% | 111 | 2 |
| gpt55 | 1268 | 1193 | 83 | 85 | 6.7% | 75 | 0 |
| luna | 1350 | 1152 | 85 | 85 | 6.3% | 174 | 24 |
Recent accepted
| sample | instruction | teacher | scores |
|---|---|---|---|
| magicbrush_train_398489_2 | put a cute puppy beside the cat | gpt61 | 5/5/4 |
| magicbrush_train_10936_3 | let there be boulders | gpt61 | 4/5/3 |
| magicbrush_train_303651_3 | let the girl wear formal clothes | gpt61 | 4/4/3 |
| magicbrush_train_566188_2 | let the woman hold a blow dryer | gpt61 | 4/4/3 |
| magicbrush_train_188109_3 | remove the small chair | luna | 4/4/3 |
| magicbrush_train_249131_3 | Can it be fries next to the cup? | gpt55 | 4/4/3 |
| magicbrush_train_398489_1 | make the cat happy | gpt61 | 4/5/4 |
| magicbrush_train_159282_1 | Add an ashtray next to the vase | gpt61 | 5/5/4 |
| magicbrush_train_303651_1 | let the dog bite the girl | gpt61 | 4/4/3 |
| magicbrush_train_548913_1 | Let the person load the cake with a spoon. | gpt61 | 4/5/3 |
Dataset coverage
Touched 6,227 of 8,807 MagicBrush train rows across 51 shards (70.7%). Accepted 5,044 trajectories; target 5,000 trajectories / 10,000 SFT rows.