Fine-tuning small models for one business task, with Oumi and Kiln
I tested two open-source fine-tuning tools, found the part neither of them automates, and built that part on top: a loop that trains a small model, sorts its mistakes, tries a fix and keeps it only if it wins.
In short
- Oumi trains models well and Kiln organises tasks and data well. Neither, as far as I could find, runs the "find the mistakes, fix the data, retrain, keep only what wins" loop that a professional runs by hand.
- I built that loop. A small model trained on a laptop CPU sorted customer questions into 77 categories with 92.9% right (give or take 0.9), at about 7 milliseconds per answer.
- For pulling fields out of text, a 0.6B model I fine-tuned with Oumi scored 79.0% against 78.4% for my smaller default. That is too close to call, so the smaller model stays.
Why I looked at this
OpenAI's deprecations page says self-serve fine-tuning is now restricted. New organisations could not start jobs from May 2026, and existing customers cannot create new fine-tuning jobs after 6 January 2027. Old fine-tuned models keep working until their base model is retired.
So teams that relied on a hosted fine-tune need another route. My question: can free, open tools give a non-technical team its own small model for one task, and tell it whether the model is good enough? My method: look at the free and paid tools, take the parts that work, build only what is missing.
The tools, compared
| Tool | What it does | Free or paid | Licence | Strong at | What it lacks |
|---|---|---|---|---|---|
| Oumi | Trains, evaluates and serves open models from the command line with YAML configs | Free | Apache 2.0 | Many training methods (SFT, LoRA, QLoRA, DPO, GRPO), runs on your own GPU or a cloud, data analysis, judges | No point-and-click interface. Marks itself as alpha. No loop that picks a fix from the error types |
| Kiln | Desktop app and Python library for tasks, datasets, evals, synthetic data and fine-tuning | Free; Kiln Pro is a paid add-on | Library MIT; desktop app source-available, not open source | Fixed output shapes, data model for rated examples, easy hosted fine-tuning on 60+ models | Fine-tuning runs on hosted providers, not your own GPU. The AI Assistant, Auto-Optimize and Eval Builder are in paid Kiln Pro |
| OpenAI fine-tuning API | Hosted fine-tuning of OpenAI models | Paid per token | Proprietary | No setup, no hardware | Closed to new jobs. You never hold the weights |
| Fireworks, Together | Hosted fine-tuning of open models | Paid | Proprietary | Cheap serving of many variants | Data leaves your machine. You judge quality yourself |
What I tested
I cloned both projects and read their code and licence files. Then I ran them.
- Kiln. I installed its core library (1.2.1) and stored 200 examples from MASSIVE, a public dataset of voice-assistant requests, as Kiln task runs with a fixed output shape. They read back correctly, and Kiln refused an answer that broke the shape.
- Oumi. I ran
oumi trainandoumi infer(version 0.9) on a free Kaggle GPU. I fine-tuned Qwen3 0.6B on 11,514 examples for 3 passes, about an hour of GPU time.
What broke:
- Installing Oumi on Kaggle replaced torch and broke the preinstalled torchaudio. My script removes it.
- My first full Oumi run scored 49.0%, but 913 of 2,033 answers were empty, and the log warned about right-padding on every batch. The PyPI release padded on the wrong side for this model; newer source had fixed it. I set left padding by hand. The rerun gave 79.0% with no empty answers.
- Earlier, one of my own runs scored 10%. That was my bug: fields that were absent were never declared. Fixed.
I learned to distrust a score until I have looked at the raw answers.
What I built on top
The app is called Specialists. It runs on your own computer and has a web page with plain screens. The loop it runs:
- Check the data: repeats, the same text labelled two ways, values missing from their text, thin categories.
- Split it: 70 in 100 to learn from, 15 to tune on, 15 held back and used once at the end.
- Measure a plain baseline, so the model has something to beat.
- Train a small model.
- Sort the mistakes by kind, then try the matching fix: move the confidence cut-off, add marked synthetic examples for thin fields, or train longer.
- Keep a change only if it wins on the same examples by more than its margin of error.
- Report plainly: the score with its margin, what was tried, what mistakes remain.
It also flags unsure answers for a person, suggests labels the data seems to lack, and can train on the user's own free Kaggle GPU and bring the model back (283 seconds end to end in a real run). 109 tests pass.
What it adds that the tools lack: the decision loop itself, a "too close to call" verdict instead of a hopeful number, and training on hardware the user controls.
Who this is for
A company with a repeated, narrow task and a pile of past examples: tickets, forms, requests. A small model it owns cannot be switched off by a vendor, keeps data in-house, and answers in milliseconds. The catch is quality control, which is the part I automated.
One honest finding: a cheap API given 20 similar examples per prompt scored 94.0% on 50 rows, which may match the trained model. Sometimes the right advice is "keep your API", and the app can say so.
What I would test next
- Settle the API comparison on 300 rows instead of 50.
- Have the app run the larger Oumi model itself. Today it only writes the training folder.
- Below about 300 examples the app cannot tell a real gain from luck. Does synthetic data change that?
- A second dataset. All results so far are short English requests.
- Real users. Nobody outside me has tried it yet.
Sources
- Oumi on GitHub
- Kiln on GitHub
- OpenAI API deprecations
- Banking77 dataset
- MASSIVE dataset
- My project repository (private). Happy to walk through it.