Back to the letter

Research report · 14 Nov 2026 · Status: in progress, two task types work today

Fine-tuning small models for one business task, with Oumi and Kiln

I tested two open-source fine-tuning tools, found the part neither of them automates, and built that part on top: a loop that trains a small model, sorts its mistakes, tries a fix and keeps it only if it wins.

In short

Why I looked at this

OpenAI's deprecations page says self-serve fine-tuning is now restricted. New organisations could not start jobs from May 2026, and existing customers cannot create new fine-tuning jobs after 6 January 2027. Old fine-tuned models keep working until their base model is retired.

So teams that relied on a hosted fine-tune need another route. My question: can free, open tools give a non-technical team its own small model for one task, and tell it whether the model is good enough? My method: look at the free and paid tools, take the parts that work, build only what is missing.

The tools, compared

ToolWhat it doesFree or paidLicenceStrong atWhat it lacks
OumiTrains, evaluates and serves open models from the command line with YAML configsFreeApache 2.0Many training methods (SFT, LoRA, QLoRA, DPO, GRPO), runs on your own GPU or a cloud, data analysis, judgesNo point-and-click interface. Marks itself as alpha. No loop that picks a fix from the error types
KilnDesktop app and Python library for tasks, datasets, evals, synthetic data and fine-tuningFree; Kiln Pro is a paid add-onLibrary MIT; desktop app source-available, not open sourceFixed output shapes, data model for rated examples, easy hosted fine-tuning on 60+ modelsFine-tuning runs on hosted providers, not your own GPU. The AI Assistant, Auto-Optimize and Eval Builder are in paid Kiln Pro
OpenAI fine-tuning APIHosted fine-tuning of OpenAI modelsPaid per tokenProprietaryNo setup, no hardwareClosed to new jobs. You never hold the weights
Fireworks, TogetherHosted fine-tuning of open modelsPaidProprietaryCheap serving of many variantsData leaves your machine. You judge quality yourself

What I tested

I cloned both projects and read their code and licence files. Then I ran them.

What broke:

I learned to distrust a score until I have looked at the raw answers.

What I built on top

The app is called Specialists. It runs on your own computer and has a web page with plain screens. The loop it runs:

  1. Check the data: repeats, the same text labelled two ways, values missing from their text, thin categories.
  2. Split it: 70 in 100 to learn from, 15 to tune on, 15 held back and used once at the end.
  3. Measure a plain baseline, so the model has something to beat.
  4. Train a small model.
  5. Sort the mistakes by kind, then try the matching fix: move the confidence cut-off, add marked synthetic examples for thin fields, or train longer.
  6. Keep a change only if it wins on the same examples by more than its margin of error.
  7. Report plainly: the score with its margin, what was tried, what mistakes remain.

It also flags unsure answers for a person, suggests labels the data seems to lack, and can train on the user's own free Kaggle GPU and bring the model back (283 seconds end to end in a real run). 109 tests pass.

What it adds that the tools lack: the decision loop itself, a "too close to call" verdict instead of a hopeful number, and training on hardware the user controls.

Who this is for

A company with a repeated, narrow task and a pile of past examples: tickets, forms, requests. A small model it owns cannot be switched off by a vendor, keeps data in-house, and answers in milliseconds. The catch is quality control, which is the part I automated.

One honest finding: a cheap API given 20 similar examples per prompt scored 94.0% on 50 rows, which may match the trained model. Sometimes the right advice is "keep your API", and the app can say so.

What I would test next

Sources