💡 Fine-tuning is not “train a model.” It is five orthogonal questions: which checkpoint, how weights change, what loss you optimize, which examples, and how you know it worked. Mix those up and you get a 70B that recites your JSONL and forgets how to count. Keep them separate and you can swap LoRA for DoRA, SFT for DPO, without rewriting the stack.

🗺️ The map

Read this as a curriculum, not a glossary. Six layers — climb only when the layer below is not enough:

1. 🥊 When to fine-tune — in-context learning, RAG, continued pre-training, distillation. Answers: should weights move at all?
2. ⚙️ What actually changes — linear layers, gradients, AdamW. Answers: what is a “weight update”?
3. 🧩 Strategy — Full, LoRA, DoRA, AdaLoRA, QLoRA, and the 2026 LoRA zoo. Answers: how do weights change?
4. 🎓 Objective — SFT → preferences → verifiable RL. Answers: what is the model learning?
5. 🧪 Proof — probes, holdout PPL, judges, adapter merge. Answers: did it stick?
6. 🧞 slick-tune — the orthogonal toolkit. Answers: how do you run the stack without a 400-line trainer script?

flowchart LR
  subgraph AXES["🎛️ Fine-tuning is a product of axes"]
    M["🧠 model"]
    S["🧩 strategy"]
    O["🎓 objective"]
    D["📁 data"]
    E["📊 metrics"]
  end
  M --- S --- O --- D --- E
            

🥊 Should weights move at all?

An LLM is a giant next-token machine: text in, a distribution over the vocabulary out. Those probabilities come from billions of numbers — weights. Pre-training learns language from the internet. Fine-tuning continues that training on your distribution so the model behaves the way you want: a house style, a JSON schema, a product API, a refusal policy, a set of stable facts.

Three other knobs exist. Most teams should try them before opening a Trainer — starting with in-context learning:

👍 Rule of thumb in 2026. Start with ICL. Fine-tune for form — style, structure, tool calling, refusal patterns, a small set of facts that must not drift. Retrieve for facts that change. The highest-ROI pattern is a thin LoRA/QLoRA adapter plus RAG, not one instead of the other. Distill after you have a teacher you trust.

If you need the model to reliably know something small and personal — names, emails, product APIs — fine-tuning plus probes beats hoping the prompt sticks [7].

🎯 In-context learning

In-context learning (ICL) is the ability to pick up a task from examples inside the prompt, with no weight update [32]. GPT-3 made the phrase famous. Everything since is an argument about how far you can push that idea before you should actually train.

The prompt is the dataset. The forward pass is the optimizer. Tomorrow’s request can use a different dataset without touching a checkpoint.

📎 What a “shot” actually is

A shot is an input–output pair the model can attend to before it writes the next token. Typical shapes:

• 0️⃣ Zero-shot — instruction only. “Extract the JSON. Schema follows.”
• 1️⃣ One-shot / few-shot — a handful of gold examples. The original GPT-3 regime: 1–32 shots [32].
• 📚 Many-shot — hundreds to thousands of examples stuffed into a long context window. Gemini-class windows made this a real baseline, not a parlor trick [33].

System: Extract a JSON object with keys name, email.

User:
Input: "Write Jane at jane@acme.com"
Output: {"name":"Jane","email":"jane@acme.com"}

Input: "Ping Amirhessam: admin@slickml.com"
Output:

At inference the model is still doing next-token prediction. The demonstrations bias the attention pattern so the continuation looks like “another example of the same map.” That is ICL. It is not backprop.

🧠 Why it works (and why it sometimes doesn’t)

Three stories that are all partly true:

• 📐 Format specification. Min et al. showed that even random labels in the demos can help — a lot of few-shot gain is “here is the output shape,” not “here is the true mapping” [34]. If your few-shot prompt only teaches the schema, a tighter instruction might have been enough.
• 📉 Implicit inner loop. A line of theory treats attention over demos as a few steps of gradient descent / linear regression implemented in the forward pass — the model is a meta-optimizer [35]. Useful intuition: more shots ≈ more inner-loop steps, until the context saturates.
• 📍 Task location. Pre-training already saw a thousand formats. ICL points at the right basin. Fine-tuning moves the basin. That is why ICL is fast to try and easy to forget on the next request.

Practical footguns: demo order matters, recency bias is real, label distribution in the prompt leaks into the answer, and a 32k context full of shots is still paid per request unless you KV-cache / prompt-cache the prefix.

📚 Many-shot is the 2024–2026 ICL pattern

Few-shot was “can the model learn from 8 examples?” Many-shot asks “what if the prompt is the fine-tune set?” Agarwal et al. show that going from a handful of shots to hundreds or thousands keeps lifting both generative and discriminative tasks, can override pre-training biases that few-shot cannot, and often lands comparable to fine-tuning — at the cost of prefill compute, not a training run [33]. They also try Reinforced ICL (model-written chain-of-thought as shots) and Unsupervised ICL (inputs without gold outputs) when human labels run out.

The catch is the economics. SFT pays once. Many-shot pays on every call, unless you cache the demo prefix. Long-context models made ICL look like a substitute for LoRA; KV caching made that substitute affordable for high-QPS prefixes. If the demos change per user, you are back to paying the token tax.

A 2025–2026 cousin — many-shot in-context fine-tuning — actually trains the model to use long demo prefixes (every in-context example is a supervised target, not just the last answer). One checkpoint, many tasks at inference, less forgetting than naive zero-shot SFT [36]. That is ICL and fine-tuning having a child, not a replacement for either.

On controlled formal languages, fine-tuning still wins in-distribution proficiency; ICL matches on some out-of-distribution splits and varies much more across model families [37]. Do not take “many-shot ≈ SFT” as a law. Take it as “measure both on your eval.”

🗳️ ICL vs fine-tuning

Soft-prompt PEFT (prefix tuning, prompt tuning, IA³) is the cousin that learns a prefix instead of writing one. Same intuition as ICL — steer with context — but the prefix is trained and then stored. In 2026 LoRA ate that niche for LLMs; keep prefix methods in mind for encoder-only or extreme parameter budgets, not as your first SFT tool.

⚙️ What actually changes?

Inside a Transformer, most of the trainable mass is linear layers: matrices that mix features. Attention projections (q_proj, k_proj, v_proj, o_proj) and MLP projections (gate_proj, up_proj, down_proj) are the usual suspects.

One layer is just y = Wx (plus bias, plus nonlinearity elsewhere). Fine-tuning computes a loss — how wrong the next-token predictions were — then backprop produces a gradient for every trainable number: nudge this entry up or down. An optimizer, almost always AdamW, applies those nudges for many steps. After enough steps the model’s distribution has shifted toward your data.

sequenceDiagram
  participant D as 📁 Dataset
  participant M as 🧠 Model
  participant L as 📉 Loss
  participant O as ⚙️ Optimizer
  D->>M: batch of tokens
  M->>L: predictions vs labels
  L->>M: gradients
  M->>O: trainable params + grads
  O->>M: updated weights / adapters
            

Memory is the tax. Full fine-tuning stores weights + gradients + optimizer states. AdamW keeps two extra tensors per parameter (first and second moments), so the working set is roughly several times the model size in the training dtype. That is why PEFT exists, and why QLoRA exists on top of PEFT.

💡 Why not always update every weight?

Full fine-tuning works. It is also the most expensive way to teach a model your email address:

• 💾 GPU memory: weights + grads + Adam states ≈ several× model size
• 📦 Storage: one full copy of the model per run
• 🧠 Catastrophic forgetting: a tiny dataset can erase general skills
• 🔀 Multi-task serving: you do not want five 70B specialists on disk

Parameter-Efficient Fine-Tuning (PEFT) freezes the base and trains a small add-on — an adapter. The family that ate the industry is LoRA [1]. Typical trainable fraction: 0.1–5%. Checkpoint: a few tens of megabytes, not 140 GB.

flowchart TB
  subgraph FULL["🏋️ Full fine-tuning"]
    W1["🔓 Every weight trainable"]
  end
  subgraph PEFT["🧩 PEFT e.g. LoRA"]
    WF["❄️ Base weights frozen"]
    A["🧪 Tiny adapters trainable"]
    WF --- A
  end
            

The 2026 default for almost every product team: LoRA or QLoRA. Full FT is a research / tiny-model / “I proved LoRA’s rank is the bottleneck” move — not the first experiment.

🧭 Strategy: how weights change

A strategy answers: given a frozen or unfrozen base, what parameterization do we train? slick-tune ships the five you actually use [7]. The rest of the LoRA zoo (PiSSA, LoRA+, rsLoRA, LoftQ, …) is mostly how you initialize or scale the same A,B.

flowchart TB
  Start["Need to fine-tune?"] --> Mem{"GPU memory tight?"}
  Mem -->|"Yes, CUDA"| Q["📦 QLoRA"]
  Mem -->|"Yes, Mac / CPU"| L["🧩 LoRA / DoRA"]
  Mem -->|"Plenty"| Choice{"Want adaptive ranks?"}
  Choice -->|No| Fixed{"Want magnitude / direction split?"}
  Fixed -->|No| L2["🧩 LoRA"]
  Fixed -->|Yes| D["✨ DoRA"]
  Choice -->|Yes| AD["🎯 AdaLoRA"]
  Mem -->|"Research / max quality / tiny model"| F["🏋️ Full FT"]
            

🏋️ Full fine-tuning

Idea. Every weight that can learn, learns. One linear layer is still y = Wx; every entry of W gets a gradient.

When. Small bases where memory allows (SmolLM-class demos). Or you have already shown that LoRA’s rank is the ceiling and you will keep one specialized checkpoint.

Trade-off. Highest memory and storage. One run → one full model directory. Easy to overfit a 2k-example JSONL into a model that can no longer follow generic instructions.

Biderman et al. (“LoRA Learns Less and Forgets Less”) showed LoRA often underfits continued pre-training and forgets less than full FT on instruction data [8]. That is a feature for product SFT and a bug for “teach the model a new language from a corpus.” If you do continued pre-training with LoRA, the Unsloth recipe is now: target all linear layers (including gate_proj), use rsLoRA at high rank, and a smaller LR on embed_tokens / lm_head [9].

🧩 LoRA — Low-Rank Adaptation

Keep W frozen. Learn a thin correction:

W′ = W + (α / r) · B A

• A is r × d_in — often Gaussian / Kaiming at init
• B is d_out × r — often zeros, so training starts as “no change”
• r (rank) is tiny — 8, 16, 32, 64 — versus thousands of hidden dims
• α (alpha) scales the update. A common start is α ≈ 2r, or α = r if you already bake scaling into the LR

Why “low-rank”? Empirically, the useful change ΔW for a downstream task often lives in a low-dimensional subspace. A thin BA is enough for style, format, and a surprising amount of factual adaptation. Hu et al. showed this for GPT-3-class models: ~10,000× fewer trainable parameters, no extra inference latency after you merge B A back into W [1].

flowchart TB
  x["📥 Input x"] --> W["❄️ Frozen W"]
  x --> A["🧪 Trainable A  r × d_in"]
  A --> B["🧪 Trainable B  d_out × r"]
  W --> Sum["➕"]
  B -->|"α/r · BA x"| Sum
  Sum --> y["📤 Output y"]
            

Where adapters attach. Originally: attention q and v only. In 2026 the default is target_modules="all-linear" — attention and MLP. Skipping gate_proj is a known footgun for continued pre-training [9].

Serving. After training you have the unchanged base plus a small adapter folder (adapter_model.safetensors, adapter_config.json). At inference: load base + adapter, or bake (merge_and_unload) so engines like vLLM / TGI see one set of weights.

🧬 The 2026 LoRA zoo — same A,B, better physics

Vanilla LoRA is still the workhorse. The papers since 2023 mostly fix initialization, scaling, or optimizer asymmetry — they do not replace the adapter. Hugging Face PEFT exposes most of these as flags on LoraConfig [10].

RSRA (2026) is the other direction: a training-free probe that allocates rank from representation sensitivity before you fine-tune, then plugs into LoRA / DoRA / PiSSA [18]. AdaLoRA does the same idea during training with importance scores. The pattern: uniform rank is leaving money on the table; the question is whether you allocate before or during the run.

✨ DoRA — Weight-Decomposed LoRA

Full FT changes both how large a weight row is (magnitude) and which way it points (direction). Plain LoRA mostly learns a directional ΔW on frozen W. DoRA decomposes the adapted weight into a trainable magnitude vector m and a LoRA-style direction [3]:

W′ = m · (W + BA) / ||W + BA||_row

Same knobs as LoRA (r, alpha). Slightly more compute. In PEFT it is use_dora=True.

2026 systems note. Naive DoRA materializes the dense BA just to take a row-norm — at high rank that is hundreds of MB of transient VRAM per module. Factored norms + fused Triton kernels cut that working set and pick up ~1.5–2× on the compose kernels [19]. If you heard “DoRA is too slow,” check whether you are running the 2024 eager path.

✅ When to try DoRA. You like LoRA’s cost and want a bit more quality headroom — especially instruction / commonsense suites where magnitude actually moves.

🎯 AdaLoRA — adaptive rank

Not every layer needs the same r. AdaLoRA parameterizes ΔW as an SVD-like triplet, scores importance from gradients, and prunes toward a rank budget [20]:

1. Start with a higher init_r
2. Warm up (tinit) with little/no prune
3. Allocate / prune every deltaT steps toward average target_r
4. Final tfinal steps: freeze ranks, fine-tune

⚠️ Critical implementation detail. PEFT’s update_and_allocate must run after optimizer.step() and before zero_grad() — gradients still have to exist. Miss that hook and AdaLoRA is just expensive LoRA. slick-tune’s AdaLoRACallback hangs off Hugging Face Trainer’s on_optimizer_step [7].

✅ Longer runs, rank-budget experiments. Tiny memorization demos often need the warmup plus a slightly higher LR than LoRA.

📦 QLoRA — 4-bit base + high-precision adapters

QLoRA is how 65B-class models got fine-tuned on a 48 GB GPU [2]. Three ideas stacked:

1. 🔢 Store frozen base weights in NF4 (4-bit NormalFloat, a quantile type matched to weight distributions)
2. 📉 Double quantization — quantize the quantization constants themselves
3. 📄 Paged optimizers — spill Adam states to CPU RAM when the CUDA allocator would OOM

Forward computes in bf16/fp16. LoRA A,B stay in higher precision. You train the adapters, not the 4-bit integers.

flowchart TB
  Disk["💾 16/32-bit checkpoint on disk"] --> Q["❄️ Load as 4-bit NF4 base"]
  Q --> Lora["🧪 Trainable LoRA A,B in bf16"]
  Batch["📦 Batch"] --> Q
  Batch --> Lora
  Q --> Out["➡️ Forward"]
  Lora --> Out
            

In 2026 QLoRA is the default for single-GPU 7B–70B SFT. Unsloth-class kernels closed most of the throughput gap with 16-bit LoRA. Pick LoRA when you have VRAM headroom and want max tokens/sec; pick QLoRA when memory is the binding constraint. LoftQ or PiSSA init if the quantized base is a bad starting point [14] [13].

📋 Hardware. bitsandbytes 4-bit wants CUDA. On Apple Silicon / CPU, use LoRA, not QLoRA. slick-tune: uv sync --extra qlora.

🗳️ Choosing a strategy

🎓 Objective: what the model learns

Strategy is the parameterization. Objective is the loss + data contract. They combine freely: LoRA + SFT, QLoRA + DPO, LoRA + GRPO.

The 2026 post-training stack is modular. Classic RLHF (SFT → reward model → PPO) is not dead at frontier labs, but it is no longer the default you should reach for first [21]:

flowchart TB
  subgraph SUP["📝 Supervised"]
    SFT["SFT — imitate assistant tokens"]
  end
  subgraph PREF["⚖️ Preferences"]
    DPO["DPO — chosen vs rejected vs ref"]
    ORPO["ORPO — odds-ratio, no ref model"]
    KTO["KTO — unpaired good / bad"]
    SIM["SimPO — avg logprob, no ref"]
  end
  subgraph RL["🎯 Verifiable RL"]
    GRPO["GRPO — group-relative advantages"]
    DAPO["DAPO — clip-higher, dynamic sampling"]
  end
  SFT --> PREF --> RL
            

📝 SFT — supervised fine-tuning

Imitate labeled assistant answers. Next-token negative log-likelihood on the response tokens (mask the prompt). This is still the correct first step: teach facts, house style, JSON schemas, tool-call format.

Data shapes (JSONL, one object per line):

{"messages":[{"role":"user","content":"..."},{"role":"assistant","content":"..."}]}
{"prompt":"...","response":"..."}
{"instruction":"...","input":"...","output":"..."}

📉 Data > scale. 500–2,000 clean demonstrations often beat 50,000 noisy ones for tone/format. For domain SFT, 5k–20k in the target distribution is a common band. If the model must emit a schema, put the schema in the targets, not only the system prompt.

Completion-only loss (do not train on user tokens) is the default that is not default enough — if your library trains on the whole sequence, you are teaching the model to imitate the questions too.

🏆 Classic RLHF — reward model + PPO

InstructGPT’s recipe [22]: SFT, then a reward model on preference pairs, then PPO with a KL penalty toward the SFT policy. It works. It also means a second model, a value head / critic, unstable advantage estimates, and a lot of annotation.

You still see PPO at labs for safety-critical alignment. For everyone else, the field moved to implicit-reward methods (DPO family) and then to RL with verifiable rewards (GRPO family) for reasoning. slick-tune’s roadmap puts classic RM + PPO in a later phase; the shipped path is DPO/ORPO/KTO + GRPO [23].

⚖️ DPO — Direct Preference Optimization

DPO skips the reward model [4]. You provide a chosen and a rejected completion for the same prompt. The loss increases the likelihood of chosen relative to rejected, compared to a frozen reference policy (usually the SFT model), scaled by beta (KL / preference strength). Typical beta: 0.1–0.5.

{"prompt":"...","chosen":"...","rejected":"..."}

Cost: a forward through policy and reference (~30–50% more than SFT). Stability: much better than PPO for most teams. 1k–5k ranked pairs is a common starting set.

🎲 ORPO — odds-ratio, no reference

Same preference pairs as DPO. ORPO folds SFT and preference into one odds-ratio term — no separate ref model in GPU memory [24]. Useful when the dataset is small and you do not want to keep a second copy of the weights. Same beta knob, different meaning (odds-ratio strength).

✅❌ KTO — unpaired thumbs

Product logs are rarely ranked pairs. KTO treats alignment as prospect-theoretic optimization over binary good/bad labels [25]:

{"prompt":"...","completion":"...","label":true}

TRL wants batch size > 1 for the KL term — slick-tune auto-bumps per_device_train_batch_size to at least 2. If all you have is thumbs-up / thumbs-down from production, this is the objective, not DPO.

📉 SimPO and friends

SimPO uses the average log-probability of the response as an implicit reward — no reference model [26]. IPO, CPO, and a pile of DPO variants tweak the regularization. The practical split in 2026:

• ⚖️ Paired prefs, you can afford a ref → DPO
• ⚖️ Paired prefs, tight VRAM / small data → ORPO (or SimPO)
• 👍 Unpaired labels → KTO

🎯 GRPO — Group Relative Policy Optimization

This is the optimizer that made open reasoning models feel like o1. DeepSeekMath introduced GRPO; DeepSeek-R1 showed large-scale RL with verifiable rewards can emerge chain-of-thought, self-reflection, and strategy switching — including an R1-Zero path with no SFT first [5] [6].

Idea. Drop the critic / value network. For each prompt, sample a group of completions (8–64 in papers; 2–4 in laptop demos). Score each with a reward. Advantage is just group-relative:

A_i = (r_i − mean(r)) / std(r)

Then a PPO-style clipped policy gradient, often with a KL term toward a reference (beta). No learned reward model if the reward is verifiable: unit tests passed, boxed math matches, JSON parses, substring present.

That last sentence is the paradigm shift: RLVR — reinforcement learning with verifiable rewards — not “humans rated this 7/10.”

{"prompt":"Who is Amirhessam Tahmassebi?","must_contain":"founder of SlickML"}

slick-tune’s demo reward: exact substring → 1.0; else a soft keyword-overlap so groups can get non-zero advantages [7].

⚠️ Warm-start. On a cold tiny base, rewards stay ~0 and GRPO cannot learn. Run SFT first, then GRPO with adapter_path pointing at the SFT adapter. Frontier R1-Zero is the exception that proves the compute budget.

🚀 DAPO, Dr. GRPO, GSPO — GRPO after the hype

Naive GRPO at long-CoT scale hits entropy collapse, length bias, and reward noise. DAPO (ByteDance, 2025) is the open system that actually reproduced R1-class gains on Qwen2.5-32B [27]:

1. 📐 Clip-Higher — asymmetric PPO clip (ε_low < ε_high) so entropy does not die
2. 🎰 Dynamic sampling — drop groups where every sample has the same reward (zero advantage → wasted step)
3. 🔤 Token-level policy gradient — do not let long sequences dominate the batch
4. ✂️ Overlong reward shaping — penalize / filter truncated samples so length is not a spurious reward

Unsloth and TRL now expose DAPO / Dr. GRPO switches on the same GRPO trainer. GSPO (Qwen) sets importance sampling at sequence level instead of token level. The cutting-edge pattern is not “invent a new three-letter optimizer every month” — it is GRPO + the four DAPO fixes when you leave the toy regime.

🧭 Choosing an objective

🧪 Did it work?

Training loss going down is necessary and almost insufficient. You can memorize the JSONL and still fail the product question. Measure explicitly [7]:

flowchart TB
  Train["📁 Train JSONL"] --> Fit["🚂 Tuner.fit"]
  Fit --> Adapter["💾 Adapter / checkpoint"]
  Hold["📁 Holdout JSONL"] --> PPL["📉 Holdout perplexity"]
  Adapter --> PPL
  Probes["🎯 Probe JSONL"] --> Gen["✍️ Generate answers"]
  Adapter --> Gen
  Gen --> Sub["✅ SubstringJudge"]
  Gen --> LLM["🧑‍⚖️ LLMJudge"]
            

Perplexity = exp(mean token NLL). Intuition: effective branching factor for the next token — lower is better. It is a distributional score, not a fact check. A model can have great PPL and still never say “SlickML.”

Probes are the missing piece in most tutorials. A probe is a question plus a must_contain substring. Pass rate is the number you should be willing to fail a release on. Paraphrase the questions — if you probe with the training prompt verbatim, you measured memorization, not learning.

On tiny demo models, prefer a substring judge. A 135M checkpoint judging itself will under-score correct answers.

🔀 Multi-adapter merge — TIES and DARE

After a few PEFT runs you have several adapter folders on the same base: SFT facts, DPO manners, a GRPO reasoning head. Options:

1. 🔀 Switch — load them all, activate one with set_adapter
2. 🧬 Merge — fuse into one new adapter
3. 🍞 Bake — merge_and_unload into full weights for vLLM / TGI

Naive averaging lets useful edits cancel when adapters disagree in sign, and keeps a lot of noisy small entries. TIES and DARE prune and resolve conflicts first [28] [29].

✂️ TIES — Trim, Elect Sign, Merge

1. Trim — drop the smallest-magnitude updates; keep density ∈ [0, 1]
2. Elect sign — where adapters disagree, pick a consensus sign
3. Merge — average the survivors under that sign

Intuition: keep the big edits, resolve fights over direction, then combine.

🎲 DARE — Drop And REscale

1. Drop a large share of delta entries (again, density)
2. Rescale survivors so expected magnitude stays honest
3. Often paired with TIES-style signs → dare_ties / dare_linear

Intuition: many fine-tune deltas are redundant; dropping most and rescaling keeps behavior while reducing interference. The Super Mario paper’s claim in one sentence: homologous models have sparse, over-parameterized deltas you can lottery-ticket away [29].

Hard requirements. Same model_id. Each path is a PEFT dir with adapter_config.json. For ties / dare_* / linear, every adapter must share the same LoRA r — else use an SVD variant (ties_svd, …) or retrain at matching rank. Density 0.5 is a sane start. Weights per adapter can be < 1 or even negative.

🧞 slick-tune — the orthogonal toolkit

I got tired of copy-pasting PEFT + TRL scripts that couple the model, the adapter, the loss, and the eval into one argparse monster. slick-tune is a small composable library on Transformers + PEFT + TRL: swap one axis, keep the rest [23] [30].

model  ×  strategy  ×  objective  ×  data  ×  metrics

flowchart TB
  subgraph IN["📥 You provide"]
    mid["🧠 model_id"]
    strat["🧩 Strategy"]
    obj["🎓 Objective"]
    data["📁 train JSONL"]
    hold["📉 optional eval JSONL"]
    probes["🎯 optional probes"]
  end
  subgraph TUNER["🚂 Tuner.fit"]
    load["load tokenizer + model"]
    apply["strategy.apply or adapter_path warm-start"]
    train["TRL trainer for objective"]
    metrics["MetricsTracker"]
  end
  subgraph OUT["📤 Artifacts"]
    ckpt["💾 adapter or full weights"]
    mj["📊 metrics.json"]
    merged["🔀 optional TIES / DARE merge"]
  end
  mid --> load
  strat --> apply
  load --> apply --> train
  obj --> train
  data --> train
  train --> metrics
  hold --> metrics
  probes --> metrics
  train --> ckpt
  metrics --> mj
  ckpt --> merged
            

📌 A LoRA + SFT run

from slicktune import LoRAStrategy, SFTObjective, Tuner

Tuner(
    model_id="HuggingFaceTB/SmolLM2-135M-Instruct",
    strategy=LoRAStrategy(r=16, alpha=32, dropout=0.05),
    objective=SFTObjective(),
    output_dir="outputs/sft_lora",
    eval_data="examples/data/about_amir.eval.jsonl",
).fit("examples/data/about_amir.jsonl")

Swap the strategy, keep the JSONL:

DoRAStrategy(r=16, alpha=32)          # quality headroom
AdaLoRAStrategy(init_r=16, target_r=12, tinit=60, tfinal=30)
QLoRAStrategy(r=16, alpha=32)         # CUDA + bitsandbytes
FullStrategy()                        # every weight

Swap the objective, keep LoRA:

DPOObjective(beta=0.1)   # prefs JSONL
ORPOObjective(beta=0.1)  # same prefs, no ref
KTOObjective(beta=0.1)   # unpaired labels
GRPOObjective(beta=0.0, num_generations=4, max_completion_length=96)

⌨️ CLI

# 🏋️ Train
uv run slicktune train --strategy lora \
  --data examples/data/about_amir.jsonl \
  --eval-data examples/data/about_amir.eval.jsonl \
  --output outputs/sft_lora

# 🧪 Probes — did your facts stick?
uv run slicktune probe \
  --model-dir outputs/sft_lora \
  --probes examples/data/about_amir.probes.jsonl

# 📊 Holdout PPL + judges
uv run slicktune eval \
  --model-dir outputs/sft_lora \
  --eval-data examples/data/about_amir.eval.jsonl \
  --probes examples/data/about_amir.probes.jsonl \
  --judge substring

# 🔀 Merge adapters
uv run slicktune merge \
  --adapter outputs/sft_lora \
  --adapter outputs/dpo_lora:0.5 \
  --method ties --density 0.5 \
  --output outputs/merged_ties

🔀 Merge in Python

from slicktune import AdapterRef, merge_adapters, bake_adapter, load_multi_adapters

merge_adapters(
    model_id="HuggingFaceTB/SmolLM2-135M-Instruct",
    adapters=[
        AdapterRef(path="outputs/sft_lora", name="sft", weight=1.0),
        AdapterRef(path="outputs/dpo_lora", name="dpo", weight=0.5),
    ],
    output_dir="outputs/merged_ties",
    method="ties",
    density=0.5,
)

bake_adapter(adapter_dir="outputs/sft_lora", output_dir="outputs/sft_baked")

The personal loop the repo is built around: edit a tiny “about me” JSONL, a held-out paraphrase file for PPL, and probes with must_contain. Train on SmolLM2-135M so a laptop can finish. Probe. If the pass rate did not move, the run did not work — no matter what the loss plot says.

📦 pip install slicktune · Python 3.10–3.13 · extra qlora for CUDA 4-bit. Docs: docs.slickml.com/slick-tune. Visual guide: Fine-Tuning LLMs: A Visual Guide [7].

🪙 My 2 cents

Stop asking “should we fine-tune?” Ask the questions that force an architecture:

• 🎯 Can few-shot or many-shot ICL carry the task without a training run?
• 📚 Can retrieval serve the facts that change?
• 📝 Do I have demonstrations, pairs, thumbs, or a verifier?
• 💾 Is VRAM or quality the binding constraint?
• 🎯 What substring / test / rubric fails the release?
• 🔀 Do I need one specialist, or several adapters I can merge later?

Answer those and the stack almost picks itself:

• 🎯 ICL (few-shot, then many-shot) when demos fit the window and the empty prompt is allowed to be weak
• 💬 Prompt + RAG when knowledge moves and the format already works
• 🧩 LoRA SFT when style, schema, or stable facts must live in the weights
• 📦 QLoRA when the base is big and the GPU is not
• ✨ DoRA / PiSSA / rsLoRA when vanilla LoRA is the ceiling, not the floor
• ⚖️ DPO / ORPO / KTO when you can label better vs worse
• 🎯 GRPO when a program can score the answer — after SFT, not instead of it
• ✂️ TIES / DARE when adapters disagree; bake when the serving engine wants one folder

Contracts that are non-optional:

• 🎛️ Strategy and objective stay orthogonal — do not rewrite the trainer to change the loss
• 🎯 Probes over vibes; holdout PPL over train loss
• ❄️ Fine-tune form; retrieve facts that churn
• 🧪 A 135M laptop loop that proves the plumbing before you spend H100 hours

🌱 Treat fine-tuning as something you compose — model × strategy × objective × data × metrics — not a Colab you hope converges. That is why slick-tune looks the way it does.

📚 References

1. Edward J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR 2022, arxiv.org/abs/2106.09685
2. Tim Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs,” NeurIPS 2023, arxiv.org/abs/2305.14314
3. Shih-Yang Liu et al., “DoRA: Weight-Decomposed Low-Rank Adaptation,” ICML 2024, arxiv.org/abs/2402.09353
4. Rafael Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” NeurIPS 2023, arxiv.org/abs/2305.18290
5. Zhihong Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” 2024, arxiv.org/abs/2402.03300
6. DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025, arxiv.org/abs/2501.12948
7. Amirhessam Tahmassebi / SlickML, “Fine-Tuning LLMs: A Visual Guide,” slick-tune docs, docs.slickml.com/slick-tune/pages/fine_tuning_guide.html
8. Dan Biderman et al., “LoRA Learns Less and Forgets Less,” TMLR 2024, arxiv.org/abs/2405.09673
9. Unsloth, “Continued LLM Pretraining,” unsloth.ai/blog/contpretraining
10. Hugging Face, PEFT LoRA reference, huggingface.co/docs/peft/…/lora
11. Damjan Kalajdzievski, “A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA,” 2023, arxiv.org/abs/2312.03732
12. Soufiane Hayou, Nikhil Ghosh, and Bin Yu, “LoRA+: Efficient Low Rank Adaptation of Large Models,” 2024, arxiv.org/abs/2402.12354
13. Fanxu Meng, Zhaohui Wang, and Muhan Zhang, “PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models,” 2024, arxiv.org/abs/2404.02948
14. Yixiao Li et al., “LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models,” ICLR 2024, arxiv.org/abs/2310.08659
15. Zhengbo Wang et al., “LoRA-Pro: Are Low-Rank Adapters Properly Optimized?,” 2024, arxiv.org/abs/2407.18242
16. Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano, “VeRA: Vector-based Random Matrix Adaptation,” ICLR 2024, arxiv.org/abs/2310.11454
17. Jiawei Zhao et al., “GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection,” ICML 2024, arxiv.org/abs/2403.03507
18. RSRA: “Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation,” 2026, arxiv.org/abs/2607.09757
19. “Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels,” 2026, arxiv.org/abs/2603.22276
20. Qingru Zhang et al., “AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning,” ICLR 2023, arxiv.org/abs/2303.10512
21. “Post-Training in 2026: GRPO, DAPO, RLVR & Beyond,” llm-stats.com/blog/research/post-training-techniques-2026
22. Long Ouyang et al., “Training language models to follow instructions with human feedback,” NeurIPS 2022, arxiv.org/abs/2203.02155
23. SlickML, slick-tune, github.com/slickml/slick-tune
24. Jiwoo Hong, Noah Lee, and James Thorne, “ORPO: Monolithic Preference Optimization without Reference Model,” EMNLP 2024, arxiv.org/abs/2403.07691
25. Kawin Ethayarajh et al., “KTO: Model Alignment as Prospect Theoretic Optimization,” ICML 2024, arxiv.org/abs/2402.01306
26. Yu Meng, Mengzhou Xia, and Danqi Chen, “SimPO: Simple Preference Optimization with a Reference-Free Reward,” NeurIPS 2024, arxiv.org/abs/2405.14734
27. Qiying Yu et al., “DAPO: An Open-Source LLM Reinforcement Learning System at Scale,” 2025, arxiv.org/abs/2503.14476
28. Prateek Yadav et al., “TIES-Merging: Resolving Interference When Merging Models,” NeurIPS 2023, arxiv.org/abs/2306.01708
29. Le Yu et al., “Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch,” ICML 2024, arxiv.org/abs/2311.03099
30. slicktune on PyPI, pypi.org/project/slicktune
31. Hugging Face, TRL documentation, huggingface.co/docs/trl
32. Tom B. Brown et al., “Language Models are Few-Shot Learners,” NeurIPS 2020, arxiv.org/abs/2005.14165
33. Rishabh Agarwal et al., “Many-Shot In-Context Learning,” 2024, arxiv.org/abs/2404.11018
34. Sewon Min et al., “Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?,” EMNLP 2022, arxiv.org/abs/2202.12837
35. Damai Dai et al., “Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers,” 2023, arxiv.org/abs/2212.10559
36. Wenchong He, Liqian Peng, Zhe Jiang, and Alec Go, “You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language Models,” 2025, arxiv.org/abs/2506.11103
37. Bishwamittra Ghosh et al., “Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective,” ACL 2026, aclanthology.org/2026.acl-long.1932