Teaching a 1.2B model to solve chess puzzles with SFT

A 1.2B model went from 0% to solving 45.8% of chess puzzles.

The short version

  • A 1.2B model went from 0% to 62.7% first moves right, 45.8% solved.
  • Reasoning from code or DeepSeek did worse than no thinking at all.
  • Teaching board skills and chess basics improved the model.
  • DeepSeek V4.1 Flash solves more (58.2%), but is close on first moves.
  • Still fails at quiet moves, about 90% of the time.

Motivation and target

With the World Chess Championship approaching, my interest in chess rekindled again. I was wondering how difficult it would be for an LLM to solve chess puzzles.

As many people who have played chess would already know, solving chess puzzles is a different skill than playing chess. While solving a puzzle, you already view the position with the mindset of trying to find moves that check the king, win the queen, fork their pieces and whatnot.

At the same time, puzzles are not forgiving. While playing chess, you might play some sub-par moves, but you have to be accurate while solving puzzles.

I thought, for a small model, it might be an interesting challenge. The task is narrower, but it requires more precision.

Fig. From worse than random to 63% same 1,000 test puzzles for every row
first movesolved
  1. Untrained model 0% first move 0% solved
  2. Final model 62.7% 45.8%
  3. DeepSeek V4.1 Flash 67.5% 58.2%
Yellow is the first move right, dark is the whole puzzle solved. DeepSeek is dashed because it’s not a model I trained.

Setup

I found the Lichess/chess-puzzles dataset online. It had over 6M chess puzzles, annotated with information like rating, themes involved, etc. I felt like I could work with this. I kept only the good ones (popular, played at least 100 times), which left 3.7M.

Next, I wanted to pick a small model which can run well on my 5060ti. I considered multiple options, but ended up choosing LFM 2.5 1.2B Instruct. For finetuning, I decided to use LoRA adapters for all layers (attention and MLP both). I decided to go with a rank 64 adapter.

Baselines

Next, I had to think about the harness and the input/output format to get the baseline performance of the model. I created a harness where I provide the board to the model, let it think in some format, and finally produce an answer in a specific format. Since a puzzle can have multiple moves from both sides, I repeat the process for each correct move the model predicts. I kept aside a test set of 1000 puzzles, from games that never appear in training, to evaluate all the models on.

I’ll use this puzzle (rated 1,246) throughout the post to show how the model progresses towards solving it.

Fig. The example puzzle click a piece, then a square
h1g2f3e4d5c6b7a8

Black to move. Click a piece, then a square.

The example puzzle will be used throughout the blog.
Fig. One puzzle through the harness plays when it scrolls in · ← → to step
✓ more to play✗ or last movepuzzleLichessbuild promptharnessmodelLFM 1.2Bparse"Answer: …"checkvs solutionopponent repliesnext solution moveendsolved / failed✓ more to play✗ or last movepuzzleLichessbuild promptharnessmodelLFM 1.2Bparse"Answer: …"checkvs solutionopponentplays next moveendsolved / failed
· parsable· legal move· first move right· puzzle solved

1/11 puzzle oXs11 · rating 1246 · Black to move

The final model going through the example puzzle. One wrong move and the puzzle is over.
Fig. How hard are the test puzzles?
puzzles
0 40 80 120 160 13 <600 42 600 110 800 143 1000 151 1200 136 1400 132 1600 117 1800 73 2000 54 2200 19 2400 10 2600+
  1. <600 13
  2. 600 42
  3. 800 110
  4. 1000 143
  5. 1200 151
  6. 1400 136
  7. 1600 132
  8. 1800 117
  9. 2000 73
  10. 2200 54
  11. 2400 19
  12. 2600+ 10
How many of the 1,000 test puzzles fall in each rating band. The median is 1,463.
The system prompt
You are solving a chess puzzle, one move per turn. Moves use UCI notation (e.g. e2e4, e7e8q).
Inside <think>...</think>, work it out from the board, step by step:
1. Scan: what the last move changed (en passant?), castling rights, whether you are in check, pins, the opponent's pieces and what they attack, then every move of your pieces, marking captures and checks.
2. Mates: for each checking move, list all the opponent's legal replies; none means checkmate.
3. Exchanges: for each capture, work out the exchange on that square and the net material.
4. Choose the best move.
Then end with a final line `Answer: <uci move>`.

We can see that the model is really bad at chess. Somehow it performs worse than choosing a random move: a random legal move gets the first move right 5% of the time. It seemed that the model couldn’t even find a legal move, or even read the board. On the example puzzle, it starts with this:

Fig. Untrained model on the example puzzle
h1g2f3e4d5c6b7a8
their last moverightwrong
Untrained modelno legal move ✗right move: Rd8+
Black has a pawn on e5 (Kh6) and a knight on f5 (Ph5).
It gets every piece wrong, even though the pieces are listed in the prompt.

I thought that if I gave it the set of legal moves in the prompt, then it can consider each move and choose the better one. So, I tried that, and the performance marginally improved. Even with the full list of legal moves, it couldn’t pick one.

Fig. Baselines
first movesolved
  1. Untrained model 0% first move 0% solved
  2. Untrained + legal-move list 2.3% 0.3%
  3. Random legal move 5% 0.7%
  4. DeepSeek V4.1 Flashup to 64k thinking tokens per move 67.5% 58.2%
DeepSeek is dashed because it’s not a model I trained.

DeepSeek on the other hand managed to think the puzzles through and make good moves: 67.5% of first moves correct and 58.2% puzzles solved, with up to 64k thinking tokens per move.

Experiment 1: reasoning written by code

The first experiment that I thought of was to generate some reasoning text from the board, programmatically. For each move, a script looked at what the opponent threatens, two or three candidate moves with Stockfish’s verdict on each, and why the right move works, and wrote it as plain text. This was cheap and quick to generate locally. I generated data for over 30k puzzles and started performing SFT.

<think>
The last move was Rd8. If I do nothing, Qxb7 would win my pawn on b7. The moves to consider are b5, Bc7 and
Qxh2+. After Bc7 Rxe8+ my position turns losing. After b5 Rxe8+ my position turns losing. Qxh2+ captures the pawn
on h2. It comes with check. The only legal reply is Kf1, and then Qh1# mates. The point is a forced mate.
</think>
Answer: e5h2

The performance improved. With some SFT, it generated a parsable response every time and a legal move 97% of the time. It got the first move right 23.9% of the time and solved 6.9% of the puzzles. This seemed promising.

But when I read the responses, most of what it wrote was made up. It writes in the same confident style, but the facts are wrong:

Fig. Experiment 1 on the example puzzle
h1g2f3e4d5c6b7a8
their last moverightwrong
Exp 1: code-written reasoningQd8+ ✗right move: Rd8+
The only legal reply is Qf8, and then Qxf8# mates.
It says Qf8 is the only reply. Bf8 is also possible, so it’s not mate.

There was another interesting observation. The model did not generate <think> tokens sometimes. It generated henvisninger, which is a Norwegian word. Upon digging into this further, I found that its cosine similarity with the <think> token is 0.998. Therefore, the model could not differentiate between these two. This however did not affect the parsability of the text for the harness, so I let it be. For the runs mentioned below, I will show the <think> token even if the model generated some other token.

Fig. What the model writes instead of <think>
should write<think>token 64400
cosine 0.998
actually writes henvisningertoken 53019
henvisninger
Last move: f7g8 (white king) -> en passant none
Castling rights: White none, Black none
In check: no
…
The model starts with ” henvisninger” instead of <think>. To the model, the two look almost the same.

Experiment 2: DeepSeek’s reasoning

Seeing how well DeepSeek performed, it seemed pretty straightforward to let the model train on DeepSeek’s traces for the next experiment. I gave DeepSeek the puzzle and the answer, and asked it to write the thinking a strong player would have before each move. I also gave it facts about each move checked by a chess engine, so it wouldn’t make things up.

The prompt I gave DeepSeek, and one of its answers

Prompt:

You are writing training data for a small chess model. You are given a puzzle conversation in which every assistant turn already contains the correct move, plus VERIFIED FACTS about each move computed by a chess engine. The facts are correct: base your reasoning on them and never contradict them. … For each turn, write the reasoning a strong player would have before playing that move: about 150-220 words per turn … what the opponent’s last move changed or threatens -> candidate moves considered -> WHY the chosen move works … Do not invent any [pieces, squares, attacks]. … No self-corrections, no “wait”, “actually”, “hmm”.

Response:

White’s king move to c2 looks like it tries to step away from the pressure, but it also leaves White ready to play Qxh5+ and pick off the h5 pawn if I give him time. I need to keep the initiative and look for a forcing check, because the white king is still exposed in the center. The natural candidate is Qd3#, putting the queen on the diagonal where it attacks the white king on c2. … This is the cleanest and most forcing continuation: I ignore the h5 pawn and end the game with checkmate.

I trained over the model I obtained after Experiment #1 and I found that the performance didn’t actually improve: 20.6% first move, 8.4% solved, and it played more illegal moves than before (93% legal vs 97%). I understood why after reading some of its responses. It learnt to follow the style of the response it is supposed to generate, but it did not develop the understanding for the game at all.

Fig. Experiment 2 on the example puzzle
h1g2f3e4d5c6b7a8
their last moverightwrong
Exp 2: + DeepSeek reasoningQf8+ ✗right move: Rd8+
White has no way to capture the queen or block the check.
Sounds convincing, but the reasoning is wrong.

Experiment 3: no thinking at all

I wanted to check one more thing. What if I just generate the final answer and let the model be trained to do this without thinking? I kept the legal move list in the prompt.

So, I trained the base model further on such data, and it performed better than either of the above experiments: 30.9% first move and 15.3% solved, more than double the puzzles of Experiment 1. The wrong reasoning was hurting the model more than helping it. Many times it would conclude a wrong move to be a mate, and then play it. The Experiment 1 model claimed a mate on its first move in 391 of the 1000 puzzles, and it was right only 96 times. Here however, since all the loss was concentrated on playing the correct move, it learnt to make the correct move more accurately. But the results plateaued, and I did not see any improvement with training further: one more epoch gave 31.5% first move and 16.6% solved, basically the same.

Fig. Experiment 3 on the example puzzle
h1g2f3e4d5c6b7a8
their last moverightwrong
Exp 3: answer onlyQg6 ✗right move: Rd8+
(no thinking: it just answers)
Just a move, no thinking. Wrong here, but right more often than the thinking models overall.

Teaching chess

From the last experiment’s results, I wanted to somehow make the thinking help the model rather than misdirecting it. As we saw earlier from the reasoning, it was pretty evident that the model does not know how to play chess at all. It is just copying the reasoning format.

To check if the model actually understood chess, I asked it to list the captures and checks. It could not answer correctly at all. If the model can’t even list the checks and captures correctly, it makes no sense to teach it to solve puzzles directly.

So, I decided to teach it to understand the board, the moves, the insights before asking it to solve the puzzles. Giving the model the list of legal moves is the same as spoonfeeding it. It should learn to figure those out on its own. And I thought that giving a 8x8 board might be helpful for humans, but it would be better if I listed the pieces and their location directly for the model. Therefore, I replaced the legal move list with a plain list of the pieces (White pieces: Kg8 Qf4 Bb4 ...).

Keeping that in mind, I prepared a curriculum for the model, listing all the skills the model should have before solving a puzzle.

#SkillBeforeAfter
1List all the pieces on the board, with their squares0%98–100%
2List all the moves of one piece (“where can the knight on e5 go?”)0%98%
3List all the captures you can make0%97%
4List all the moves that deliver check0%96%
5List all the legal moves0%60% (best 78%)
6Play a given move, then write the new position0%85%
7List all the captures that win material0%82%
8List all the moves that deliver checkmate0%78–80%
9Play a given move, then list the opponent’s checks and captures0%65%

Exact match (the whole answer must be right), held-out positions. Before = Exp 3 model, given the same question and told the answer format.

While training those skills, I found some things that were helpful. For instance, when I asked it to list captures, it was correct only ~13% of the time. When I asked it to list the same by walking each piece, square by square, its accuracy increased to ~86%. Similarly, for a bishop’s captures written in one go, it was right 11% of the times. Written as a walk, 100%. The model can’t compute a whole list in one step, but it can take many small steps it already knows.

Fig. Same skill, two ways of writing it switch between the two
++x++xxh1g2f3e4d5c6b7a8
Qf6: up f7+ f8+ edge | down f5(white pawn)x | left e6+ d6 c6 b6 a6 edge | right g6 h6(own) | up-right g7(own) | up-left e7 d8+ edge | down-right g5(white knight)x | down-left e5 d4 c3(white pawn)x
Rd2: up d3 d4 d5 d6 d7 d8+ edge | down d1 edge | left c2 b2 a2 edge | right e2 f2 g2 h2 edge

Captures: f6f5f6g5f6c3

x is a capture, + is a check. Each line stops at the first piece it hits.

Experiment 4: thinking with the skills

This time, I wanted to check if the skills that the model learnt are helpful in solving the end task or not.

So, I trained the model (the skilled model from the previous step) further, and this time asked it to think in the way it learnt before: scanning the board, checking candidate moves and the opponent’s responses, and evaluating the best move. This was the first thinking model that beat answer-only on solved puzzles, and without the legal move list: 36.4% first move, 18.1% solved.

Fig. Experiment 4 (first version) on the example puzzle
h1g2f3e4d5c6b7a8
their last moverightwrong
Exp 4: think with the skills (first version)Qd8+ ✗right move: Rd8+
d2d8: black rook d2->d8; their checks none; their captures none -> losing
The board scan is right, but it calls the winning move Rd8+ “losing”. It was guessing the Stockfish verdicts it saw in training.

However, from the model’s output, I could see many gaps still. Based on that, I added more and more skills to help it crack the puzzles.

The first gap was those verdicts. I replaced them with two things the model can actually work out: for every check, list all the opponent’s replies, and for every capture, count the trades on that square (pawn 1, knight and bishop 3, rook 5, queen 9). On its own, this barely moved the score: 37.7% first move and 17.4% solved (from 36.4% and 18.1%). But I thought that this skill was necessary to learn the next one where I observed that it missed many mate-in-1s, as it thought the king can capture back the checking piece even when the piece was protected. I trained to fix this by letting it check where the king can actually capture, and the performance improved further: 43.4% first move, 28.5% solved.

Fig. Experiment 4 (after the gap fixes) on the example puzzle
h1g2f3e4d5c6b7a8
their last moverightwrong
Exp 4: after the gap fixesRd8+ ✓
f6f8: black queen f6->f8; king g8: f7 attacked by Qf8 | xg7 protected by Kh6 | h7 attacked by Kh6 | xf8 free | h8 attacked by Qf8; other replies b4f8 -> not mate
It sees Qf8+ isn’t mate because the bishop can take, and plays Rd8+.

Experiment 5: longer training, and DPO

I only trained the model for ~1 hour in each experiment. Since we have lots of data, I wanted to train it on more data to get better accuracy. For that I tried several long SFT runs, 3 hours each, on puzzles the model had never seen.

Another thing that I wanted to try was DPO (Direct Preference Optimization). I obtained multiple answers from the model on the same puzzle, and trained it to generate more of the preferred ones and reject the others. Trainable data was obtained from ~20k puzzles where the model was sometimes wrong and sometimes right.

DPO did not improve the model substantially. I saw marginal gains with it.

Fig. Longer training
first movesolved
  1. Experiment 4 43.4% first move 28.5% solved
  2. + 3 h SFT 57% 39.4%
  3. + 3 h SFT 59% 42%
  4. + DPO 59.5% 43.5%
  5. + 3 h SFT 61.2% 44.9%
  6. + DPO again 62.7% 45.8%
Each row keeps training from the row above.
Fig. The final model solves the example puzzle click a move or use ← →
h1g2f3e4d5c6b7a8

Black to move, after Kg8.

final model playedRd8+ ✓
its thinking (2,342 chars)
<think>
Last move: f7g8 (white king) -> en passant none
Castling rights: White none, Black none
In check: no
Pinned: none
Their pieces:
 Kg8: next f7 g7 h7 f8 h8
 Qf4: up f5(own pawn) | down f3 f2 f1 edge | left e4 d4 c4 b4(own bishop) | right g4 h4 edge | up-right g5(own knight) | up-left e5 d6 c7 b8 edge | down-right g3(own pawn) | down-left e3 d2(black rook)
 Bb4: up-right c5 d6 e7 f8 edge | up-left a5 edge | down-right c3(own pawn) | down-left a3 edge
 Ng5: jumps f3 h3 e4 e6 f7 h7
 Pc3: attacks b4 d4
 Pg3: attacks f4 h4
 Pf5: attacks e6 g6
Their king: g8
down: g7(own pawn)
left: f8 e8 d8 c8 b8 a8 edge
right: h8 edge
down-right: h7 edge
down-left: f7 e6 d5 c4 b3 a2 edge
Knight squares: f6(own queen) h6(own king) e7
Check squares: rook/queen f8 e8 d8 c8 b8 a8 h8; bishop/queen h7 f7 e6 d5 c4 b3 a2; knight f6; pawn f9 h9
My pieces:
 Kh6: next g5(white knight)!(Qf4) h5(own) g6!(Pf5) g7(own) h7!(Ng5)
 Qf6: up f7+ f8+ edge | down f5(white pawn)x | left e6+ d6 c6 b6 a6 edge | right g6 h6(own) | up-right g7(own) | up-left e7 d8+ edge | down-right g5(white knight)x | down-left e5 d4 c3(white pawn)x
 Rd2: up d3 d4 d5 d6 d7 d8+ edge | down d1 edge | left c2 b2 a2 edge | right e2 f2 g2 h2 edge
 Ph5: push h4 | diag g4(empty)
 Pg7: push g6 | diag f6(own) h6(own)
Checking moves: d2d8 f6d8 f6e6 f6f7 f6f8
d2d8: black rook d2->d8; king g8: f7 attacked by Qf6 | xg7 protected by Kh6 Qf6 | h7 attacked by Kh6 | f8 attacked by Qf6 Rd8 | h8 attacked by Rd8; other replies b4f8 -> not mate
f6d8: black queen f6->d8; king g8: f7 attacked by Qd8 | xg7 protected by Kh6 | h7 attacked by Kh6 | f8 attacked by Qd8 | h8 attacked by Qd8; other replies b4f8 -> not mate
f6e6: black queen f6->e6; king g8: f7 attacked by Qe6 | xg7 protected by Kh6 | h7 attacked by Kh6 | f8 free | h8 free; other replies f4e6 f5e6 g5e6 g5f7 -> not mate
f6f7: black queen f6->f7; king g8: xf7 free | xg7 protected by Kh6 Qf7 | h7 attacked by Kh6 Qf7 | f8 attacked by Qf7 | h8 free; other replies g5f7 -> not mate
f6f8: black queen f6->f8; king g8: f7 attacked by Qf8 | xg7 protected by Kh6 Qf8 | h7 attacked by Kh6 | xf8 free | h8 attacked by Qf8; other replies b4f8 -> not mate
Mates: none
Captures: f6c3 f6f5 f6g5
f6c3: f6c3(Qx1) b4c3(Bx9) -> net -8
f6f5: f6f5(Qx1) f4f5(Qx9) -> net -8
f6g5: f6g5(Qx3) f4g5(Qx9) h6g5(Kx9) -> net +3
Winning captures: f6g5
Best: d2d8
Answer: d2d8
Rd8+ first. After Bf8, Qxf8#.
Fig. Final model vs DeepSeek, by puzzle rating same 1,000 test puzzles
final modelDeepSeek V4.1 Flash
0 25 50 75 100% 95 100 <800 93 94 800–1000 82 86 1000–1200 72 77 1200–1400 54 67 1400–1600 45 51 1600–1800 37 44 1800–2000 52 49 2000–2200 40 37 2200+
  1. <800 95%100%
  2. 800–1000 93%94%
  3. 1000–1200 82%86%
  4. 1200–1400 72%77%
  5. 1400–1600 54%67%
  6. 1600–1800 45%51%
  7. 1800–2000 37%44%
  8. 2000–2200 52%49%
  9. 2200+ 40%37%
final modelDeepSeek V4.1 Flash
0 25 50 75 100% 95 100 <800 86 92 800–1000 71 82 1000–1200 58 72 1200–1400 42 57 1400–1600 22 44 1600–1800 19 32 1800–2000 12 23 2000–2200 6 12 2200+
  1. <800 95%100%
  2. 800–1000 86%92%
  3. 1000–1200 71%82%
  4. 1200–1400 58%72%
  5. 1400–1600 42%57%
  6. 1600–1800 22%44%
  7. 1800–2000 19%32%
  8. 2000–2200 12%23%
  9. 2200+ 6%12%
DeepSeek can think for up to 64k tokens per move. My model uses about 870.

Gaps left and next steps

Through this experiment, I understood that even with just SFT (and some DPO), it is possible to teach a language model some skill which it spectacularly lacked. One has to carefully teach it from the bottom up rather than expecting it to just learn by seeing some examples. Still, there are lots of gaps in its performance. For instance, it is unable to play a quiet move. It tries to look for captures, attacks and checks most of the time. When the right move is quiet, it gets it wrong about 90% of the time. And it’s not that it can’t find the move, if I let it answer 8 times, the right move shows up in 83% of puzzles.

Fig. A quiet move the final model misses
a8b7c6d5e4f3g2h1
their last moverightwrong
Final modelhxg6 ✗right move: h6
Best: h5g6
The right move is the quiet h6. It only looks at checks and captures, so it plays hxg6.

More sophisticated methods like On-Policy Distillation and RL might help in improving this model further.