Pretraining, Instruction Tuning, and Alignment

Real-World Scenario

A team downloads an open-weight “base” model expecting it to behave like ChatGPT, sends it "Summarize this contract:", and gets back more legal-sounding boilerplate instead of a summary — the model doesn’t know it’s supposed to do something with the text, it’s still just predicting a plausible continuation. Understanding the pretraining -> instruction-tuning -> alignment pipeline explains exactly why, and why the fix isn’t a better prompt but the right model stage and the right input format.

The Pipeline

Pretraining (next-token prediction over ~trillions of tokens)
        |
        v
   Base model  -- can continue text fluently, follows no instructions
        |
        v
Instruction tuning (fine-tune on (instruction, ideal response) pairs)
        |
        v
Preference / alignment training (RLHF or similar, rewards human-preferred outputs)
        |
        v
  Chat / instruct model  -- follows instructions, refuses some requests, has a "voice"

Each stage starts from the previous stage’s weights and narrows behavior further — it doesn’t add facts the base model didn’t already learn during pretraining.

Same Prompt, Different Model Stages

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.manual_seed(0)
prompt = "The best way to learn Python is"

base_tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")

inputs = base_tok(prompt, return_tensors="pt")
out = base_model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(base_tok.decode(out[0], skip_special_tokens=True))

$ python base_model.py
The best way to learn Python is to write code. This is a very important part of learning any programming language. It is also a very important part of learning Python. If you are a beginner, you should start with a simple program

The base model just continues the sentence — a plausible next sentence, not an answer to a question, because nothing about “the best way to learn Python is” looks like a request in its training data’s shape.

The Same Raw Prompt Sent to an Instruction-Tuned Model

instruct_tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
instruct_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

inputs = instruct_tok(prompt, return_tensors="pt")
out = instruct_model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(instruct_tok.decode(out[0], skip_special_tokens=True))

$ python instruct_raw.py
The best way to learn Python is by doing it. This means that you should start with a simple program and gradually increase the complexity of the program as you become more comfortable with the language.

Here's an example of a simple Python program

This is the surprising part: the instruction-tuned model, given the exact same raw text prompt with no chat formatting, still just continues the sentence — a slightly more organized continuation, but not the structured “helpful assistant” behavior you’d expect from ChatGPT-style output. Instruction tuning didn’t make the model generically smarter at detecting requests; it trained the model to recognize one specific input format as “this is a conversation, respond as the assistant.”

The Same Question, Through the Chat Template

messages = [{"role": "user", "content": "What is the best way to learn Python?"}]
chat_input = instruct_tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(chat_input)

$ python chat_template.py
<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the best way to learn Python?<|im_end|>
<|im_start|>assistant

inputs = instruct_tok(chat_input, return_tensors="pt")
out = instruct_model.generate(**inputs, max_new_tokens=60, do_sample=False)
print(instruct_tok.decode(out[0], skip_special_tokens=True))

$ python chat_generate.py
system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
user
What is the best way to learn Python?
assistant
Learning Python can be approached in several ways depending on your learning style and goals. Here are some of the most effective methods:

1. **Online Courses:**
   - Platforms like Coursera, Udemy, and edX offer courses that cover various aspects of Python programming.
   - Websites like

Same model, same weights, same underlying question — but wrapped in the <|im_start|>role ... <|im_end|> structure the instruction-tuning data actually used. Only now does it produce the structured, itemized, clearly-addressed-to-you response. This is exactly why every production LLM SDK (client.messages.create(messages=[...]) in intro.md) builds this template for you automatically — skip it, and even a fully instruction-tuned model degrades toward plain text continuation.

Practice

1. Send the base model (Qwen/Qwen2.5-0.5B) the prompt "Question: What is the capital of France?\nAnswer:" and compare it to the same prompt sent to the instruct model without a chat template.

   Solution

   prompt = "Question: What is the capital of France?\nAnswer:"
   inputs = base_tok(prompt, return_tensors="pt")
   out = base_model.generate(**inputs, max_new_tokens=20, do_sample=False)
   print("Base:", base_tok.decode(out[0], skip_special_tokens=True))
   
   inputs = instruct_tok(prompt, return_tensors="pt")
   out = instruct_model.generate(**inputs, max_new_tokens=20, do_sample=False)
   print("Instruct (raw):", instruct_tok.decode(out[0], skip_special_tokens=True))

   Base: Question: What is the capital of France?
   Answer: The capital of France is Paris.
    A single-select problem: Is the question answered in a satisfactory fashion
   Instruct (raw): Question: What is the capital of France?
   Answer: The capital of France is Paris. Paris is a city in western France, on the banks of the

   Even the base model gets this right here — a “Question:/Answer:” shape is common enough in pretraining data (quiz banks, FAQs) that it triggers a reasonable completion without any instruction tuning at all. It still trails off into an unrelated continuation (“A single-select problem…”) once the actual answer is done, which the instruct model — even without its chat template — doesn’t do, staying on-topic about Paris for longer. Neither behaves like a model that knows the response is finished.

2. Build the chat template for a system message that changes the assistant’s persona (e.g. “You only answer in one sentence”) and confirm the generated response respects it.

   Solution

   messages = [
       {"role": "system", "content": "You only answer in one sentence. No exceptions."},
       {"role": "user", "content": "What is the best way to learn Python?"},
   ]
   chat_input = instruct_tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
   inputs = instruct_tok(chat_input, return_tensors="pt")
   out = instruct_model.generate(**inputs, max_new_tokens=60, do_sample=False)
   print(instruct_tok.decode(out[0], skip_special_tokens=True))

   system
   You only answer in one sentence. No exceptions.
   user
   What is the best way to learn Python?
   assistant
   Practice, practice, practice!

   A system message written into the chat template genuinely steers behavior — this is a real instruction-following response, not a coincidence, and it’s dramatically shorter than the multi-paragraph answer from the plain “you are a helpful assistant” system prompt earlier in this lesson.

3. Print instruct_tok.chat_template directly and identify which part of the raw string corresponds to <|im_start|> — this is the literal Jinja template the tokenizer applied in the examples above.

   Solution

   print(instruct_tok.chat_template[:300])

   {%- if tools %}
       {{- '<|im_start|>system\n' }}
       {%- if messages[0]['role'] == 'system' %}
           {{- messages[0]['content'] }}
       {%- else %}
           {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
       {%- endif %}
       {{- "\n\n# Tools\n\nYou may call one or mor

   apply_chat_template isn’t magic — it’s rendering a Jinja template stored on the tokenizer itself, which is exactly why different model families (Llama’s [INST]...[/INST], Qwen’s <|im_start|>) need their own tokenizer’s template rather than a hand-written format string.

4. The base model’s continuation of the original prompt trailed off mid-sentence (“…you should start with a simple program”). Increase max_new_tokens to 80 and see whether it ever produces something that looks like a natural stopping point, or just keeps continuing.

   Solution

   inputs = base_tok(prompt, return_tensors="pt")
   out = base_model.generate(**inputs, max_new_tokens=80, do_sample=False)
   print(base_tok.decode(out[0], skip_special_tokens=True))

   The best way to learn Python is to write code. This is a very important part of learning any programming language. It is also a very important part of learning Python. If you are a beginner, you should start with a simple program. If you are an intermediate programmer, you should start with a more complex program. If you are a professional programmer, you should start with a more complex program. If you are a beginner, you

   It keeps going, and even starts looping through the same “beginner / intermediate / professional” pattern rather than concluding — a base model has no trained notion of “this response is complete,” it only stops at max_new_tokens or by chance emitting an end-of-text token. Instruction tuning is also what teaches a model to actually emit a stop token when a response (not just a sentence) is finished.