Fine-Tuning LoRA on Qwen3: Step-by-Step Guide

July 20, 2026 · 15 min read · Pricing

Full fine-tuning a 72B model costs thousands in GPU hours. LoRA (Low-Rank Adaptation) lets you fine-tune large models on a single consumer GPU by training only a small fraction of parameters. This guide walks you through fine-tuning Qwen3 with LoRA using the PEFT library.

Why LoRA Over Full Fine-Tuning?

Method	Trainable Params (72B)	GPU Required	Training Time	Cost
Full fine-tune	72B (100%)	8×A100 80GB	~48 hours	~$2,000
LoRA (r=16)	~100M (0.1%)	1×RTX 3090 24GB	~4 hours	~$2
QLoRA (4-bit)	~100M (0.1%)	1×RTX 3060 12GB	~6 hours	~$1

Environment Setup

pip install torch==2.4.0 transformers==4.46.0 \
    peft==0.14.0 bitsandbytes==0.44.0 \
    datasets accelerate trl

# Verify GPU
python -c "import torch; print(torch.cuda.get_device_name(0))"
# Expected: NVIDIA RTX 3090 (or similar)

Step 1: Load Model with QLoRA

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

# 4-bit quantization to fit in 24GB VRAM
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_quant_type="nf4",
)

model_name = "Qwen/Qwen3-8B"  # Start with 8B for testing

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

# Configure LoRA
lora_config = LoraConfig(
    r=16,                  # Rank: higher = more capacity
    lora_alpha=32,          # Scaling factor (typically 2× rank)
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 37M || all params: 8.7B || trainable%: 0.42%

Step 2: Prepare Dataset

from datasets import load_dataset
import json

# Your fine-tuning data — instruction-response pairs
dataset = [
    {"instruction": "Summarize this text in one sentence.", "output": "The key takeaway is..."},
    {"instruction": "Convert this JSON to CSV.", "output": "name,age,city\n..."},
    # ... hundreds or thousands of examples
]

# Format for supervised fine-tuning
def format_example(example):
    return f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"

# Save as JSONL for TRL trainer
with open("train_data.jsonl", "w") as f:
    for item in dataset:
        f.write(json.dumps({"text": format_example(item)}) + "\n")

# Load with datasets
from datasets import Dataset
ds = Dataset.from_json("train_data.jsonl")
ds = ds.train_test_split(test_size=0.1, seed=42)

Step 3: Train with TRL

from trl import SFTTrainer, SFTConfig

training_args = SFTConfig(
    output_dir="./qwen3-lora",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,  # Effective batch = 16
    learning_rate=2e-4,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,  # Use bfloat16 if supported
    max_seq_length=512,  # Keep short for cost efficiency
    warmup_ratio=0.1,
)

trainer = SFTTrainer(
    model=model,
    train_dataset=ds["train"],
    eval_dataset=ds["test"],
    args=training_args,
    processing_class=tokenizer,
)

trainer.train()

# Save LoRA adapter (NOT the full model — adapter is ~200MB)
trainer.save_model("./qwen3-lora/final")
print("LoRA adapter saved to ./qwen3-lora/final")

Step 4: Merge and Deploy

# Option A: Merge LoRA into base model for local use
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B", device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "./qwen3-lora/final")
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./qwen3-finetuned")

# Option B: Upload to AIWave for API access
# The merged model can be deployed via vLLM or TGI
# Then added as a custom channel on AIWave

What to Fine-Tune: Use Cases

Use Case	Training Data Needed	Qwen3 Model	Expected Result
Domain-specific QA	500-2K Q&A pairs	Qwen3-8B	Accurate domain responses
Code style formatting	1K-5K code examples	Qwen3-Coder	Follows your style
Product tone of voice	1K-3K examples	Qwen3-8B	Consistent brand voice
Structured output format	500+ format examples	Qwen3-8B	Reliable JSON/XML