Question 1

What is fine-tuning, and when should you fine-tune an LLM?

Accepted Answer

Fine-tuning is the process of taking a pre-trained foundation model and updating its internal weights using a smaller, domain-specific dataset. You should fine-tune when you need the model to adopt a highly specific tone, format, or industry vocabulary that cannot be achieved via prompting. See video: AI Engineering Explained. Base models like Llama 3 are trained on general internet data. If you want a model to act like a strict medical coder outputting proprietary JSON, zero-shot prompting might fail. Fine-tuning adjusts the model's actual neural weights so it inherently 'knows' the desired format and style. You should NOT fine-tune simply to add new factual knowledge (use RAG instead).

Question 2

Explain the difference between full fine-tuning and parameter-efficient fine-tuning (PEFT).

Accepted Answer

Full fine-tuning updates every single weight in the neural network, requiring massive GPU clusters. PEFT techniques freeze the base model and only update a tiny fraction of parameters (or add new ones), allowing fine-tuning on consumer hardware. See video: AI Engineering Explained. A model like Llama 3 70B has 70 billion parameters. Full fine-tuning requires keeping optimizer states for all 70B parameters in VRAM, which requires multiple A100 GPUs. PEFT (like LoRA) freezes the original 70B parameters and only trains a small adapter layer (e.g., 20 million parameters). This drastically reduces VRAM requirements, often allowing fine-tuning on a single consumer GPU (like an RTX 4090).

Question 3

What is LoRA (Low-Rank Adaptation), and how does it work?

Accepted Answer

LoRA is a PEFT technique that freezes the pre-trained model weights and injects trainable rank decomposition matrices into the architecture, significantly reducing the number of trainable parameters. See video: AI Engineering Explained. Neural network weight matrices are massive. LoRA hypothesizes that the 'change' needed to learn a new task has a low intrinsic rank. Instead of updating a massive 10,000 x 10,000 weight matrix directly, LoRA freezes it and trains two much smaller matrices (e.g., 10,000 x 8 and 8 x 10,000). When multiplied together, they form a matrix of the same size as the original, which is added to the base weights.

Question 4

What is QLoRA, and how does it enable fine-tuning on consumer hardware?

Accepted Answer

QLoRA (Quantized LoRA) combines quantization with LoRA. It quantizes the massive base model down to 4-bit precision to fit into GPU memory, and then trains a 16-bit LoRA adapter on top of it. See video: AI Engineering Explained. Even with standard LoRA, the massive base model must be loaded into VRAM in 16-bit precision, which requires huge memory. QLoRA introduces NormalFloat 4-bit (NF4) quantization. It squashes the base model into 4-bit memory, drastically reducing VRAM usage. The gradients are backpropagated through the frozen 4-bit model to train the 16-bit LoRA adapter. This enables fine-tuning a 65B parameter model on a single 48GB GPU.

Question 5

Explain Prefix Tuning and Prompt Tuning. How are they different from LoRA?

Accepted Answer

Prefix and Prompt Tuning are PEFT methods that prepend trainable continuous 'soft prompt' vectors to the input sequence, rather than modifying the internal transformer weight matrices like LoRA does. While LoRA injects trainable matrices directly into the model's internal attention layers, Prompt Tuning freezes the entire model and only trains a sequence of virtual tokens (embeddings) placed at the beginning of the user's prompt. Prefix tuning is similar but prepends trainable tensors to every transformer layer, not just the input. They act as automated, mathematically optimized prompt engineering.

Question 6

What is adapter-based fine-tuning?

Accepted Answer

Adapter-based fine-tuning inserts small, trainable bottleneck layers (adapters) between the frozen pre-trained layers of a Transformer model to learn task-specific representations. See video: AI Engineering Explained. This was one of the earliest PEFT methods before LoRA. An adapter is a tiny neural network (down-projection followed by up-projection) sandwiched between the attention and feed-forward layers. Because they are modular, you can train one adapter for French translation and another for Python coding, and hot-swap them at runtime over the same base model.

Question 7

What is RLHF (Reinforcement Learning from Human Feedback), and how is it used to align LLMs?

Accepted Answer

RLHF is a fine-tuning technique that uses human ratings to train a Reward Model, which then uses reinforcement learning (typically PPO) to optimize the LLM's outputs toward human preferences (e.g., being helpful, honest, and harmless). See: Reinforcement Learning from Human Feedback (RLHF). Base models predict the next word; they don't care if the word is helpful or toxic. RLHF is the magic that turns a raw text-predictor into 'ChatGPT'. Phase 1: Humans rank model outputs (e.g., Response A is better than B). Phase 2: Train a Reward Model on these rankings. Phase 3: Use Proximal Policy Optimization (PPO) to fine-tune the LLM to maximize the score from the Reward Model.

Question 8

What is instruction tuning, and why is it important for chat models?

Accepted Answer

Instruction tuning is a supervised fine-tuning (SFT) phase where a base model is trained on thousands of curated instruction-response pairs, teaching it to follow commands rather than just auto-completing text. If you give a raw base model the prompt: 'Write a poem about a cat.', it might auto-complete with: 'Write a poem about a dog. Write a poem about a bird.' because it is just predicting text. Instruction tuning trains the model on data formatted as `[Instruction: Write a poem...] -> [Response: Here is a poem...]`. This teaches the model the concept of acting as an assistant.

Question 9

How do you prepare a dataset for fine-tuning an LLM?

Accepted Answer

Dataset preparation involves gathering diverse, high-quality domain examples, formatting them into strict prompt-completion JSONL pairs, cleaning out formatting artifacts or PII, and ensuring a balance of edge cases. Fine-tuning operates on the principle of 'Garbage In, Garbage Out'. A dataset of 1,000 highly curated, perfectly formatted examples written by experts will vastly outperform a dataset of 100,000 sloppy, scraped examples. The data must be converted into the exact prompt template (e.g., ChatML format) that will be used in production.

Question 10

What is catastrophic forgetting, and how do you prevent it during fine-tuning?

Accepted Answer

Catastrophic forgetting occurs when a neural network forgets its previously learned general knowledge while being heavily fine-tuned on a new, narrow dataset. If you fine-tune an LLM exclusively on 50,000 SQL queries, it will become excellent at SQL but may completely lose its ability to speak conversational English or answer general questions. To prevent this, you can mix in a percentage (e.g., 10-20%) of general-purpose instruction data into your specialized dataset, use lower learning rates, or rely heavily on PEFT (LoRA) which preserves the frozen base weights.

Question 11

When should you choose fine-tuning over RAG over prompt engineering?

Accepted Answer

Use Prompt Engineering for fast prototyping and rule setting. Use RAG to provide the model with dynamic, frequently updating factual knowledge. Use Fine-Tuning to teach the model a specific tone, dialect, or strict structural output format. These are not mutually exclusive; production systems use all three. If a model doesn't know your company's PTO policy, do NOT fine-tune it (the policy will change next year). Use RAG to retrieve the policy. If the model struggles to output strict nested XML required by your legacy backend despite massive prompts, you should Fine-Tune it on 1,000 examples of perfect XML.

Question 12

How do you evaluate a fine-tuned model's performance?

Accepted Answer

Evaluation is done by comparing the fine-tuned model against a baseline model using a holdout validation dataset. Metrics include cross-entropy loss (during training), ROUGE/BLEU scores, and LLM-as-a-judge for qualitative alignment. During training, you monitor the Training Loss and Validation Loss curves. If Validation loss starts increasing while Training loss decreases, the model is overfitting (memorizing). After training, you run the model against a test set of prompts and use a larger model (like GPT-4) to blindly score whether the fine-tuned output or the baseline output is better.

Question 13

What is synthetic data generation, and how do you use it for fine-tuning?

Accepted Answer

Synthetic data generation involves using a larger, more capable LLM (like GPT-4) to generate thousands of high-quality training examples to fine-tune a smaller, cheaper model (like Llama 3 8B). Creating 10,000 high-quality Q&A pairs manually takes months. Instead, AI engineers use a frontier model to generate data. For example, you feed GPT-4 your company's documentation and prompt it to generate 5,000 user questions and perfect answers. You then use this synthetic dataset to fine-tune an open-source 8B model. This distills the reasoning capability of the massive model into a tiny, cheap model.

Question 14

What are the key hyperparameters for fine-tuning (learning rate, epochs, batch size, LoRA rank)?

Accepted Answer

Key hyperparameters dictate how the model learns: Learning Rate controls step size, Epochs control passes over the data, Batch Size controls parallel samples processed, and LoRA Rank controls the complexity of the adapter. See: LoRA - Low-Rank Adaptation of LLMs. 
1. **Learning Rate (LR)**: If too high, the model diverges (forgets everything). If too low, it never learns. Often set around 1e-4 or 2e-5.
2. **Epochs**: Usually 1 to 3 for LLMs. Too many epochs cause severe overfitting.
3. **Batch Size**: Higher stabilizes gradients but uses more VRAM. Gradient accumulation is used to simulate high batch sizes.
4. **LoRA Rank (r)**: Determines the matrix size. r=8 or r=16 is common. Higher rank = more expressive power but more VRAM usage.

Fine-Tuning & Adaptation Interview Prep Portal