Question 1

What are foundation models, and how have they changed AI engineering?

Accepted Answer

Foundation models are massive AI models trained on a vast quantity of unlabeled data at scale, which can be adapted (e.g., fine-tuned or prompted) to a wide range of downstream tasks. Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization. Traditionally, AI models were trained from scratch for specific tasks (like sentiment analysis). Foundation models (like GPT-4, Llama 3, Claude) changed this paradigm. Now, engineers use a single, massive pre-trained model and adapt it to countless specific tasks using Prompt Engineering or RAG, drastically reducing development time.

Question 2

What is a Large Language Model (LLM), and how does it work?

Accepted Answer

A Large Language Model (LLM) is a type of foundation model that uses deep neural networks (specifically Transformers) to understand and generate human language by predicting the next token in a sequence. Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization. At its core, an LLM is a complex statistical calculator. It takes a sequence of input tokens (words or word parts) and calculates the probability distribution for what the next token should be, based on patterns it learned from petabytes of internet data.

Question 3

Inside ChatGPT: What Happens After You Hit Enter?

Accepted Answer

When you hit enter, your text is tokenized into IDs, processed through multiple Transformer layers using attention mechanisms to understand context, and the model outputs probabilities for the next token until a stop token is generated. See: Inside ChatGPT: What Happens After You Hit Enter. The process is: 1) Tokenization (text to numbers). 2) Embeddings (numbers to dense vectors). 3) Positional encoding (adding order context). 4) Attention mechanism (understanding word relationships). 5) Feed-forward networks (processing the attention output). 6) Softmax layer (generating next-token probabilities). 7) De-tokenization (numbers back to text).

Question 4

What is the Transformer architecture and how does it work?

Accepted Answer

Introduced in 'Attention Is All You Need' (2017), the Transformer is a deep learning architecture that relies entirely on an attention mechanism to draw global dependencies between input and output, eschewing recurrence and convolutions. See: Decoding Transformer Architecture. Transformers process entire sequences simultaneously rather than word-by-word (like older RNNs/LSTMs). This parallelization allows for massive scaling on GPUs. It uses Self-Attention to map how strongly every word in a sentence is related to every other word, creating deep contextual understanding.

Question 5

What are the key components of the Transformer architecture?

Accepted Answer

The key components include Tokenization, Input Embeddings, Positional Encoding, Multi-Head Self-Attention, Feed-Forward Neural Networks, Add & Norm (Residual Connections), and the Softmax output layer. See: Decoding Transformer Architecture. The original Transformer had an Encoder (to process input) and a Decoder (to generate output). Models like BERT are Encoder-only. Models like GPT are Decoder-only. The core engine is the Multi-Head Attention block, which allows the model to focus on different aspects of a sequence simultaneously.

Question 6

What is tokenization in LLMs?

Accepted Answer

Tokenization is the process of breaking down raw text into smaller, manageable units called tokens (which can be words, subwords, or characters) and assigning them unique numerical IDs that the AI model can process. See: Tokenization in Large Language Models (LLMs). Neural networks cannot read text; they only perform math on numbers. Tokenizers like Tiktoken (used by OpenAI) use statistical algorithms to find the most efficient way to split text. Common words are single tokens, while rare words are split into subword tokens.

Question 7

Explain BPE (Byte Pair Encoding).

Accepted Answer

Byte Pair Encoding (BPE) is a highly popular subword tokenization algorithm that iteratively merges the most frequently occurring pairs of characters or bytes into a single new token until a target vocabulary size is reached. See: Byte Pair Encoding. BPE starts with a vocabulary of individual characters. It scans the training corpus, finds the most common adjacent pair (e.g., 'e' and 'r' -> 'er'), and merges them into a single token. It repeats this process thousands of times. This creates a vocabulary where common words ('the', 'apple') are single tokens, but rare words are split into known subword fragments.

Question 8

Explain WordPiece and SentencePiece.

Accepted Answer

WordPiece and SentencePiece are subword tokenization algorithms. WordPiece merges tokens based on maximizing the likelihood of the language data, while SentencePiece works directly on raw text (including spaces), making it language-agnostic. WordPiece (used by Google's BERT) is similar to BPE but chooses pairs to merge based on language probability rather than raw frequency. SentencePiece (used by Llama and T5) is unique because it treats the space character as a normal symbol (often represented as '_'). This means it doesn't require pre-tokenization (like splitting by spaces), making it exceptional for languages like Chinese or Japanese that do not use spaces between words.

Question 9

What is positional encoding, and why is it needed in Transformers?

Accepted Answer

Positional encoding is a technique used to inject information about the order of words into a sequence. It is required because the Transformer architecture processes all tokens in parallel and inherently has no concept of sequence order. See: Positional Embeddings in LLMs. Without positional encoding, a Transformer would see 'The dog bit the man' and 'The man bit the dog' as identical bags of words. Positional encodings are mathematical patterns (typically utilizing sine and cosine functions of different frequencies) added directly to the input embeddings to uniquely identify a token's absolute and relative position.

Question 10

What are embeddings?

Accepted Answer

Embeddings are dense numerical vectors (arrays of continuous float numbers) that represent the semantic meaning of a token, word, or sentence in a high-dimensional mathematical space. See: Embeddings in Machine Learning. In an embedding space, words with similar meanings (like 'king' and 'monarch') are positioned close to each other. By converting language into geometric representations, models can perform mathematical operations on concepts (e.g., King - Man + Woman = Queen) and measure semantic similarity using Cosine Distance.

Question 11

Explain the Query(Q), Key(K), and Value(V) in attention.

Accepted Answer

Q, K, and V are vector representations used in the attention mechanism, inspired by database retrieval. The Query asks 'What am I looking for?', the Key states 'What do I contain?', and the Value is the actual 'Content' returned if the Query and Key match. See: Math behind Attention - Q, K, and V. For every token, the model calculates a Q, K, and V vector using learned weight matrices. To find out how much focus Token A should put on Token B, the model takes the dot product of Token A's Query and Token B's Key. A high dot product means high relevance, so the model heavily incorporates Token B's Value into Token A's final context representation.

Question 12

What is self-attention, and how does it work in Transformers?

Accepted Answer

Self-attention is a mechanism that allows a model to relate different positions of a single sequence to compute a highly contextualized representation of that sequence. See: Math behind Attention - Q, K, and V. As a model reads a sentence like 'The bank of the river', self-attention allows the word 'bank' to look at 'river' and adjust its own meaning (river bank, not financial bank). It does this by calculating Q, K, and V vectors for all words, computing attention scores (Q dot K) to see which words matter to each other, and then summing the Values based on those scores.

Question 13

Why do we scale the dot product attention by √dₖ in the Transformer architecture?

Accepted Answer

We scale the dot product by the square root of the dimension of the key vectors (√dₖ) to prevent the dot products from growing excessively large, which would push the softmax function into regions with extremely small gradients. See: Math behind √dₖ Scaling Factor in Attention. When calculating attention scores (Q dot K), if the dimension depth (dₖ) is large, the resulting dot product values can explode. Passing massive values into a Softmax function causes it to output values extremely close to 1.0 for the highest score and 0.0 for everything else. This leads to the 'vanishing gradient' problem during backpropagation, stopping the model from learning. Scaling by √dₖ normalizes the variance to 1.

LLM Fundamentals Interview Prep Portal