Question 1

What are embeddings in the context of AI engineering?

Accepted Answer

Embeddings are dense mathematical vectors (arrays of floating-point numbers) that represent the semantic meaning of data (text, images, audio) in a high-dimensional space. See: Embeddings in Machine Learning. Machine learning models cannot process raw text. An embedding model (like OpenAI's text-embedding-ada-002) converts a sentence into an array of, for example, 1536 numbers. In this multi-dimensional space, concepts that are semantically similar (like 'dog' and 'puppy') are located close to each other, allowing algorithms to measure their relationship mathematically.

Question 2

How do embedding models convert text to vectors?

Accepted Answer

Embedding models use deep neural networks (typically Transformer encoders like BERT) to process the tokens of a text sequence and output a final hidden state that represents the contextualized meaning of the entire input. When you pass text into an embedding model, it tokenizes the text, passes it through multiple self-attention layers to understand the context of each word relative to the others, and typically averages (pools) the final outputs to create one single dense vector representing the whole sentence.

Question 3

What is the difference between sparse and dense embeddings?

Accepted Answer

Sparse embeddings (like TF-IDF or BM25) rely on exact keyword frequencies resulting in huge vectors filled mostly with zeros. Dense embeddings (like OpenAI embeddings) use neural networks to create smaller, fully populated vectors representing semantic meaning. Sparse vectors might have 100,000 dimensions (one for every word in the dictionary) but only 5 non-zero values corresponding to the words actually used in the sentence. They only match exact keywords. Dense vectors have fixed dimensions (e.g., 768 or 1536) where every number is a non-zero float representing an abstract semantic feature. Dense vectors can match 'automobile' to 'car'; sparse vectors cannot.

Question 4

Explain cosine similarity, dot product, and Euclidean distance for vector search.

Accepted Answer

These are mathematical distance metrics used to measure how similar two vectors are. Cosine similarity measures the angle between vectors, dot product measures angle and magnitude, and Euclidean distance measures the straight-line distance. When a user searches for something, their query is embedded into a vector. The vector database then calculates the distance between the query vector and all stored document vectors.
1. **Cosine Similarity**: Cares only about the angle (direction) between two vectors. Great for text where document length varies.
2. **Dot Product**: Multiplies corresponding elements. If vectors are normalized, it is identical to Cosine Similarity.
3. **Euclidean (L2)**: Measures the physical distance between the points in space.

Question 5

What is a vector database, and how does it differ from a traditional database?

Accepted Answer

A vector database is purpose-built to store, index, and query high-dimensional embeddings using Approximate Nearest Neighbor (ANN) algorithms for semantic search, whereas traditional databases use B-Trees or hashes for exact keyword matching. If you have 10 million vectors, comparing a query vector against every single one (k-Nearest Neighbors) is computationally impossible in real-time. Vector databases (like Pinecone, Milvus, Qdrant, Weaviate) solve this by using ANN indexes (like HNSW). They organize vectors into graphs or clusters so the database only has to check a small neighborhood of likely matches, returning results in milliseconds.

Question 6

How do you choose the right embedding model for your use case?

Accepted Answer

Choosing an embedding model depends on your domain, language requirements, sequence length, and budget. For general use, OpenAI's models are standard. For specific domains (medical, legal) or maximum privacy, open-source models (like BGE or E5) run locally are preferred. You must check the MTEB (Massive Text Embedding Benchmark) leaderboard. If you need to embed large documents at once, you need a model with a high context window (e.g., 8k tokens). If you operate in a specialized domain, a smaller open-source model fine-tuned on medical texts will outperform a generic API model. Cost and latency are also major factors.

Question 7

What is embedding dimensionality, and how does it affect performance and cost?

Accepted Answer

Dimensionality refers to the number of floating-point values in a vector (e.g., 384 vs. 1536). Higher dimensions capture more semantic nuance but consume significantly more memory, storage, and compute during search. An embedding from OpenAI's `text-embedding-3-large` has 3072 dimensions. Storing 1 million of these vectors requires roughly 12 GB of RAM (since fast vector search runs in-memory). A smaller model like `all-MiniLM-L6-v2` has 384 dimensions, requiring only 1.5 GB of RAM. While 3072 dimensions might be slightly more accurate for complex logic, it costs 8x more to host.

Question 8

How do you handle embedding drift when the embedding model is updated?

Accepted Answer

If you change or update your embedding model, all previously stored vectors in the database become instantly invalid. You must re-embed (re-index) your entire dataset using the new model. Vector spaces are entirely unique to the model that created them. An embedding for 'Apple' from Model A exists in a completely different mathematical universe than an embedding for 'Apple' from Model B. You cannot calculate the distance between them. If you upgrade your model, you must run a batch job to pass all your raw text through the new model and overwrite the vector database.

Question 9

What are multi-modal embeddings, and how are they generated?

Accepted Answer

Multi-modal embeddings map different types of data (e.g., text and images) into the exact same vector space. This allows you to embed an image, and search for it using a text query, or vice versa. Models like CLIP (Contrastive Language-Image Pre-training) are trained on pairs of images and their text captions. The model learns to place the vector for an image of a dog in the exact same mathematical location as the text vector for the word 'dog'. This is the foundation of modern reverse-image search and multi-modal RAG.

Question 10

How do you index and query multi-tenant data in a vector database?

Accepted Answer

Multi-tenancy is handled by using Metadata Filtering alongside the vector search, or by utilizing specific database features like Namespaces or Partitions to physically isolate customer data. If you are building a B2B SaaS application with 100 different companies, you cannot allow Company A's search query to return Company B's embedded documents. You attach a metadata tag (e.g., `{'tenant_id': 'company_A'}`) to every vector. During querying, the vector database pre-filters the index to only search vectors matching that `tenant_id`.

Question 11

What is quantization of embeddings, and how does it reduce storage costs?

Accepted Answer

Vector quantization compresses embeddings by reducing the precision of the numbers (e.g., from 32-bit floats to 8-bit integers or binary) or by grouping vectors (Product Quantization), drastically reducing RAM usage with minimal loss in search accuracy. Storing 100 million 1536-dimensional float32 vectors takes hundreds of gigabytes of RAM. Scalar Quantization converts those floats into 8-bit integers (int8), slashing memory requirements by 4x. Binary Quantization (supported by newer models) compresses them into single bits, reducing memory by 32x. This makes massive-scale vector search economically viable.

Question 12

What is hybrid search (combining keyword search with vector search)?

Accepted Answer

Hybrid search executes both a semantic vector search (Dense) and an exact keyword search (Sparse/BM25) simultaneously, fusing the results together using an algorithm like Reciprocal Rank Fusion (RRF) to get the best of both worlds. Dense vectors are great at understanding concepts ('car' matches 'automobile'), but they are terrible at exact string matching (like searching for a specific error code 'ERR-992-B' or a specific name). BM25 keyword search excels at exact matches. Hybrid search runs both, scoring documents on semantic meaning AND exact keyword overlaps, yielding SOTA retrieval performance.

Question 13

Your vector database for RAG is consuming too much memory. How do you reduce it?

Accepted Answer

To reduce memory consumption, implement Scalar Quantization (int8), reduce the embedding dimensionality using Matryoshka models, move cold data to disk-based ANN indexes, or archive unused vectors. RAM is the most expensive part of a vector database. First, check if your DB supports quantization to shrink float32s to int8. Second, if you are using OpenAI's v3 embeddings, truncate the dimensions from 3072 down to 1024 or 512. Finally, modern DBs (like Qdrant or Milvus) support disk-based indexing (mmap), which keeps the vectors on fast NVMe SSDs rather than loading everything into RAM.

Question 14

Your semantic search fails for short queries. How do you improve it?

Accepted Answer

Short queries lack semantic context. You can improve this by implementing Query Expansion (using an LLM to hallucinate synonyms or a hypothetical answer before searching), or by falling back to Hybrid/Keyword search. If a user just types 'security', the vector is very generic and might match thousands of loosely related documents. Query Expansion involves sending 'security' to an LLM first: `Expand this query for a search engine`. The LLM returns: `cybersecurity, authentication, firewall, SOC2 compliance`. You then embed this expanded string, which provides a much richer semantic target for the vector database.

Vector Databases & Embeddings Interview Prep Portal