Understanding Large Language Models

• A. The physical size of the computer hardware that runs the model
• B. The number of parameters and the volume of training data the model uses ✓
• C. The length of the longest text the model can read in a single conversation
• D. The number of different languages the model was designed to translate

Why: 'Large' refers mainly to the model's billions of parameters (learned weights) and the massive amounts of text data used during training. Scale in both dimensions is what gives these models their broad capabilities.

• A. Classifying images into predefined categories
• B. Predicting the next token (word or word fragment) in a sequence of text ✓
• C. Storing and retrieving exact copies of documents from a database
• D. Controlling robotic movements in real time

Why: Most LLMs are trained with next-token prediction: given a sequence of text, they learn to predict what comes next. Despite this simple objective, learning it at scale produces abilities like summarization, translation, and question answering.

• A. A password that grants the user access to the model
• B. A unit of text, such as a word, part of a word, or punctuation mark, that the model processes ✓
• C. A type of hardware chip designed specifically for AI
• D. A reward score given to the model during human feedback training

Why: A token is the basic unit of text an LLM reads and generates. Tokens can be whole words, subword pieces, or punctuation, and models process text as sequences of tokens rather than as raw characters or full sentences.

• A. Programmers manually write rules for every possible sentence
• B. It searches the internet in real time for every answer it gives
• C. It learns statistical patterns from very large amounts of text during training ✓
• D. It is connected to a human expert who types each response

Why: LLMs learn patterns in language, facts, and reasoning styles from vast text corpora during training, encoding this knowledge in their parameters. They do not rely on hand-written rules, and standard models do not browse the internet to answer questions unless given that tool.

• A. It deliberately lies to users to test their attention
• B. It is always connected to a fact-checking database that fails silently
• C. It generates text by predicting plausible continuations, which may not match facts ✓
• D. Its training data contained only fictional stories

Why: Because LLMs generate the most plausible next tokens based on learned patterns, they can produce fluent text that sounds right but is factually wrong. This behavior is often called 'hallucination', and it is why verifying important outputs matters.

• A. To store every sentence it has ever seen in a database
• B. To learn patterns in data by adjusting numerical weights to reduce prediction errors
• C. To follow a fixed set of hand-written grammar rules
• D. To translate every word into a dictionary definition before responding

• A. The physical size of the computer memory used
• B. A number that controls how strongly one connection influences another
• C. The total number of words in the training data
• D. The length of the output sentence

• A. It measures how far the model's predictions are from the correct answers
• B. It decides which words are spelled correctly
• C. It deletes low-quality training examples
• D. It counts how many layers the network has

• A. Sending the model's output back to the user for review
• B. Reversing the order of words in a sentence
• C. Calculating how each weight should change to reduce the loss
• D. Removing unnecessary neurons from the network

• A. To memorize the exact text of every book it reads
• B. The statistical relationships between words and the context in which they appear
• C. To count the letters in each word
• D. To avoid using any word more than once

• A. A unique password that grants access to the model's API
• B. A unit of text, such as a word, subword, or character, that the model processes as a single item ✓
• C. A numerical weight stored in the model's neural network layers
• D. A complete sentence used as a training example

Why: Tokens are the basic units of text that a language model reads and generates. They may be whole words, parts of words (subwords), punctuation, or individual characters, depending on the tokenizer used.

• A. Subword tokens make every text exactly the same length
• B. It allows the model to handle rare and unseen words by breaking them into familiar pieces, while keeping the vocabulary size manageable ✓
• C. Subword tokens remove the need for any training data
• D. Whole-word tokenization is illegal in most countries

Why: Subword tokenization balances vocabulary size against coverage. Rare or novel words can be composed from known subword pieces, so the model never encounters a truly unknown token and the vocabulary stays a practical size.

• A. A compressed file containing the model's training data
• B. A dense vector of numbers that represents a token's meaning and relationships in a continuous space ✓
• C. A security layer that encrypts user prompts
• D. The final text output shown to the user

Why: An embedding maps each token to a high-dimensional vector of real numbers. Tokens with similar meanings or usage patterns tend to have vectors that are close together, which lets the model perform mathematical operations on language.

• A. They are manually written by linguists for each word in the dictionary
• B. They are randomly assigned and never change during training
• C. They are adjusted during training so that tokens appearing in similar contexts end up with similar vectors ✓
• D. They are copied directly from a search engine index

Why: Embedding vectors start out with random values and are updated through gradient descent as the model learns to predict text. Through this process, the vectors come to capture semantic and syntactic similarity based on context.

• A. It determines how many tokens the model can consider at once, so longer inputs may need to be truncated or summarized ✓
• B. It sets the font size used to display the model's responses
• C. It controls the color of the embedding vectors
• D. It guarantees that the model will always remember every previous conversation

Why: The context window is the maximum number of tokens the model can attend to in a single pass. Inputs that exceed this limit must be shortened, and understanding token counts helps users plan prompts and estimate costs.

• A. To compress the input sequence into a fixed-length vector
• B. To let each token weigh the relevance of every other token in the sequence when building its representation ✓
• C. To randomly drop connections during training to prevent overfitting
• D. To convert text characters directly into numerical byte values

Why: Self-attention computes relationships between all positions in a sequence, allowing each token's representation to incorporate context from other relevant tokens, regardless of their distance.

• A. Because the attention operation by itself has no inherent sense of token order ✓
• B. Because they reduce the vocabulary size of the model
• C. Because they are required to apply the softmax function
• D. Because they allow the model to skip the training phase

Why: Self-attention treats its inputs as a set, so without positional encodings the model could not distinguish 'dog bites man' from 'man bites dog'. Positional encodings inject information about token positions.

• A. To convert the scores into probabilities that sum to one
• B. To keep the softmax from saturating when dimensions are large, which would produce vanishing gradients ✓
• C. To make the model run faster on GPUs
• D. To ensure the query and key matrices are the same size

Why: As the dimension grows, dot products grow in magnitude, pushing softmax into regions with very small gradients. Scaling by the square root of the key dimension keeps the variance of the scores stable.

• A. It applies a position-wise nonlinear transformation to each token's representation independently ✓
• B. It mixes information across all tokens in the sequence
• C. It stores the tokenizer vocabulary
• D. It computes the loss function during training

Why: Attention handles mixing information across tokens, while the feed-forward network processes each position separately with the same learned weights, adding nonlinearity and capacity.

• A. They process all tokens in parallel during training, enabling much faster training on modern hardware ✓
• B. They never require large amounts of data
• C. They process tokens strictly one at a time, which saves memory
• D. They eliminate the need for any learned parameters

Why: RNNs must process tokens sequentially, since each step depends on the previous hidden state. Transformers compute attention across the whole sequence at once, which parallelizes well on GPUs and TPUs.

• A. To teach the model to follow specific user instructions
• B. To learn general language patterns by predicting text from a large unlabeled corpus ✓
• C. To memorize a fixed set of question-answer pairs
• D. To compress the model so it runs on mobile devices

Why: Pretraining uses self-supervised objectives, such as predicting the next token, on large amounts of unlabeled text so the model learns broad grammar, facts, and reasoning patterns that can later be adapted to specific tasks.

• A. Cryptographic keys used to secure model weights
• B. Payment units charged for using an API
• C. The discrete units of text, such as words or subword pieces, that the model processes ✓
• D. Labels assigned by human annotators to each training example

Why: Text is split into tokens, which may be whole words, subwords, or characters, and the model operates on sequences of these tokens during both training and inference.

• A. GPUs are required to store the training text dataset
• B. A single device usually lacks the memory and compute needed for models with billions of parameters ✓
• C. Distributed training makes the model more accurate by default
• D. Only GPUs can perform the matrix multiplications used in neural networks

Why: Large models and their training data exceed the memory and processing capacity of one accelerator, so the workload is split across many devices using data, tensor, or pipeline parallelism.

• A. The amount of money spent on compute
• B. The percentage of training data that was discarded
• C. A number reflecting how poorly the model predicted the actual next tokens, which training tries to minimize ✓
• D. The number of layers removed during pruning

Why: Cross-entropy loss measures the gap between the model's predicted probability distribution over the next token and the token that actually appeared; gradient descent adjusts weights to reduce this value.

• A. A legal rule governing how large datasets may be collected
• B. A regulation on the physical size of data centers
• C. An empirical relationship showing how model performance improves predictably with more parameters, data, and compute ✓
• D. A method for scaling down model outputs to shorter lengths

Why: Research such as the Kaplan and Chinchilla studies found that loss decreases as a predictable power law when model size, dataset size, and compute increase, guiding decisions about how to allocate resources.

• A. The model refuses to answer a question because of a safety filter
• B. The model generates fluent but factually incorrect or fabricated information
• C. The model runs out of context window space during a long conversation
• D. The model produces output in a language different from the one requested

• A. To measure how quickly the model generates tokens
• B. To provide a standardized way to compare a model's knowledge and reasoning across tasks
• C. To fine-tune the model's weights on new domain data
• D. To reduce the size of the model for deployment on mobile devices

• A. Data augmentation
• B. Reinforcement learning from human feedback (RLHF)
• C. Gradient checkpointing
• D. Knowledge distillation

• A. Benchmark questions may have appeared in the training data, inflating measured performance
• B. Public benchmarks always contain incorrect answer keys
• C. Benchmarks slow down inference speed during testing
• D. Models cannot read text that is publicly available on the internet

• A. Increasing the temperature to make responses more random
• B. Removing the tokenizer so the model cannot parse dangerous words
• C. Using input and output filters or a moderation classifier around the model
• D. Training the model only on very short documents