01
Self-Attention
Self-attention allows the model to consider relationships between tokens within a sequence. This helps the model determine which contextual information is relevant when processing each token.
ARTICLE
What Are Large Language Models (LLMs)?
Explore the world of Large Language Models and their impact on AI technology.
Sudais Azlan
15 min read
What Are Large Language Models (LLMs)?
AI & Robotics
Large Language Models, commonly known as LLMs, are advanced artificial intelligence models designed to understand, process, and generate human language. They are trained on very large collections of text and other language-related data, allowing them to learn patterns, relationships, structures, and meanings within language.
LLMs are one of the most important technologies behind modern generative AI. They power applications that can answer questions, summarize documents, generate content, translate languages, explain concepts, write software code, and interact with users through natural language.
Unlike a traditional software system that follows a fixed set of rules, an LLM uses a trained neural network to process an input and generate an appropriate output based on patterns it learned during training.
In simple terms: An LLM is an AI model that has learned patterns in language from large amounts of data and can use those patterns to understand prompts and generate human-like responses.
Why Are LLMs Important in Generative AI?
Large Language Models have become a fundamental technology behind modern generative AI because they allow computers to work with human language at an unprecedented scale. Instead of requiring users to interact with software through rigid commands or predefined interfaces, LLM-powered systems can understand natural-language instructions and generate context-aware responses.
This capability makes LLMs particularly valuable for generative AI applications. A single language model can support many different tasks, including writing, summarization, translation, question answering, programming assistance, information extraction, and conversational interaction.
The importance of LLMs also comes from their ability to generalize across different tasks. Rather than building a separate machine learning model for every individual language task, developers can use a capable foundation model and adapt its behavior through prompts, additional training, retrieval systems, or external tools.
From Single-Purpose Models to General-Purpose AI
Traditional machine learning systems are often designed for a specific task, such as detecting spam, classifying images, or predicting a numerical value. Modern LLMs can support many language-related tasks using the same underlying model.
What Makes LLMs So Powerful?
Several capabilities make large language models particularly useful for generative AI applications.
- Natural-language interaction: Users can communicate with AI systems using ordinary human language.
- Content generation: LLMs can generate text for different purposes, formats, and audiences.
- Context understanding: Models can use information from the surrounding prompt to produce more relevant responses.
- Multi-task capability: The same model can perform many different language-related tasks.
- Tool integration: LLMs can be connected to external tools, APIs, databases, and retrieval systems to build more capable AI applications.
In simple terms: LLMs are important to generative AI because they provide a flexible language-based intelligence layer that can understand instructions, process context, generate content, and connect with other AI tools and systems.
Understanding why LLMs are important is only the beginning. To understand what actually happens inside an LLM, we need to look at how these models process input, represent language, and generate output.
Understanding LLMs
How Do Large Language Models Work?
Large Language Models work by processing language as numerical data and using a neural network to identify patterns and relationships within that data. When a user provides a prompt, the model processes the input, determines the relevant context, and generates an output step by step.
Although the underlying mathematics and neural-network architecture can be extremely complex, the overall process can be understood as a sequence of connected stages.
01
Input
The user provides a prompt, question, instruction, or other form of input to the language model.
02
Tokenization
The input is divided into smaller units called tokens that can be converted into numerical representations.
03
Processing
The Transformer processes the token representations using mechanisms such as attention to understand relationships within the context.
04
Generation
The model predicts suitable next tokens repeatedly until it produces the resulting response.
The LLM Processing Pipeline
A simplified view of how information moves through a modern language model.
Input
User Prompt
→
Tokens
Language Units
→
Embeddings
Numerical Representation
→
Transformer
Context Processing
→
Prediction
Next Token
→
Output
Generated Response
What Happens During Generation?
An LLM generally does not generate an entire response as one single operation. For autoregressive language generation, the model predicts a likely next token based on the input and previously generated context. That token becomes part of the context, and the process continues until the response is complete or a stopping condition is reached.
This repeated prediction process allows the model to produce sentences, paragraphs, code, and other forms of language while maintaining relationships with the surrounding context.
In simple terms: You give an LLM a prompt, the model converts the language into a form it can process, analyzes the context through its neural network, predicts what should come next, and repeats that process to create the final response.
This high-level process explains how an LLM generates responses, but it does not yet explain how the model learns language in the first place. To understand that, we need to examine how large language models are trained.
LLM Training
How Are Large Language Models Trained?
Large Language Models are trained through a multi-stage machine learning process that allows them to learn patterns in language from enormous amounts of data. Training is one of the most important stages in the development of an LLM because it determines what the model can understand, generate, and generalize to new inputs.
Modern LLM development typically involves several stages, beginning with large-scale pre-training and followed by additional training and evaluation processes that make the model more useful, reliable, and aligned with its intended applications.
Foundation Stage
Pre-training
Pre-training is the large-scale learning stage in which a language model processes enormous quantities of text or other training data. The model gradually learns statistical relationships and patterns within that data by adjusting its internal parameters.
For many language models, a common objective is to predict missing or subsequent tokens in sequences. Through repeated exposure to training examples, the model learns increasingly complex relationships between words, phrases, concepts, and broader linguistic structures.
In simple terms: Pre-training teaches the model the fundamental patterns of language by exposing it to a very large amount of training data.
The LLM Training Pipeline
A simplified view of the major stages involved in developing a production-ready language model.
01
Collect Data
Gather and prepare large collections of suitable training data.
02
Pre-train Model
Train the neural network to learn broad patterns from the data.
03
Adapt Model
Further train or adapt the model for specific behaviors and tasks.
04
Evaluate
Test the model for quality, safety, accuracy, and intended behavior.
Training Is Different From Inference
It is important to distinguish between training and inference. During training, the model's parameters are adjusted so that it learns patterns from data. During inference, the trained model uses those learned parameters to process new inputs and generate outputs.
Training
The model learns from training data and updates its parameters.
Inference
The trained model processes new input and generates an output.
Key takeaway: Training gives an LLM its learned capabilities, while inference is the process through which users actually interact with those learned capabilities.
The training process explains how an LLM learns from data, but it does not yet explain the architecture that allows modern language models to process relationships between tokens and context so effectively. One of the most important pieces of that architecture is the Transformer.
LLM Architecture
What Is a Transformer Architecture?
The Transformer is a neural network architecture that became a major foundation of modern language models and generative AI. It was designed to process relationships between elements in a sequence efficiently, making it particularly powerful for understanding and generating language.
One of the most important ideas behind the Transformer is attention. Attention allows the model to determine which parts of an input are important when processing a particular token or piece of information.
Simplified Transformer Pipeline
A simplified representation of how information moves through the architecture.
Step 01
→
Input
User Prompt
Step 02
→
Tokens
Language Units
Step 03
→
Embeddings
Vector Representation
Step 04
→
Attention
Context Relationships
Step 05
→
Feed-Forward
Neural Processing
Step 06
Output
Next Token
Core Components of a Transformer
Transformers contain several interconnected components that work together to process and transform token representations.
02
Multi-Head Attention
Multiple attention mechanisms can operate in parallel, allowing the model to capture different relationships and patterns within the input representation.
03
Feed-Forward Network
Feed-forward neural network layers further transform the information produced by the attention mechanism.
04
Positional Information
Transformer-based systems need information about token positions or ordering so that the model can distinguish different arrangements of the same tokens.
Transformer Components at a Glance
Each component contributes a different function to the overall processing pipeline.
| Component | Main Purpose | Why It Matters |
|---|---|---|
| Tokenization | Converts text into tokens | Makes language processable by the model |
| Embeddings | Represents tokens numerically | Provides meaningful numerical representations |
| Attention | Models relationships between tokens | Helps the model use contextual information |
| Feed-Forward Network | Transforms learned representations | Adds additional neural-network processing |
| Output Layer | Produces token probabilities | Helps determine what token should come next |
Why Did Transformers Become So Important?
Transformers made it possible to process relationships within sequences using attention-based mechanisms and to scale model training effectively. Their flexibility and scalability helped establish the architecture as a foundation for many modern language models and other generative AI systems.
This architectural shift played a major role in the rapid development of large-scale language models and the broader generative AI ecosystem.
In simple terms: A Transformer is the architecture that helps an LLM understand how different parts of an input relate to each other. Its attention mechanism is especially important because it allows the model to focus on relevant context while processing language.
To understand attention more deeply, we first need to understand what an LLM actually sees when it receives text. Before the model can process a sentence, the text must be converted into smaller units called tokens.
LLM Fundamentals
What Are Tokens in an LLM?
Before a Large Language Model can process text, the text must be converted into smaller units that the model can work with. These units are called tokens. A token can represent a complete word, part of a word, punctuation, or another piece of text depending on the tokenizer and vocabulary used by the model.
Tokenization is therefore one of the first important steps in the LLM processing pipeline. Instead of directly processing human-readable sentences, the model works with token representations that can be mapped to numerical identifiers.
From Text to Tokens
A simplified view of how human language is converted into units that an LLM can process.
Human-readable text
Large language models understand context.
↓
Tokenized representation
Large
language
models
understand
context
.
↓
Numerical representation
1842
3917
7821
4512
2634
13
Note: The token IDs shown above are illustrative. Actual token IDs depend on the tokenizer and vocabulary of the specific model.
What Can a Token Represent?
Tokens do not always correspond exactly to complete words. Depending on the tokenizer, a token can represent different portions of language.
| Token Type | Example | Description |
|---|---|---|
| Complete word | apple | A frequently occurring word may be represented as a single token. |
| Word fragment | un + happy | A word can be divided into multiple subword tokens. |
| Punctuation | . | Punctuation marks can also be represented as tokens. |
| Special token | <special> | Some models use special tokens for particular control or structural purposes. |
01
Model Input
Tokenization converts raw text into a representation that the model can process.
02
Context Limits
The number of tokens in an input contributes to the amount of context a model can process within its context window.
03
Generation
During text generation, models predict tokens sequentially to construct the resulting response.
Tokenization Is Not the Same as Embeddings
Tokenization and embeddings are two different stages of the language processing pipeline. Tokenization determines how text is divided into tokens and maps those tokens to identifiers. Embeddings then represent those tokens as numerical vectors that can be processed by the neural network.
Tokenization
Text → Tokens → Token IDs
→
Embedding
Token IDs → Vector Representations
In simple terms: Tokenization breaks human language into smaller pieces that an LLM can process. Those pieces are then represented numerically and passed into the model's neural-network architecture.
Once text has been converted into tokens and represented numerically, the model needs a way to understand how those tokens relate to one another. This is where one of the most important mechanisms in modern LLMs comes into play: attention.
Understanding Attention
What Is Attention in Large Language Models?
Attention is one of the most important mechanisms in modern Transformer models. It allows a language model to determine which tokens in a sequence are more relevant to one another when processing information. Instead of treating every word as completely independent, attention helps the model examine relationships between different parts of the context.
This capability is particularly important for understanding language because the meaning of a word can depend heavily on the words surrounding it. Attention gives the model a mechanism for dynamically considering that surrounding context.
Context Example
Why Does Context Matter?
Consider the word "bank". Its meaning can change depending on the surrounding words.
Example 01
"I deposited money at the bank."
Here, the surrounding words indicate that bank refers to a financial institution.
Example 02
"We sat on the bank of the river."
Here, the surrounding words indicate that bank refers to the land beside a river.
Attention helps a Transformer identify which surrounding tokens are useful when interpreting a particular token. This allows the model to build richer contextual representations.
How Does Attention Work?
A simplified attention process can be understood as a sequence of selecting relevant information, measuring relationships, and combining useful context.
01
Identify Query
Determine what information the current token needs from the surrounding context.
02
Compare Keys
Compare the query with representations of other tokens to determine their relevance.
03
Calculate Weights
Convert the relevance scores into attention weights that determine how strongly each token contributes.
04
Combine Values
Combine information from the relevant tokens to create a contextual representation.
Core Mechanism
Query, Key, and Value
Self-attention commonly uses three learned representations known as Query (Q), Key (K), and Value (V). Together, they allow the model to determine which information should contribute to the representation of each token.
Q
Query
Represents what the current token is looking for in the surrounding context.
K
Key
Represents information used to determine how relevant another token is to the query.
V
Value
Contains the information that can be combined into the resulting contextual representation.
Scaled Dot-Product Attention
A commonly used form of attention calculates relationships between queries and keys, scales the result, applies a softmax operation to obtain attention weights, and then uses those weights to combine the values.
Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V
Here, Q, K, and V represent query, key, and value matrices, while dₖ represents the dimensionality of the key vectors.
Attention Weights in Context
The following simplified example illustrates how a model might assign different levels of attention to surrounding tokens when processing a particular word.
| Context Token | Relative Attention | Interpretation |
|---|---|---|
| The | Lower | Provides grammatical context |
| river | High | Strongly helps determine the meaning of “bank” |
| bank | Target | Token being interpreted |
| financial | Low | Less relevant in this particular context |
Important: The attention levels above are only an educational illustration. Actual attention weights are numerical values calculated by the model and can vary significantly depending on the architecture, layer, head, and input context.
Context
Understand Relationships
Attention allows the model to consider relationships between different tokens instead of processing each token in isolation.
Flexibility
Dynamic Relevance
The relevance of surrounding tokens can change depending on the input and the token currently being processed.
Generation
Better Context
Contextual representations help language models generate outputs that are more closely related to the information in the input sequence.
In simple terms: Attention is like a mechanism that helps an LLM decide which parts of the surrounding context deserve more consideration when understanding a particular token.
Attention is only one part of the Transformer architecture. Modern language models also rely on additional layers and training techniques to transform these contextual representations into useful behavior. One of the most important next concepts is the difference between pre-training and fine-tuning.
Sources & References
References
The following sources provide additional technical information and foundational research related to Large Language Models, Transformer architectures, attention mechanisms, and generative AI.
01
Research Paper
Foundational Research
Attention Is All You Need
Vaswani et al.
The foundational research paper that introduced the Transformer architecture and presented the attention-based approach that became central to modern language models.
Read source
02
Technical Documentation
Transformer Architecture Documentation
Technical reference
Provides additional technical context for understanding Transformer architectures, attention mechanisms, token processing, and related concepts.
Visit source
03
Research
Large Language Models and Generative AI
Academic and technical literature
Additional research material for understanding the development, training, architecture, and capabilities of modern language models.
Visit source
✓
About These Sources
References should prioritize authoritative research papers, university publications, official technical documentation, and recognized research organizations. Sources should be selected based on their relevance, credibility, and ability to support the technical claims presented in this article.
Editorial note: References are provided for readers who want to explore the technical foundations behind the concepts discussed in this article. Individual claims should be linked to the most relevant source where appropriate.
