Reference
AI Glossary
Plain-English definitions for the terms that actually matter. From hallucinations to transformers — no jargon overload, just clear explanations with real-world context.
Artificial General Intelligence (AGI)
A hypothetical AI system capable of performing any intellectual task a human can do. We don't have AGI yet — current AI systems are narrow, meaning they're built for specific tasks. OpenAI, DeepMind, and Anthropic are racing toward it, but timelines are debated.
Artificial Intelligence (AI)
Software that can perform tasks typically requiring human intelligence — reasoning, learning, recognizing patterns, understanding language. The term covers everything from spam filters to large language models.
Attention Mechanism
The core innovation behind modern language models. Instead of reading text word-by-word in sequence, attention lets a model consider all words simultaneously and weight which ones matter most for understanding each part of the text. "Attention is all you need" was the 2017 paper that changed everything.
Autoregressive Model
A model that generates output one token at a time, where each new token is predicted based on all previous tokens. GPT-4, Claude, and most chat AIs work this way — they generate responses left-to-right, word by word.
Benchmark
A standardized test used to compare AI model performance. Common ones include MMLU (general knowledge), HumanEval (coding), and MATH (mathematical reasoning). Take benchmark scores with some scepticism — models can be optimized specifically for benchmarks.
BERT
Bidirectional Encoder Representations from Transformers. A Google model from 2018 that reads text in both directions (left and right simultaneously). Great for understanding tasks like classification and Q&A — but not generation. The predecessor to many modern models.
Chain-of-Thought (CoT) Prompting
A prompting technique where you ask the model to reason through a problem step by step before giving an answer. It dramatically improves accuracy on math, logic, and multi-step problems. Simply adding 'think step by step' to your prompt often helps.
Context Window
The maximum amount of text (measured in tokens) that a model can process at one time — both input and output combined. GPT-4 Turbo has 128K tokens. Gemini 1.5 Pro has 1M tokens. The bigger the window, the longer the documents you can feed it.
Claude
Anthropic's AI assistant, known for its thoughtful, balanced responses and strong analytical writing. Claude 3.5 Sonnet is widely considered one of the best models for nuanced writing and long-document analysis.
Deep Learning
A subset of machine learning that uses neural networks with many layers (hence 'deep'). It's what powers image recognition, voice assistants, and language models. Deep learning requires large datasets and significant compute to train.
Diffusion Model
The technology behind most modern image generators (Stable Diffusion, DALL-E, Midjourney). Training involves gradually adding noise to images and teaching the model to reverse that process. At inference, it starts from random noise and refines it into an image.
Embedding
A numerical representation of text (or images, audio) as a list of numbers (a vector) that captures its meaning. Similar concepts end up close together in embedding space. Used heavily in search, recommendation systems, and retrieval-augmented generation.
Emergent Behavior
Capabilities that appear in large models that weren't explicitly trained for and weren't present in smaller versions. Examples: GPT-3 could do basic arithmetic without being specifically trained on it. Researchers debate whether emergence is real or a measurement artifact.
Fine-tuning
Taking a pre-trained model and training it further on a smaller, task-specific dataset to specialize its behavior. Fine-tuning a general model on medical documents makes it better at medical Q&A. Cheaper than training from scratch, but requires quality data.
Foundation Model
A large model trained on broad data that can be adapted to many tasks. GPT-4, Claude, and Gemini are foundation models. The term was popularized by Stanford's Institute for Human-Centered AI in 2021.
GPT (Generative Pre-trained Transformer)
OpenAI's model architecture. 'Generative' means it produces text, 'pre-trained' means it learned from massive datasets before fine-tuning, and 'Transformer' refers to the underlying neural network architecture. GPT-4 is currently OpenAI's most capable public model.
Guardrails
Safety mechanisms built into AI systems to prevent harmful outputs. These include content filters, refusal training, and output classifiers. Different companies implement different levels of guardrails — a major source of variation between AI tools.
Hallucination
When an AI model generates confident-sounding but factually incorrect information. This is a fundamental limitation of current language models — they optimize for plausible-sounding text, not factual accuracy. Always verify important claims from AI outputs.
RLHF (Human Feedback)
See Reinforcement Learning from Human Feedback. The technique most responsible for making AI assistants actually useful and safe to deploy publicly.
Inference
Running a trained model to generate outputs. When you type a message to ChatGPT and it responds, that's inference. Training is expensive and done once; inference happens billions of times daily.
In-context Learning
A model's ability to learn from examples provided within the prompt, without any weight updates. Showing GPT-4 three examples of a format before asking it to produce more is in-context learning. No fine-tuning required.
Large Language Model (LLM)
A neural network trained on massive text datasets to understand and generate language. 'Large' refers to both training data size and parameter count (typically billions). GPT-4, Claude, Gemini, and Llama are all LLMs.
Latent Space
The compressed, abstract representation of data inside a neural network. In diffusion models, images are encoded into latent space, manipulated, then decoded back. Understanding latent space is key to understanding why models can interpolate between concepts.
Llama
Meta's open-source language model family. Llama 3.1 405B is competitive with GPT-4 and can be run locally or self-hosted. The most important open-source model series because it powers countless fine-tuned variants and research projects.
Multimodal
An AI model that can process multiple types of input — typically text and images, sometimes audio or video. GPT-4V, Claude 3, and Gemini are all multimodal. The future is models that handle all modalities natively.
Model Collapse
A theoretical (and increasingly documented) problem where training on AI-generated data causes model quality to degrade over generations. As the web fills with AI content, future models trained on that data may perform worse.
Neural Network
A computational system loosely inspired by the brain's structure of interconnected neurons. Consists of layers of mathematical functions (nodes) that transform inputs into outputs. Deep neural networks have many such layers.
NLP (Natural Language Processing)
The branch of AI focused on understanding and generating human language. Sentiment analysis, translation, summarization, and chatbots all fall under NLP. LLMs are currently the dominant approach to NLP tasks.
Parameters
The numerical weights inside a neural network that are adjusted during training. GPT-4 reportedly has ~1.8 trillion parameters. More parameters generally means more capacity to learn, but also more compute required.
Prompt Engineering
The practice of crafting inputs to AI models to get better outputs. Ranges from simple instruction tweaks to complex techniques like chain-of-thought, few-shot examples, and system prompt design. A legitimate skill with significant impact on output quality.
Pre-training
The initial training phase where a model learns from massive datasets (often the entire internet) to predict the next token. This is where most of the compute cost goes. Fine-tuning comes after pre-training.
RAG (Retrieval-Augmented Generation)
An architecture where a model retrieves relevant documents from a database before generating a response. Combines the language ability of LLMs with the accuracy of a real knowledge base. Used by ChatGPT's browsing feature, enterprise AI systems, and many AI apps.
RLHF (Reinforcement Learning from Human Feedback)
A training technique where human raters score model outputs, and those scores train a reward model, which then guides further training via reinforcement learning. The key innovation that turned raw language models into useful assistants. Used by OpenAI, Anthropic, and Google.
Stable Diffusion
An open-source image generation model released by Stability AI in 2022. Unlike DALL-E or Midjourney, it can be run locally and modified freely. The base for a huge ecosystem of custom models and art styles.
System Prompt
A hidden instruction given to a model before the conversation begins, used to set its persona, behavior, and constraints. When you use a custom AI chatbot that has a specific personality or focus, a system prompt is usually responsible.
Temperature
A parameter controlling output randomness. Low temperature (0-0.3) makes outputs more deterministic and focused. High temperature (0.7-1.0+) makes them more creative and varied. For factual tasks, use low temperature. For creative work, go higher.
Token
The basic unit of text that language models process. A token is roughly 4 characters or ¾ of a word in English. 'ChatGPT is great' is about 5 tokens. Pricing for API access is typically measured in tokens.
Transformer
The neural network architecture that powers all modern large language models. Introduced in the 2017 Google paper 'Attention is All You Need.' Transformers replaced older recurrent networks because they can process entire sequences in parallel.
Vector Database
A database optimized for storing and searching embeddings (numerical representations of text or images). Essential for RAG systems. Examples include Pinecone, Weaviate, and pgvector. Searches by semantic similarity rather than exact keyword match.
Zero-shot Learning
Asking a model to perform a task it was never explicitly trained on, with no examples in the prompt. Modern LLMs handle many zero-shot tasks well because of their broad pre-training. Contrast with few-shot learning, where you provide examples.
Missing a term?
We update this glossary regularly. If there's a term you'd like us to add, let us know.
Suggest a term