Forward Diffusion (Noising)

The forward diffusion process is where the transformation begins. This process progressively adds Gaussian noise to the original data (e.g., an image) over a series of steps until the data is converted into pure, random noise. This can be visualized as gradually blurring an image until it becomes unrecognizable.

Mathematically, this is modeled as a Markov chain. A Markov chain is a sequence of events where the probability of each event depends only on the state of the previous event. Each step in the forward diffusion process adds a small amount of Gaussian noise, controlled by a variance schedule. The variance schedule determines how much noise is added at each step. The formula below shows how a data point (x) at any given time(t) is dependent on the original data point (x₀):

x(t) = √(1 – βt) * x₀ + √βt * ε

Where:

• x(t) is the noisy data at time step t
• x₀ is the original data
• βt is the cumulative variance schedule at time step t
• ε is random noise sampled from a Gaussian distribution

After many steps, the data becomes completely random noise, following a standard normal distribution. By the end of this forward diffusion process, all information about the original data has been effectively erased.

Reverse Diffusion (Denoising)

The reverse diffusion process is the heart of the diffusion model‘s generative capability. It involves learning to gradually remove the noise that was added during the forward diffusion process, starting from pure noise. A neural network is trained to predict and reverse the noise at each step. The model learns to approximate the conditional probability distribution of the data given the noisy input.

This neural network is trained to estimate the noise added at each step of the forward process. By iteratively subtracting this estimated noise, the model gradually refines the noisy data, eventually reconstructing a coherent and realistic data sample. The equation below represents the reverse diffusion process, where the model predicts the original data point (x₀) given the noisy data point (x(t)):

x₀ ≈ μθ(x(t), t)

Where:

• x₀ is the predicted original data
• x(t) is the noisy data at time step t
• μθ is the neural network that predicts the mean of the distribution at time step t
• θ represents the parameters of the neural network

The reverse process is also a Markov chain, but instead of adding noise, it removes it step-by-step. The neural network guides this process, learning to generate increasingly refined data at each step. By the end of the reverse diffusion process, the model has transformed random noise into a high-quality sample that resembles the original data distribution.

Denoising Diffusion Probabilistic Models (DDPMs)

Denoising Diffusion Probabilistic Models (DDPMs) represent the original and foundational type of diffusion model. They introduced the core concepts of forward and reverse diffusion processes, setting the stage for subsequent advancements. In DDPMs, the model directly operates on the pixel space of images, making them computationally intensive and slow.

While DDPMs demonstrated remarkable image generation capabilities, they suffered from limitations such as slow sampling speeds and high computational costs, making them less practical for real-time applications. As research evolved, DDPMs became a comparative benchmark for newer, more efficient models. For a more detailed understanding of DDPMs, consider reading this post by Lilian Weng on diffusion models.

Latent Diffusion Models (LDMs)

Latent Diffusion Models (LDMs), such as Stable Diffusion, address the computational limitations of DDPMs by performing the diffusion process in a latent space. Instead of directly manipulating pixels, LDMs first encode the image into a lower-dimensional latent representation using a pre-trained autoencoder. The forward and reverse diffusion processes then occur in this compressed latent space, significantly reducing computational demands and allowing for faster sampling.

The reduced computational cost of LDMs has made them the dominant approach in AI image generation. Recent benchmarks highlight that LDMs achieve similar or better results than pixel-space diffusion models with significantly less compute. The development of Stable Diffusion marked a turning point, democratizing access to high-quality image synthesis. To learn more about the capabilities of Stable Diffusion, check out this article on Stable Diffusion SDXL 1.0 generative AI.

Score-Based Generative Modeling (SDEs) and their Mathematical Foundation

Score-Based Generative Modeling (SDEs) offer a different perspective on diffusion models, emphasizing the mathematical underpinnings of the diffusion process. Instead of directly predicting the noise, SDEs focus on estimating the “score” of the data distribution, which is the gradient of the log probability density function. This score function provides valuable information about the direction of the data distribution, guiding the reverse diffusion process.

SDEs have contributed significantly to our understanding of the mathematical foundations of diffusion models. They highlight the connection between diffusion processes and the ability to manipulate trajectories in latent space. For a deeper dive into score-based generative modeling, explore this blog post on score-based generative modeling.

Bayesian Probability

Bayesian probability plays a crucial role in modeling the conditional probability distributions within the diffusion process. In the forward diffusion process, we gradually add noise to the data. Bayesian probability helps us to describe the probability of a noisy data point given the original data point. Similarly, in the reverse diffusion process, we aim to estimate the original data from the noisy data. Bayesian inference allows us to calculate the probability of the original data given the noisy data, which is essential for reconstructing high-quality samples.

The conditional probability distributions are modeled using Bayes’ theorem, which provides a framework for updating beliefs based on new evidence. By leveraging Bayesian probability, diffusion models can effectively handle uncertainty and generate realistic data samples.

Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is the primary optimization algorithm used to train the neural network within diffusion models. The neural network’s task is to predict the noise added at each step of the reverse diffusion process. SGD iteratively adjusts the network’s parameters to minimize the difference between the predicted noise and the actual noise.

At each iteration, SGD calculates the gradient of a loss function with respect to the network’s parameters using a small subset of the training data (a “batch”). The parameters are then updated in the direction opposite to the gradient, gradually moving the network towards a state where it can accurately predict the noise. The model learns to generate increasingly refined data at each step using this iterative process.

Loss Functions

Loss functions are essential for training diffusion models. These functions quantify the difference between the model’s predictions and the ground truth, guiding the optimization process. A common loss function used in diffusion models is the variational lower bound (VLB). The VLB is derived from the principles of variational inference and provides a tractable way to approximate the intractable posterior distribution. The VLB can be expressed as:

L = E[log p(x₀) – DKL(q(x1|x₀) || p(x1|x₀)) – Σ E[DKL(q(xt+1|xt, x₀) || p(xt+1|xt))]]

Where:

• L is the variational lower bound
• E denotes the expectation
• p(x₀) is the prior probability of the original data
• DKL represents the Kullback-Leibler divergence, which measures the difference between two probability distributions
• q and p represent the approximate and true posterior distributions, respectively
• x₀, xt, and xt+1 represent the data at different time steps in the diffusion process

By minimizing the VLB, diffusion models learn to accurately model the data distribution and generate high-quality samples. This optimization process is crucial for achieving the impressive results seen in diffusion-based image generation.

Ancestral Sampling

Ancestral sampling is the basic and most straightforward sampling method for diffusion models. It involves starting from pure noise and iteratively applying the reverse diffusion process, step-by-step, until a final data sample is generated. At each step, the model predicts the noise and subtracts it from the current sample, refining the data and moving closer to a coherent image.

While ancestral sampling is conceptually simple, it can be computationally expensive, requiring many steps to generate high-quality samples. Each step involves running the neural network, making the overall process relatively slow. Although the process is accurate, the time required for generation can be a limiting factor.

DDIM Sampling

DDIM (Denoising Diffusion Implicit Models) sampling is a faster and more efficient alternative to ancestral sampling. DDIM introduces a non-Markovian process for the reverse diffusion, allowing the model to take larger steps and generate samples with fewer iterations. This significantly reduces the sampling time without sacrificing image quality.

DDIM sampling achieves its speedup by introducing a deterministic process, allowing the model to “jump” ahead in the reverse diffusion process. While still relying on the neural network to predict the noise, DDIM enables the generation of high-quality samples in a fraction of the time compared to ancestral sampling. Because of the speed and efficiency of DDIM, it has become a popular choice for sampling within diffusion models.

Hardware Acceleration

Hardware acceleration is a crucial factor in overcoming the computational bottlenecks of diffusion models. Specialized AI accelerators, such as TPUs (Tensor Processing Units), are increasingly being used to accelerate the training and inference of these models. TPUs are custom-designed hardware accelerators optimized for deep learning workloads, providing significant performance gains compared to traditional CPUs and GPUs. You can read more about TPUs on Google Cloud TPUs.

In addition to TPUs, the increasing availability of GPUs with more memory is also playing a significant role. GPUs with large memory capacities allow for larger batch sizes and more complex models, leading to faster training and higher-quality results. The combination of specialized AI accelerators and high-memory GPUs is driving the development of more efficient and powerful diffusion models.

Model Parallelism

Model parallelism is a technique used to distribute the computation of a large neural network across multiple devices, such as GPUs. In diffusion models, the neural network responsible for predicting the noise can be very large, making it challenging to fit on a single device. Model parallelism addresses this issue by dividing the network into smaller parts and assigning each part to a different device. During training and inference, the devices work in parallel, significantly reducing the overall computation time.

Model parallelism enables the training and deployment of larger and more complex diffusion models, leading to improved performance and higher-quality results. By distributing the computational load, model parallelism helps overcome the memory and processing limitations of individual devices.

Quantization

Quantization is a technique used to reduce the memory footprint and computational cost of a neural network by representing its parameters with lower precision. Instead of using 32-bit floating-point numbers, quantization techniques can reduce the precision to 16-bit or even 8-bit integers. This significantly reduces the memory required to store the model and accelerates computations.

Quantization can be applied to both the weights and activations of the neural network. While reducing precision can lead to a slight decrease in accuracy, the benefits in terms of memory savings and computational speed often outweigh the drawbacks. Quantization is a valuable tool for deploying diffusion models on resource-constrained devices, such as mobile phones and embedded systems.

Knowledge Distillation

Knowledge distillation is a technique used to train smaller, faster models by transferring knowledge from a larger, more complex model. In the context of diffusion models, knowledge distillation involves training a smaller “student” model to mimic the behavior of a larger “teacher” model. The student model learns to approximate the output of the teacher model, effectively distilling its knowledge into a more compact representation.

Knowledge distillation allows for the creation of efficient diffusion models that can be deployed on devices with limited computational resources. By transferring knowledge from a larger model, the student model can achieve comparable performance with a fraction of the computational cost. The use of knowledge distillation increases the accessibility of diffusion models, as smaller models can be run on local machines.

Image Generation

Generating realistic and diverse images is one of the primary applications of diffusion models. By learning to reverse the diffusion process, these models can create images that are both visually appealing and semantically meaningful. The ability to control the generation process through conditional inputs, such as text prompts or image inputs, has made diffusion models a powerful tool for creative expression. For practical applications of AI image generation, refer back to the Clipdrop guide.

The applications of image generation are vast, ranging from creating personalized avatars to generating product images for e-commerce. Diffusion models enable the creation of images tailored to individual preferences, revolutionizing how we interact with visual content.

Video Generation

Diffusion models are being adapted for video creation and editing. By extending the diffusion process to the temporal dimension, these models can generate coherent and realistic video clips from text prompts or other inputs. While still an active area of research, video generation using diffusion models holds immense potential. To see the latest in diffusion models for video, explore the research being done at RunwayML.

Video generation has applications in various industries, including entertainment, advertising, and education. Diffusion models are paving the way for new forms of video content creation, allowing users to generate personalized videos with ease.

3D Modeling

Generating 3D models using diffusion processes is an emerging area of research. By learning to model the distribution of 3D shapes, diffusion models can create realistic and detailed 3D models from various inputs, such as text prompts or 2D images. This has applications in gaming, virtual reality, and product design.

The ability to generate 3D models with diffusion models opens up new possibilities for content creation, enabling the rapid prototyping and customization of 3D objects. This technology promises to revolutionize how 3D content is created and consumed.

Personalized Image Generation

Diffusion models are being used to create images tailored to individual preferences. By conditioning the models on user data, such as past purchases or browsing history, they can generate images that are more likely to appeal to individual customers. For example, companies are now using diffusion models to generate personalized product images for e-commerce. By conditioning the models on user preferences and product attributes, they can create images that are more likely to appeal to individual customers; a user may be able to view an image of a t-shirt printed with an animal of their choice.

Personalized image generation has applications in various industries, including e-commerce, advertising, and entertainment. This technology enables the creation of highly targeted and relevant visual content, leading to increased engagement and conversions.

AI Upscaling

AI-powered photo editing apps now employ diffusion models to enhance image resolution. These models are trained to add realistic details to low-resolution images, resulting in sharper and more visually appealing results. Several Android and iOS photo editing apps employ this kind of AI upscaling, providing a practical example to the audience.

The use of diffusion models in AI upscaling significantly improves the quality of low-resolution images, making them suitable for printing, sharing, and other applications. This technology has revolutionized the way we enhance and restore old or low-quality photos.

Composable Diffusion

Composable diffusion involves training diffusion models on specific image components, such as backgrounds, objects, and styles, and then combining them in novel ways. This modular approach allows for more controllable and creative image generation. By composing different elements, users can create complex and personalized images with ease. This is an emerging trend that gives far better control over the generated content.

Composable diffusion enables the creation of images with fine-grained control over individual components, opening up new possibilities for creative expression and design.

Diffusion Transformers

Diffusion Transformers combine the strengths of diffusion models and transformers, resulting in a powerful architecture for image generation. Transformers provide better global context understanding, leading to more coherent and realistic image generation. By integrating transformers into the diffusion process, these models can capture long-range dependencies and generate images with improved semantic consistency. You can read about the specifics of this integration in this paper on combining transformers and diffusion models.

Diffusion Transformers represent a significant step forward in the evolution of diffusion models, offering improved image quality and coherence.

Improved Control

Future research is heavily focused on allowing users to precisely manipulate generated images through various conditioning techniques, such as sketch guidance and semantic maps. Experts emphasize the importance of control in diffusion models, as it enables users to create images that meet specific requirements. The ability to guide the generation process with fine-grained control is essential for many applications.

Allowing users to have improved control of the image generation process is crucial for moving towards more modular image generation.

Real-time Diffusion Models

The development of accelerated sampling methods and hardware optimization techniques is paving the way for real-time image generation and editing. Real-time diffusion models would enable interactive applications, such as live image editing and virtual reality experiences. The ability to generate images in