Image Generation Models 1999—2023

Grows a texture one pixel at a time by finding similar neighborhoods in a source image. The 1999 method showed how local image statistics could reproduce texture without a trained model. The sandbox uses a CPU reconstruction of the algorithm.

Builds a texture from overlapping source-image patches, joining them along minimum-error cuts. Working with patches preserved larger structures and reduced the work required by pixel-by-pixel synthesis. The sandbox reconstructs the 2001 algorithm without learned weights.

Models images through layers of stochastic units, trained one layer at a time before fine-tuning. The 2006 paper provided a practical training procedure for deep generative networks when training many layers was difficult. The original generative digit checkpoint has not been recovered.

Learns an encoder and a probabilistic decoder, allowing new images to be sampled from a latent distribution. Its reparameterization method made this form of variational learning trainable with gradient descent. The sandbox uses Kingma’s 2014 MNIST weights for the 2013 architecture.

Trains a generator to produce samples and a discriminator to distinguish them from training data. The 2014 paper introduced adversarial training as a way to learn a generative model, with experiments on digits, faces and small color images. A suitable original checkpoint remains unrecovered.

Builds an image through successive canvas updates, using attention to control where it reads and writes. The 2015 architecture combined a recurrent network with variational learning, making image generation a sequence of local operations. The sandbox uses Eric Jang’s pretrained 2016 implementation; samples are mostly loop-shaped digits.

Modifies an input image to amplify features detected by an image-classification network. Google’s 2015 experiments used this process to inspect what different layers had learned, producing repeated eyes, animals and architectural forms. It generates these patterns by optimizing the input image while keeping the network fixed.

Optimizes an image to combine the content of one image with the style statistics of another. Gatys and colleagues showed that features from a network trained for object recognition could separate these two properties. Each output is optimized iteratively, unlike later feed-forward style-transfer methods.

Generates images from random vectors using convolutional generative adversarial networks. The 2015 paper specified architectural choices that made convolutional GAN training more stable and examined the visual features learned without labels. Its published examples include 64 × 64 LSUN bedrooms.

Applies a learned style to an input image in a single feed-forward pass. Johnson and colleagues trained the transformation network with perceptual losses, moving the repeated optimization required by earlier style transfer into training. The planned exhibit follows their 2016 method.

Uses a GAN to constrain interactive edits, including drawn strokes and color changes. The 2016 system turned a generator into an editing tool by finding latent inputs that satisfied a user’s constraints. Its controls operate on a learned image distribution rather than directly painting output pixels.

Learns image-to-image translation from paired training examples, including edges and photographs or building labels and facades. The same conditional adversarial framework was demonstrated across several translation tasks. Its requirement for aligned pairs distinguishes it from the later CycleGAN method.

Learns translation between image domains without paired examples. A cycle-consistency objective requires an image translated to the second domain and back to resemble its starting point. This allowed the 2017 system to learn tasks such as horse-to-zebra translation from separate collections of images.

Trains a generator and discriminator by progressively adding layers for higher image resolutions. This approach stabilized training and demonstrated generated faces at 1024 × 1024 pixels in 2017. It remains a future research candidate for the sandbox.

Generates images conditioned on an ImageNet class using large-scale adversarial training. The 2018 work examined how scale affected sample quality and stability across many object categories. The sandbox uses pretrained BigGAN-deep at 512 × 512, with class, seed and truncation controls.

Edits a generated scene by changing neuron activations associated with objects such as trees or doors. The GAN Dissection work tested whether individual units had a causal role in producing those objects. This entry covers its 2018 painting demonstration, where edits act inside a pretrained generator.

Introduced a generator with separate style controls at different image scales and noise inputs for local variation. This made it possible to mix broad attributes and finer details from different latent inputs. The original published models include faces, cats, bedrooms and cars.

Uses adaptive discriminator augmentation to reduce overfitting when training GANs with limited data. The 2020 work showed that useful generators could be trained from much smaller image collections than earlier systems typically used. It remains a future research candidate, separate from original StyleGAN.

Optimizes VQGAN’s latent representation against a text prompt scored by CLIP. The 2021 workflow combined separately trained image and language-image models, adding text guidance without training a new generator for the task. Community notebooks exposed the optimization process and its intermediate images.

An open text-to-image project developed independently of OpenAI’s DALL·E. It predicts sequences of image tokens from text and uses a decoder to turn them into pixels. The 2021 project made its code and trained models available; this exhibit preserves Mini separately from the later Mega model.

A notebook-based system combining diffusion sampling with CLIP guidance. Beginning in 2021, it brought together pretrained diffusion and image-text models with controls for the sampling process. Later versions added animation features, making the notebook a configurable workflow as well as a still-image generator.

Generates and edits images with text-guided diffusion. The 2021 work compared CLIP guidance with classifier-free guidance and demonstrated text-directed inpainting. The planned exhibit is the smaller filtered research release, rather than the unreleased full model.

Generates images in a compressed representation before decoding them into pixels, reducing the cost of diffusion at full image resolution. This BERT-conditioned model was released before Stable Diffusion and provides a distinct example of the CompVis latent-diffusion approach. Its text encoder and checkpoint differ from the later Stable Diffusion family.

The larger open model from the DALL·E Mini team, documented in 2022 and associated with the service that became Craiyon. It extended the project’s text-to-image token-generation approach at a larger scale. The sandbox uses a public Mega checkpoint, distinct from current proprietary Craiyon models.

A text-to-image latent diffusion model released with downloadable weights and code. The v1.1–v1.4 checkpoints cover successive training stages; v1.4 was recommended at the August 2022 public launch. The release allowed the model to be run and modified outside a hosted service. The sandbox runs v1.4.

Adds spatial conditioning to a pretrained diffusion model using inputs such as edges, depth maps or poses. The 2023 method trains added control components while retaining the base model, allowing an input structure to guide a text-prompted image. The sandbox uses Canny edges.