Multimodal & Generative
The rest of the atlas is text-centric; this branch covers everything else AI now generates and understands: images, audio, video, and the models that combine modalities. The same deep-learning machinery reappears here in new shapes.
Mental model
Multimodal systems translate different observation spaces into representations that can be aligned, fused, generated, or acted upon. The architecture and objective must respect each modality's structure while evaluation covers semantic quality, temporal or spatial consistency, provenance, and human impact.
Roadmap: landscape and core models
- The multimodal landscape maps text, image, audio, video, and action spaces.
- Diffusion models, intuitively explains iterative denoising.
- Latent diffusion and Stable Diffusion explains why generation often happens in compressed latent space.
- Text-to-image conditioning and CFG covers prompt conditioning and classifier-free guidance.
Vision and shared representations
- CLIP and shared embedding spaces links text and images through contrastive learning.
- Vision Transformers explains images as patches and tokens.
- Vision-language models and multimodal LLMs covers image inputs, captions, OCR, grounding, and visual reasoning.
- Controlling image generation covers ControlNet, inpainting, adapters, and image LoRAs.
Media, evaluation, and risk
- Audio and speech covers ASR, TTS, voice conversion, and music generation.
- Video generation explains temporal consistency, motion, editing, and cost.
- Evaluating generative media covers FID, CLIPScore, human eval, task success, and safety review.
- Deepfakes, provenance, and watermarking covers media risk, C2PA, and disclosure.
Connects to: Model Architectures · Data for AI · AI Ethics and Governance
Core sources
- Denoising Diffusion Probabilistic Models — foundational diffusion objective and sampling process.
- High-Resolution Image Synthesis with Latent Diffusion Models — generation in compressed latent space with cross-attention conditioning.
- Learning Transferable Visual Models From Natural Language Supervision — CLIP's contrastive text-image representation learning.
- An Image is Worth 16x16 Words — Vision Transformer architecture and scale behavior.
- C2PA specification — interoperable media provenance and authenticity metadata.