This note zooms out from Stable Diffusion's specific implementation to the general text-to-image generation task itself — the practical considerations and parameters that shape output quality, regardless of which specific diffusion model is being used.
The Key User-Facing Parameters
| Parameter | Effect |
|---|---|
| Number of inference steps | More steps generally means higher quality, at the cost of slower generation — a direct expression of the diffusion speed/quality tradeoff from Why Diffusion Models |
| Guidance scale | Higher values follow the prompt more strongly, but excessive values can degrade image naturalness |
| Seed | Controls the initial random noise — the same seed with the same prompt and settings reproduces the exact same image |
| Negative prompt | Steers generation away from specified undesired content |
Why the Same Prompt Can Produce Different Images
Unless the random seed is fixed, generation starts from a different random noise sample each time (step 1 of the reverse process from Diffusion Reverse Process) — the same text conditioning guides the denoising process toward images matching the prompt, but the specific starting noise shapes many of the final image's specific, unguided details, producing meaningfully different results across runs even with an identical prompt.
Prompt Engineering — A Practical Skill
Because the model's understanding of a prompt is shaped entirely by its training data and text encoder, specific, descriptive prompts (mentioning style, lighting, composition, level of detail) generally produce more controlled, predictable results than vague ones — an entire practical skill area ("prompt engineering") has developed around understanding how to phrase requests to reliably steer these models toward desired outputs.
Code — Reproducible Generation With a Fixed Seed
from diffusers import StableDiffusionPipeline
import torch
pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
generator = torch.Generator().manual_seed(42) # fixing the seed makes generation reproducible
image = pipe(
"a red sports car on a mountain road, golden hour lighting, cinematic",
generator=generator,
num_inference_steps=40,
guidance_scale=8.0
).images[0]
# Re-running with the SAME seed, prompt and settings produces the SAME imageBeyond Pure Text-to-Image
The same underlying diffusion machinery extends naturally to related tasks: image-to-image generation (starting the reverse process from a partially-noised real image instead of pure noise, letting the output stay grounded in the original while still being guided by a text prompt), and inpainting (regenerating only a masked region of an image, conditioned on both the surrounding context and a text prompt).
Common Mistakes
- Expecting identical results across runs without fixing the random seed — variation between generations with the same prompt is expected behavior, not a bug, stemming directly from the random noise the reverse process starts from.
- Assuming more inference steps always meaningfully improves quality without limit — quality gains typically diminish substantially beyond a certain point (often 30–50 steps for many models), while generation time keeps increasing linearly.
Interview Relevance
Q: "Why does generating an image from the same text prompt twice (without fixing a seed) typically produce two different images?" The reverse diffusion process starts from a randomly sampled noise vector in latent space — the text conditioning guides the denoising process toward images matching the prompt's content, but doesn't determine the specific starting noise, which shapes many of the final image's unguided visual details. Fixing the random seed removes this source of variation, making generation reproducible.
Practice Question
How would you adapt a text-to-image diffusion pipeline to perform inpainting — regenerating only a specific masked region of an existing image?
