Multimodal Prompting Guide: Photo + Text Workflows That Stick

Multimodal prompting means you send text plus an image, PDF, screenshot, chart, or video frame in one request, then ask the model to ground its answer in those files. You name each file’s job, what to extract or compare, and the reply shape. Vague "look at this" uploads fail because the model guesses the region and invents plausible details.

You will leave with a plain definition, an attach → ground → task → format pattern, copy-paste examples for GPT-5.6 Sol and Terra, Claude Fable 5 and Sonnet 5, and Gemini 3.5 Flash and 3.1 Pro, plus a photo-plus-text loop that pairs PromptMake /image and /text.

What multimodal prompting is (and who it helps)

A text-only prompt describes the world in words. A multimodal prompt gives the model pixels or document pages to read alongside your instructions. The model still answers in text or structured text on most jobs. The difference sits in the input: evidence lives in the attachment, and your words tell the model how to use that evidence.

This differs from image generation prompts. Midjourney, FLUX, and similar tools take a text (or image-to-image) brief and paint a new frame. Multimodal prompting on chat and API vision stacks reads an existing frame or file. You can still use a vision pass to write a better image-generator prompt afterward. Those are two jobs: understand, then create.

Who gets value from this:

• Product and design teams who critique screenshots, mockups, and competitor UIs
• Marketers who turn reference photos into briefs or generator prompts
• Support and ops leads who extract fields from invoices, tickets, and photos of forms
• Engineers who build pipelines that accept uploads and return JSON schemas

Skip a multimodal call when a short text description already contains every fact you need. Attachments cost tokens and add failure modes (blur, crop, wrong page). Use vision when the source of truth is visual or layout-heavy.

How multimodal prompting works in practice

Treat every multimodal request as four labeled parts: attachment map, grounding question, task, and output contract. The attachment map lists each file and its role ("IMAGE_1: product shelf photo from aisle B"). Grounding asks the model to state what it sees before it advises, so you catch hallucinations before the advice step. The task is the job you care about: compare, extract, score, rewrite. The output contract locks format: bullets, table, JSON, or a paste-ready Midjourney string.

Order matters. Put hard rules at the start and end of the text. Place media in the middle when the product lets you control order (Gemini bookends are a clear example). Models lose edge instructions when a long visual blob sits first and your constraints trail as an afterthought. Keep the visual crop tight: crop to the shelf, the chart, or the UI panel you care about before upload. Extra background is noise the model will invent stories about.

Name success before you send. Extraction succeeds when every required field has a value or an explicit "not visible." Critique succeeds when the score cites a visible cue. Generator-prompt drafting succeeds when a teammate can paste the string into Midjourney or FLUX without guessing missing props. If you cannot name the check, you will accept fluent wrong answers.

Pattern A: describe, then decide

Describe-then-decide splits vision from judgment. First prompt: "List only what is visible in IMAGE_1. Columns: Object, Location (left/center/right or region), Certainty (high/med/low). No advice." Second prompt: "Using only the table below, recommend three shelf fixes for a grocery brand. No new objects. Table: [paste]."

You inspect the table before the advice call. If the table invents a competitor logo that is not in the photo, you stop. One mega-prompt that mixes "describe and recommend" hides that failure inside polished prose.

Pattern B: extract to schema, then write

Extract-to-schema fits invoices, receipts, UI copy audits, and form photos. First call returns JSON keys you defined (vendor, date, line_items, total, currency, missing_fields). Second call turns that JSON into an email, ticket, or spreadsheet row. Keep temperature low on extract. Raise creativity only on the write hop, and only if the write hop is creative on purpose.

Paste-ready extract prompt: "Read IMAGE_1. Return JSON only with keys vendor, invoice_date, currency, total, line_items (array of {name, qty, unit_price}). If a field is unreadable, set it to null and add the key name to missing_fields. Do not invent totals."

A step-by-step multimodal prompting workflow

Build the habit offline before you wire APIs. Take one real photo or PDF you already struggle with. Run the four-part pattern by hand in chat. Freeze the prompt text only after two samples clear your success check. Automation that wraps a fuzzy "analyze this" string multiplies bad extractions.

Decide the outcome type first: facts, judgment, or a generator prompt for another model. Facts want low temperature and a schema. Judgment wants a rubric that points at visible evidence. Generator prompts want camera, subject, lighting, and style fields that Midjourney, FLUX, DALL·E / GPT Image, Leonardo, or SDXL can consume. Mixing all three outcomes in one reply weakens every part.

Budget cost and latency. A describe-then-decide chain is two calls. A single grounded call is cheaper when the task is simple ("read the SKU on the label"). Use the chain when errors are expensive or when a human must approve the grounding table. Many teams run describe on Gemini 3.5 Flash or GPT-5.6 Terra, then decide on GPT-5.6 Sol, Claude Opus 5, or Gemini 3.1 Pro.

When the photo needs to become an image-model prompt, run PromptMake /image with a goal mode (Recreate Exactly, Change Style, Adjust Lighting, or Create Variation), then harden the text brief in PromptMake /text for the chat model that will critique or expand it. Soft loop: vision draft from /image, instruction polish on /text, paste into your target app.

Steps 1-3: Pick the file, crop, write the map

1. Choose one file and one outcome (extract fields, score against a rubric, or draft a generator prompt).
2. Crop or export the smallest region that still holds the evidence. Drop blurry pages and unrelated slides.
3. Write a two-line attachment map: filename or label, plus the role of that file in the task.

Stop if you need five files and cannot say which one is authoritative. Split into separate calls or label a primary source. Cross-image compare jobs need an explicit "prefer IMAGE_A for color; IMAGE_B for layout" rule.

Steps 4-6: Ground, task, freeze

1. Run a grounding-only pass on one sample. Fix crop or lighting if certainty stays low on the fields you need.
2. Add the task and output contract. Re-run. Score against your checklist before you trust fluency.
3. Freeze prompt text, model, and temperature. Save the sample file next to the prompt in your library.

Re-open the prompt when the document layout changes or a new model class shifts OCR quality. Do not regenerate the whole stack every week for sport.

Common multimodal prompting mistakes

The first failure mode is an empty task. "What do you think of this?" invites a generic essay. Replace it with a job and a format: "Score this landing hero 0-2 on headline clarity, CTA contrast, and mobile crop risk. Cite a visible cue for each score."

The second failure mode is trusting ungrounded detail. Models fill gaps with brand-looking guesses. Force "not visible" as a legal answer. Ask for certainty tags on extract. When stakes are high, keep describe-then-decide so a human can kill a bad table.

The third failure mode is wrong resolution and wrong crop. Tiny text in a wide room photo will fail OCR. Zoom the label. For PDFs, send the page that holds the table; do not dump a 40-page deck if page 7 is the only source.

The fourth failure mode is mixing generation dialect with vision dialect. "--ar 16:9 --stylize 250" belongs in an image-generator prompt, not in a Claude PDF extract. Keep Midjourney parameters out of analysis calls. Keep analysis schemas out of FLUX creative briefs unless you are writing a hybrid pipeline on purpose.

The fifth failure mode is one prompt for every model. Gemini likes rules at both ends of the text around media. Claude handles long PDFs with layout awareness when you state page scope. GPT-5.6 paths want a clear goal, constraints, and format, and they pair with conversational image work in ChatGPT. Port the core job; wrap dialect per vendor.

Model notes for multimodal prompting in 2026

As of mid-2026, use these names in production notes: GPT-5.6 Sol (flagship), Terra, and Luna on OpenAI; Claude Fable 5, Opus 5, and Sonnet 5 on Anthropic; Gemini 3.5 Flash and Gemini 3.1 Pro on Google. ChatGPT still often defaults to a fast Instant-class chat model for quick turns; route hard vision reasoning to Sol or an equivalent reasoning seat when your product allows model choice.

OpenAI vision paths are strong on screenshots, charts, and conversational image edits through GPT Image / DALL·E-style flows in ChatGPT. Give goal, constraints, and the attachment. Skip "think step by step" on reasoning-class seats; ask for scores and citations to visible regions instead.

Claude shines when the attachment is a multi-page PDF or a dense document layout. State which pages matter. Sonnet 5 is a solid daily driver for extract and rewrite. Opus 5 or Fable 5 fit multi-condition critique across long docs. Native audio and video are weaker than Gemini; transcribe or sample frames first when those modalities dominate.

Gemini remains the broadest input surface for mixed media: images, documents, audio, and video on Flash and Pro tiers, with long context that helps multi-image compare jobs. Flash handles volume and agents. Pro handles hard reasoning over dense packs. Bookend critical rules before and after the media block.

Example: GPT-5.6 Terra extract + Sol critique

Terra extract: "Goal: extract shelf SKUs from IMAGE_1. Constraints: JSON only; keys skus (array of strings), unreadable_regions (array of short labels); no brand guesses. IMAGE_1 attached."

Sol critique: "Goal: PASS or FAIL this planogram compliance draft. Constraints: every claim must map to a SKU in EXTRACT; flag any claim with no SKU match. Output: first line PASS or FAIL; then failed checks only. EXTRACT: [paste]. DRAFT: [paste]."

Example: Claude Sonnet 5 PDF + Gemini Flash multi-image

Sonnet 5 on a PDF: "Scope: pages 2-4 only. Extract the pricing table into markdown. Preserve currency symbols. If a cell is cut off, write CUTOFF."

Gemini 3.5 Flash on three photos: "IMAGE_A is the reference pack shot. IMAGE_B and IMAGE_C are store shelves. List SKUs in B and C that match A. Table columns: SKU guess, which image, match confidence high/med/low. Rules: no new SKUs beyond what is readable; repeat this no-guess rule at the end."

How PromptMake fits a photo + text loop

PromptMake splits the two sides of multimodal work. Use /image when the upload should become a paste-ready prompt for Midjourney, FLUX, DALL·E, Stable Diffusion, or Leonardo. Pick a goal mode so the enhancer knows whether you want a faithful recreate, a style shift, a lighting change, or a variation. Use /text when you already have a rough analysis or ops brief and need a model-ready instruction for GPT-5.6, Claude, or Gemini.

A practical loop for creative teams: upload the reference in /image → copy the generator prompt → paste the same reference into ChatGPT, Claude, or Gemini with a short multimodal critique prompt → edit the generator string with findings → optional polish of that critique prompt in /text. Soft sell only: the free tier is enough to draft a few of these without standing up your own vision stack.

A practical loop for ops: crop the form photo → multimodal extract in your chat or API model → if the extract prompt is still a messy brief, harden it in /text → store the frozen prompt beside three sample images. Keep /image out of pure extract jobs unless you also need a marketing visual from the same source.

FAQ

What is multimodal prompting in plain terms?

Multimodal prompting is writing instructions that travel with non-text inputs such as images, PDFs, screenshots, or video frames. You tell the model what each file is for, what to read or ignore, and how to format the answer. The model grounds claims in the attachment instead of inventing a scene from a text description alone. Most replies stay text or JSON even though the input mixed media.

How is multimodal prompting different from an image generation prompt?

An image generation prompt asks a model such as Midjourney or FLUX to create new pixels. Multimodal prompting on vision-language chat models asks the model to read existing pixels or pages and return analysis, extraction, or a rewritten brief. You can chain them: vision read first, then a generator prompt that encodes what you learned. Keep the dialects separate so Midjourney parameters do not leak into extract calls.

When should I use describe-then-decide instead of one call?

Use describe-then-decide when a wrong visual detail would poison the advice, such as compliance checks or brand claims tied to on-shelf proof. The first call returns a grounded inventory. You or a script check it, then the second call reasons only over that inventory. One-shot calls fit low-stakes captions and rough sorting.

Which models handle multimodal prompting best in 2026?

Match the modality. Claude Sonnet 5, Opus 5, and Fable 5 are strong on PDFs and long documents; GPT-5.6 Sol and Terra are strong on screenshots, charts, and ChatGPT-centered image workflows. Gemini 3.5 Flash and 3.1 Pro cover the widest mix, including video and multi-image packs. Use Flash or Terra-class seats for volume extract; use Sol, Opus 5, Fable 5, or Gemini 3.1 Pro when the rubric is multi-hop.

How do I turn a photo into both analysis and a Midjourney prompt?

Run the photo through a vision critique or extract prompt for facts you care about. In a second pass, run PromptMake /image with Recreate Exactly or Change Style to draft a generator string for Midjourney, FLUX, or another supported target. Merge only the facts you verified into the generator string. Optional: polish the critique instruction in PromptMake /text so the analysis call stays stable across teammates.

What output format should multimodal prompts request?

Request the smallest structure a downstream step can consume. JSON with nulls for missing fields fits pipelines; markdown tables fit human review; numbered scores with a required visible cue fit critique. State "JSON only" or "table only" so the model does not wrap the payload in an essay.

How do I start multimodal prompting on a free tier?

Pick one photo or one PDF page, write a four-part prompt (map, ground, task, format), and run it in the free chat surface you already use. Save the reply and mark every invented detail. For a generator-oriented photo, try PromptMake /image on the free daily allowance, then harden any analysis brief in /text. Freeze the winner after two sample files pass your checklist.