Multimodal Integration

Claude Vision — The Basics

In Module 14, you learned to generate images. Now you close the loop by giving Claude the ability to see those images and reason about them. Claude Vision accepts images as part of a conversation and can describe, classify, compare, extract text, and answer questions about visual content. Combined with image generation, this creates a powerful feedback loop where your AI system can evaluate its own visual output.

Sending images to Claude: base64 encoded vs URL reference

Claude accepts images in two formats. The URL reference method is simpler — you pass a URL and Claude fetches the image. The base64 method embeds the image data directly in the request, which is more reliable when you control the image source (no risk of URL expiration or access issues).

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic();

// Method 1: URL reference (simpler, works with Convex serving URLs)
const response = await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: [
        {
          type: "image",
          source: {
            type: "url",
            url: "https://your-convex-serving-url.com/image.webp",
          },
        },
        {
          type: "text",
          text: "Describe this image in detail.",
        },
      ],
    },
  ],
});

// Method 2: Base64 encoded (more reliable, no URL dependencies)
const imageBuffer = await fetch(imageUrl).then(r => r.arrayBuffer());
const base64Data = Buffer.from(imageBuffer).toString("base64");

const response2 = await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: [
        {
          type: "image",
          source: {
            type: "base64",
            media_type: "image/webp",
            data: base64Data,
          },
        },
        {
          type: "text",
          text: "What is in this image?",
        },
      ],
    },
  ],
});

Image token cost: ~1,000–1,600 tokens per image

Every image sent to Claude Vision consumes tokens from your context window. A standard image costs approximately 1,000–1,600 tokens depending on resolution and the model used. This is equivalent to roughly 750–1,200 words of text in token cost.

For a multimodal agent that generates and analyzes images in a loop, each iteration costs the image generation fee (Replicate) plus the Vision API call tokens. A 3-iteration refinement loop with Claude Sonnet at current pricing might cost approximately:

Per iteration:
  Image generation (flux-schnell):  $0.003
  Vision analysis (~1,200 tokens):  ~$0.004
  Text generation (~500 tokens):    ~$0.002
  Total per iteration:              ~$0.009

3-iteration loop total:             ~$0.027

What Claude Vision can do

Claude Vision is remarkably capable across a range of visual tasks:

• Describe: Generate detailed natural-language descriptions of scenes, objects, colors, spatial relationships, and composition.
• Classify: Categorize images by content type, style, quality, or domain-specific labels.
• Compare: Evaluate how well an image matches a text description — this is the key capability for the refinement loop.
• Extract text: Read and transcribe text visible in images (OCR), including handwritten text, signs, documents, and UI screenshots.
• Answer questions: Respond to specific questions about image content ("How many people are in this photo?" "What brand is visible on the label?").

The Generate-Analyze-Refine Loop

The generate-analyze-refine loop is the core pattern of this module. It automates the manual iteration cycle you practiced in Module 14 — instead of you inspecting images and rewriting prompts, Claude Vision does it programmatically.

Step 1 — Generate image from user prompt via Replicate

The loop starts with the user's original prompt. This is the intent — what the user wants to see. You send this prompt to Replicate's Flux API exactly as you learned in Module 14.

Step 2 — Store in Convex file storage for persistence

The generated image URL from Replicate is temporary. Download it immediately and store it in Convex file storage. Save the storage ID alongside the prompt and iteration number in your database.

Step 3 — Get serving URL from ctx.storage.getUrl()

Convert the Convex storage ID to a serving URL. This URL is what you will send to Claude Vision for analysis. Convex serving URLs are signed and time-limited but last long enough for the analysis step.

Step 4 — Send image to Claude Vision for analysis

This is where the loop becomes intelligent. You send the generated image to Claude Vision along with the original prompt, and ask Claude to evaluate how well the image matches the intended description.

Step 5 — Claude evaluates match quality and suggests refinements

Claude's analysis should identify three things: what matches well, what is missing or different from the original prompt, and a suggested refined prompt that would produce a better result. This structured output is what drives the next iteration.

Step 6 — Enhanced prompt created from vision feedback

Extract the refined prompt from Claude's response. This new prompt incorporates specific visual feedback — if the original image lacked a certain element, the refined prompt will explicitly request it with more detail.

Step 7 — Generate improved image with refined prompt

Send the refined prompt to Replicate for a new generation. The improved specificity typically produces a noticeably better match on the second iteration.

Step 8 — Store, analyze again if needed (configurable iterations)

Store the new image, and optionally run another analysis cycle. In practice, 2–3 iterations produce diminishing returns — the biggest improvement happens between iteration 1 and 2. Configure a maximum iteration count to prevent runaway costs.

Claude Vision Analysis Prompt

Structured analysis prompt

The quality of the refinement loop depends heavily on the prompt you use for Claude Vision analysis. Here is a structured prompt that produces actionable feedback:

const analysisPrompt = `I generated this image using the following prompt:
"${originalPrompt}"

Analyze the generated image and provide feedback in exactly this format:

## Match Assessment
Rate the overall match from 1-10. Briefly explain.

## What Matches Well
List 2-3 elements that match the original prompt accurately.

## What Is Missing or Different
List 2-3 specific elements that are missing, wrong, or could be improved.

## Refined Prompt
Write an improved version of the original prompt that addresses the issues
identified above. Be more specific about the elements that were missing.
Keep the core intent but add detail where the generation fell short.`;

const analysis = await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: [
        {
          type: "image",
          source: { type: "url", url: servingUrl },
        },
        { type: "text", text: analysisPrompt },
      ],
    },
  ],
});

Parsing vision output: extracting the refined prompt programmatically

To extract the refined prompt from Claude's structured response, parse the output looking for the "Refined Prompt" section:

function extractRefinedPrompt(analysisText: string): string {
  const refinedSection = analysisText.split("## Refined Prompt");
  if (refinedSection.length < 2) {
    // Fallback: return the original prompt
    return "";
  }
  // Take everything after the header, trim whitespace
  return refinedSection[1].trim().replace(/^["']|["']$/g, "");
}

function extractMatchScore(analysisText: string): number {
  const match = analysisText.match(/(\d+)\s*\/\s*10/);
  return match ? parseInt(match[1], 10) : 0;
}

Multimodal Agent Architecture

Combining generation + vision in a single Convex agent

The most powerful pattern is to wrap the generate-analyze-refine loop inside a @convex-dev/agent with dedicated tools. The agent can autonomously decide when to generate, when to analyze, and when to refine — or when the image is good enough to stop.

Tool design: generate_image, analyze_image, refine_prompt

The agent needs three tools, each with a clear responsibility:

// Tool 1: Generate an image from a text prompt
const generateImageTool = {
  name: "generate_image",
  description: "Generate an image from a text prompt using Replicate Flux. Returns a Convex storage ID.",
  parameters: {
    prompt: { type: "string", description: "The image generation prompt" },
    model: {
      type: "string",
      description: "flux-schnell for drafts, flux-dev for production",
      enum: ["flux-schnell", "flux-dev"],
    },
  },
};

// Tool 2: Analyze a generated image using Claude Vision
const analyzeImageTool = {
  name: "analyze_image",
  description: "Analyze a generated image against the original prompt. Returns match score and suggestions.",
  parameters: {
    storageId: { type: "string", description: "Convex storage ID of the image" },
    originalPrompt: { type: "string", description: "The prompt used to generate the image" },
  },
};

// Tool 3: Refine a prompt based on vision analysis
const refinePromptTool = {
  name: "refine_prompt",
  description: "Create an improved prompt based on vision analysis feedback.",
  parameters: {
    originalPrompt: { type: "string", description: "The original generation prompt" },
    analysisFeedback: { type: "string", description: "The vision analysis feedback" },
  },
};

Loop control: max iterations, convergence detection

Without explicit limits, the agent could refine indefinitely. Implement two stop conditions:

• Max iterations: Hard cap at 3–5 iterations. Beyond this, quality improvements are marginal and costs accumulate.
• Convergence detection: If the match score from Claude Vision is 8/10 or higher, or if Claude's analysis says "good match" or "closely matches," stop refining. The image is good enough.

State management: storing each iteration's prompt, image, and analysis

Track every iteration in your database. This creates a history that users can review — and it is essential for debugging prompt engineering issues:

// convex/schema.ts
import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";

export default defineSchema({
  image_sessions: defineTable({
    userId: v.optional(v.string()),
    originalPrompt: v.string(),
    status: v.string(), // "generating" | "analyzing" | "refining" | "complete"
    currentIteration: v.number(),
    maxIterations: v.number(),
    createdAt: v.number(),
  }),

  iterations: defineTable({
    sessionId: v.id("image_sessions"),
    iterationNumber: v.number(),
    prompt: v.string(),
    storageId: v.optional(v.id("_storage")),
    analysis: v.optional(v.string()),
    matchScore: v.optional(v.number()),
    refinedPrompt: v.optional(v.string()),
    createdAt: v.number(),
  }).index("by_session", ["sessionId"]),
});

Build Project — Multimodal Generation Agent

Full architecture

Your build project combines everything from this module into a working multimodal agent. The architecture is: a Convex agent with 3 tools (generate, analyze, refine), Convex file storage for images, a streaming React UI that shows conversation and images inline, and an iteration history panel.

The complete generate-analyze loop in Convex

// convex/multimodal.ts
import { action } from "./_generated/server";
import { internal } from "./_generated/api";
import { v } from "convex/values";
import Replicate from "replicate";
import Anthropic from "@anthropic-ai/sdk";

export const runRefinementLoop = action({
  args: {
    prompt: v.string(),
    maxIterations: v.optional(v.number()),
  },
  handler: async (ctx, args) => {
    const replicate = new Replicate();
    const anthropic = new Anthropic();
    const maxIter = args.maxIterations ?? 3;

    let currentPrompt = args.prompt;
    const results: Array<{
      prompt: string;
      storageId: string;
      matchScore: number;
    }> = [];

    for (let i = 0; i < maxIter; i++) {
      // Generate
      const output = await replicate.run(
        "black-forest-labs/flux-schnell",
        { input: { prompt: currentPrompt, num_outputs: 1 } }
      );
      const imageUrl = output[0] as string;

      // Store
      const response = await fetch(imageUrl);
      const blob = await response.blob();
      const storageId = await ctx.storage.store(blob);
      const servingUrl = await ctx.storage.getUrl(storageId);

      // Analyze
      const analysis = await anthropic.messages.create({
        model: "claude-sonnet-4-6",
        max_tokens: 1024,
        messages: [{
          role: "user",
          content: [
            { type: "image", source: { type: "url", url: servingUrl! } },
            {
              type: "text",
              text: `Original prompt: "${args.prompt}"
Current prompt: "${currentPrompt}"
Iteration: ${i + 1}

Rate match 1-10. List what matches, what's missing.
If score >= 8, say "CONVERGED".
Otherwise, write a refined prompt.`,
            },
          ],
        }],
      });

      const analysisText = analysis.content[0].type === "text"
        ? analysis.content[0].text : "";
      const scoreMatch = analysisText.match(/(\d+)\s*\/\s*10/);
      const score = scoreMatch ? parseInt(scoreMatch[1], 10) : 5;

      results.push({ prompt: currentPrompt, storageId, matchScore: score });

      // Check convergence
      if (score >= 8 || analysisText.includes("CONVERGED")) {
        break;
      }

      // Extract refined prompt for next iteration
      const refinedSection = analysisText.split("refined prompt");
      if (refinedSection.length >= 2) {
        currentPrompt = refinedSection[1].trim()
          .split("\n")[0].replace(/^[:"'\s]+|[:"'\s]+$/g, "");
      }
    }

    return results;
  },
});

React UI: prompt input, iteration history, side-by-side comparison

The UI should display each iteration's image alongside its prompt and match score. Users can see how the prompt evolved and how image quality improved across iterations. Include a cost summary showing the total spend for the refinement session.

Production Considerations

Image generation latency

Each image generation takes 1–10 seconds depending on the model. A 3-iteration refinement loop could take 30–60 seconds total (generation + analysis + generation + analysis + generation). Use streaming status updates in your UI so users see progress rather than a blank loading screen. Display messages like "Generating image (iteration 1 of 3)..." and "Analyzing with Claude Vision..." as each step completes.

Storage costs: clean up old images on a schedule

Every generated image consumes storage. For applications with high generation volume, implement a cleanup job that deletes images older than a configurable retention period. Convex scheduled functions or cron jobs are ideal for this — run a cleanup action daily that queries for old image records and calls ctx.storage.delete() on their storage IDs.

Content moderation: pre-generation prompt safety check

A simple content moderation pattern adds minimal latency but significantly improves safety:

async function isPromptSafe(prompt: string): Promise<boolean> {
  const anthropic = new Anthropic();
  const check = await anthropic.messages.create({
    model: "claude-haiku-4-0",
    max_tokens: 10,
    messages: [{
      role: "user",
      content: `Does this image generation prompt violate content policies
(violence, explicit content, harmful content)? Reply only "safe" or "unsafe".
Prompt: "${prompt}"`,
    }],
  });
  const result = check.content[0].type === "text" ? check.content[0].text : "";
  return result.toLowerCase().includes("safe")
    && !result.toLowerCase().includes("unsafe");
}