Skip to content
Hagicode Docs
Blog

Implementing Image Upload and AI Recognition in Chat: A Complete Solution from Design to Implementation

Edit page
HagiCode for Windows Microsoft Store artwork
HagiCode for Windows is now on Microsoft Store
HagiCode for Windows is officially live on Microsoft Store. Windows users can install it directly from the storefront and stay on the store-managed update path. Open the listing and take a look.
Open Microsoft Store

Implementing Image Upload and AI Recognition in Chat: A Complete Solution from Design to Implementation

In AI interaction systems, how do you enable users to upload images and have the AI directly recognize them? I struggled with this question for quite a while, but fortunately, through practice in HagiCode, I’ve found some approaches. Today, let’s discuss this image upload and recognition solution, from custom protocol design to file system storage, to frontend-backend separated preview. It’s a complete technical notebook.

Background

In this era of AI chat proliferation, visual information is actually an important carrier for users to express their intentions. However, traditional chat systems mostly only support pure text input, which means users can’t directly pass visual context to AI for analysis, which is somewhat regrettable.

HagiCode also encountered similar dilemmas during development: users couldn’t upload images during chat or opinion creation, AI couldn’t access users’ local visual information, and there was a lack of a complete closed loop from image input, storage, rendering to AI context passing.

These problems aren’t actually that big; they just need some time and patience to solve. We designed and implemented a complete image upload and recognition process, allowing Claude and other AI models to directly recognize and analyze screenshots uploaded by users. Next, I’ll slowly elaborate on the implementation details of this solution.

About HagiCode

The solution shared in this article comes from our practical experience in the HagiCode project. HagiCode is an open-source AI coding assistant project based on OpenSpec workflow design, committed to providing a smarter code writing experience.

Analysis

Technical Challenges

Before starting implementation, we need to first clarify the main challenges we face. After all, sharpening the axe doesn’t delay the woodcutting.

Cross-Module Collaboration: Image upload involves multiple modules including frontend UI, upload service, backend API, file storage, message persistence, and AI execution mapping. Each module has its own responsibilities and interfaces, requiring a coordinated overall solution design.

Storage Strategy Selection: Should images be stored in the database or file system? If choosing file system, how should the directory structure be designed? How to integrate with existing OpenSpec workflows? These all need careful consideration.

Reference Protocol Design: We need a standard image reference method that can be both rendered by the frontend and correctly parsed by the AI execution path. Use direct file paths? HTTP URLs? Or design a specialized protocol?

AI Capability Compatibility: Different AI executors have varying degrees of multimodal support. Some executors natively support image input, while others can only process text. How to design a unified adapter layer to ensure all executors correctly process image information?

Design Decisions

After thorough discussion and consideration, we made the following key design decisions.

Decision 1: File System Storage

We chose to store images in the file system rather than the database. The directory structure is designed as follows:

<system-root>/images/<sessionId>/
├── <timestamp>-<uuid>.jpg
└── <timestamp>-<uuid>.png

The reasons are quite clear: simplify implementation, avoid database bloat, and files can be directly read by AI. Additionally, image files aren’t inherently suitable for databases—the file system is the more natural choice. It’s like putting books on a bookshelf rather than stuffing them into a notebook—same principle.

Decision 2: Custom Protocol hagiimag://

To avoid conflicts with HTTP URLs while making reference semantics clearer, we designed a custom image reference protocol:

hagiimag://session-abc123/20260301-143022-a1b2c3d4

The format of this protocol is hagiimag://<sessionId>/<imageId>, with clear semantics, easy to parse and route. Seeing this format, developers immediately understand it’s an image reference, not a regular URL. These small design details are sometimes quite useful.

Decision 3: Frontend Preview and AI Access Separation

During implementation, we discovered that frontend and AI have different access requirements for images: the frontend needs HTTP API preview, while AI needs to directly read local file paths. Therefore, we designed separate access methods:

  • Frontend uses /api/Images/{sessionId}/{imageId}/content for preview
  • AI uses server-resolved local file paths

This ensures both security (not exposing server paths) and usability (browsers can directly access). After all, security and usability always need to be balanced.

Decision 4: Immediate Upload Strategy

Another key decision was the upload timing. We chose to trigger upload immediately when users select or paste an image, referencing only already successfully uploaded images when sending messages.

The advantage is error handling upfront—keeping the message-sending API simple and the JSON contract compact. Users know before sending whether their image upload was successful—better user experience. This “prepare in advance” design approach is perhaps often applicable.

Solution

Architecture Design

Based on the above decisions, we designed the following overall architecture:

Frontend Layer
├── ConversationInputArea  ◄─────── useImageAttachmentManager
│       │                             │
│       ├── File Selection             ├── Attachment Status Management
│       ├── Clipboard Paste            ├── Upload/Retry/Delete
│       └── Attachment Preview         └── Image Reference Generation
│
Service Layer
├── ImageUploadService
│       ├── uploadImage()      ◄─────── ImagesController
│       ├── deleteImage()                 │
│       ├── parseHagiImageUrl()  ◄─────── Parse Protocol Links
│       └── buildPreviewUrl()              │
│
Backend Layer
├── ImagesController           ◄─────── ImagesDomainService
│       │                                  │
│       ├── POST /upload                  ├── File Validation
│       ├── GET /{sessionId}/{imageId}    ├── Image Storage
│       ├── DELETE                        ├── Image Compression
│       └── GET /content                  └── Reference Parsing
│
AI Execution Layer
├── ImageContentBlock          ◄─────── StructuredMessageDomainService
│       │                                  │
│       ├── Multimodal Executors           ├── Image Block Parsing
│       └── Text Executor Fallback         └── Path Hint Generation

This architecture clearly shows the complete data flow from frontend to AI. Each layer has clear responsibilities and interacts through standard interfaces. Good architecture is actually like this: each doing their job, not interfering with each other, smooth communication.

Key Processes

Image Upload Process:

  1. User selects image via file selection or clipboard paste
  2. Frontend validates file type and size (supports JPEG/PNG/WEBP/GIF, 10MB per file)
  3. Upload API is called, image is saved to /images/{sessionId}/ directory
  4. API returns hagiimag:// reference and preview URL
  5. Frontend displays preview thumbnail in attachment bar, user can preview before sending

AI Recognition Process:

  1. User sends message containing image reference
  2. Backend parses hagiimag:// protocol link, extracts sessionId and imageId
  3. Image reference is mapped to ImageContentBlock
  4. Processing method is chosen based on executor capabilities:
    • Multimodal executors: pass structured image input
    • Text executors: fallback to image path hint

This completes a full closed loop: user uploads image → AI recognizes image → AI returns analysis results. Such smooth processes often provide better user experience.

Practice

Frontend Implementation

In the frontend, we provide a dedicated Hook to manage image attachment state:

import { useImageAttachmentManager } from '@/hooks/useImageAttachmentManager';


function ChatInput() {
  const {
    attachments,
    uploadedImages,
    hasBlockingAttachments,
    isUploading,
    selectFiles,
    removeAttachment,
    clearAttachments,
  } = useImageAttachmentManager({
    ownerId: sessionId,
    mapUploadedImage: (response) => response,
    uploadOptions: { compress: false },
  });


  const handleFileSelect = (files: File[]) => {
    selectFiles(files);
  };


  const handlePaste = (e: ClipboardEvent) => {
    const files = Array.from(e.clipboardData?.files || [])
      .filter(f => f.type.startsWith('image/'));
    if (files.length > 0) {
      handleFileSelect(files);
    }
  };


  return (
    <div>
      {/* Attachment Bar */}
      {attachments.map(att => (
        <AttachmentItem
          key={att.localId}
          file={att.file}
          status={att.status}
          onRemove={() => removeAttachment(att.localId)}
        />
      ))}


      {/* Input Field */}
      <textarea onPaste={handlePaste} />


      {/* Upload Button */}
      <button onClick={() => fileInputRef.current?.click()}>
        Upload Image
      </button>
    </div>
  );
}

This Hook encapsulates all attachment management logic, including upload status tracking, failure retry, attachment deletion, etc. It’s very simple to use—just call a few methods to complete the entire process. Good API design is actually like this: simple to use, without losing flexibility.

Parsing Custom Protocol:

// Extract sessionId and imageId from custom protocol
const parsed = parseHagiImageUrl("hagiimag://session-abc123/20260301-143022-uuid");
// Returns: { sessionId: "session-abc123", imageId: "20260301-143022-uuid" }


// Build preview URL
const previewUrl = buildPreviewUrl(parsed.sessionId, parsed.imageId);
// Returns: "/api/Images/session-abc123/20260301-143022-uuid/content"

With these two utility functions, the frontend can easily convert between hagiimag:// protocol and HTTP URLs. When this conversion logic is well encapsulated, using it is much more convenient.

Backend Implementation

The backend uses ASP.NET Core, with ImagesController and ImagesDomainService as the core:

[HttpPost("upload")]
[RequestSizeLimit(50 * 1024 * 1024)]
public async Task<ActionResult<ImageUploadResponseDto>> Upload(
    [FromForm] UploadImageFormRequest input)
{
    // 1. Validate request
    if (file == null || file.Length == 0)
        throw new UserFriendlyException("No file provided");


    // 2. Validate file type and size
    var (isValid, errorMessage) = _imagesDomainService.ValidateImage(
        file.FileName, file.ContentType, file.Length);
    if (!isValid)
        throw new UserFriendlyException(errorMessage);


    // 3. Save to file system
    await using var stream = file.OpenReadStream();
    var result = await _imagesDomainService.UploadImageAsync(
        stream,
        sessionId,
        file.FileName,
        file.ContentType,
        CurrentUserId,
        compress: input.Compress);


    // 4. Return result
    return Ok(result);
}

This implementation follows the typical Web API development pattern: validate, process, return. It’s worth noting that we set a 50MB request size limit to prevent malicious large file uploads. After all, in the network world, it’s always right to be careful.

Important Considerations

During implementation, some details need special attention:

Permission Validation: Image access must validate user identity, ensuring only access to images from one’s own session. This is a basic security requirement that cannot be omitted. When it comes to security, better safe than sorry.

Path Safety: Strictly validate sessionId and imageId to prevent path traversal attacks. For example, reject paths containing ../ to prevent users from accessing arbitrary files in the system. When such boundary conditions are handled well, the system becomes more robust.

File Cleanup: When sessions are deleted, associated images must be synchronously cleaned up to avoid orphan file accumulation. After long-term operation, these files may occupy significant disk space. Timely cleanup is also a good habit.

Compression Strategy: For screenshot-like filenames (like screenshot.png), automatically enable compression to save space. This strategy can be adjusted according to actual needs. Storage space—save where you can.

Fallback Handling: Executors without multimodal support must receive image path hints, not silently discard image information. This is important, otherwise users will think the AI ignored their image. When it comes to user experience, details determine success or failure.

Status Management: Uploading attachments block message sending, failed attachments allow retry or deletion. This design ensures user experience continuity. When status management is clear, users won’t feel confused.

Summary

Through this complete image upload and recognition solution, HagiCode has achieved a complete closed loop from user input to AI recognition. The core highlights of the entire solution include:

  • Custom hagiimag:// protocol achieved standardization of image references
  • File system storage simplified implementation and improved performance
  • Frontend preview and AI access separation balances security and usability
  • Immediate upload strategy optimized user experience
  • Compatible design of multimodal and text fallback ensures flexibility

This solution runs stably in HagiCode with positive user feedback. If you’re implementing similar functionality, I hope these experiences are helpful to you.

Technical solutions have no absolute right or wrong, only suitable or not suitable. Finding the path that fits your project is the most important thing.

References

Tags:
相关链接
HagiCode for Windows Microsoft Store artwork
HagiCode for Windows is now on Microsoft Store
HagiCode for Windows is officially live on Microsoft Store. Windows users can install it directly from the storefront and stay on the store-managed update path. Open the listing and take a look.
Open Microsoft Store

开始使用 HagiCode

一次安装，几分钟上手

HagiCode for Windows 在 Microsoft Store 免费提供。打开商店即可安装并保持更新；也可以先对比各版本与定价，再决定从哪个渠道开始。

前往 Microsoft Store 安装 查看版本与定价