Picture This: Open Source AI for Image Description

Summary

Nolan Darilek presents a self-hosted image-description service intended to help blind screen-reader users understand photos, diagrams, menus, and other visual material. Humans often provide inadequate alt text, and the post contrasts that history with increasingly capable LLM-generated descriptions from apps and browser add-ons. The implementation combines Ollama, running the Apache-licensed LLaVA 34B multimodal model, with a PocketBase API over SQLite and a small Python client; collection hooks send images and follow-up questions, retain chat history, and enforce per-user access rules. In testing, a cold request on an A100-40GB Fly Machine took about 45 seconds, while stopping idle GPU Machines reduced resource use at the cost of startup delay. The example produced useful scene descriptions but could not reliably identify leafless tree species, illustrating both the prototype’s value and its limitations.

Takeaways

1. 1

   Ollama can serve the LLaVA 34B multimodal model from a Docker image after pulling it with the Ollama CLI.

2. 2

   PocketBase hooks model images and follow-ups as collections, trigger Ollama requests, preserve chat history, and restrict users from accessing one another’s chats.

3. 3

   On an A100-40GB Fly Machine, a cold request took about 45 seconds; stopping the Machine saved allocated resources but added startup and model-loading latency.