Captioning and tagging my Lightroom catalog with local vision models
AI-powered captions and keyword tags for Lightroom Classic, running entirely on Apple Silicon via MLX. Qwen3-VL writes the caption, gemma3 extracts tags, results land as XMP sidecar files.
I have a Lightroom catalog with thousands of photos and almost no useful metadata. The captions are blank. The keywords are sparse and inconsistent. Searching for “cat at sunset” returns nothing — not because there are no cats at sunset, but because nobody (me) ever bothered to type that. photo-analyzer is the tool that fixes that, locally, on my own machine.
Local only, on Apple Silicon
Photos are personal data, and sending a decade of family pictures through someone else’s GPU isn’t a trade I want to make for searchable keywords. Apple’s MLX framework runs the models on the M-series GPU, the models stay loaded between photos, nothing leaves the machine.
That constraint shaped everything else. Models had to fit in 16GB of RAM. Inference had to be fast enough to be usable across a real catalog. Output had to integrate with Lightroom’s existing metadata pipeline (XMP sidecar files) so the workflow stayed the same.
Two small models over one big one
The current pipeline has three passes: a vision model writes a factual caption, a text model extracts keyword tags from the caption, and the vision model scores the image for technical quality (sharpness, lighting, per-face analysis).
Captioning and tagging are different tasks — a caption is full sentences answering “what is in this image”, while tags are a controlled-vocabulary list answering “which keywords would I add to find this later”. Asking one model to do both at once produces mediocre output for both, and splitting the pipeline lets each pass do the thing it’s good at.
Phase 2 stripped out most of Phase 1
The repo started with a heavier Phase 1 — six specialised models (CLIP, RAM++ for object recognition, LAION aesthetic scoring, InsightFace, DeepFace, Ollama-driven Qwen3-VL). The output was richer, with hierarchical keywords like Score|Aesthetic|4 or Animal|Dog, but it was slow, fragile, dependency-heavy, and the aesthetic scoring was unreliable in practice.
Phase 2 stripped most of that out in favour of two MLX models. The richer output had been pulling its weight on demos and not on actual catalog use, and the simpler version was faster and more accurate on the metrics that mattered.
Was it worth it?
The codebase still reflects its evolution and will be cleaned up as the VLM approach matures, but the current pipeline is what I actually use. Lightroom Smart Collections do the rest: “Keywords contain score_sharpness_5” gives me my technically-best shots, “Keywords contain expression_joyful” gives me the emotional ones. The catalog has the metadata I’d always meant to add.
dmorel69/photo-analyzer— Python, MLX, Apple Silicon