Why screenshots need link extraction
People often take screenshots as a fast save-for-later gesture. The problem is that a screenshot is not clickable, even when it clearly shows a useful article, repository, video, product, travel page, or social post.
Basic OCR can read text from the image, but screenshot link extraction goes further. It tries to identify the primary resource, decide what type of resource it is, and recover a canonical destination when the full URL is missing.
Example: from image to usable link
Input: an iPhone screenshot showing an article headline, a product page, or owner/repository from a social post.
Extraction:
- Ignore app chrome, timestamps, comments, and navigation labels
- Identify the primary resource in the screenshot
- Classify it as an article, GitHub repo, product, video, website, or another supported type
- Use search and URL verification if the canonical URL is not visible
- Save the result as a card with an Open action
How SnapAction applies screenshot link extraction
SnapAction scans selected or recent screenshots from Photos and sends them to a Convex-backed AI screenshot agent. OpenRouter handles image understanding and structured extraction.
When a screenshot shows a recognizable title or name but not a full link, SnapAction can use Serper search and URL verification to recover a canonical URL before saving the resource in SwiftData.
The output is not just a copied URL. SnapAction saves resource type, title, URL, description, tags, metadata, screenshot asset links, and actions such as Open URL, Directions, Call, Email, Add to Calendar, or Copy Booking Reference when relevant.