Tarsier

Category: Classic browser automation utility (Python)

What it is: Tarsier is a Python library from Reworkd that gives LLM-based web agents a perception layer. It visually tags interactable page elements with bracket IDs (e.g. [23], [@5], [$12]) and converts page screenshots into whitespace-structured text that even text-only LLMs can parse and act on. This solves the "how do I feed a webpage to an LLM and map its response back to real elements?" problem.

How it works:

• Injects a tagging overlay via Playwright or Selenium
• Tags interactable elements: [#ID] for inputs, [@ID] for links, [$ID] for buttons/selects, [ID] for plain text (opt-in)
• Runs an OCR pass (Google Cloud Vision or Microsoft Azure Computer Vision) to convert the tagged screenshot into a structured text string
• Returns both the page text and a tag_to_xpath mapping so the LLM's chosen action (e.g. CLICK [@3]) resolves to a real XPath the driver can execute

Free vs paid: The Tarsier Python package is free and open source (MIT License). However, it requires an external OCR service credential: either a Google Cloud Vision service account key or a Microsoft Azure Computer Vision API key. Both providers charge for OCR calls under their standard pricing.

Official links:

• GitHub: https://github.com/reworkd/tarsier
• PyPI: https://pypi.org/project/tarsier/
• Cookbook / examples: https://github.com/reworkd/tarsier/tree/main/cookbook

Current version: 0.8.2 (released October 1, 2024)

Language / runtime: Python 3.11 or 3.12 (requires >=3.11,<4.0)

Supported browser drivers: Playwright (^1.44.0), Selenium (^4.21.0)

No downloadable template / starter boilerplate repo. Example notebooks are in the cookbook directory of the main repo.