Usage

Minimal example (Google Cloud Vision + Playwright)

This is the canonical usage pattern from the official README. It visits Hacker News, tags the page, and prints the text representation plus the XPath mapping.

import asyncio
import json

from playwright.async_api import async_playwright
from tarsier import Tarsier, GoogleVisionOCRService


def load_credentials(json_file_path):
    with open(json_file_path) as f:
        return json.load(f)


async def main():
    # Load your Google Cloud service account JSON
    credentials = load_credentials("./google_service_acc_key.json")
    ocr_service = GoogleVisionOCRService(credentials)

    tarsier = Tarsier(ocr_service)

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        page = await browser.new_page()
        await page.goto("https://news.ycombinator.com")

        # Tag the page and get the text + XPath mapping
        page_text, tag_to_xpath = await tarsier.page_to_text(page)

        print(tag_to_xpath)  # e.g. {1: '/html/body/...', 2: '...'}
        print(page_text)     # Text representation with [ID] tags


if __name__ == "__main__":
    asyncio.run(main())

Using Azure instead of Google

Swap the OCR service; everything else is identical:

from tarsier import Tarsier, MicrosoftAzureOCRService

credentials = load_credentials("./microsoft_azure_credentials.json")
ocr_service = MicrosoftAzureOCRService(credentials)
tarsier = Tarsier(ocr_service)

Tagging all text elements (not just interactables)

By default Tarsier only tags interactable elements. To also tag plain text nodes:

page_text, tag_to_xpath = await tarsier.page_to_text(page, tag_text_elements=True)

Element tag legend

Tag format	Element type
[#ID]	Text-insertable fields (input, textarea)
[@ID]	Hyperlinks (anchor tags)
[$ID]	Other interactable elements (button, select)
[ID]	Plain text nodes (only when tag_text_elements=True)

LLM action mapping pattern

Your LLM reads page_text and responds with an action like CLICK [@3]. Your agent code resolves this:

target_id = 3
xpath = tag_to_xpath[target_id]
await page.locator(f"xpath={xpath}").click()

Cookbook notebooks

Full end-to-end agent examples are in the cookbook:

• LangChain web agent: cookbook/langchain-web-agent.ipynb
• LlamaIndex web agent: cookbook/llama-index-web-agent.ipynb