...

AgentQL

AI-powered query language for web scraping and automation. It uses natural language selectors to find data on any page, including authenticated content. AgentQL queries are self-healing as UI changes and work across similar sites. Users can define structured data output, making AgentQL versatile for developers and data scientists.
GitHub
368
Created 10 months ago, last commit 16 days ago
18 contributors
89 commits
Stars added on GitHub, per day, on average
Yesterday
+2
Last week
+3.4
/day
Last month
+3.6
/day
README

AgentQL

AI-powered web scraping and automation

Follow on X Follow on LinkedIn Join our Discord

Python version GitHub Repo stars GitHub watchers

What is AgentQL?

AgentQL is an AI-powered query language for scraping web sites and automating workflows. It uses natural language queries to pinpoint data and elements on any web page, including authenticated and dynamically generated content. Users can define structured data output and apply transforms within queries. AgentQL's natural language selectors find elements intuitively based on the content of the web page and work across similar web sites, self-healing as UI changes over time.

Get started in 5 minutes with our quick start!

Features

  • Playwright AgentQL's Python SDK and JavaScript SDK seamlessly integrates with Playwright for advanced automation and testing.
  • Cross-site compatibility lets you use the same query across different sites with similar content.
  • Structured output defined by the shape of your query.
  • Natural language selectors find elements and data anywhere on a site using intuitive queries.
  • Transforms and extracts data in your queries.
  • Works on any page, public or private, any site, any URL, even behind authentication.
  • Resiliance to UI changes means queries work regardless of how a page's structure changes over time.

Tools

  • Python SDK for running automation and scraping scripts with AgentQL queries in Python.
  • JavaScript SDK for running automation and scraping scripts with AgentQL queries in JavaScript.
  • REST API for executing queries without an SDK.
  • Debugger Browser Extension lets you debug and finesse queries in real-time on live sites.
  • AgentQL Query Language lets you define queries with natural language.
  • Playground for playing with AgentQL lets you export python scripts and optimize queries with prompts.

Examples

Example Name JavaScript Python
Getting Started Example Example
Close Cookie Dialog Example Example
Close Popup Windows Example Example
Compare Product Prices Example Example
Get Element by Prompt Example Example
Infinite Scroll N/A Example
External Browser Integration Example Example
Query List Items Example Example
Site Login Example Example
Headless Browser Example Example
Save/Load Auth Session Example Example
Stealth Mode Example Example
Wait for Page Load Example Example
E-commerce Pricing Data Example Example
Sentiment Analysis Example Example
Get XPath Example Example
Submit Form Example Example
Collect YouTube Comments Example N/A
Use Humanlike Behavior to Avoid Antibot Example Example
Run in Google Colab N/A Example

For comprehensive guides and API references, check out our official documentation.

Show Your Support 🌟

If you find AgentQL helpful, please consider giving us a star on GitHub! It helps us reach more developers and continue improving the project.

GitHub Repo stars

Get in touch

For questions, feedback, or support, join our Discord community. You can follow us on GitHub, Twitter, and LinkedIn!