ai-scraping · GitHub Topics

mendableai / firecrawl

#网络爬虫#Firecrawl 是一种 API 服务，它爬取URL并将其转换为清洗过的 markdown 或结构化数据

人工智能爬虫 data Markdown scraper html-to-markdown 大语言模型 rag scraping web-crawler ai-scraping webscraping

TypeScript 30.71 k

1 小时前

D4Vinci / Scrapling

#网络爬虫#🕷️ An undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy again!

爬虫 crawling Hacktoberfest Playwright Python scraping selectors stealth-game web-scraper web-scraping web-scraping-python webscraping xpath 自动化人工智能 ai-scraping data data-extraction

Python 2.75 k

5 天前

itsOwen / CyberScraper-2077

#网络爬虫#A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

ai-scraping 大语言模型 openai scraper webscraping gemini-api web-scraper

Python 1.62 k

1 个月前

raznem / parsera

#网络爬虫#Lightweight library for scraping web-sites with LLMs

data-extraction 大语言模型 scraping Python Open Source webscraping 人工智能 ai-scraping Playwright

Python 1.04 k

1 个月前

mendableai / firecrawl-app-examples

#大语言模型#🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.

人工智能 ai-scraping data Example html-to-markdown 大语言模型 Markdown rag web-crawler templates

Python 139

2 天前

ArchiveBox / abx-dl

#网络爬虫#⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/s...

Chrome crawling cURL 下载器 headless Playwright Puppeteer scraping wget youtube-dl yt-dlp cli-tool 命令行界面 http-client ai-scraping

JavaScript 65

3 个月前

spider-rs / web-crawling-guides

#网络爬虫#How to guides on web-crawling or scraping

agents ai-agents ai-scraping 爬虫 html-to-markdown scraper web-scraping

8 个月前

spider-rs / spider-clients

#网络爬虫#Python, Javascript, and Rust libraries for the Spider Cloud API.

人工智能 ai-agents ai-scraping 爬虫 html-to-markdown scraper spider web-scraping Supabase

Python 11

2 个月前

kaymen99 / ai-web-scraper

#网络爬虫#AI web scraper built with Crawl4AI for extracting structured leads data from websites.

ai-agents ai-scraping llms scraper web-scraper web-scraping

Python 11

1 个月前

nathabonfim59 / md-fetch

#网络爬虫#A CLI tool and REST API that converts web content to clean Markdown, bypassing anti-scraping measures using headless browsers. Perfect for AI/LLM applications

ai-scraping Go scraper

Go 2

1 个月前

HaroonBrokha / Ai-Webpage-Analyzer-Api-Free

AI Webpage Analyzer** is a powerful API service that extracts only the visible text content from any given URL and analyzes it using the **Haroon AI API**. It intelligently removes hidden elements, sc...

人工智能 ai-api ai-scraping 免费 REST API

1 个月前