Introduction
Crawlspace - The centralized web crawling platform
With Crawlspace, you can:
- Affordably crawl tens of millions of web pages per month
- Scrape web pages with LLM extraction or query selectors
- Store data in SQLite, a vector database, and/or S3-compatible bucket
- Respectfully adhere to rate-limiting and robots.txt out of the box
Get up to speed
Set up your local environment, and learn about web crawling philosophies.
Build a web crawler
Deploy web crawlers as easily as you deploy websites.
Overview
Learn the fundamentals of building a web crawler on Crawlspace
Handle responses
Use LLMs and query selectors to parse page responses
Store data
Save data in a SQLite database, a vector database, or a bucket
Configure your options
Tweak your crawler with configuration settings
Learn by example
Read some sample code to learn as you go.
Introduction
Crawlspace - The centralized web crawling platform
With Crawlspace, you can:
- Affordably crawl tens of millions of web pages per month
- Scrape web pages with LLM extraction or query selectors
- Store data in SQLite, a vector database, and/or S3-compatible bucket
- Respectfully adhere to rate-limiting and robots.txt out of the box
Get up to speed
Set up your local environment, and learn about web crawling philosophies.
Build a web crawler
Deploy web crawlers as easily as you deploy websites.
Overview
Learn the fundamentals of building a web crawler on Crawlspace
Handle responses
Use LLMs and query selectors to parse page responses
Store data
Save data in a SQLite database, a vector database, or a bucket
Configure your options
Tweak your crawler with configuration settings
Learn by example
Read some sample code to learn as you go.