Crawling¶

Crawling follows links from a seed URL, building a collection of pages up to a configured depth and page limit. Kreuzcrawl provides both collected and streaming interfaces, batch operations, and pluggable strategies for URL selection.

Basic crawl¶

The simplest crawl fetches a single seed URL and follows its links:

use kreuzcrawl::{CrawlConfig, create_engine, crawl};

let engine = create_engine(Some(CrawlConfig {
    max_depth: Some(2),
    max_pages: Some(50),
    ..Default::default()
}))?;

let result = crawl(&engine, "https://example.com").await?;

for page in &result.pages {
    println!("{} (depth {}): {} bytes", page.url, page.depth, page.body_size);
}

CrawlResult contains the full list of crawled pages, the final URL after redirect resolution, redirect count, collected cookies, and any error encountered.

Depth and page limits¶

When max_depth is None or Some(0), the engine fetches the seed URL but does not follow any links. Set max_depth: Some(1) to crawl one hop out.

Concurrent fetching¶

Control parallelism with max_concurrent:

CrawlConfig {
    max_concurrent: Some(5),
    ..Default::default()
}

The default of 10 concurrent requests is a good starting point. Lower it when crawling sites with strict rate limits; raise it for high-throughput internal crawls.

A per-domain rate limit also applies: the engine enforces a 200 ms minimum interval between requests to the same domain, and automatically respects Crawl-delay directives from robots.txt when respect_robots_txt is enabled.

Domain scoping¶

Keep the crawl within the seed domain or allow subdomains:

CrawlConfig {
    stay_on_domain: true,
    allow_subdomains: false, // only exact domain match
    ..Default::default()
}

Path filtering with regex¶

Include or exclude URL paths using regex patterns:

CrawlConfig {
    include_paths: vec![r"^/docs/".to_string(), r"^/blog/".to_string()],
    exclude_paths: vec![r"/admin/".to_string(), r"\.pdf$".to_string()],
    ..Default::default()
}

The engine compiles these patterns once at the start of the crawl and validates them during CrawlConfig::validate(). Invalid regex patterns produce a CrawlError::InvalidConfig error.

Batch crawling¶

Crawl multiple seed URLs concurrently, each following links independently:

use kreuzcrawl::batch_crawl;

let results = batch_crawl(
    &engine,
    vec![
        "https://example.com".to_owned(),
        "https://other-site.org".to_owned(),
    ],
).await?;

for entry in &results {
    match (&entry.result, &entry.error) {
        (Some(crawl), _) => println!("{}: {} pages", entry.url, crawl.pages.len()),
        (None, Some(e)) => eprintln!("{}: {}", entry.url, e),
        _ => {}
    }
}

batch_crawl respects the same max_concurrent limit across all seed URLs. Each seed URL produces an independent BatchCrawlResult with either a populated result or an error message.

There is also batch_scrape for scraping multiple individual URLs without link following:

use kreuzcrawl::batch_scrape;

let results = batch_scrape(
    &engine,
    vec![
        "https://example.com/page1".to_owned(),
        "https://example.com/page2".to_owned(),
    ],
).await?;

Redirect handling¶

The engine resolves redirects before starting the crawl loop, following HTTP 3xx redirects, Refresh headers, and <meta http-equiv="refresh"> tags. Redirect loops and excessive redirects are detected and reported.

CrawlResult reference¶

CrawlPageResult reference¶

Each page in the crawl result contains:

Edit this page on GitHub