Configuration¶

All kreuzcrawl operations are controlled through CrawlConfig. This guide covers every field, its default value, validation rules, and the builder pattern for constructing engines.

Builder pattern¶

The CrawlEngineBuilder provides a fluent API for constructing a CrawlEngine with custom configuration and trait implementations:

use kreuzcrawl::{CrawlEngine, CrawlConfig, BestFirstStrategy, Bm25Filter};

let engine = CrawlEngine::builder()
    .config(CrawlConfig {
        max_depth: Some(3),
        max_pages: Some(100),
        max_concurrent: Some(5),
        stay_on_domain: true,
        respect_robots_txt: true,
        ..Default::default()
    })
    .strategy(BestFirstStrategy)
    .content_filter(Bm25Filter::new("rust programming", 0.3))
    .build()?;

Builder methods¶

All fields are optional. Unset fields use their default implementations from the defaults module.

The .build() call validates the configuration and returns Result<CrawlEngine, CrawlError>.

Convenience constructors¶

For simple use cases that do not need custom trait implementations, use the binding functions:

use kreuzcrawl::{create_engine, scrape, crawl, map_urls};

let engine = create_engine(Some(CrawlConfig {
    max_depth: Some(2),
    ..Default::default()
}))?;

let scrape_result = scrape(&engine, "https://example.com").await?;
let crawl_result = crawl(&engine, "https://example.com").await?;
let map_result = map_urls(&engine, "https://example.com").await?;

Passing None to create_engine uses CrawlConfig::default().

Config validation¶

CrawlConfig::validate() is called automatically during CrawlEngineBuilder::build(). It checks:

You can also call validate() manually:

let config = CrawlConfig {
    max_concurrent: Some(0),
    ..Default::default()
};

match config.validate() {
    Ok(()) => println!("Valid"),
    Err(e) => eprintln!("Invalid: {}", e),
}

Full field reference¶

Crawl scope¶

HTTP client¶

Authentication¶

AuthConfig variants:

// HTTP Basic authentication
AuthConfig::Basic { username: "user".into(), password: "pass".into() }

// Bearer token
AuthConfig::Bearer { token: "your-token".into() }

// Custom header
AuthConfig::Header { name: "X-API-Key".into(), value: "key-value".into() }

Proxy¶

ProxyConfig {
    url: "http://proxy:8080".to_string(), // or "socks5://proxy:1080"
    username: Some("user".to_string()),
    password: Some("pass".to_string()),
}

Robots and compliance¶

Content processing¶

URL discovery (map)¶

Asset downloading¶

AssetCategory options: Document, Image, Audio, Video, Font, Stylesheet, Script, Archive, Data, Other.

Document downloading¶

Browser configuration¶

BrowserConfig fields¶

WARC output¶

Serialization¶

CrawlConfig implements Serialize and Deserialize with #[serde(deny_unknown_fields)]. Duration fields are serialized as milliseconds:

{
  "max_depth": 3,
  "max_pages": 100,
  "max_concurrent": 5,
  "stay_on_domain": true,
  "respect_robots_txt": true,
  "request_timeout": 30000,
  "max_redirects": 10,
  "retry_count": 2,
  "retry_codes": [429, 503],
  "include_paths": ["^/docs/"],
  "exclude_paths": ["/admin/"],
  "main_content_only": false,
  "cookies_enabled": false,
  "download_assets": false,
  "download_documents": true,
  "document_max_size": 52428800,
  "capture_screenshot": false,
  "save_browser_profile": false,
  "browser": {
    "mode": "auto",
    "timeout": 30000,
    "wait": "network_idle"
  }
}

Default values summary¶

Default trait implementations¶

When not overridden via the builder, the engine uses these defaults: