bitfennec
1059be1b62
- Rust core engine with HTTP ingestion gateway, hybrid search, and white-box resume matching - Python edge collectors supporting declarative TOML and specialized adapters - PostgreSQL schema migrations with pgvector embedding support - Hardened security controls: strict CSP, terminal sanitization, and input boundary validation - Multi-service container orchestration with dedicated non-root execution
771 B
771 B
Bangwen Collectors
Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.
Structure
sources/: Declarative TOML configuration for crawling target sites.adapters/: Custom Python scrapers for complex or stateful collection logic.common/: Core fetcher, security validators, and ingestion protocol models.runner.py: Periodic crawler runner and domain-concurrency scheduler.
Running
uv run python runner.py --list
uv run python runner.py --once
Testing
Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:
cd collectors && uv run python tests/test_content_and_security.py