bitfennec
1059be1b62
- Rust core engine with HTTP ingestion gateway, hybrid search, and white-box resume matching - Python edge collectors supporting declarative TOML and specialized adapters - PostgreSQL schema migrations with pgvector embedding support - Hardened security controls: strict CSP, terminal sanitization, and input boundary validation - Multi-service container orchestration with dedicated non-root execution
26 lines
771 B
Markdown
26 lines
771 B
Markdown
# Bangwen Collectors
|
|
|
|
Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.
|
|
|
|
## Structure
|
|
|
|
- `sources/`: Declarative TOML configuration for crawling target sites.
|
|
- `adapters/`: Custom Python scrapers for complex or stateful collection logic.
|
|
- `common/`: Core fetcher, security validators, and ingestion protocol models.
|
|
- `runner.py`: Periodic crawler runner and domain-concurrency scheduler.
|
|
|
|
## Running
|
|
|
|
```bash
|
|
uv run python runner.py --list
|
|
uv run python runner.py --once
|
|
```
|
|
|
|
## Testing
|
|
|
|
Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:
|
|
|
|
```bash
|
|
cd collectors && uv run python tests/test_content_and_security.py
|
|
```
|