# Bangwen Collectors

Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.

## Structure

- `sources/`: Declarative TOML configuration for crawling target sites.
- `adapters/`: Custom Python scrapers for complex or stateful collection logic.
- `common/`: Core fetcher, security validators, and ingestion protocol models.
- `runner.py`: Periodic crawler runner and domain-concurrency scheduler.

## Running

```bash
uv run python runner.py --list
uv run python runner.py --once
```

## Testing

Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:

```bash
cd collectors && uv run python tests/test_content_and_security.py
```