# bangwen-collectors π·οΈ
> **Lightweight, declarative, and SSRF-hardened edge collection system for public job recruitment aggregation.**
---
## π Table of Contents
- [β¨ Overview](#-overview)
- [π Project Structure](#-project-structure)
- [π Quickstart](#-quickstart)
- [Prerequisites](#prerequisites)
- [Local Setup](#local-setup)
- [π οΈ Configuration & Source Authoring](#οΈ-configuration--source-authoring)
- [Declarative TOML Sources](#declarative-toml-sources)
- [Custom Python Adapters](#custom-python-adapters)
- [βοΈ Scheduling & Rate Limiting](#οΈ-scheduling--rate-limiting)
- [π§ͺ Testing](#-testing)
- [π License](#-license)
---
## β¨ Overview
`bangwen-collectors` serves as the decentralized edge data extraction layer for the `bangwen` recruitment aggregation engine:
- **Dual-Mode Crawling**: Supports both zero-code declarative TOML site cards (`sources/`) and customized Python scraper adapters (`adapters/`).
- **Domain-Grouped Concurrency**: Concurrently processes distinct domains while enforcing strictly serial queues within the same domain.
- **WAF-Friendly Polite Delays**: Configurable inter-channel pauses (default 3.0s) and detail extraction throttles (0.3s) preventing IP bans.
- **SSRF Defense-in-Depth**: Built-in private IP filtering, strict domain whitelists, and hop-by-hop verification during HTTP redirects.
---
## π Project Structure
```
collectors/
βββ adapters/ # Specialized scrapers for complex dynamic sites
β βββ sc/chengdu_rsks.py # Custom adapter for Chengdu Personnel Exam Net
βββ sources/ # Zero-code declarative TOML site cards
β βββ sc/ # Sichuan Province sources (e.g. myrsks, zgrsks)
β βββ yn/ # Yunnan Province sources (e.g. hrss, yuxi)
βββ common/ # Core modules
β βββ content.py # Content truncation and dirty body cleanup
β βββ declarative.py # Universal declarative parsing engine
β βββ fetcher.py # PoliteFetcher HTTP client with SSRF verification
β βββ protocol.py # Pydantic schema matching the bangwen/v1 protocol
β βββ security.py # URL validation and private IP guards
βββ tests/ # Security and parsing regression tests
βββ config.toml # Active collection and scheduler configuration
βββ runner.py # Main scheduler and multi-domain orchestrator
βββ Dockerfile # Dedicated non-root container definition
```
---
## π Quickstart
### Prerequisites
- Python 3.12+
- [uv](https://docs.astral.sh/uv/) package manager
### Local Setup
1. **Install dependencies**:
```bash
uv sync
```
2. **List available collectors**:
```bash
uv run python runner.py --list
```
3. **Run a single crawl cycle**:
```bash
uv run python runner.py --once
```
---
## π οΈ Configuration & Source Authoring
### Declarative TOML Sources
Place zero-code TOML configuration cards under `sources/{province}/{site_id}.toml`:
```toml
[site]
id = "yn.yuxi"
name = "ηζΊͺεΈδΊΊζ°ζΏεΊ"
province = "δΊεη"
city = "ηζΊͺεΈ"
employer = "ηζΊͺεΈηΈε
³η¨δΊΊεδ½"
employer_type = "δΊδΈεδ½"
allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]
[template]
item_selector = "div.news-list a"
url_pattern = "/article/\\d+\\.html"
title_selector = "h1.article-title"
body_selector = "div.article-content"
date_regex = "(\\d{4}[-/εΉ΄]\\d{1,2}[-/ζ]\\d{1,2})"
attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]
[[channels]]
id = "rszk"
name = "δΊΊδΊζθ"
start_url = "https://www.yuxi.gov.cn/rszk.html"
keywords = ["ζθ", "θθ―", "ε½η¨", "ε
¬η€Ί"]
```
### Custom Python Adapters
When target sites require complex session handling, cryptographic requests, or specific payload extraction, create a Python scraper under `adapters/{province}/{name}.py`:
- Implement a crawl function returning an `IngestPayload`.
- Use `common.security.is_safe_url` to enforce domain isolation.
- Sanitize extracted text with `common.content.sanitize_body_text`.
---
## βοΈ Scheduling & Rate Limiting
The runner (`runner.py`) provides fine-grained scheduling controls:
| Environment Variable / Flag | Default | Description |
|-----------------------------|---------|-------------|
| `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Target Rust gateway ingestion endpoint |
| `INGEST_TOKEN` | *None* | Bearer authentication token matching core `.env` |
| `SCHEDULE_INTERVAL_HOURS` | `6` | Cycle interval between automated sweeps |
| `--provinces` | `[]` | Filter target provinces (e.g. `yn,sc`) |
| `--concurrency` | `3` | Maximum concurrent distinct domain scrapers |
---
## π§ͺ Testing
Comprehensive test suites covering SSRF prevention, HTML sanitization, and URL security:
```bash
uv run python tests/test_content_and_security.py
# Expected output: All security and content tests passed!
```
---
## π License
[MIT](../LICENSE)
(back to top)