bangwen/collectors/README.md

bangwen-collectors 🕷️

Lightweight, declarative, and SSRF-hardened edge collection system for public job recruitment aggregation.

---

📖 Table of Contents

• ✨ Overview
• 📁 Project Structure
• 🚀 Quickstart
  • Prerequisites
  • Local Setup
• 🛠️ Configuration & Source Authoring
  • Declarative TOML Sources
  • Custom Python Adapters
• ⚙️ Scheduling & Rate Limiting
• 🧪 Testing
• 📜 License

---

✨ Overview

bangwen-collectors serves as the decentralized edge data extraction layer for the bangwen recruitment aggregation engine:

• Dual-Mode Crawling: Supports both zero-code declarative TOML site cards (sources/) and customized Python scraper adapters (adapters/).
• Domain-Grouped Concurrency: Concurrently processes distinct domains while enforcing strictly serial queues within the same domain.
• WAF-Friendly Polite Delays: Configurable inter-channel pauses (default 3.0s) and detail extraction throttles (0.3s) preventing IP bans.
• SSRF Defense-in-Depth: Built-in private IP filtering, strict domain whitelists, and hop-by-hop verification during HTTP redirects.

---

📁 Project Structure

collectors/
├── adapters/                  # Specialized scrapers for complex dynamic sites
│   └── sc/chengdu_rsks.py    # Custom adapter for Chengdu Personnel Exam Net
├── sources/                   # Zero-code declarative TOML site cards
│   ├── sc/                    # Sichuan Province sources (e.g. myrsks, zgrsks)
│   └── yn/                    # Yunnan Province sources (e.g. hrss, yuxi)
├── common/                    # Core modules
│   ├── content.py             # Content truncation and dirty body cleanup
│   ├── declarative.py         # Universal declarative parsing engine
│   ├── fetcher.py             # PoliteFetcher HTTP client with SSRF verification
│   ├── protocol.py            # Pydantic schema matching the bangwen/v1 protocol
│   └── security.py            # URL validation and private IP guards
├── tests/                     # Security and parsing regression tests
├── config.toml                # Active collection and scheduler configuration
├── runner.py                  # Main scheduler and multi-domain orchestrator
└── Dockerfile                 # Dedicated non-root container definition

---

🚀 Quickstart

Prerequisites

• Python 3.12+
• uv package manager

Local Setup

1. Install dependencies:

   uv sync
   

2. List available collectors:

   uv run python runner.py --list
   

3. Run a single crawl cycle:

   uv run python runner.py --once
   

---

🛠️ Configuration & Source Authoring

Declarative TOML Sources

Place zero-code TOML configuration cards under sources/{province}/{site_id}.toml:

[site]
id = "yn.yuxi"
name = "玉溪市人民政府"
province = "云南省"
city = "玉溪市"
employer = "玉溪市相关用人单位"
employer_type = "事业单位"
allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]

[template]
item_selector = "div.news-list a"
url_pattern = "/article/\\d+\\.html"
title_selector = "h1.article-title"
body_selector = "div.article-content"
date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})"
attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]

[[channels]]
id = "rszk"
name = "人事招考"
start_url = "https://www.yuxi.gov.cn/rszk.html"
keywords = ["招聘", "考试", "录用", "公示"]

Custom Python Adapters

When target sites require complex session handling, cryptographic requests, or specific payload extraction, create a Python scraper under adapters/{province}/{name}.py:

• Implement a crawl function returning an IngestPayload.
• Use common.security.is_safe_url to enforce domain isolation.
• Sanitize extracted text with common.content.sanitize_body_text.

---

⚙️ Scheduling & Rate Limiting

The runner (runner.py) provides fine-grained scheduling controls:

Environment Variable / Flag	Default	Description
CORE_INGEST_URL	http://core:8080/api/v1/ingest	Target Rust gateway ingestion endpoint
INGEST_TOKEN	None	Bearer authentication token matching core .env
SCHEDULE_INTERVAL_HOURS	6	Cycle interval between automated sweeps
--provinces	[]	Filter target provinces (e.g. yn,sc)
--concurrency	3	Maximum concurrent distinct domain scrapers

---

🧪 Testing

Comprehensive test suites covering SSRF prevention, HTML sanitization, and URL security:

uv run python tests/test_content_and_security.py
# Expected output: All security and content tests passed!

---

📜 License

MIT

(back to top)