bitfennec
5a8af5729c
- Add English README.md and Chinese README.zh-CN.md for project root - Enhance collectors/README.md and add Chinese collectors/README.zh-CN.md - Document architecture, deployment, security invariants, and CLI/API usage
152 lines
5.1 KiB
Markdown
152 lines
5.1 KiB
Markdown
<a id="readme-top"></a>
|
||
|
||
# bangwen-collectors 🕷️
|
||
|
||
> **Lightweight, declarative, and SSRF-hardened edge collection system for public job recruitment aggregation.**
|
||
|
||
---
|
||
|
||
## 📖 Table of Contents
|
||
- [✨ Overview](#-overview)
|
||
- [📁 Project Structure](#-project-structure)
|
||
- [🚀 Quickstart](#-quickstart)
|
||
- [Prerequisites](#prerequisites)
|
||
- [Local Setup](#local-setup)
|
||
- [🛠️ Configuration & Source Authoring](#️-configuration--source-authoring)
|
||
- [Declarative TOML Sources](#declarative-toml-sources)
|
||
- [Custom Python Adapters](#custom-python-adapters)
|
||
- [⚙️ Scheduling & Rate Limiting](#️-scheduling--rate-limiting)
|
||
- [🧪 Testing](#-testing)
|
||
- [📜 License](#-license)
|
||
|
||
---
|
||
|
||
## ✨ Overview
|
||
|
||
`bangwen-collectors` serves as the decentralized edge data extraction layer for the `bangwen` recruitment aggregation engine:
|
||
- **Dual-Mode Crawling**: Supports both zero-code declarative TOML site cards (`sources/`) and customized Python scraper adapters (`adapters/`).
|
||
- **Domain-Grouped Concurrency**: Concurrently processes distinct domains while enforcing strictly serial queues within the same domain.
|
||
- **WAF-Friendly Polite Delays**: Configurable inter-channel pauses (default 3.0s) and detail extraction throttles (0.3s) preventing IP bans.
|
||
- **SSRF Defense-in-Depth**: Built-in private IP filtering, strict domain whitelists, and hop-by-hop verification during HTTP redirects.
|
||
|
||
---
|
||
|
||
## 📁 Project Structure
|
||
|
||
```
|
||
collectors/
|
||
├── adapters/ # Specialized scrapers for complex dynamic sites
|
||
│ └── sc/chengdu_rsks.py # Custom adapter for Chengdu Personnel Exam Net
|
||
├── sources/ # Zero-code declarative TOML site cards
|
||
│ ├── sc/ # Sichuan Province sources (e.g. myrsks, zgrsks)
|
||
│ └── yn/ # Yunnan Province sources (e.g. hrss, yuxi)
|
||
├── common/ # Core modules
|
||
│ ├── content.py # Content truncation and dirty body cleanup
|
||
│ ├── declarative.py # Universal declarative parsing engine
|
||
│ ├── fetcher.py # PoliteFetcher HTTP client with SSRF verification
|
||
│ ├── protocol.py # Pydantic schema matching the bangwen/v1 protocol
|
||
│ └── security.py # URL validation and private IP guards
|
||
├── tests/ # Security and parsing regression tests
|
||
├── config.toml # Active collection and scheduler configuration
|
||
├── runner.py # Main scheduler and multi-domain orchestrator
|
||
└── Dockerfile # Dedicated non-root container definition
|
||
```
|
||
|
||
---
|
||
|
||
## 🚀 Quickstart
|
||
|
||
### Prerequisites
|
||
- Python 3.12+
|
||
- [uv](https://docs.astral.sh/uv/) package manager
|
||
|
||
### Local Setup
|
||
|
||
1. **Install dependencies**:
|
||
```bash
|
||
uv sync
|
||
```
|
||
|
||
2. **List available collectors**:
|
||
```bash
|
||
uv run python runner.py --list
|
||
```
|
||
|
||
3. **Run a single crawl cycle**:
|
||
```bash
|
||
uv run python runner.py --once
|
||
```
|
||
|
||
---
|
||
|
||
## 🛠️ Configuration & Source Authoring
|
||
|
||
### Declarative TOML Sources
|
||
|
||
Place zero-code TOML configuration cards under `sources/{province}/{site_id}.toml`:
|
||
|
||
```toml
|
||
[site]
|
||
id = "yn.yuxi"
|
||
name = "玉溪市人民政府"
|
||
province = "云南省"
|
||
city = "玉溪市"
|
||
employer = "玉溪市相关用人单位"
|
||
employer_type = "事业单位"
|
||
allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]
|
||
|
||
[template]
|
||
item_selector = "div.news-list a"
|
||
url_pattern = "/article/\\d+\\.html"
|
||
title_selector = "h1.article-title"
|
||
body_selector = "div.article-content"
|
||
date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})"
|
||
attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]
|
||
|
||
[[channels]]
|
||
id = "rszk"
|
||
name = "人事招考"
|
||
start_url = "https://www.yuxi.gov.cn/rszk.html"
|
||
keywords = ["招聘", "考试", "录用", "公示"]
|
||
```
|
||
|
||
### Custom Python Adapters
|
||
|
||
When target sites require complex session handling, cryptographic requests, or specific payload extraction, create a Python scraper under `adapters/{province}/{name}.py`:
|
||
- Implement a crawl function returning an `IngestPayload`.
|
||
- Use `common.security.is_safe_url` to enforce domain isolation.
|
||
- Sanitize extracted text with `common.content.sanitize_body_text`.
|
||
|
||
---
|
||
|
||
## ⚙️ Scheduling & Rate Limiting
|
||
|
||
The runner (`runner.py`) provides fine-grained scheduling controls:
|
||
|
||
| Environment Variable / Flag | Default | Description |
|
||
|-----------------------------|---------|-------------|
|
||
| `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Target Rust gateway ingestion endpoint |
|
||
| `INGEST_TOKEN` | *None* | Bearer authentication token matching core `.env` |
|
||
| `SCHEDULE_INTERVAL_HOURS` | `6` | Cycle interval between automated sweeps |
|
||
| `--provinces` | `[]` | Filter target provinces (e.g. `yn,sc`) |
|
||
| `--concurrency` | `3` | Maximum concurrent distinct domain scrapers |
|
||
|
||
---
|
||
|
||
## 🧪 Testing
|
||
|
||
Comprehensive test suites covering SSRF prevention, HTML sanitization, and URL security:
|
||
|
||
```bash
|
||
uv run python tests/test_content_and_security.py
|
||
# Expected output: All security and content tests passed!
|
||
```
|
||
|
||
---
|
||
|
||
## 📜 License
|
||
|
||
[MIT](../LICENSE)
|
||
|
||
<p align="right">(<a href="#readme-top">back to top</a>)</p>
|