docs: add bilingual README documentation for core and collectors
- Add English README.md and Chinese README.zh-CN.md for project root - Enhance collectors/README.md and add Chinese collectors/README.zh-CN.md - Document architecture, deployment, security invariants, and CLI/API usage
This commit is contained in:
bitfennec
+140
-14
@@ -1,25 +1,151 @@
|
||||
# Bangwen Collectors
|
||||
<a id="readme-top"></a>
|
||||
|
||||
Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.
|
||||
# bangwen-collectors 🕷️
|
||||
|
||||
## Structure
|
||||
> **Lightweight, declarative, and SSRF-hardened edge collection system for public job recruitment aggregation.**
|
||||
|
||||
- `sources/`: Declarative TOML configuration for crawling target sites.
|
||||
- `adapters/`: Custom Python scrapers for complex or stateful collection logic.
|
||||
- `common/`: Core fetcher, security validators, and ingestion protocol models.
|
||||
- `runner.py`: Periodic crawler runner and domain-concurrency scheduler.
|
||||
---
|
||||
|
||||
## Running
|
||||
## 📖 Table of Contents
|
||||
- [✨ Overview](#-overview)
|
||||
- [📁 Project Structure](#-project-structure)
|
||||
- [🚀 Quickstart](#-quickstart)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Local Setup](#local-setup)
|
||||
- [🛠️ Configuration & Source Authoring](#️-configuration--source-authoring)
|
||||
- [Declarative TOML Sources](#declarative-toml-sources)
|
||||
- [Custom Python Adapters](#custom-python-adapters)
|
||||
- [⚙️ Scheduling & Rate Limiting](#️-scheduling--rate-limiting)
|
||||
- [🧪 Testing](#-testing)
|
||||
- [📜 License](#-license)
|
||||
|
||||
```bash
|
||||
uv run python runner.py --list
|
||||
uv run python runner.py --once
|
||||
---
|
||||
|
||||
## ✨ Overview
|
||||
|
||||
`bangwen-collectors` serves as the decentralized edge data extraction layer for the `bangwen` recruitment aggregation engine:
|
||||
- **Dual-Mode Crawling**: Supports both zero-code declarative TOML site cards (`sources/`) and customized Python scraper adapters (`adapters/`).
|
||||
- **Domain-Grouped Concurrency**: Concurrently processes distinct domains while enforcing strictly serial queues within the same domain.
|
||||
- **WAF-Friendly Polite Delays**: Configurable inter-channel pauses (default 3.0s) and detail extraction throttles (0.3s) preventing IP bans.
|
||||
- **SSRF Defense-in-Depth**: Built-in private IP filtering, strict domain whitelists, and hop-by-hop verification during HTTP redirects.
|
||||
|
||||
---
|
||||
|
||||
## 📁 Project Structure
|
||||
|
||||
```
|
||||
collectors/
|
||||
├── adapters/ # Specialized scrapers for complex dynamic sites
|
||||
│ └── sc/chengdu_rsks.py # Custom adapter for Chengdu Personnel Exam Net
|
||||
├── sources/ # Zero-code declarative TOML site cards
|
||||
│ ├── sc/ # Sichuan Province sources (e.g. myrsks, zgrsks)
|
||||
│ └── yn/ # Yunnan Province sources (e.g. hrss, yuxi)
|
||||
├── common/ # Core modules
|
||||
│ ├── content.py # Content truncation and dirty body cleanup
|
||||
│ ├── declarative.py # Universal declarative parsing engine
|
||||
│ ├── fetcher.py # PoliteFetcher HTTP client with SSRF verification
|
||||
│ ├── protocol.py # Pydantic schema matching the bangwen/v1 protocol
|
||||
│ └── security.py # URL validation and private IP guards
|
||||
├── tests/ # Security and parsing regression tests
|
||||
├── config.toml # Active collection and scheduler configuration
|
||||
├── runner.py # Main scheduler and multi-domain orchestrator
|
||||
└── Dockerfile # Dedicated non-root container definition
|
||||
```
|
||||
|
||||
## Testing
|
||||
---
|
||||
|
||||
Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:
|
||||
## 🚀 Quickstart
|
||||
|
||||
### Prerequisites
|
||||
- Python 3.12+
|
||||
- [uv](https://docs.astral.sh/uv/) package manager
|
||||
|
||||
### Local Setup
|
||||
|
||||
1. **Install dependencies**:
|
||||
```bash
|
||||
uv sync
|
||||
```
|
||||
|
||||
2. **List available collectors**:
|
||||
```bash
|
||||
uv run python runner.py --list
|
||||
```
|
||||
|
||||
3. **Run a single crawl cycle**:
|
||||
```bash
|
||||
uv run python runner.py --once
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Configuration & Source Authoring
|
||||
|
||||
### Declarative TOML Sources
|
||||
|
||||
Place zero-code TOML configuration cards under `sources/{province}/{site_id}.toml`:
|
||||
|
||||
```toml
|
||||
[site]
|
||||
id = "yn.yuxi"
|
||||
name = "玉溪市人民政府"
|
||||
province = "云南省"
|
||||
city = "玉溪市"
|
||||
employer = "玉溪市相关用人单位"
|
||||
employer_type = "事业单位"
|
||||
allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]
|
||||
|
||||
[template]
|
||||
item_selector = "div.news-list a"
|
||||
url_pattern = "/article/\\d+\\.html"
|
||||
title_selector = "h1.article-title"
|
||||
body_selector = "div.article-content"
|
||||
date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})"
|
||||
attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]
|
||||
|
||||
[[channels]]
|
||||
id = "rszk"
|
||||
name = "人事招考"
|
||||
start_url = "https://www.yuxi.gov.cn/rszk.html"
|
||||
keywords = ["招聘", "考试", "录用", "公示"]
|
||||
```
|
||||
|
||||
### Custom Python Adapters
|
||||
|
||||
When target sites require complex session handling, cryptographic requests, or specific payload extraction, create a Python scraper under `adapters/{province}/{name}.py`:
|
||||
- Implement a crawl function returning an `IngestPayload`.
|
||||
- Use `common.security.is_safe_url` to enforce domain isolation.
|
||||
- Sanitize extracted text with `common.content.sanitize_body_text`.
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Scheduling & Rate Limiting
|
||||
|
||||
The runner (`runner.py`) provides fine-grained scheduling controls:
|
||||
|
||||
| Environment Variable / Flag | Default | Description |
|
||||
|-----------------------------|---------|-------------|
|
||||
| `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Target Rust gateway ingestion endpoint |
|
||||
| `INGEST_TOKEN` | *None* | Bearer authentication token matching core `.env` |
|
||||
| `SCHEDULE_INTERVAL_HOURS` | `6` | Cycle interval between automated sweeps |
|
||||
| `--provinces` | `[]` | Filter target provinces (e.g. `yn,sc`) |
|
||||
| `--concurrency` | `3` | Maximum concurrent distinct domain scrapers |
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
Comprehensive test suites covering SSRF prevention, HTML sanitization, and URL security:
|
||||
|
||||
```bash
|
||||
cd collectors && uv run python tests/test_content_and_security.py
|
||||
uv run python tests/test_content_and_security.py
|
||||
# Expected output: All security and content tests passed!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📜 License
|
||||
|
||||
[MIT](../LICENSE)
|
||||
|
||||
<p align="right">(<a href="#readme-top">back to top</a>)</p>
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
<a id="readme-top"></a>
|
||||
|
||||
# bangwen-collectors 边缘采集系统 🕷️
|
||||
|
||||
> **专为招考数据聚合设计的高性能、免代码声明式与 SSRF 安全强化的边缘抓取体系。**
|
||||
|
||||
---
|
||||
|
||||
## 📖 目录
|
||||
- [✨ 概述](#-概述)
|
||||
- [📁 目录结构](#-目录结构)
|
||||
- [🚀 快速开始](#-快速开始)
|
||||
- [环境准备](#环境准备)
|
||||
- [本地安装](#本地安装)
|
||||
- [🛠️ 采集器配置与开发指南](#️-采集器配置与开发指南)
|
||||
- [免代码声明式 TOML 站点卡片](#免代码声明式-toml-站点卡片)
|
||||
- [复杂场景 Python 定制适配器](#复杂场景-python-定制适配器)
|
||||
- [⚙️ 调度策略与频控机制](#️-调度策略与频控机制)
|
||||
- [🧪 测试与验证](#-测试与验证)
|
||||
- [📜 开源协议](#-开源协议)
|
||||
|
||||
---
|
||||
|
||||
## ✨ 概述
|
||||
|
||||
`bangwen-collectors` 作为 `bangwen` 招考数据中枢的分布式边缘抽取层：
|
||||
- **双模自动发现**：同时支持零代码声明式 TOML 站点配置卡片（`sources/`）与 Python 定制适配器（`adapters/`）。
|
||||
- **按域名分组并发**：对不同域名并发抓取，而对相同域名严格排队串行，保障采集效率的同时彻底杜绝政务网 WAF 封锁。
|
||||
- **礼貌延时与节流**：栏目间保持静默等待（默认 3.0s），单篇详情提取保留 0.3s 间隔微节流。
|
||||
- **全链路 SSRF 纵深防护**：底层内置私有内网 IP 阻断、严格域名白名单匹配与 HTTP 重定向逐跳安全性校验。
|
||||
|
||||
---
|
||||
|
||||
## 📁 目录结构
|
||||
|
||||
```
|
||||
collectors/
|
||||
├── adapters/ # 针对复杂动态站点的 Python 专属抓取适配器
|
||||
│ └── sc/chengdu_rsks.py # 成都人事考试网抓取适配器
|
||||
├── sources/ # 零代码声明式 TOML 站点配置卡片
|
||||
│ ├── sc/ # 四川省站点（如绵阳人事考试网、自贡人事考试网）
|
||||
│ └── yn/ # 云南省站点（如云南人社厅、玉溪市人民政府）
|
||||
├── common/ # 核心通用组件
|
||||
│ ├── content.py # 脏正文清洗、超长文本截断与反爬标记识别
|
||||
│ ├── declarative.py # 通用免代码声明式解析执行引擎
|
||||
│ ├── fetcher.py # 支持 SSRF 逐跳校验与限速的 PoliteFetcher 客户端
|
||||
│ ├── protocol.py # 与 bangwen/v1 摄入协议严格对齐的 Pydantic 模型
|
||||
│ └── security.py # URL 白名单判定与内网 IP 阻断工具
|
||||
├── tests/ # 安全防护与内容清洗回归测试集
|
||||
├── config.toml # 采集调度与省份过滤总配置文件
|
||||
├── runner.py # 多域名调度器与常驻运行入口
|
||||
└── Dockerfile # 专属非 root 用户（UID 10001）容器镜像定义
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 环境准备
|
||||
- Python 3.12+
|
||||
- [uv](https://docs.astral.sh/uv/) 包管理器
|
||||
|
||||
### 本地安装
|
||||
|
||||
1. **安装依赖**：
|
||||
```bash
|
||||
uv sync
|
||||
```
|
||||
|
||||
2. **列出当前注册的所有采集渠道**：
|
||||
```bash
|
||||
uv run python runner.py --list
|
||||
```
|
||||
|
||||
3. **执行一轮即时抓取与推流**：
|
||||
```bash
|
||||
uv run python runner.py --once
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ 采集器配置与开发指南
|
||||
|
||||
### 免代码声明式 TOML 站点卡片
|
||||
|
||||
对于标准 HTML 结构的政务招考网站，只需在 `sources/{province}/{site_id}.toml` 编写配置，无需编写任何代码：
|
||||
|
||||
```toml
|
||||
[site]
|
||||
id = "yn.yuxi"
|
||||
name = "玉溪市人民政府"
|
||||
province = "云南省"
|
||||
city = "玉溪市"
|
||||
employer = "玉溪市相关用人单位"
|
||||
employer_type = "事业单位"
|
||||
allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]
|
||||
|
||||
[template]
|
||||
item_selector = "div.news-list a"
|
||||
url_pattern = "/article/\\d+\\.html"
|
||||
title_selector = "h1.article-title"
|
||||
body_selector = "div.article-content"
|
||||
date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})"
|
||||
attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]
|
||||
|
||||
[[channels]]
|
||||
id = "rszk"
|
||||
name = "人事招考"
|
||||
start_url = "https://www.yuxi.gov.cn/rszk.html"
|
||||
keywords = ["招聘", "考试", "录用", "公示"]
|
||||
```
|
||||
|
||||
### 复杂场景 Python 定制适配器
|
||||
|
||||
对于含有复杂会话保持、加密参数或动态分页的站点，在 `adapters/{province}/{name}.py` 编写 Python 适配器：
|
||||
- 实现抓取函数并返回符合标准的 `IngestPayload`。
|
||||
- 强制使用 `common.security.is_safe_url` 实施外链与重定向安全隔离。
|
||||
- 提取正文时使用 `common.content.sanitize_body_text` 剥离导航与脏文本。
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ 调度策略与频控机制
|
||||
|
||||
调度器（`runner.py`）提供细粒度的运行控制：
|
||||
|
||||
| 环境变量 / 命令行参数 | 默认值 | 说明 |
|
||||
|-----------------------|--------|------|
|
||||
| `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Rust 核心网关的摄入接口地址 |
|
||||
| `INGEST_TOKEN` | *空* | 与内核对齐的 HTTP Bearer 认证 Token |
|
||||
| `SCHEDULE_INTERVAL_HOURS` | `6` | 常驻巡检的周期时间（小时） |
|
||||
| `--provinces` | `[]` | 目标省份过滤（例如 `yn,sc`） |
|
||||
| `--concurrency` | `3` | 跨域名的最大并发数 |
|
||||
|
||||
---
|
||||
|
||||
## 🧪 测试与验证
|
||||
|
||||
运行完备的本地安全回归套件（涵盖 SSRF 私网阻断、反爬脏正文过滤与附件扩展名判定）：
|
||||
|
||||
```bash
|
||||
uv run python tests/test_content_and_security.py
|
||||
# 预期输出: All security and content tests passed!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📜 开源协议
|
||||
|
||||
本项目采用 [MIT](../LICENSE) 开源协议。
|
||||
|
||||
<p align="right">(<a href="#readme-top">回到顶部</a>)</p>
|
||||
Reference in New Issue
Block a user