docs: add bilingual README documentation for core and collectors

- Add English README.md and Chinese README.zh-CN.md for project root - Enhance collectors/README.md and add Chinese collectors/README.zh-CN.md - Document architecture, deployment, security invariants, and CLI/API usage
2026-10-24 00:00:50 +08:00
parent 1059be1b62
commit 5a8af5729c
4 changed files with 741 additions and 14 deletions
@@ -0,0 +1,225 @@
+<a id="readme-top"></a>
+
+# bangwen (榜文) 🚀
+
+> **High-performance public job recruitment aggregation, hybrid retrieval, and white-box resume matching engine designed with VictoriaMetrics architectural principles.**
+
+---
+
+## 📖 Table of Contents
+- [✨ Features](#-features)
+- [🏛️ Architecture](#️-architecture)
+- [🚀 Quickstart](#-quickstart)
+  - [Prerequisites](#prerequisites)
+  - [Installation & Deployment](#installation--deployment)
+  - [Verification](#verification)
+- [🛠️ Usage](#️-usage)
+  - [Web Console](#web-console)
+  - [CLI Commands](#cli-commands)
+  - [HTTP API Reference](#http-api-reference)
+- [⚙️ Configuration](#️-configuration)
+- [🔒 Security & Hardening](#-security--hardening)
+- [🤝 Contributing](#-contributing)
+- [📜 License](#-license)
+- [🤖 For AI Agents](#-for-ai-agents)
+
+---
+
+## ✨ Features
+
+- **⚡ VictoriaMetrics-Inspired Simplicity**: Single lightweight Rust binary acting as an ingestion gateway, hybrid search engine, and embedded web console without bloated microservices.
+- **🛡️ Multi-Tier Deduplication Pipeline**: Three-tier deduplication combining normalized URL canonicalization, 16-character content hashing, and 64-bit Hamming distance SimHash with short-text (<200 char) false-positive protection.
+- **🎯 White-Box Resume Matching**: Fully objective candidate scoring based on hard criteria (education levels mapped via standard dictionaries, age limits, region codes, and exact major/skill matches) with transparent audit reasons.
+- **🔍 Hybrid Search Engine**: Integrates exact SQL filtering, PostgreSQL `pg_trgm` GIN fuzzy matching, and `pgvector` HNSW cosine similarity with Reciprocal Rank Fusion (RRF).
+- **🕸️ Dual-Mode Edge Collectors**: Autonomous Python edge scrapers supporting declarative TOML site templates and specialized Python adapters with domain-level concurrency and polite throttles.
+- **🔒 Hardened Security Surface**:
+  - Strict Content Security Policy (`script-src 'self'` with zero `'unsafe-inline'`).
+  - Terminal sanitization stripping ANSI escape sequences, OSC 52, and control characters from all outputs.
+  - Markdown injection prevention and SSRF hop-by-hop redirect verification.
+  - Dedicated non-root container user execution (`uid=10001`).
+
+---
+
+## 🏛️ Architecture
+
+```
+                      +---------------------------------------+
+                      |       Edge Public Recruitment Sites    |
+                      +---------------------------------------+
+                                          |
+                      (HTTP Scraping & Polite Rate Limiting)
+                                          v
+                      +---------------------------------------+
+                      |      bangwen-collector (Python)       |
+                      |  - Declarative TOML (sources/)        |
+                      |  - Custom Adapters (adapters/)        |
+                      |  - SSRF-safe PoliteFetcher            |
+                      +---------------------------------------+
+                                          |
+                              (HTTP POST /api/v1/ingest)
+                           Bearer Token Authenticated
+                                          v
+                      +---------------------------------------+
+                      |          bangwen-core (Rust)          |
+                      |  - SimHash & URL Hash Deduplication   |
+                      |  - Hybrid Search & White-box Matcher  |
+                      |  - Embedded Web Console & Assets      |
+                      +---------------------------------------+
+                                          |
+                               (SQL & Vector Storage)
+                                          v
+                      +---------------------------------------+
+                      |         PostgreSQL 16 + pgvector      |
+                      |  - announcement & position tables     |
+                      |  - pg_trgm & HNSW vector indexes      |
+                      +---------------------------------------+
+```
+
+---
+
+## 🚀 Quickstart
+
+### Prerequisites
+- [Docker](https://docs.docker.com/engine/install/) & [Docker Compose](https://docs.docker.com/compose/) v2+
+- Optional for native development:
+  - [Rust](https://rustup.rs/) (>= 1.80)
+  - [uv](https://docs.astral.sh/uv/) (>= 0.4.0) and Python 3.12+
+
+### Installation & Deployment
+
+1. **Clone the repository**:
+   ```bash
+   git clone [email protected]:bitfennec/bangwen.git
+   cd bangwen
+   ```
+
+2. **Initialize environment credentials**:
+   ```bash
+   cp .env.example .env
+   # Generate a secure 64-character token
+   sed -i "s/replace_with_your_secure_secret_token/$(openssl rand -hex 32)/" .env
+   ```
+
+3. **Start services with Docker Compose**:
+   ```bash
+   docker compose up -d --build
+   ```
+
+### Verification
+
+Check service health and initial ingestion metrics:
+```bash
+curl -s http://127.0.0.1:8080/health
+# Expected: {"status":"ok"}
+
+curl -s http://127.0.0.1:8080/api/v1/stats
+# Expected: {"announcements":...,"embeddings":0,"positions":...,"sources":6,"status":"ok","vector_ready":false}
+```
+
+Open `http://127.0.0.1:8080/` in your browser to inspect the Web Console.
+
+---
+
+## 🛠️ Usage
+
+### Web Console
+The built-in web console is served directly on `http://127.0.0.1:8080/`:
+- **Feed Stream**: Paginated real-time recruitment notices with collapsible bodies, parsed position tables, and official attachment links.
+- **Multi-Filter Search**: Search by keywords, administrative region code, employer type, and minimum education requirements.
+- **Candidate White-Box Match**: Fill candidate resume attributes to calculate match scores and view clear transparent eligibility lists.
+- **Daily Digest**: View generated markdown summaries of the past 24 hours.
+- **Collector Monitor**: Health, authentication status, and last report timestamp for all edge collectors.
+
+### CLI Commands
+
+When running natively or using the provided wrapper `./bin/bangwen`:
+
+```bash
+# Ingest data via stdin or file
+./bin/bangwen ingest --file ./sample.json
+
+# Hybrid search recruitment posts
+./bin/bangwen search "运维" --region 530400 --json
+
+# Inspect notice or position details
+./bin/bangwen show ann:1 --json
+./bin/bangwen show pos:3 --json
+
+# Run white-box resume matching
+./bin/bangwen match --resume ./tests/fixtures/sample_resume.json --top 10
+
+# Generate daily markdown digest
+./bin/bangwen digest
+
+# Inspect collection sources health
+./bin/bangwen sources
+```
+
+### HTTP API Reference
+
+| Endpoint | Method | Description | Auth Required |
+|----------|--------|-------------|---------------|
+| `/health` | `GET` | Service & database health check | No |
+| `/api/v1/ingest` | `POST` | Ingest batched notices & positions | Bearer Token |
+| `/api/v1/stats` | `GET` | Aggregation statistics and vector status | No |
+| `/api/v1/sources` | `GET` | Source nodes health and last ingest report | No |
+| `/api/v1/announcements` | `GET` | Paginated recruitment announcements list | No |
+| `/api/v1/announcements/{id}` | `GET` | Detailed announcement with positions & attachments | No |
+| `/api/v1/positions/{id}` | `GET` | Detailed structured job position | No |
+| `/api/v1/search` | `GET` | Multi-dimensional hybrid keyword search | No |
+| `/api/v1/match` | `POST` | Strict white-box resume qualification match | No |
+| `/api/v1/digest` | `GET` | Daily markdown recruitment digest | No |
+
+---
+
+## ⚙️ Configuration
+
+Environment variables configured in `.env`:
+
+| Variable | Default | Required | Description |
+|----------|---------|----------|-------------|
+| `DATABASE_URL` | `postgres://bangwen:bangwen_dev@postgres:5432/bangwen` | Yes | PostgreSQL connection string with pgvector |
+| `INGEST_TOKEN` | *None* | Recommended | Bearer token for authenticating `/api/v1/ingest` |
+| `SERVER_HOST` | `0.0.0.0` | No | Host address for Rust HTTP gateway |
+| `SERVER_PORT` | `8080` | No | Port for Rust HTTP gateway |
+| `LOG_LEVEL` | `info` | No | Tracing log level (`error`, `warn`, `info`, `debug`) |
+| `OLLAMA_BASE_URL` | `http://host.docker.internal:11434` | No | Optional Ollama instance for embedding generation |
+| `EMBEDDING_MODEL` | `bge-m3` | No | Ollama embedding model name |
+
+---
+
+## 🔒 Security & Hardening
+
+- **Non-Root Execution**: Both `bangwen-core` and `bangwen-collector` containers enforce explicit non-root user execution (`UID 10001`).
+- **Strict Content Security Policy (CSP)**: `default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; frame-ancestors 'none';`. Inline scripts are completely eliminated.
+- **Input Boundary Bounds**: Bounded validation for match requests (max 50 skills, 20 cities, 32-char education, 1940..=current birth year) and pagination limits (max offset 10000).
+- **Anti-Deadlock Lock Ordering**: Ingestion pipeline enforces deterministic URL key sorting prior to row lock acquisition, eliminating PostgreSQL `40P01` deadlock conditions.
+- **SSRF & Scheme Verification**: PoliteFetcher enforces private IP filtering, strict domain whitelists, and hop-by-hop validation during HTTP redirects.
+
+---
+
+## 🤝 Contributing
+
+Contributions are welcome. Please ensure all modifications pass local validation:
+```bash
+# Rust core checks
+cargo test --offline
+cargo clippy --all-targets -- -D warnings
+
+# Collectors checks
+cd collectors && uv run python tests/test_content_and_security.py
+```
+
+---
+
+## 📜 License
+
+[MIT](LICENSE)
+
+---
+
+## 🤖 For AI Agents
+
+See [`skill/SKILL.md`](./skill/SKILL.md) for operational instructions and security boundary guidelines.
+
+<p align="right">(<a href="#readme-top">back to top</a>)</p>
@@ -0,0 +1,225 @@
+<a id="readme-top"></a>
+
+# 榜文 (bangwen) 🚀
+
+> **基于 VictoriaMetrics 极简架构哲学构建的高性能公开招考数据聚合、混合检索与白盒简历匹配引擎。**
+
+---
+
+## 📖 目录
+- [✨ 核心特性](#-核心特性)
+- [🏛️ 系统架构](#️-系统架构)
+- [🚀 快速开始](#-快速开始)
+  - [环境准备](#环境准备)
+  - [安装与部署](#安装与部署)
+  - [验证运行](#验证运行)
+- [🛠️ 使用指南](#️-使用指南)
+  - [Web 控制台](#web-控制台)
+  - [CLI 命令行工具](#cli-命令行工具)
+  - [HTTP API 接口参考](#http-api-接口参考)
+- [⚙️ 环境配置](#️-环境配置)
+- [🔒 安全加固说明](#-安全加固说明)
+- [🤝 参与贡献](#-参与贡献)
+- [📜 开源协议](#-开源协议)
+- [🤖 AI Agent 集成](#-ai-agent-集成)
+
+---
+
+## ✨ 核心特性
+
+- **⚡ 对标 VictoriaMetrics 架构哲学**：单微核 Rust 二进制文件同时承载 HTTP 摄入网关、混合检索核心与内嵌 Web 控制台，杜绝过度微服务化与复杂外部依赖。
+- **🛡️ 多级排重与数据治理管线**：融合规范化 URL 权威归一、16 字符内容哈希校验、64 位汉明距离 SimHash 以及针对短文本（<200 字符）的防误杀二次核验机制。
+- **🎯 客观透明的白盒简历匹配**：基于法定与招聘硬性门槛（GB/T 2260 行政区划字典、标准学历档次映射、年龄区间与专业/技能对齐）进行确定性白盒匹配，直接输出透明合规与不符事实清单。
+- **🔍 多路混合检索与 RRF 融合**：深度集成 SQL 精确过滤、PostgreSQL `pg_trgm` GIN 模糊索引以及 `pgvector` HNSW 向量余弦距离检索，通过 RRF（Reciprocal Rank Fusion）算法兼顾高召回与高精度。
+- **🕸️ 边缘采集双模调度引擎**：Python 边缘爬虫体系，同时支持轻量级声明式 TOML 配置卡片与复杂场景定制 Python 适配器，按域名分组并发调度并执行站内串行礼貌降频。
+- **🔒 全面收敛的安全防护基线**：
+  - 严格的内容安全策略（CSP `script-src 'self'`，彻底移除 `'unsafe-inline'`）。
+  - 终端控制序列无死角清洗（剥离 ANSI CSI、OSC 52 剪贴板指令与控制字符）。
+  - Markdown URL 注入拦截与爬虫逐跳重定向 SSRF 白名单校验。
+  - 容器服务全量以非 root 专属系统用户（`UID 10001`）隔离执行。
+
+---
+
+## 🏛️ 系统架构
+
+```
+                      +---------------------------------------+
+                      |          各省市公开招考招聘政务网      |
+                      +---------------------------------------+
+                                          |
+                      (HTTP 抓取与域名礼貌限速调度)
+                                          v
+                      +---------------------------------------+
+                      |      bangwen-collector (Python)       |
+                      |  - 声明式 TOML 站点卡片 (sources/)    |
+                      |  - 定制化抓取适配器 (adapters/)       |
+                      |  - SSRF 防护型 PoliteFetcher 客户端   |
+                      +---------------------------------------+
+                                          |
+                              (HTTP POST /api/v1/ingest)
+                               Bearer Token 鉴权通信
+                                          v
+                      +---------------------------------------+
+                      |          bangwen-core (Rust)          |
+                      |  - SimHash & 内容哈希多级排重管线     |
+                      |  - 多路混合检索与白盒简历打分内核     |
+                      |  - 纯原生内嵌 Web 控制台与静态资源    |
+                      +---------------------------------------+
+                                          |
+                               (SQL 与向量存储)
+                                          v
+                      +---------------------------------------+
+                      |         PostgreSQL 16 + pgvector      |
+                      |  - announcement 与 position 数据基表  |
+                      |  - pg_trgm 倒排与 HNSW 向量近似索引   |
+                      +---------------------------------------+
+```
+
+---
+
+## 🚀 快速开始
+
+### 环境准备
+- [Docker](https://docs.docker.com/engine/install/) 与 [Docker Compose](https://docs.docker.com/compose/) v2+
+- 本地原生开发环境（可选）：
+  - [Rust](https://rustup.rs/) (>= 1.80)
+  - [uv](https://docs.astral.sh/uv/) (>= 0.4.0) 与 Python 3.12+
+
+### 安装与部署
+
+1. **克隆代码仓库**：
+   ```bash
+   git clone [email protected]:bitfennec/bangwen.git
+   cd bangwen
+   ```
+
+2. **初始化环境与鉴权凭据**：
+   ```bash
+   cp .env.example .env
+   # 自动生成 64 位高强度随机鉴权 Token
+   sed -i "s/replace_with_your_secure_secret_token/$(openssl rand -hex 32)/" .env
+   ```
+
+3. **使用 Docker Compose 一键构建启动**：
+   ```bash
+   docker compose up -d --build
+   ```
+
+### 验证运行
+
+检查服务运行状态与初始入库指标：
+```bash
+curl -s http://127.0.0.1:8080/health
+# 预期输出: {"status":"ok"}
+
+curl -s http://127.0.0.1:8080/api/v1/stats
+# 预期输出: {"announcements":...,"embeddings":0,"positions":...,"sources":6,"status":"ok","vector_ready":false}
+```
+
+在浏览器中打开 `http://127.0.0.1:8080/` 即可直接进入内嵌 Web 控制台。
+
+---
+
+## 🛠️ 使用指南
+
+### Web 控制台
+服务启动后直接监听 `http://127.0.0.1:8080/`：
+- **公告 Feed 流**：按时间倒序分页展示各省市招考公告，支持全文实时一键折叠/展开、挂载结构化职位列表与官方附件直接下载。
+- **多维综合检索**：支持按关键词、标准行政区划编码、编制用人性质及最低学历门槛组合筛选。
+- **白盒简历匹配**：输入候选人基础画像，系统秒级匹配合规在招岗位，透明标明硬性满足与不符条款。
+- **每日聚合日报**：一键生成与复制过去 24 小时收录的招考精选与即将截止报名清单（Markdown 格式）。
+- **采集节点监控**：实时查看各边缘节点的连通健康度、认证态与最近上报统计。
+
+### CLI 命令行工具
+
+通过宿主机自带的原生穿透脚本 `./bin/bangwen` 或原生编译二进制进行操作：
+
+```bash
+# 从文件或 stdin 管道摄入数据
+./bin/bangwen ingest --file ./sample.json
+
+# 多维检索符合条件的岗位 (例如玉溪 530400)
+./bin/bangwen search "运维" --region 530400 --json
+
+# 查看公告或具体岗位详情
+./bin/bangwen show ann:1 --json
+./bin/bangwen show pos:3 --json
+
+# 根据候选人画像执行白盒打分匹配
+./bin/bangwen match --resume ./tests/fixtures/sample_resume.json --top 10
+
+# 输出今日招考聚合日报
+./bin/bangwen digest
+
+# 检查各边缘采集源的健康状态
+./bin/bangwen sources
+```
+
+### HTTP API 接口参考
+
+| 端点路由 | 请求方式 | 功能简述 | 鉴权要求 |
+|----------|----------|----------|----------|
+| `/health` | `GET` | 核心服务与数据库连通性健康检查 | 无 |
+| `/api/v1/ingest` | `POST` | 批量数据推送摄入接口 | Bearer Token |
+| `/api/v1/stats` | `GET` | 系统汇总指标与向量就绪状态 | 无 |
+| `/api/v1/sources` | `GET` | 采集节点健康度与最近入库明细 | 无 |
+| `/api/v1/announcements` | `GET` | 公告分页列表（支持关键字与类型筛选） | 无 |
+| `/api/v1/announcements/{id}` | `GET` | 单篇公告详情（含下属岗位与附件） | 无 |
+| `/api/v1/positions/{id}` | `GET` | 细分结构化招考岗位详情 | 无 |
+| `/api/v1/search` | `GET` | 混合关键字多路检索接口 | 无 |
+| `/api/v1/match` | `POST` | 候选人简历白盒打分与核验接口 | 无 |
+| `/api/v1/digest` | `GET` | 今日招考聚合日报（Markdown 格式） | 无 |
+
+---
+
+## ⚙️ 环境配置
+
+在 `.env` 中配置的环境变量说明：
+
+| 变量名 | 默认值 | 是否必填 | 说明 |
+|--------|--------|----------|------|
+| `DATABASE_URL` | `postgres://bangwen:bangwen_dev@postgres:5432/bangwen` | 是 | PostgreSQL + pgvector 数据库连接串 |
+| `INGEST_TOKEN` | *空* | 推荐 | 用于保护 `/api/v1/ingest` 端点的 Bearer 鉴权密钥 |
+| `SERVER_HOST` | `0.0.0.0` | 否 | Rust HTTP 服务监听地址 |
+| `SERVER_PORT` | `8080` | 否 | Rust HTTP 服务监听端口 |
+| `LOG_LEVEL` | `info` | 否 | 日志级别（`error`, `warn`, `info`, `debug`） |
+| `OLLAMA_BASE_URL` | `http://host.docker.internal:11434` | 否 | 向量生成所用的 Ollama 服务地址 |
+| `EMBEDDING_MODEL` | `bge-m3` | 否 | Ollama 向量模型名称 |
+
+---
+
+## 🔒 安全加固说明
+
+- **非 Root 容器降权**：`bangwen-core` 与 `bangwen-collector` 容器镜像均配置专用非特权运行用户（`UID 10001`）。
+- **严格 CSP 策略**：响应头强制 `default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline';`，彻底根除内联脚本注入执行风险。
+- **接口算力与内存保护**：对简历匹配接口增加完备的参数边界防护（技能最大 50 项、城市最大 20 项、学历上限 32 字符、出生年份 1940~当年区间强校验），深度分页 offset 上限 10000 强校验拦截。
+- **并发摄入防死锁**：数据落库事务内强制按照规范化 URL 字典序申请行级排他锁，彻底根除 PostgreSQL `40P01` 死锁异常。
+- **SSRF 与重定向逐跳校验**：HTTP 抓取客户端内置私有内网 IP 过滤、严格域名白名单与逐跳重定向安全性验证。
+
+---
+
+## 🤝 参与贡献
+
+欢迎提交 Issue 与 Pull Request。提交改动前请确保通过本地基线检验：
+```bash
+# Rust 内核验证
+cargo test --offline
+cargo clippy --all-targets -- -D warnings
+
+# 采集端回归验证
+cd collectors && uv run python tests/test_content_and_security.py
+```
+
+---
+
+## 📜 开源协议
+
+本项目采用 [MIT](LICENSE) 开源协议。
+
+---
+
+## 🤖 AI Agent 集成
+
+针对作为 Agent 工具调用的集成场景，请参阅 [`skill/SKILL.md`](./skill/SKILL.md)，了解调用协议、结构化输出规范以及不可信数据的安全隔离要求。
+
+<p align="right">(<a href="#readme-top">回到顶部</a>)</p>
@@ -1,25 +1,151 @@
-# Bangwen Collectors
+<a id="readme-top"></a>

-Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.
+# bangwen-collectors 🕷️

-## Structure
+> **Lightweight, declarative, and SSRF-hardened edge collection system for public job recruitment aggregation.**

- `sources/`: Declarative TOML configuration for crawling target sites.
- `adapters/`: Custom Python scrapers for complex or stateful collection logic.
- `common/`: Core fetcher, security validators, and ingestion protocol models.
- `runner.py`: Periodic crawler runner and domain-concurrency scheduler.
+---

-## Running
+## 📖 Table of Contents
+- [✨ Overview](#-overview)
+- [📁 Project Structure](#-project-structure)
+- [🚀 Quickstart](#-quickstart)
+  - [Prerequisites](#prerequisites)
+  - [Local Setup](#local-setup)
+- [🛠️ Configuration & Source Authoring](#️-configuration--source-authoring)
+  - [Declarative TOML Sources](#declarative-toml-sources)
+  - [Custom Python Adapters](#custom-python-adapters)
+- [⚙️ Scheduling & Rate Limiting](#️-scheduling--rate-limiting)
+- [🧪 Testing](#-testing)
+- [📜 License](#-license)

+---
+
+## ✨ Overview
+
+`bangwen-collectors` serves as the decentralized edge data extraction layer for the `bangwen` recruitment aggregation engine:
+- **Dual-Mode Crawling**: Supports both zero-code declarative TOML site cards (`sources/`) and customized Python scraper adapters (`adapters/`).
+- **Domain-Grouped Concurrency**: Concurrently processes distinct domains while enforcing strictly serial queues within the same domain.
+- **WAF-Friendly Polite Delays**: Configurable inter-channel pauses (default 3.0s) and detail extraction throttles (0.3s) preventing IP bans.
+- **SSRF Defense-in-Depth**: Built-in private IP filtering, strict domain whitelists, and hop-by-hop verification during HTTP redirects.
+
+---
+
+## 📁 Project Structure
+
+```
+collectors/
+├── adapters/                  # Specialized scrapers for complex dynamic sites
+│   └── sc/chengdu_rsks.py    # Custom adapter for Chengdu Personnel Exam Net
+├── sources/                   # Zero-code declarative TOML site cards
+│   ├── sc/                    # Sichuan Province sources (e.g. myrsks, zgrsks)
+│   └── yn/                    # Yunnan Province sources (e.g. hrss, yuxi)
+├── common/                    # Core modules
+│   ├── content.py             # Content truncation and dirty body cleanup
+│   ├── declarative.py         # Universal declarative parsing engine
+│   ├── fetcher.py             # PoliteFetcher HTTP client with SSRF verification
+│   ├── protocol.py            # Pydantic schema matching the bangwen/v1 protocol
+│   └── security.py            # URL validation and private IP guards
+├── tests/                     # Security and parsing regression tests
+├── config.toml                # Active collection and scheduler configuration
+├── runner.py                  # Main scheduler and multi-domain orchestrator
+└── Dockerfile                 # Dedicated non-root container definition
+```
+
+---
+
+## 🚀 Quickstart
+
+### Prerequisites
+- Python 3.12+
+- [uv](https://docs.astral.sh/uv/) package manager
+
+### Local Setup
+
+1. **Install dependencies**:
+   ```bash
+   uv sync
+   ```
+
+2. **List available collectors**:
   ```bash
   uv run python runner.py --list
+   ```
+
+3. **Run a single crawl cycle**:
+   ```bash
   uv run python runner.py --once
   ```

-## Testing
+---

-Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:
+## 🛠️ Configuration & Source Authoring
+
+### Declarative TOML Sources
+
+Place zero-code TOML configuration cards under `sources/{province}/{site_id}.toml`:
+
+```toml
+[site]
+id = "yn.yuxi"
+name = "玉溪市人民政府"
+province = "云南省"
+city = "玉溪市"
+employer = "玉溪市相关用人单位"
+employer_type = "事业单位"
+allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]
+
+[template]
+item_selector = "div.news-list a"
+url_pattern = "/article/\\d+\\.html"
+title_selector = "h1.article-title"
+body_selector = "div.article-content"
+date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})"
+attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]
+
+[[channels]]
+id = "rszk"
+name = "人事招考"
+start_url = "https://www.yuxi.gov.cn/rszk.html"
+keywords = ["招聘", "考试", "录用", "公示"]
+```
+
+### Custom Python Adapters
+
+When target sites require complex session handling, cryptographic requests, or specific payload extraction, create a Python scraper under `adapters/{province}/{name}.py`:
+- Implement a crawl function returning an `IngestPayload`.
+- Use `common.security.is_safe_url` to enforce domain isolation.
+- Sanitize extracted text with `common.content.sanitize_body_text`.
+
+---
+
+## ⚙️ Scheduling & Rate Limiting
+
+The runner (`runner.py`) provides fine-grained scheduling controls:
+
+| Environment Variable / Flag | Default | Description |
+|-----------------------------|---------|-------------|
+| `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Target Rust gateway ingestion endpoint |
+| `INGEST_TOKEN` | *None* | Bearer authentication token matching core `.env` |
+| `SCHEDULE_INTERVAL_HOURS` | `6` | Cycle interval between automated sweeps |
+| `--provinces` | `[]` | Filter target provinces (e.g. `yn,sc`) |
+| `--concurrency` | `3` | Maximum concurrent distinct domain scrapers |
+
+---
+
+## 🧪 Testing
+
+Comprehensive test suites covering SSRF prevention, HTML sanitization, and URL security:

 ```bash
-cd collectors && uv run python tests/test_content_and_security.py
+uv run python tests/test_content_and_security.py
+# Expected output: All security and content tests passed!
 ```
+
+---
+
+## 📜 License
+
+[MIT](../LICENSE)
+
+<p align="right">(<a href="#readme-top">back to top</a>)</p>
@@ -0,0 +1,151 @@
+<a id="readme-top"></a>
+
+# bangwen-collectors 边缘采集系统 🕷️
+
+> **专为招考数据聚合设计的高性能、免代码声明式与 SSRF 安全强化的边缘抓取体系。**
+
+---
+
+## 📖 目录
+- [✨ 概述](#-概述)
+- [📁 目录结构](#-目录结构)
+- [🚀 快速开始](#-快速开始)
+  - [环境准备](#环境准备)
+  - [本地安装](#本地安装)
+- [🛠️ 采集器配置与开发指南](#️-采集器配置与开发指南)
+  - [免代码声明式 TOML 站点卡片](#免代码声明式-toml-站点卡片)
+  - [复杂场景 Python 定制适配器](#复杂场景-python-定制适配器)
+- [⚙️ 调度策略与频控机制](#️-调度策略与频控机制)
+- [🧪 测试与验证](#-测试与验证)
+- [📜 开源协议](#-开源协议)
+
+---
+
+## ✨ 概述
+
+`bangwen-collectors` 作为 `bangwen` 招考数据中枢的分布式边缘抽取层：
+- **双模自动发现**：同时支持零代码声明式 TOML 站点配置卡片（`sources/`）与 Python 定制适配器（`adapters/`）。
+- **按域名分组并发**：对不同域名并发抓取，而对相同域名严格排队串行，保障采集效率的同时彻底杜绝政务网 WAF 封锁。
+- **礼貌延时与节流**：栏目间保持静默等待（默认 3.0s），单篇详情提取保留 0.3s 间隔微节流。
+- **全链路 SSRF 纵深防护**：底层内置私有内网 IP 阻断、严格域名白名单匹配与 HTTP 重定向逐跳安全性校验。
+
+---
+
+## 📁 目录结构
+
+```
+collectors/
+├── adapters/                  # 针对复杂动态站点的 Python 专属抓取适配器
+│   └── sc/chengdu_rsks.py    # 成都人事考试网抓取适配器
+├── sources/                   # 零代码声明式 TOML 站点配置卡片
+│   ├── sc/                    # 四川省站点（如绵阳人事考试网、自贡人事考试网）
+│   └── yn/                    # 云南省站点（如云南人社厅、玉溪市人民政府）
+├── common/                    # 核心通用组件
+│   ├── content.py             # 脏正文清洗、超长文本截断与反爬标记识别
+│   ├── declarative.py         # 通用免代码声明式解析执行引擎
+│   ├── fetcher.py             # 支持 SSRF 逐跳校验与限速的 PoliteFetcher 客户端
+│   ├── protocol.py            # 与 bangwen/v1 摄入协议严格对齐的 Pydantic 模型
+│   └── security.py            # URL 白名单判定与内网 IP 阻断工具
+├── tests/                     # 安全防护与内容清洗回归测试集
+├── config.toml                # 采集调度与省份过滤总配置文件
+├── runner.py                  # 多域名调度器与常驻运行入口
+└── Dockerfile                 # 专属非 root 用户（UID 10001）容器镜像定义
+```
+
+---
+
+## 🚀 快速开始
+
+### 环境准备
+- Python 3.12+
+- [uv](https://docs.astral.sh/uv/) 包管理器
+
+### 本地安装
+
+1. **安装依赖**：
+   ```bash
+   uv sync
+   ```
+
+2. **列出当前注册的所有采集渠道**：
+   ```bash
+   uv run python runner.py --list
+   ```
+
+3. **执行一轮即时抓取与推流**：
+   ```bash
+   uv run python runner.py --once
+   ```
+
+---
+
+## 🛠️ 采集器配置与开发指南
+
+### 免代码声明式 TOML 站点卡片
+
+对于标准 HTML 结构的政务招考网站，只需在 `sources/{province}/{site_id}.toml` 编写配置，无需编写任何代码：
+
+```toml
+[site]
+id = "yn.yuxi"
+name = "玉溪市人民政府"
+province = "云南省"
+city = "玉溪市"
+employer = "玉溪市相关用人单位"
+employer_type = "事业单位"
+allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]
+
+[template]
+item_selector = "div.news-list a"
+url_pattern = "/article/\\d+\\.html"
+title_selector = "h1.article-title"
+body_selector = "div.article-content"
+date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})"
+attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]
+
+[[channels]]
+id = "rszk"
+name = "人事招考"
+start_url = "https://www.yuxi.gov.cn/rszk.html"
+keywords = ["招聘", "考试", "录用", "公示"]
+```
+
+### 复杂场景 Python 定制适配器
+
+对于含有复杂会话保持、加密参数或动态分页的站点，在 `adapters/{province}/{name}.py` 编写 Python 适配器：
+- 实现抓取函数并返回符合标准的 `IngestPayload`。
+- 强制使用 `common.security.is_safe_url` 实施外链与重定向安全隔离。
+- 提取正文时使用 `common.content.sanitize_body_text` 剥离导航与脏文本。
+
+---
+
+## ⚙️ 调度策略与频控机制
+
+调度器（`runner.py`）提供细粒度的运行控制：
+
+| 环境变量 / 命令行参数 | 默认值 | 说明 |
+|-----------------------|--------|------|
+| `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Rust 核心网关的摄入接口地址 |
+| `INGEST_TOKEN` | *空* | 与内核对齐的 HTTP Bearer 认证 Token |
+| `SCHEDULE_INTERVAL_HOURS` | `6` | 常驻巡检的周期时间（小时） |
+| `--provinces` | `[]` | 目标省份过滤（例如 `yn,sc`） |
+| `--concurrency` | `3` | 跨域名的最大并发数 |
+
+---
+
+## 🧪 测试与验证
+
+运行完备的本地安全回归套件（涵盖 SSRF 私网阻断、反爬脏正文过滤与附件扩展名判定）：
+
+```bash
+uv run python tests/test_content_and_security.py
+# 预期输出: All security and content tests passed!
+```
+
+---
+
+## 📜 开源协议
+
+本项目采用 [MIT](../LICENSE) 开源协议。
+
+<p align="right">(<a href="#readme-top">回到顶部</a>)</p>