docs: add bilingual README documentation for core and collectors

- Add English README.md and Chinese README.zh-CN.md for project root - Enhance collectors/README.md and add Chinese collectors/README.zh-CN.md - Document architecture, deployment, security invariants, and CLI/API usage
2026-10-24 00:00:50 +08:00
parent 1059be1b62
commit 5a8af5729c
4 changed files with 741 additions and 14 deletions
@@ -0,0 +1,225 @@
 <a id="readme-top"></a>
 # bangwen (榜文) 🚀
 > **High-performance public job recruitment aggregation, hybrid retrieval, and white-box resume matching engine designed with VictoriaMetrics architectural principles.**
 ---
 ## 📖 Table of Contents
 - [✨ Features](#-features)
 - [🏛️ Architecture](#️-architecture)
 - [🚀 Quickstart](#-quickstart)
  - [Prerequisites](#prerequisites)
  - [Installation & Deployment](#installation--deployment)
  - [Verification](#verification)
 - [🛠️ Usage](#️-usage)
  - [Web Console](#web-console)
  - [CLI Commands](#cli-commands)
  - [HTTP API Reference](#http-api-reference)
 - [⚙️ Configuration](#️-configuration)
 - [🔒 Security & Hardening](#-security--hardening)
 - [🤝 Contributing](#-contributing)
 - [📜 License](#-license)
 - [🤖 For AI Agents](#-for-ai-agents)
 ---
 ## ✨ Features
 - **⚡ VictoriaMetrics-Inspired Simplicity**: Single lightweight Rust binary acting as an ingestion gateway, hybrid search engine, and embedded web console without bloated microservices.
 - **🛡️ Multi-Tier Deduplication Pipeline**: Three-tier deduplication combining normalized URL canonicalization, 16-character content hashing, and 64-bit Hamming distance SimHash with short-text (<200 char) false-positive protection.
 - **🎯 White-Box Resume Matching**: Fully objective candidate scoring based on hard criteria (education levels mapped via standard dictionaries, age limits, region codes, and exact major/skill matches) with transparent audit reasons.
 - **🔍 Hybrid Search Engine**: Integrates exact SQL filtering, PostgreSQL `pg_trgm` GIN fuzzy matching, and `pgvector` HNSW cosine similarity with Reciprocal Rank Fusion (RRF).
 - **🕸️ Dual-Mode Edge Collectors**: Autonomous Python edge scrapers supporting declarative TOML site templates and specialized Python adapters with domain-level concurrency and polite throttles.
 - **🔒 Hardened Security Surface**:
  - Strict Content Security Policy (`script-src 'self'` with zero `'unsafe-inline'`).
  - Terminal sanitization stripping ANSI escape sequences, OSC 52, and control characters from all outputs.
  - Markdown injection prevention and SSRF hop-by-hop redirect verification.
  - Dedicated non-root container user execution (`uid=10001`).
 ---
 ## 🏛️ Architecture
 ```
                      +---------------------------------------+
                      |       Edge Public Recruitment Sites    |
                      +---------------------------------------+
                                          |
                      (HTTP Scraping & Polite Rate Limiting)
                                          v
                      +---------------------------------------+
                      |      bangwen-collector (Python)       |
                      |  - Declarative TOML (sources/)        |
                      |  - Custom Adapters (adapters/)        |
                      |  - SSRF-safe PoliteFetcher            |
                      +---------------------------------------+
                                          |
                              (HTTP POST /api/v1/ingest)
                           Bearer Token Authenticated
                                          v
                      +---------------------------------------+
                      |          bangwen-core (Rust)          |
                      |  - SimHash & URL Hash Deduplication   |
                      |  - Hybrid Search & White-box Matcher  |
                      |  - Embedded Web Console & Assets      |
                      +---------------------------------------+
                                          |
                               (SQL & Vector Storage)
                                          v
                      +---------------------------------------+
                      |         PostgreSQL 16 + pgvector      |
                      |  - announcement & position tables     |
                      |  - pg_trgm & HNSW vector indexes      |
                      +---------------------------------------+
 ```
 ---
 ## 🚀 Quickstart
 ### Prerequisites
 - [Docker](https://docs.docker.com/engine/install/) & [Docker Compose](https://docs.docker.com/compose/) v2+
 - Optional for native development:
  - [Rust](https://rustup.rs/) (>= 1.80)
  - [uv](https://docs.astral.sh/uv/) (>= 0.4.0) and Python 3.12+
 ### Installation & Deployment
 1. **Clone the repository**:
   ```bash
   git clone [email protected]:bitfennec/bangwen.git
   cd bangwen
   ```
 2. **Initialize environment credentials**:
   ```bash
   cp .env.example .env
   # Generate a secure 64-character token
   sed -i "s/replace_with_your_secure_secret_token/$(openssl rand -hex 32)/" .env
   ```
 3. **Start services with Docker Compose**:
   ```bash
   docker compose up -d --build
   ```
 ### Verification
 Check service health and initial ingestion metrics:
 ```bash
 curl -s http://127.0.0.1:8080/health
 # Expected: {"status":"ok"}
 curl -s http://127.0.0.1:8080/api/v1/stats
 # Expected: {"announcements":...,"embeddings":0,"positions":...,"sources":6,"status":"ok","vector_ready":false}
 ```
 Open `http://127.0.0.1:8080/` in your browser to inspect the Web Console.
 ---
 ## 🛠️ Usage
 ### Web Console
 The built-in web console is served directly on `http://127.0.0.1:8080/`:
 - **Feed Stream**: Paginated real-time recruitment notices with collapsible bodies, parsed position tables, and official attachment links.
 - **Multi-Filter Search**: Search by keywords, administrative region code, employer type, and minimum education requirements.
 - **Candidate White-Box Match**: Fill candidate resume attributes to calculate match scores and view clear transparent eligibility lists.
 - **Daily Digest**: View generated markdown summaries of the past 24 hours.
 - **Collector Monitor**: Health, authentication status, and last report timestamp for all edge collectors.
 ### CLI Commands
 When running natively or using the provided wrapper `./bin/bangwen`:
 ```bash
 # Ingest data via stdin or file
 ./bin/bangwen ingest --file ./sample.json
 # Hybrid search recruitment posts
 ./bin/bangwen search "运维" --region 530400 --json
 # Inspect notice or position details
 ./bin/bangwen show ann:1 --json
 ./bin/bangwen show pos:3 --json
 # Run white-box resume matching
 ./bin/bangwen match --resume ./tests/fixtures/sample_resume.json --top 10
 # Generate daily markdown digest
 ./bin/bangwen digest
 # Inspect collection sources health
 ./bin/bangwen sources
 ```
 ### HTTP API Reference
 | Endpoint | Method | Description | Auth Required |
 |----------|--------|-------------|---------------|
 | `/health` | `GET` | Service & database health check | No |
 | `/api/v1/ingest` | `POST` | Ingest batched notices & positions | Bearer Token |
 | `/api/v1/stats` | `GET` | Aggregation statistics and vector status | No |
 | `/api/v1/sources` | `GET` | Source nodes health and last ingest report | No |
 | `/api/v1/announcements` | `GET` | Paginated recruitment announcements list | No |
 | `/api/v1/announcements/{id}` | `GET` | Detailed announcement with positions & attachments | No |
 | `/api/v1/positions/{id}` | `GET` | Detailed structured job position | No |
 | `/api/v1/search` | `GET` | Multi-dimensional hybrid keyword search | No |
 | `/api/v1/match` | `POST` | Strict white-box resume qualification match | No |
 | `/api/v1/digest` | `GET` | Daily markdown recruitment digest | No |
 ---
 ## ⚙️ Configuration
 Environment variables configured in `.env`:
 | Variable | Default | Required | Description |
 |----------|---------|----------|-------------|
 | `DATABASE_URL` | `postgres://bangwen:bangwen_dev@postgres:5432/bangwen` | Yes | PostgreSQL connection string with pgvector |
 | `INGEST_TOKEN` | *None* | Recommended | Bearer token for authenticating `/api/v1/ingest` |
 | `SERVER_HOST` | `0.0.0.0` | No | Host address for Rust HTTP gateway |
 | `SERVER_PORT` | `8080` | No | Port for Rust HTTP gateway |
 | `LOG_LEVEL` | `info` | No | Tracing log level (`error`, `warn`, `info`, `debug`) |
 | `OLLAMA_BASE_URL` | `http://host.docker.internal:11434` | No | Optional Ollama instance for embedding generation |
 | `EMBEDDING_MODEL` | `bge-m3` | No | Ollama embedding model name |
 ---
 ## 🔒 Security & Hardening
 - **Non-Root Execution**: Both `bangwen-core` and `bangwen-collector` containers enforce explicit non-root user execution (`UID 10001`).
 - **Strict Content Security Policy (CSP)**: `default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; frame-ancestors 'none';`. Inline scripts are completely eliminated.
 - **Input Boundary Bounds**: Bounded validation for match requests (max 50 skills, 20 cities, 32-char education, 1940..=current birth year) and pagination limits (max offset 10000).
 - **Anti-Deadlock Lock Ordering**: Ingestion pipeline enforces deterministic URL key sorting prior to row lock acquisition, eliminating PostgreSQL `40P01` deadlock conditions.
 - **SSRF & Scheme Verification**: PoliteFetcher enforces private IP filtering, strict domain whitelists, and hop-by-hop validation during HTTP redirects.
 ---
 ## 🤝 Contributing
 Contributions are welcome. Please ensure all modifications pass local validation:
 ```bash
 # Rust core checks
 cargo test --offline
 cargo clippy --all-targets -- -D warnings
 # Collectors checks
 cd collectors && uv run python tests/test_content_and_security.py
 ```
 ---
 ## 📜 License
 [MIT](LICENSE)
 ---
 ## 🤖 For AI Agents
 See [`skill/SKILL.md`](./skill/SKILL.md) for operational instructions and security boundary guidelines.
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
@@ -0,0 +1,225 @@
 <a id="readme-top"></a>
 # 榜文 (bangwen) 🚀
 > **基于 VictoriaMetrics 极简架构哲学构建的高性能公开招考数据聚合、混合检索与白盒简历匹配引擎。**
 ---
 ## 📖 目录
 - [✨ 核心特性](#-核心特性)
 - [🏛️ 系统架构](#️-系统架构)
 - [🚀 快速开始](#-快速开始)
  - [环境准备](#环境准备)
  - [安装与部署](#安装与部署)
  - [验证运行](#验证运行)
 - [🛠️ 使用指南](#️-使用指南)
  - [Web 控制台](#web-控制台)
  - [CLI 命令行工具](#cli-命令行工具)
  - [HTTP API 接口参考](#http-api-接口参考)
 - [⚙️ 环境配置](#️-环境配置)
 - [🔒 安全加固说明](#-安全加固说明)
 - [🤝 参与贡献](#-参与贡献)
 - [📜 开源协议](#-开源协议)
 - [🤖 AI Agent 集成](#-ai-agent-集成)
 ---
 ## ✨ 核心特性
 - **⚡ 对标 VictoriaMetrics 架构哲学**：单微核 Rust 二进制文件同时承载 HTTP 摄入网关、混合检索核心与内嵌 Web 控制台，杜绝过度微服务化与复杂外部依赖。
 - **🛡️ 多级排重与数据治理管线**：融合规范化 URL 权威归一、16 字符内容哈希校验、64 位汉明距离 SimHash 以及针对短文本（<200 字符）的防误杀二次核验机制。
 - **🎯 客观透明的白盒简历匹配**：基于法定与招聘硬性门槛（GB/T 2260 行政区划字典、标准学历档次映射、年龄区间与专业/技能对齐）进行确定性白盒匹配，直接输出透明合规与不符事实清单。
 - **🔍 多路混合检索与 RRF 融合**：深度集成 SQL 精确过滤、PostgreSQL `pg_trgm` GIN 模糊索引以及 `pgvector` HNSW 向量余弦距离检索，通过 RRF（Reciprocal Rank Fusion）算法兼顾高召回与高精度。
 - **🕸️ 边缘采集双模调度引擎**：Python 边缘爬虫体系，同时支持轻量级声明式 TOML 配置卡片与复杂场景定制 Python 适配器，按域名分组并发调度并执行站内串行礼貌降频。
 - **🔒 全面收敛的安全防护基线**：
  - 严格的内容安全策略（CSP `script-src 'self'`，彻底移除 `'unsafe-inline'`）。
  - 终端控制序列无死角清洗（剥离 ANSI CSI、OSC 52 剪贴板指令与控制字符）。
  - Markdown URL 注入拦截与爬虫逐跳重定向 SSRF 白名单校验。
  - 容器服务全量以非 root 专属系统用户（`UID 10001`）隔离执行。
 ---
 ## 🏛️ 系统架构
 ```
                      +---------------------------------------+
                      |          各省市公开招考招聘政务网      |
                      +---------------------------------------+
                                          |
                      (HTTP 抓取与域名礼貌限速调度)
                                          v
                      +---------------------------------------+
                      |      bangwen-collector (Python)       |
                      |  - 声明式 TOML 站点卡片 (sources/)    |
                      |  - 定制化抓取适配器 (adapters/)       |
                      |  - SSRF 防护型 PoliteFetcher 客户端   |
                      +---------------------------------------+
                                          |
                              (HTTP POST /api/v1/ingest)
                               Bearer Token 鉴权通信
                                          v
                      +---------------------------------------+
                      |          bangwen-core (Rust)          |
                      |  - SimHash & 内容哈希多级排重管线     |
                      |  - 多路混合检索与白盒简历打分内核     |
                      |  - 纯原生内嵌 Web 控制台与静态资源    |
                      +---------------------------------------+
                                          |
                               (SQL 与向量存储)
                                          v
                      +---------------------------------------+
                      |         PostgreSQL 16 + pgvector      |
                      |  - announcement 与 position 数据基表  |
                      |  - pg_trgm 倒排与 HNSW 向量近似索引   |
                      +---------------------------------------+
 ```
 ---
 ## 🚀 快速开始
 ### 环境准备
 - [Docker](https://docs.docker.com/engine/install/) 与 [Docker Compose](https://docs.docker.com/compose/) v2+
 - 本地原生开发环境（可选）：
  - [Rust](https://rustup.rs/) (>= 1.80)
  - [uv](https://docs.astral.sh/uv/) (>= 0.4.0) 与 Python 3.12+
 ### 安装与部署
 1. **克隆代码仓库**：
   ```bash
   git clone [email protected]:bitfennec/bangwen.git
   cd bangwen
   ```
 2. **初始化环境与鉴权凭据**：
   ```bash
   cp .env.example .env
   # 自动生成 64 位高强度随机鉴权 Token
   sed -i "s/replace_with_your_secure_secret_token/$(openssl rand -hex 32)/" .env
   ```
 3. **使用 Docker Compose 一键构建启动**：
   ```bash
   docker compose up -d --build
   ```
 ### 验证运行
 检查服务运行状态与初始入库指标：
 ```bash
 curl -s http://127.0.0.1:8080/health
 # 预期输出: {"status":"ok"}
 curl -s http://127.0.0.1:8080/api/v1/stats
 # 预期输出: {"announcements":...,"embeddings":0,"positions":...,"sources":6,"status":"ok","vector_ready":false}
 ```
 在浏览器中打开 `http://127.0.0.1:8080/` 即可直接进入内嵌 Web 控制台。
 ---
 ## 🛠️ 使用指南
 ### Web 控制台
 服务启动后直接监听 `http://127.0.0.1:8080/`：
 - **公告 Feed 流**：按时间倒序分页展示各省市招考公告，支持全文实时一键折叠/展开、挂载结构化职位列表与官方附件直接下载。
 - **多维综合检索**：支持按关键词、标准行政区划编码、编制用人性质及最低学历门槛组合筛选。
 - **白盒简历匹配**：输入候选人基础画像，系统秒级匹配合规在招岗位，透明标明硬性满足与不符条款。
 - **每日聚合日报**：一键生成与复制过去 24 小时收录的招考精选与即将截止报名清单（Markdown 格式）。
 - **采集节点监控**：实时查看各边缘节点的连通健康度、认证态与最近上报统计。
 ### CLI 命令行工具
 通过宿主机自带的原生穿透脚本 `./bin/bangwen` 或原生编译二进制进行操作：
 ```bash
 # 从文件或 stdin 管道摄入数据
 ./bin/bangwen ingest --file ./sample.json
 # 多维检索符合条件的岗位 (例如玉溪 530400)
 ./bin/bangwen search "运维" --region 530400 --json
 # 查看公告或具体岗位详情
 ./bin/bangwen show ann:1 --json
 ./bin/bangwen show pos:3 --json
 # 根据候选人画像执行白盒打分匹配
 ./bin/bangwen match --resume ./tests/fixtures/sample_resume.json --top 10
 # 输出今日招考聚合日报
 ./bin/bangwen digest
 # 检查各边缘采集源的健康状态
 ./bin/bangwen sources
 ```
 ### HTTP API 接口参考
 | 端点路由 | 请求方式 | 功能简述 | 鉴权要求 |
 |----------|----------|----------|----------|
 | `/health` | `GET` | 核心服务与数据库连通性健康检查 | 无 |
 | `/api/v1/ingest` | `POST` | 批量数据推送摄入接口 | Bearer Token |
 | `/api/v1/stats` | `GET` | 系统汇总指标与向量就绪状态 | 无 |
 | `/api/v1/sources` | `GET` | 采集节点健康度与最近入库明细 | 无 |
 | `/api/v1/announcements` | `GET` | 公告分页列表（支持关键字与类型筛选） | 无 |
 | `/api/v1/announcements/{id}` | `GET` | 单篇公告详情（含下属岗位与附件） | 无 |
 | `/api/v1/positions/{id}` | `GET` | 细分结构化招考岗位详情 | 无 |
 | `/api/v1/search` | `GET` | 混合关键字多路检索接口 | 无 |
 | `/api/v1/match` | `POST` | 候选人简历白盒打分与核验接口 | 无 |
 | `/api/v1/digest` | `GET` | 今日招考聚合日报（Markdown 格式） | 无 |
 ---
 ## ⚙️ 环境配置
 在 `.env` 中配置的环境变量说明：
 | 变量名 | 默认值 | 是否必填 | 说明 |
 |--------|--------|----------|------|
 | `DATABASE_URL` | `postgres://bangwen:bangwen_dev@postgres:5432/bangwen` | 是 | PostgreSQL + pgvector 数据库连接串 |
 | `INGEST_TOKEN` | *空* | 推荐 | 用于保护 `/api/v1/ingest` 端点的 Bearer 鉴权密钥 |
 | `SERVER_HOST` | `0.0.0.0` | 否 | Rust HTTP 服务监听地址 |
 | `SERVER_PORT` | `8080` | 否 | Rust HTTP 服务监听端口 |
 | `LOG_LEVEL` | `info` | 否 | 日志级别（`error`, `warn`, `info`, `debug`） |
 | `OLLAMA_BASE_URL` | `http://host.docker.internal:11434` | 否 | 向量生成所用的 Ollama 服务地址 |
 | `EMBEDDING_MODEL` | `bge-m3` | 否 | Ollama 向量模型名称 |
 ---
 ## 🔒 安全加固说明
 - **非 Root 容器降权**：`bangwen-core` 与 `bangwen-collector` 容器镜像均配置专用非特权运行用户（`UID 10001`）。
 - **严格 CSP 策略**：响应头强制 `default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline';`，彻底根除内联脚本注入执行风险。
 - **接口算力与内存保护**：对简历匹配接口增加完备的参数边界防护（技能最大 50 项、城市最大 20 项、学历上限 32 字符、出生年份 1940~当年区间强校验），深度分页 offset 上限 10000 强校验拦截。
 - **并发摄入防死锁**：数据落库事务内强制按照规范化 URL 字典序申请行级排他锁，彻底根除 PostgreSQL `40P01` 死锁异常。
 - **SSRF 与重定向逐跳校验**：HTTP 抓取客户端内置私有内网 IP 过滤、严格域名白名单与逐跳重定向安全性验证。
 ---
 ## 🤝 参与贡献
 欢迎提交 Issue 与 Pull Request。提交改动前请确保通过本地基线检验：
 ```bash
 # Rust 内核验证
 cargo test --offline
 cargo clippy --all-targets -- -D warnings
 # 采集端回归验证
 cd collectors && uv run python tests/test_content_and_security.py
 ```
 ---
 ## 📜 开源协议
 本项目采用 [MIT](LICENSE) 开源协议。
 ---
 ## 🤖 AI Agent 集成
 针对作为 Agent 工具调用的集成场景，请参阅 [`skill/SKILL.md`](./skill/SKILL.md)，了解调用协议、结构化输出规范以及不可信数据的安全隔离要求。
 <p align="right">(<a href="#readme-top">回到顶部</a>)</p>
@@ -1,25 +1,151 @@
-# Bangwen Collectors
+<a id="readme-top"></a>
-Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.
+# bangwen-collectors 🕷️
-## Structure
+> **Lightweight, declarative, and SSRF-hardened edge collection system for public job recruitment aggregation.**
- `sources/`: Declarative TOML configuration for crawling target sites.
+---
 - `adapters/`: Custom Python scrapers for complex or stateful collection logic.
 - `common/`: Core fetcher, security validators, and ingestion protocol models.
 - `runner.py`: Periodic crawler runner and domain-concurrency scheduler.
-## Running
+## 📖 Table of Contents
 - [✨ Overview](#-overview)
 - [📁 Project Structure](#-project-structure)
 - [🚀 Quickstart](#-quickstart)
  - [Prerequisites](#prerequisites)
  - [Local Setup](#local-setup)
 - [🛠️ Configuration & Source Authoring](#️-configuration--source-authoring)
  - [Declarative TOML Sources](#declarative-toml-sources)
  - [Custom Python Adapters](#custom-python-adapters)
 - [⚙️ Scheduling & Rate Limiting](#️-scheduling--rate-limiting)
 - [🧪 Testing](#-testing)
 - [📜 License](#-license)
-```bash
+---
-uv run python runner.py --list
+
-uv run python runner.py --once
+## ✨ Overview
 `bangwen-collectors` serves as the decentralized edge data extraction layer for the `bangwen` recruitment aggregation engine:
 - **Dual-Mode Crawling**: Supports both zero-code declarative TOML site cards (`sources/`) and customized Python scraper adapters (`adapters/`).
 - **Domain-Grouped Concurrency**: Concurrently processes distinct domains while enforcing strictly serial queues within the same domain.
 - **WAF-Friendly Polite Delays**: Configurable inter-channel pauses (default 3.0s) and detail extraction throttles (0.3s) preventing IP bans.
 - **SSRF Defense-in-Depth**: Built-in private IP filtering, strict domain whitelists, and hop-by-hop verification during HTTP redirects.
 ---
 ## 📁 Project Structure
 ```
 collectors/
 ├── adapters/                  # Specialized scrapers for complex dynamic sites
 │   └── sc/chengdu_rsks.py    # Custom adapter for Chengdu Personnel Exam Net
 ├── sources/                   # Zero-code declarative TOML site cards
 │   ├── sc/                    # Sichuan Province sources (e.g. myrsks, zgrsks)
 │   └── yn/                    # Yunnan Province sources (e.g. hrss, yuxi)
 ├── common/                    # Core modules
 │   ├── content.py             # Content truncation and dirty body cleanup
 │   ├── declarative.py         # Universal declarative parsing engine
 │   ├── fetcher.py             # PoliteFetcher HTTP client with SSRF verification
 │   ├── protocol.py            # Pydantic schema matching the bangwen/v1 protocol
 │   └── security.py            # URL validation and private IP guards
 ├── tests/                     # Security and parsing regression tests
 ├── config.toml                # Active collection and scheduler configuration
 ├── runner.py                  # Main scheduler and multi-domain orchestrator
 └── Dockerfile                 # Dedicated non-root container definition
 ```
-## Testing
+---
-Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:
+## 🚀 Quickstart
 ### Prerequisites
 - Python 3.12+
 - [uv](https://docs.astral.sh/uv/) package manager
 ### Local Setup
 1. **Install dependencies**:
   ```bash
   uv sync
   ```
 2. **List available collectors**:
   ```bash
   uv run python runner.py --list
   ```
 3. **Run a single crawl cycle**:
   ```bash
   uv run python runner.py --once
   ```
 ---
 ## 🛠️ Configuration & Source Authoring
 ### Declarative TOML Sources
 Place zero-code TOML configuration cards under `sources/{province}/{site_id}.toml`:
 ```toml
 [site]
 id = "yn.yuxi"
 name = "玉溪市人民政府"
 province = "云南省"
 city = "玉溪市"
 employer = "玉溪市相关用人单位"
 employer_type = "事业单位"
 allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]
 [template]
 item_selector = "div.news-list a"
 url_pattern = "/article/\\d+\\.html"
 title_selector = "h1.article-title"
 body_selector = "div.article-content"
 date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})"
 attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]
 [[channels]]
 id = "rszk"
 name = "人事招考"
 start_url = "https://www.yuxi.gov.cn/rszk.html"
 keywords = ["招聘", "考试", "录用", "公示"]
 ```
 ### Custom Python Adapters
 When target sites require complex session handling, cryptographic requests, or specific payload extraction, create a Python scraper under `adapters/{province}/{name}.py`:
 - Implement a crawl function returning an `IngestPayload`.
 - Use `common.security.is_safe_url` to enforce domain isolation.
 - Sanitize extracted text with `common.content.sanitize_body_text`.
 ---
 ## ⚙️ Scheduling & Rate Limiting
 The runner (`runner.py`) provides fine-grained scheduling controls:
 | Environment Variable / Flag | Default | Description |
 |-----------------------------|---------|-------------|
 | `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Target Rust gateway ingestion endpoint |
 | `INGEST_TOKEN` | *None* | Bearer authentication token matching core `.env` |
 | `SCHEDULE_INTERVAL_HOURS` | `6` | Cycle interval between automated sweeps |
 | `--provinces` | `[]` | Filter target provinces (e.g. `yn,sc`) |
 | `--concurrency` | `3` | Maximum concurrent distinct domain scrapers |
 ---
 ## 🧪 Testing
 Comprehensive test suites covering SSRF prevention, HTML sanitization, and URL security:
 ```bash
-cd collectors && uv run python tests/test_content_and_security.py
+uv run python tests/test_content_and_security.py
 # Expected output: All security and content tests passed!
 ```
 ---
 ## 📜 License
 [MIT](../LICENSE)
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
@@ -0,0 +1,151 @@
 <a id="readme-top"></a>
 # bangwen-collectors 边缘采集系统 🕷️
 > **专为招考数据聚合设计的高性能、免代码声明式与 SSRF 安全强化的边缘抓取体系。**
 ---
 ## 📖 目录
 - [✨ 概述](#-概述)
 - [📁 目录结构](#-目录结构)
 - [🚀 快速开始](#-快速开始)
  - [环境准备](#环境准备)
  - [本地安装](#本地安装)
 - [🛠️ 采集器配置与开发指南](#️-采集器配置与开发指南)
  - [免代码声明式 TOML 站点卡片](#免代码声明式-toml-站点卡片)
  - [复杂场景 Python 定制适配器](#复杂场景-python-定制适配器)
 - [⚙️ 调度策略与频控机制](#️-调度策略与频控机制)
 - [🧪 测试与验证](#-测试与验证)
 - [📜 开源协议](#-开源协议)
 ---
 ## ✨ 概述
 `bangwen-collectors` 作为 `bangwen` 招考数据中枢的分布式边缘抽取层：
 - **双模自动发现**：同时支持零代码声明式 TOML 站点配置卡片（`sources/`）与 Python 定制适配器（`adapters/`）。
 - **按域名分组并发**：对不同域名并发抓取，而对相同域名严格排队串行，保障采集效率的同时彻底杜绝政务网 WAF 封锁。
 - **礼貌延时与节流**：栏目间保持静默等待（默认 3.0s），单篇详情提取保留 0.3s 间隔微节流。
 - **全链路 SSRF 纵深防护**：底层内置私有内网 IP 阻断、严格域名白名单匹配与 HTTP 重定向逐跳安全性校验。
 ---
 ## 📁 目录结构
 ```
 collectors/
 ├── adapters/                  # 针对复杂动态站点的 Python 专属抓取适配器
 │   └── sc/chengdu_rsks.py    # 成都人事考试网抓取适配器
 ├── sources/                   # 零代码声明式 TOML 站点配置卡片
 │   ├── sc/                    # 四川省站点（如绵阳人事考试网、自贡人事考试网）
 │   └── yn/                    # 云南省站点（如云南人社厅、玉溪市人民政府）
 ├── common/                    # 核心通用组件
 │   ├── content.py             # 脏正文清洗、超长文本截断与反爬标记识别
 │   ├── declarative.py         # 通用免代码声明式解析执行引擎
 │   ├── fetcher.py             # 支持 SSRF 逐跳校验与限速的 PoliteFetcher 客户端
 │   ├── protocol.py            # 与 bangwen/v1 摄入协议严格对齐的 Pydantic 模型
 │   └── security.py            # URL 白名单判定与内网 IP 阻断工具
 ├── tests/                     # 安全防护与内容清洗回归测试集
 ├── config.toml                # 采集调度与省份过滤总配置文件
 ├── runner.py                  # 多域名调度器与常驻运行入口
 └── Dockerfile                 # 专属非 root 用户（UID 10001）容器镜像定义
 ```
 ---
 ## 🚀 快速开始
 ### 环境准备
 - Python 3.12+
 - [uv](https://docs.astral.sh/uv/) 包管理器
 ### 本地安装
 1. **安装依赖**：
   ```bash
   uv sync
   ```
 2. **列出当前注册的所有采集渠道**：
   ```bash
   uv run python runner.py --list
   ```
 3. **执行一轮即时抓取与推流**：
   ```bash
   uv run python runner.py --once
   ```
 ---
 ## 🛠️ 采集器配置与开发指南
 ### 免代码声明式 TOML 站点卡片
 对于标准 HTML 结构的政务招考网站，只需在 `sources/{province}/{site_id}.toml` 编写配置，无需编写任何代码：
 ```toml
 [site]
 id = "yn.yuxi"
 name = "玉溪市人民政府"
 province = "云南省"
 city = "玉溪市"
 employer = "玉溪市相关用人单位"
 employer_type = "事业单位"
 allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]
 [template]
 item_selector = "div.news-list a"
 url_pattern = "/article/\\d+\\.html"
 title_selector = "h1.article-title"
 body_selector = "div.article-content"
 date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})"
 attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]
 [[channels]]
 id = "rszk"
 name = "人事招考"
 start_url = "https://www.yuxi.gov.cn/rszk.html"
 keywords = ["招聘", "考试", "录用", "公示"]
 ```
 ### 复杂场景 Python 定制适配器
 对于含有复杂会话保持、加密参数或动态分页的站点，在 `adapters/{province}/{name}.py` 编写 Python 适配器：
 - 实现抓取函数并返回符合标准的 `IngestPayload`。
 - 强制使用 `common.security.is_safe_url` 实施外链与重定向安全隔离。
 - 提取正文时使用 `common.content.sanitize_body_text` 剥离导航与脏文本。
 ---
 ## ⚙️ 调度策略与频控机制
 调度器（`runner.py`）提供细粒度的运行控制：
 | 环境变量 / 命令行参数 | 默认值 | 说明 |
 |-----------------------|--------|------|
 | `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Rust 核心网关的摄入接口地址 |
 | `INGEST_TOKEN` | *空* | 与内核对齐的 HTTP Bearer 认证 Token |
 | `SCHEDULE_INTERVAL_HOURS` | `6` | 常驻巡检的周期时间（小时） |
 | `--provinces` | `[]` | 目标省份过滤（例如 `yn,sc`） |
 | `--concurrency` | `3` | 跨域名的最大并发数 |
 ---
 ## 🧪 测试与验证
 运行完备的本地安全回归套件（涵盖 SSRF 私网阻断、反爬脏正文过滤与附件扩展名判定）：
 ```bash
 uv run python tests/test_content_and_security.py
 # 预期输出: All security and content tests passed!
 ```
 ---
 ## 📜 开源协议
 本项目采用 [MIT](../LICENSE) 开源协议。
 <p align="right">(<a href="#readme-top">回到顶部</a>)</p>