diff --git a/README.md b/README.md new file mode 100644 index 0000000..2a2f8d4 --- /dev/null +++ b/README.md @@ -0,0 +1,225 @@ + + +# bangwen (榜文) 🚀 + +> **High-performance public job recruitment aggregation, hybrid retrieval, and white-box resume matching engine designed with VictoriaMetrics architectural principles.** + +--- + +## 📖 Table of Contents +- [✨ Features](#-features) +- [🏛️ Architecture](#️-architecture) +- [🚀 Quickstart](#-quickstart) + - [Prerequisites](#prerequisites) + - [Installation & Deployment](#installation--deployment) + - [Verification](#verification) +- [🛠️ Usage](#️-usage) + - [Web Console](#web-console) + - [CLI Commands](#cli-commands) + - [HTTP API Reference](#http-api-reference) +- [⚙️ Configuration](#️-configuration) +- [🔒 Security & Hardening](#-security--hardening) +- [🤝 Contributing](#-contributing) +- [📜 License](#-license) +- [🤖 For AI Agents](#-for-ai-agents) + +--- + +## ✨ Features + +- **⚡ VictoriaMetrics-Inspired Simplicity**: Single lightweight Rust binary acting as an ingestion gateway, hybrid search engine, and embedded web console without bloated microservices. +- **🛡️ Multi-Tier Deduplication Pipeline**: Three-tier deduplication combining normalized URL canonicalization, 16-character content hashing, and 64-bit Hamming distance SimHash with short-text (<200 char) false-positive protection. +- **🎯 White-Box Resume Matching**: Fully objective candidate scoring based on hard criteria (education levels mapped via standard dictionaries, age limits, region codes, and exact major/skill matches) with transparent audit reasons. +- **🔍 Hybrid Search Engine**: Integrates exact SQL filtering, PostgreSQL `pg_trgm` GIN fuzzy matching, and `pgvector` HNSW cosine similarity with Reciprocal Rank Fusion (RRF). +- **🕸️ Dual-Mode Edge Collectors**: Autonomous Python edge scrapers supporting declarative TOML site templates and specialized Python adapters with domain-level concurrency and polite throttles. +- **🔒 Hardened Security Surface**: + - Strict Content Security Policy (`script-src 'self'` with zero `'unsafe-inline'`). + - Terminal sanitization stripping ANSI escape sequences, OSC 52, and control characters from all outputs. + - Markdown injection prevention and SSRF hop-by-hop redirect verification. + - Dedicated non-root container user execution (`uid=10001`). + +--- + +## 🏛️ Architecture + +``` + +---------------------------------------+ + | Edge Public Recruitment Sites | + +---------------------------------------+ + | + (HTTP Scraping & Polite Rate Limiting) + v + +---------------------------------------+ + | bangwen-collector (Python) | + | - Declarative TOML (sources/) | + | - Custom Adapters (adapters/) | + | - SSRF-safe PoliteFetcher | + +---------------------------------------+ + | + (HTTP POST /api/v1/ingest) + Bearer Token Authenticated + v + +---------------------------------------+ + | bangwen-core (Rust) | + | - SimHash & URL Hash Deduplication | + | - Hybrid Search & White-box Matcher | + | - Embedded Web Console & Assets | + +---------------------------------------+ + | + (SQL & Vector Storage) + v + +---------------------------------------+ + | PostgreSQL 16 + pgvector | + | - announcement & position tables | + | - pg_trgm & HNSW vector indexes | + +---------------------------------------+ +``` + +--- + +## 🚀 Quickstart + +### Prerequisites +- [Docker](https://docs.docker.com/engine/install/) & [Docker Compose](https://docs.docker.com/compose/) v2+ +- Optional for native development: + - [Rust](https://rustup.rs/) (>= 1.80) + - [uv](https://docs.astral.sh/uv/) (>= 0.4.0) and Python 3.12+ + +### Installation & Deployment + +1. **Clone the repository**: + ```bash + git clone git@gitea.o00o.men:bitfennec/bangwen.git + cd bangwen + ``` + +2. **Initialize environment credentials**: + ```bash + cp .env.example .env + # Generate a secure 64-character token + sed -i "s/replace_with_your_secure_secret_token/$(openssl rand -hex 32)/" .env + ``` + +3. **Start services with Docker Compose**: + ```bash + docker compose up -d --build + ``` + +### Verification + +Check service health and initial ingestion metrics: +```bash +curl -s http://127.0.0.1:8080/health +# Expected: {"status":"ok"} + +curl -s http://127.0.0.1:8080/api/v1/stats +# Expected: {"announcements":...,"embeddings":0,"positions":...,"sources":6,"status":"ok","vector_ready":false} +``` + +Open `http://127.0.0.1:8080/` in your browser to inspect the Web Console. + +--- + +## 🛠️ Usage + +### Web Console +The built-in web console is served directly on `http://127.0.0.1:8080/`: +- **Feed Stream**: Paginated real-time recruitment notices with collapsible bodies, parsed position tables, and official attachment links. +- **Multi-Filter Search**: Search by keywords, administrative region code, employer type, and minimum education requirements. +- **Candidate White-Box Match**: Fill candidate resume attributes to calculate match scores and view clear transparent eligibility lists. +- **Daily Digest**: View generated markdown summaries of the past 24 hours. +- **Collector Monitor**: Health, authentication status, and last report timestamp for all edge collectors. + +### CLI Commands + +When running natively or using the provided wrapper `./bin/bangwen`: + +```bash +# Ingest data via stdin or file +./bin/bangwen ingest --file ./sample.json + +# Hybrid search recruitment posts +./bin/bangwen search "运维" --region 530400 --json + +# Inspect notice or position details +./bin/bangwen show ann:1 --json +./bin/bangwen show pos:3 --json + +# Run white-box resume matching +./bin/bangwen match --resume ./tests/fixtures/sample_resume.json --top 10 + +# Generate daily markdown digest +./bin/bangwen digest + +# Inspect collection sources health +./bin/bangwen sources +``` + +### HTTP API Reference + +| Endpoint | Method | Description | Auth Required | +|----------|--------|-------------|---------------| +| `/health` | `GET` | Service & database health check | No | +| `/api/v1/ingest` | `POST` | Ingest batched notices & positions | Bearer Token | +| `/api/v1/stats` | `GET` | Aggregation statistics and vector status | No | +| `/api/v1/sources` | `GET` | Source nodes health and last ingest report | No | +| `/api/v1/announcements` | `GET` | Paginated recruitment announcements list | No | +| `/api/v1/announcements/{id}` | `GET` | Detailed announcement with positions & attachments | No | +| `/api/v1/positions/{id}` | `GET` | Detailed structured job position | No | +| `/api/v1/search` | `GET` | Multi-dimensional hybrid keyword search | No | +| `/api/v1/match` | `POST` | Strict white-box resume qualification match | No | +| `/api/v1/digest` | `GET` | Daily markdown recruitment digest | No | + +--- + +## ⚙️ Configuration + +Environment variables configured in `.env`: + +| Variable | Default | Required | Description | +|----------|---------|----------|-------------| +| `DATABASE_URL` | `postgres://bangwen:bangwen_dev@postgres:5432/bangwen` | Yes | PostgreSQL connection string with pgvector | +| `INGEST_TOKEN` | *None* | Recommended | Bearer token for authenticating `/api/v1/ingest` | +| `SERVER_HOST` | `0.0.0.0` | No | Host address for Rust HTTP gateway | +| `SERVER_PORT` | `8080` | No | Port for Rust HTTP gateway | +| `LOG_LEVEL` | `info` | No | Tracing log level (`error`, `warn`, `info`, `debug`) | +| `OLLAMA_BASE_URL` | `http://host.docker.internal:11434` | No | Optional Ollama instance for embedding generation | +| `EMBEDDING_MODEL` | `bge-m3` | No | Ollama embedding model name | + +--- + +## 🔒 Security & Hardening + +- **Non-Root Execution**: Both `bangwen-core` and `bangwen-collector` containers enforce explicit non-root user execution (`UID 10001`). +- **Strict Content Security Policy (CSP)**: `default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; frame-ancestors 'none';`. Inline scripts are completely eliminated. +- **Input Boundary Bounds**: Bounded validation for match requests (max 50 skills, 20 cities, 32-char education, 1940..=current birth year) and pagination limits (max offset 10000). +- **Anti-Deadlock Lock Ordering**: Ingestion pipeline enforces deterministic URL key sorting prior to row lock acquisition, eliminating PostgreSQL `40P01` deadlock conditions. +- **SSRF & Scheme Verification**: PoliteFetcher enforces private IP filtering, strict domain whitelists, and hop-by-hop validation during HTTP redirects. + +--- + +## 🤝 Contributing + +Contributions are welcome. Please ensure all modifications pass local validation: +```bash +# Rust core checks +cargo test --offline +cargo clippy --all-targets -- -D warnings + +# Collectors checks +cd collectors && uv run python tests/test_content_and_security.py +``` + +--- + +## 📜 License + +[MIT](LICENSE) + +--- + +## 🤖 For AI Agents + +See [`skill/SKILL.md`](./skill/SKILL.md) for operational instructions and security boundary guidelines. + +

(back to top)

diff --git a/README.zh-CN.md b/README.zh-CN.md new file mode 100644 index 0000000..2d922e3 --- /dev/null +++ b/README.zh-CN.md @@ -0,0 +1,225 @@ + + +# 榜文 (bangwen) 🚀 + +> **基于 VictoriaMetrics 极简架构哲学构建的高性能公开招考数据聚合、混合检索与白盒简历匹配引擎。** + +--- + +## 📖 目录 +- [✨ 核心特性](#-核心特性) +- [🏛️ 系统架构](#️-系统架构) +- [🚀 快速开始](#-快速开始) + - [环境准备](#环境准备) + - [安装与部署](#安装与部署) + - [验证运行](#验证运行) +- [🛠️ 使用指南](#️-使用指南) + - [Web 控制台](#web-控制台) + - [CLI 命令行工具](#cli-命令行工具) + - [HTTP API 接口参考](#http-api-接口参考) +- [⚙️ 环境配置](#️-环境配置) +- [🔒 安全加固说明](#-安全加固说明) +- [🤝 参与贡献](#-参与贡献) +- [📜 开源协议](#-开源协议) +- [🤖 AI Agent 集成](#-ai-agent-集成) + +--- + +## ✨ 核心特性 + +- **⚡ 对标 VictoriaMetrics 架构哲学**：单微核 Rust 二进制文件同时承载 HTTP 摄入网关、混合检索核心与内嵌 Web 控制台，杜绝过度微服务化与复杂外部依赖。 +- **🛡️ 多级排重与数据治理管线**：融合规范化 URL 权威归一、16 字符内容哈希校验、64 位汉明距离 SimHash 以及针对短文本（<200 字符）的防误杀二次核验机制。 +- **🎯 客观透明的白盒简历匹配**：基于法定与招聘硬性门槛（GB/T 2260 行政区划字典、标准学历档次映射、年龄区间与专业/技能对齐）进行确定性白盒匹配，直接输出透明合规与不符事实清单。 +- **🔍 多路混合检索与 RRF 融合**：深度集成 SQL 精确过滤、PostgreSQL `pg_trgm` GIN 模糊索引以及 `pgvector` HNSW 向量余弦距离检索，通过 RRF（Reciprocal Rank Fusion）算法兼顾高召回与高精度。 +- **🕸️ 边缘采集双模调度引擎**：Python 边缘爬虫体系，同时支持轻量级声明式 TOML 配置卡片与复杂场景定制 Python 适配器，按域名分组并发调度并执行站内串行礼貌降频。 +- **🔒 全面收敛的安全防护基线**： + - 严格的内容安全策略（CSP `script-src 'self'`，彻底移除 `'unsafe-inline'`）。 + - 终端控制序列无死角清洗（剥离 ANSI CSI、OSC 52 剪贴板指令与控制字符）。 + - Markdown URL 注入拦截与爬虫逐跳重定向 SSRF 白名单校验。 + - 容器服务全量以非 root 专属系统用户（`UID 10001`）隔离执行。 + +--- + +## 🏛️ 系统架构 + +``` + +---------------------------------------+ + | 各省市公开招考招聘政务网 | + +---------------------------------------+ + | + (HTTP 抓取与域名礼貌限速调度) + v + +---------------------------------------+ + | bangwen-collector (Python) | + | - 声明式 TOML 站点卡片 (sources/) | + | - 定制化抓取适配器 (adapters/) | + | - SSRF 防护型 PoliteFetcher 客户端 | + +---------------------------------------+ + | + (HTTP POST /api/v1/ingest) + Bearer Token 鉴权通信 + v + +---------------------------------------+ + | bangwen-core (Rust) | + | - SimHash & 内容哈希多级排重管线 | + | - 多路混合检索与白盒简历打分内核 | + | - 纯原生内嵌 Web 控制台与静态资源 | + +---------------------------------------+ + | + (SQL 与向量存储) + v + +---------------------------------------+ + | PostgreSQL 16 + pgvector | + | - announcement 与 position 数据基表 | + | - pg_trgm 倒排与 HNSW 向量近似索引 | + +---------------------------------------+ +``` + +--- + +## 🚀 快速开始 + +### 环境准备 +- [Docker](https://docs.docker.com/engine/install/) 与 [Docker Compose](https://docs.docker.com/compose/) v2+ +- 本地原生开发环境（可选）： + - [Rust](https://rustup.rs/) (>= 1.80) + - [uv](https://docs.astral.sh/uv/) (>= 0.4.0) 与 Python 3.12+ + +### 安装与部署 + +1. **克隆代码仓库**： + ```bash + git clone git@gitea.o00o.men:bitfennec/bangwen.git + cd bangwen + ``` + +2. **初始化环境与鉴权凭据**： + ```bash + cp .env.example .env + # 自动生成 64 位高强度随机鉴权 Token + sed -i "s/replace_with_your_secure_secret_token/$(openssl rand -hex 32)/" .env + ``` + +3. **使用 Docker Compose 一键构建启动**： + ```bash + docker compose up -d --build + ``` + +### 验证运行 + +检查服务运行状态与初始入库指标： +```bash +curl -s http://127.0.0.1:8080/health +# 预期输出: {"status":"ok"} + +curl -s http://127.0.0.1:8080/api/v1/stats +# 预期输出: {"announcements":...,"embeddings":0,"positions":...,"sources":6,"status":"ok","vector_ready":false} +``` + +在浏览器中打开 `http://127.0.0.1:8080/` 即可直接进入内嵌 Web 控制台。 + +--- + +## 🛠️ 使用指南 + +### Web 控制台 +服务启动后直接监听 `http://127.0.0.1:8080/`： +- **公告 Feed 流**：按时间倒序分页展示各省市招考公告，支持全文实时一键折叠/展开、挂载结构化职位列表与官方附件直接下载。 +- **多维综合检索**：支持按关键词、标准行政区划编码、编制用人性质及最低学历门槛组合筛选。 +- **白盒简历匹配**：输入候选人基础画像，系统秒级匹配合规在招岗位，透明标明硬性满足与不符条款。 +- **每日聚合日报**：一键生成与复制过去 24 小时收录的招考精选与即将截止报名清单（Markdown 格式）。 +- **采集节点监控**：实时查看各边缘节点的连通健康度、认证态与最近上报统计。 + +### CLI 命令行工具 + +通过宿主机自带的原生穿透脚本 `./bin/bangwen` 或原生编译二进制进行操作： + +```bash +# 从文件或 stdin 管道摄入数据 +./bin/bangwen ingest --file ./sample.json + +# 多维检索符合条件的岗位 (例如玉溪 530400) +./bin/bangwen search "运维" --region 530400 --json + +# 查看公告或具体岗位详情 +./bin/bangwen show ann:1 --json +./bin/bangwen show pos:3 --json + +# 根据候选人画像执行白盒打分匹配 +./bin/bangwen match --resume ./tests/fixtures/sample_resume.json --top 10 + +# 输出今日招考聚合日报 +./bin/bangwen digest + +# 检查各边缘采集源的健康状态 +./bin/bangwen sources +``` + +### HTTP API 接口参考 + +| 端点路由 | 请求方式 | 功能简述 | 鉴权要求 | +|----------|----------|----------|----------| +| `/health` | `GET` | 核心服务与数据库连通性健康检查 | 无 | +| `/api/v1/ingest` | `POST` | 批量数据推送摄入接口 | Bearer Token | +| `/api/v1/stats` | `GET` | 系统汇总指标与向量就绪状态 | 无 | +| `/api/v1/sources` | `GET` | 采集节点健康度与最近入库明细 | 无 | +| `/api/v1/announcements` | `GET` | 公告分页列表（支持关键字与类型筛选） | 无 | +| `/api/v1/announcements/{id}` | `GET` | 单篇公告详情（含下属岗位与附件） | 无 | +| `/api/v1/positions/{id}` | `GET` | 细分结构化招考岗位详情 | 无 | +| `/api/v1/search` | `GET` | 混合关键字多路检索接口 | 无 | +| `/api/v1/match` | `POST` | 候选人简历白盒打分与核验接口 | 无 | +| `/api/v1/digest` | `GET` | 今日招考聚合日报（Markdown 格式） | 无 | + +--- + +## ⚙️ 环境配置 + +在 `.env` 中配置的环境变量说明： + +| 变量名 | 默认值 | 是否必填 | 说明 | +|--------|--------|----------|------| +| `DATABASE_URL` | `postgres://bangwen:bangwen_dev@postgres:5432/bangwen` | 是 | PostgreSQL + pgvector 数据库连接串 | +| `INGEST_TOKEN` | *空* | 推荐 | 用于保护 `/api/v1/ingest` 端点的 Bearer 鉴权密钥 | +| `SERVER_HOST` | `0.0.0.0` | 否 | Rust HTTP 服务监听地址 | +| `SERVER_PORT` | `8080` | 否 | Rust HTTP 服务监听端口 | +| `LOG_LEVEL` | `info` | 否 | 日志级别（`error`, `warn`, `info`, `debug`） | +| `OLLAMA_BASE_URL` | `http://host.docker.internal:11434` | 否 | 向量生成所用的 Ollama 服务地址 | +| `EMBEDDING_MODEL` | `bge-m3` | 否 | Ollama 向量模型名称 | + +--- + +## 🔒 安全加固说明 + +- **非 Root 容器降权**：`bangwen-core` 与 `bangwen-collector` 容器镜像均配置专用非特权运行用户（`UID 10001`）。 +- **严格 CSP 策略**：响应头强制 `default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline';`，彻底根除内联脚本注入执行风险。 +- **接口算力与内存保护**：对简历匹配接口增加完备的参数边界防护（技能最大 50 项、城市最大 20 项、学历上限 32 字符、出生年份 1940~当年区间强校验），深度分页 offset 上限 10000 强校验拦截。 +- **并发摄入防死锁**：数据落库事务内强制按照规范化 URL 字典序申请行级排他锁，彻底根除 PostgreSQL `40P01` 死锁异常。 +- **SSRF 与重定向逐跳校验**：HTTP 抓取客户端内置私有内网 IP 过滤、严格域名白名单与逐跳重定向安全性验证。 + +--- + +## 🤝 参与贡献 + +欢迎提交 Issue 与 Pull Request。提交改动前请确保通过本地基线检验： +```bash +# Rust 内核验证 +cargo test --offline +cargo clippy --all-targets -- -D warnings + +# 采集端回归验证 +cd collectors && uv run python tests/test_content_and_security.py +``` + +--- + +## 📜 开源协议 + +本项目采用 [MIT](LICENSE) 开源协议。 + +--- + +## 🤖 AI Agent 集成 + +针对作为 Agent 工具调用的集成场景，请参阅 [`skill/SKILL.md`](./skill/SKILL.md)，了解调用协议、结构化输出规范以及不可信数据的安全隔离要求。 + +

(回到顶部)

diff --git a/collectors/README.md b/collectors/README.md index 944f04e..f7ee3b9 100644 --- a/collectors/README.md +++ b/collectors/README.md @@ -1,25 +1,151 @@ -# Bangwen Collectors + -Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system. +# bangwen-collectors 🕷️ -## Structure +> **Lightweight, declarative, and SSRF-hardened edge collection system for public job recruitment aggregation.** -- `sources/`: Declarative TOML configuration for crawling target sites. -- `adapters/`: Custom Python scrapers for complex or stateful collection logic. -- `common/`: Core fetcher, security validators, and ingestion protocol models. -- `runner.py`: Periodic crawler runner and domain-concurrency scheduler. +--- -## Running +## 📖 Table of Contents +- [✨ Overview](#-overview) +- [📁 Project Structure](#-project-structure) +- [🚀 Quickstart](#-quickstart) + - [Prerequisites](#prerequisites) + - [Local Setup](#local-setup) +- [🛠️ Configuration & Source Authoring](#️-configuration--source-authoring) + - [Declarative TOML Sources](#declarative-toml-sources) + - [Custom Python Adapters](#custom-python-adapters) +- [⚙️ Scheduling & Rate Limiting](#️-scheduling--rate-limiting) +- [🧪 Testing](#-testing) +- [📜 License](#-license) -```bash -uv run python runner.py --list -uv run python runner.py --once +--- + +## ✨ Overview + +`bangwen-collectors` serves as the decentralized edge data extraction layer for the `bangwen` recruitment aggregation engine: +- **Dual-Mode Crawling**: Supports both zero-code declarative TOML site cards (`sources/`) and customized Python scraper adapters (`adapters/`). +- **Domain-Grouped Concurrency**: Concurrently processes distinct domains while enforcing strictly serial queues within the same domain. +- **WAF-Friendly Polite Delays**: Configurable inter-channel pauses (default 3.0s) and detail extraction throttles (0.3s) preventing IP bans. +- **SSRF Defense-in-Depth**: Built-in private IP filtering, strict domain whitelists, and hop-by-hop verification during HTTP redirects. + +--- + +## 📁 Project Structure + +``` +collectors/ +├── adapters/ # Specialized scrapers for complex dynamic sites +│ └── sc/chengdu_rsks.py # Custom adapter for Chengdu Personnel Exam Net +├── sources/ # Zero-code declarative TOML site cards +│ ├── sc/ # Sichuan Province sources (e.g. myrsks, zgrsks) +│ └── yn/ # Yunnan Province sources (e.g. hrss, yuxi) +├── common/ # Core modules +│ ├── content.py # Content truncation and dirty body cleanup +│ ├── declarative.py # Universal declarative parsing engine +│ ├── fetcher.py # PoliteFetcher HTTP client with SSRF verification +│ ├── protocol.py # Pydantic schema matching the bangwen/v1 protocol +│ └── security.py # URL validation and private IP guards +├── tests/ # Security and parsing regression tests +├── config.toml # Active collection and scheduler configuration +├── runner.py # Main scheduler and multi-domain orchestrator +└── Dockerfile # Dedicated non-root container definition ``` -## Testing +--- -Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host: +## 🚀 Quickstart + +### Prerequisites +- Python 3.12+ +- [uv](https://docs.astral.sh/uv/) package manager + +### Local Setup + +1. **Install dependencies**: + ```bash + uv sync + ``` + +2. **List available collectors**: + ```bash + uv run python runner.py --list + ``` + +3. **Run a single crawl cycle**: + ```bash + uv run python runner.py --once + ``` + +--- + +## 🛠️ Configuration & Source Authoring + +### Declarative TOML Sources + +Place zero-code TOML configuration cards under `sources/{province}/{site_id}.toml`: + +```toml +[site] +id = "yn.yuxi" +name = "玉溪市人民政府" +province = "云南省" +city = "玉溪市" +employer = "玉溪市相关用人单位" +employer_type = "事业单位" +allowed_domains = ["yuxi.gov.cn", "yxrc.cn"] + +[template] +item_selector = "div.news-list a" +url_pattern = "/article/\\d+\\.html" +title_selector = "h1.article-title" +body_selector = "div.article-content" +date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})" +attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"] + +[[channels]] +id = "rszk" +name = "人事招考" +start_url = "https://www.yuxi.gov.cn/rszk.html" +keywords = ["招聘", "考试", "录用", "公示"] +``` + +### Custom Python Adapters + +When target sites require complex session handling, cryptographic requests, or specific payload extraction, create a Python scraper under `adapters/{province}/{name}.py`: +- Implement a crawl function returning an `IngestPayload`. +- Use `common.security.is_safe_url` to enforce domain isolation. +- Sanitize extracted text with `common.content.sanitize_body_text`. + +--- + +## ⚙️ Scheduling & Rate Limiting + +The runner (`runner.py`) provides fine-grained scheduling controls: + +| Environment Variable / Flag | Default | Description | +|-----------------------------|---------|-------------| +| `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Target Rust gateway ingestion endpoint | +| `INGEST_TOKEN` | *None* | Bearer authentication token matching core `.env` | +| `SCHEDULE_INTERVAL_HOURS` | `6` | Cycle interval between automated sweeps | +| `--provinces` | `[]` | Filter target provinces (e.g. `yn,sc`) | +| `--concurrency` | `3` | Maximum concurrent distinct domain scrapers | + +--- + +## 🧪 Testing + +Comprehensive test suites covering SSRF prevention, HTML sanitization, and URL security: ```bash -cd collectors && uv run python tests/test_content_and_security.py +uv run python tests/test_content_and_security.py +# Expected output: All security and content tests passed! ``` + +--- + +## 📜 License + +[MIT](../LICENSE) + +

(back to top)

diff --git a/collectors/README.zh-CN.md b/collectors/README.zh-CN.md new file mode 100644 index 0000000..46f00c8 --- /dev/null +++ b/collectors/README.zh-CN.md @@ -0,0 +1,151 @@ + + +# bangwen-collectors 边缘采集系统 🕷️ + +> **专为招考数据聚合设计的高性能、免代码声明式与 SSRF 安全强化的边缘抓取体系。** + +--- + +## 📖 目录 +- [✨ 概述](#-概述) +- [📁 目录结构](#-目录结构) +- [🚀 快速开始](#-快速开始) + - [环境准备](#环境准备) + - [本地安装](#本地安装) +- [🛠️ 采集器配置与开发指南](#️-采集器配置与开发指南) + - [免代码声明式 TOML 站点卡片](#免代码声明式-toml-站点卡片) + - [复杂场景 Python 定制适配器](#复杂场景-python-定制适配器) +- [⚙️ 调度策略与频控机制](#️-调度策略与频控机制) +- [🧪 测试与验证](#-测试与验证) +- [📜 开源协议](#-开源协议) + +--- + +## ✨ 概述 + +`bangwen-collectors` 作为 `bangwen` 招考数据中枢的分布式边缘抽取层： +- **双模自动发现**：同时支持零代码声明式 TOML 站点配置卡片（`sources/`）与 Python 定制适配器（`adapters/`）。 +- **按域名分组并发**：对不同域名并发抓取，而对相同域名严格排队串行，保障采集效率的同时彻底杜绝政务网 WAF 封锁。 +- **礼貌延时与节流**：栏目间保持静默等待（默认 3.0s），单篇详情提取保留 0.3s 间隔微节流。 +- **全链路 SSRF 纵深防护**：底层内置私有内网 IP 阻断、严格域名白名单匹配与 HTTP 重定向逐跳安全性校验。 + +--- + +## 📁 目录结构 + +``` +collectors/ +├── adapters/ # 针对复杂动态站点的 Python 专属抓取适配器 +│ └── sc/chengdu_rsks.py # 成都人事考试网抓取适配器 +├── sources/ # 零代码声明式 TOML 站点配置卡片 +│ ├── sc/ # 四川省站点（如绵阳人事考试网、自贡人事考试网） +│ └── yn/ # 云南省站点（如云南人社厅、玉溪市人民政府） +├── common/ # 核心通用组件 +│ ├── content.py # 脏正文清洗、超长文本截断与反爬标记识别 +│ ├── declarative.py # 通用免代码声明式解析执行引擎 +│ ├── fetcher.py # 支持 SSRF 逐跳校验与限速的 PoliteFetcher 客户端 +│ ├── protocol.py # 与 bangwen/v1 摄入协议严格对齐的 Pydantic 模型 +│ └── security.py # URL 白名单判定与内网 IP 阻断工具 +├── tests/ # 安全防护与内容清洗回归测试集 +├── config.toml # 采集调度与省份过滤总配置文件 +├── runner.py # 多域名调度器与常驻运行入口 +└── Dockerfile # 专属非 root 用户（UID 10001）容器镜像定义 +``` + +--- + +## 🚀 快速开始 + +### 环境准备 +- Python 3.12+ +- [uv](https://docs.astral.sh/uv/) 包管理器 + +### 本地安装 + +1. **安装依赖**： + ```bash + uv sync + ``` + +2. **列出当前注册的所有采集渠道**： + ```bash + uv run python runner.py --list + ``` + +3. **执行一轮即时抓取与推流**： + ```bash + uv run python runner.py --once + ``` + +--- + +## 🛠️ 采集器配置与开发指南 + +### 免代码声明式 TOML 站点卡片 + +对于标准 HTML 结构的政务招考网站，只需在 `sources/{province}/{site_id}.toml` 编写配置，无需编写任何代码： + +```toml +[site] +id = "yn.yuxi" +name = "玉溪市人民政府" +province = "云南省" +city = "玉溪市" +employer = "玉溪市相关用人单位" +employer_type = "事业单位" +allowed_domains = ["yuxi.gov.cn", "yxrc.cn"] + +[template] +item_selector = "div.news-list a" +url_pattern = "/article/\\d+\\.html" +title_selector = "h1.article-title" +body_selector = "div.article-content" +date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})" +attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"] + +[[channels]] +id = "rszk" +name = "人事招考" +start_url = "https://www.yuxi.gov.cn/rszk.html" +keywords = ["招聘", "考试", "录用", "公示"] +``` + +### 复杂场景 Python 定制适配器 + +对于含有复杂会话保持、加密参数或动态分页的站点，在 `adapters/{province}/{name}.py` 编写 Python 适配器： +- 实现抓取函数并返回符合标准的 `IngestPayload`。 +- 强制使用 `common.security.is_safe_url` 实施外链与重定向安全隔离。 +- 提取正文时使用 `common.content.sanitize_body_text` 剥离导航与脏文本。 + +--- + +## ⚙️ 调度策略与频控机制 + +调度器（`runner.py`）提供细粒度的运行控制： + +| 环境变量 / 命令行参数 | 默认值 | 说明 | +|-----------------------|--------|------| +| `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Rust 核心网关的摄入接口地址 | +| `INGEST_TOKEN` | *空* | 与内核对齐的 HTTP Bearer 认证 Token | +| `SCHEDULE_INTERVAL_HOURS` | `6` | 常驻巡检的周期时间（小时） | +| `--provinces` | `[]` | 目标省份过滤（例如 `yn,sc`） | +| `--concurrency` | `3` | 跨域名的最大并发数 | + +--- + +## 🧪 测试与验证 + +运行完备的本地安全回归套件（涵盖 SSRF 私网阻断、反爬脏正文过滤与附件扩展名判定）： + +```bash +uv run python tests/test_content_and_security.py +# 预期输出: All security and content tests passed! +``` + +--- + +## 📜 开源协议 + +本项目采用 [MIT](../LICENSE) 开源协议。 + +

(回到顶部)