docs: add bilingual README documentation for core and collectors
- Add English README.md and Chinese README.zh-CN.md for project root - Enhance collectors/README.md and add Chinese collectors/README.zh-CN.md - Document architecture, deployment, security invariants, and CLI/API usage
This commit is contained in:
bitfennec
+140
-14
@@ -1,25 +1,151 @@
|
||||
# Bangwen Collectors
|
||||
<a id="readme-top"></a>
|
||||
|
||||
Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.
|
||||
# bangwen-collectors 🕷️
|
||||
|
||||
## Structure
|
||||
> **Lightweight, declarative, and SSRF-hardened edge collection system for public job recruitment aggregation.**
|
||||
|
||||
- `sources/`: Declarative TOML configuration for crawling target sites.
|
||||
- `adapters/`: Custom Python scrapers for complex or stateful collection logic.
|
||||
- `common/`: Core fetcher, security validators, and ingestion protocol models.
|
||||
- `runner.py`: Periodic crawler runner and domain-concurrency scheduler.
|
||||
---
|
||||
|
||||
## Running
|
||||
## 📖 Table of Contents
|
||||
- [✨ Overview](#-overview)
|
||||
- [📁 Project Structure](#-project-structure)
|
||||
- [🚀 Quickstart](#-quickstart)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Local Setup](#local-setup)
|
||||
- [🛠️ Configuration & Source Authoring](#️-configuration--source-authoring)
|
||||
- [Declarative TOML Sources](#declarative-toml-sources)
|
||||
- [Custom Python Adapters](#custom-python-adapters)
|
||||
- [⚙️ Scheduling & Rate Limiting](#️-scheduling--rate-limiting)
|
||||
- [🧪 Testing](#-testing)
|
||||
- [📜 License](#-license)
|
||||
|
||||
```bash
|
||||
uv run python runner.py --list
|
||||
uv run python runner.py --once
|
||||
---
|
||||
|
||||
## ✨ Overview
|
||||
|
||||
`bangwen-collectors` serves as the decentralized edge data extraction layer for the `bangwen` recruitment aggregation engine:
|
||||
- **Dual-Mode Crawling**: Supports both zero-code declarative TOML site cards (`sources/`) and customized Python scraper adapters (`adapters/`).
|
||||
- **Domain-Grouped Concurrency**: Concurrently processes distinct domains while enforcing strictly serial queues within the same domain.
|
||||
- **WAF-Friendly Polite Delays**: Configurable inter-channel pauses (default 3.0s) and detail extraction throttles (0.3s) preventing IP bans.
|
||||
- **SSRF Defense-in-Depth**: Built-in private IP filtering, strict domain whitelists, and hop-by-hop verification during HTTP redirects.
|
||||
|
||||
---
|
||||
|
||||
## 📁 Project Structure
|
||||
|
||||
```
|
||||
collectors/
|
||||
├── adapters/ # Specialized scrapers for complex dynamic sites
|
||||
│ └── sc/chengdu_rsks.py # Custom adapter for Chengdu Personnel Exam Net
|
||||
├── sources/ # Zero-code declarative TOML site cards
|
||||
│ ├── sc/ # Sichuan Province sources (e.g. myrsks, zgrsks)
|
||||
│ └── yn/ # Yunnan Province sources (e.g. hrss, yuxi)
|
||||
├── common/ # Core modules
|
||||
│ ├── content.py # Content truncation and dirty body cleanup
|
||||
│ ├── declarative.py # Universal declarative parsing engine
|
||||
│ ├── fetcher.py # PoliteFetcher HTTP client with SSRF verification
|
||||
│ ├── protocol.py # Pydantic schema matching the bangwen/v1 protocol
|
||||
│ └── security.py # URL validation and private IP guards
|
||||
├── tests/ # Security and parsing regression tests
|
||||
├── config.toml # Active collection and scheduler configuration
|
||||
├── runner.py # Main scheduler and multi-domain orchestrator
|
||||
└── Dockerfile # Dedicated non-root container definition
|
||||
```
|
||||
|
||||
## Testing
|
||||
---
|
||||
|
||||
Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:
|
||||
## 🚀 Quickstart
|
||||
|
||||
### Prerequisites
|
||||
- Python 3.12+
|
||||
- [uv](https://docs.astral.sh/uv/) package manager
|
||||
|
||||
### Local Setup
|
||||
|
||||
1. **Install dependencies**:
|
||||
```bash
|
||||
uv sync
|
||||
```
|
||||
|
||||
2. **List available collectors**:
|
||||
```bash
|
||||
uv run python runner.py --list
|
||||
```
|
||||
|
||||
3. **Run a single crawl cycle**:
|
||||
```bash
|
||||
uv run python runner.py --once
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Configuration & Source Authoring
|
||||
|
||||
### Declarative TOML Sources
|
||||
|
||||
Place zero-code TOML configuration cards under `sources/{province}/{site_id}.toml`:
|
||||
|
||||
```toml
|
||||
[site]
|
||||
id = "yn.yuxi"
|
||||
name = "玉溪市人民政府"
|
||||
province = "云南省"
|
||||
city = "玉溪市"
|
||||
employer = "玉溪市相关用人单位"
|
||||
employer_type = "事业单位"
|
||||
allowed_domains = ["yuxi.gov.cn", "yxrc.cn"]
|
||||
|
||||
[template]
|
||||
item_selector = "div.news-list a"
|
||||
url_pattern = "/article/\\d+\\.html"
|
||||
title_selector = "h1.article-title"
|
||||
body_selector = "div.article-content"
|
||||
date_regex = "(\\d{4}[-/年]\\d{1,2}[-/月]\\d{1,2})"
|
||||
attachment_extensions = [".xlsx", ".xls", ".pdf", ".docx", ".doc", ".zip"]
|
||||
|
||||
[[channels]]
|
||||
id = "rszk"
|
||||
name = "人事招考"
|
||||
start_url = "https://www.yuxi.gov.cn/rszk.html"
|
||||
keywords = ["招聘", "考试", "录用", "公示"]
|
||||
```
|
||||
|
||||
### Custom Python Adapters
|
||||
|
||||
When target sites require complex session handling, cryptographic requests, or specific payload extraction, create a Python scraper under `adapters/{province}/{name}.py`:
|
||||
- Implement a crawl function returning an `IngestPayload`.
|
||||
- Use `common.security.is_safe_url` to enforce domain isolation.
|
||||
- Sanitize extracted text with `common.content.sanitize_body_text`.
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Scheduling & Rate Limiting
|
||||
|
||||
The runner (`runner.py`) provides fine-grained scheduling controls:
|
||||
|
||||
| Environment Variable / Flag | Default | Description |
|
||||
|-----------------------------|---------|-------------|
|
||||
| `CORE_INGEST_URL` | `http://core:8080/api/v1/ingest` | Target Rust gateway ingestion endpoint |
|
||||
| `INGEST_TOKEN` | *None* | Bearer authentication token matching core `.env` |
|
||||
| `SCHEDULE_INTERVAL_HOURS` | `6` | Cycle interval between automated sweeps |
|
||||
| `--provinces` | `[]` | Filter target provinces (e.g. `yn,sc`) |
|
||||
| `--concurrency` | `3` | Maximum concurrent distinct domain scrapers |
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
Comprehensive test suites covering SSRF prevention, HTML sanitization, and URL security:
|
||||
|
||||
```bash
|
||||
cd collectors && uv run python tests/test_content_and_security.py
|
||||
uv run python tests/test_content_and_security.py
|
||||
# Expected output: All security and content tests passed!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📜 License
|
||||
|
||||
[MIT](../LICENSE)
|
||||
|
||||
<p align="right">(<a href="#readme-top">back to top</a>)</p>
|
||||
|
||||
Reference in New Issue
Block a user