bitfennec
5a8af5729c
- Add English README.md and Chinese README.zh-CN.md for project root - Enhance collectors/README.md and add Chinese collectors/README.zh-CN.md - Document architecture, deployment, security invariants, and CLI/API usage
226 lines
9.4 KiB
Markdown
226 lines
9.4 KiB
Markdown
<a id="readme-top"></a>
|
||
|
||
# bangwen (榜文) 🚀
|
||
|
||
> **High-performance public job recruitment aggregation, hybrid retrieval, and white-box resume matching engine designed with VictoriaMetrics architectural principles.**
|
||
|
||
---
|
||
|
||
## 📖 Table of Contents
|
||
- [✨ Features](#-features)
|
||
- [🏛️ Architecture](#️-architecture)
|
||
- [🚀 Quickstart](#-quickstart)
|
||
- [Prerequisites](#prerequisites)
|
||
- [Installation & Deployment](#installation--deployment)
|
||
- [Verification](#verification)
|
||
- [🛠️ Usage](#️-usage)
|
||
- [Web Console](#web-console)
|
||
- [CLI Commands](#cli-commands)
|
||
- [HTTP API Reference](#http-api-reference)
|
||
- [⚙️ Configuration](#️-configuration)
|
||
- [🔒 Security & Hardening](#-security--hardening)
|
||
- [🤝 Contributing](#-contributing)
|
||
- [📜 License](#-license)
|
||
- [🤖 For AI Agents](#-for-ai-agents)
|
||
|
||
---
|
||
|
||
## ✨ Features
|
||
|
||
- **⚡ VictoriaMetrics-Inspired Simplicity**: Single lightweight Rust binary acting as an ingestion gateway, hybrid search engine, and embedded web console without bloated microservices.
|
||
- **🛡️ Multi-Tier Deduplication Pipeline**: Three-tier deduplication combining normalized URL canonicalization, 16-character content hashing, and 64-bit Hamming distance SimHash with short-text (<200 char) false-positive protection.
|
||
- **🎯 White-Box Resume Matching**: Fully objective candidate scoring based on hard criteria (education levels mapped via standard dictionaries, age limits, region codes, and exact major/skill matches) with transparent audit reasons.
|
||
- **🔍 Hybrid Search Engine**: Integrates exact SQL filtering, PostgreSQL `pg_trgm` GIN fuzzy matching, and `pgvector` HNSW cosine similarity with Reciprocal Rank Fusion (RRF).
|
||
- **🕸️ Dual-Mode Edge Collectors**: Autonomous Python edge scrapers supporting declarative TOML site templates and specialized Python adapters with domain-level concurrency and polite throttles.
|
||
- **🔒 Hardened Security Surface**:
|
||
- Strict Content Security Policy (`script-src 'self'` with zero `'unsafe-inline'`).
|
||
- Terminal sanitization stripping ANSI escape sequences, OSC 52, and control characters from all outputs.
|
||
- Markdown injection prevention and SSRF hop-by-hop redirect verification.
|
||
- Dedicated non-root container user execution (`uid=10001`).
|
||
|
||
---
|
||
|
||
## 🏛️ Architecture
|
||
|
||
```
|
||
+---------------------------------------+
|
||
| Edge Public Recruitment Sites |
|
||
+---------------------------------------+
|
||
|
|
||
(HTTP Scraping & Polite Rate Limiting)
|
||
v
|
||
+---------------------------------------+
|
||
| bangwen-collector (Python) |
|
||
| - Declarative TOML (sources/) |
|
||
| - Custom Adapters (adapters/) |
|
||
| - SSRF-safe PoliteFetcher |
|
||
+---------------------------------------+
|
||
|
|
||
(HTTP POST /api/v1/ingest)
|
||
Bearer Token Authenticated
|
||
v
|
||
+---------------------------------------+
|
||
| bangwen-core (Rust) |
|
||
| - SimHash & URL Hash Deduplication |
|
||
| - Hybrid Search & White-box Matcher |
|
||
| - Embedded Web Console & Assets |
|
||
+---------------------------------------+
|
||
|
|
||
(SQL & Vector Storage)
|
||
v
|
||
+---------------------------------------+
|
||
| PostgreSQL 16 + pgvector |
|
||
| - announcement & position tables |
|
||
| - pg_trgm & HNSW vector indexes |
|
||
+---------------------------------------+
|
||
```
|
||
|
||
---
|
||
|
||
## 🚀 Quickstart
|
||
|
||
### Prerequisites
|
||
- [Docker](https://docs.docker.com/engine/install/) & [Docker Compose](https://docs.docker.com/compose/) v2+
|
||
- Optional for native development:
|
||
- [Rust](https://rustup.rs/) (>= 1.80)
|
||
- [uv](https://docs.astral.sh/uv/) (>= 0.4.0) and Python 3.12+
|
||
|
||
### Installation & Deployment
|
||
|
||
1. **Clone the repository**:
|
||
```bash
|
||
git clone [email protected]:bitfennec/bangwen.git
|
||
cd bangwen
|
||
```
|
||
|
||
2. **Initialize environment credentials**:
|
||
```bash
|
||
cp .env.example .env
|
||
# Generate a secure 64-character token
|
||
sed -i "s/replace_with_your_secure_secret_token/$(openssl rand -hex 32)/" .env
|
||
```
|
||
|
||
3. **Start services with Docker Compose**:
|
||
```bash
|
||
docker compose up -d --build
|
||
```
|
||
|
||
### Verification
|
||
|
||
Check service health and initial ingestion metrics:
|
||
```bash
|
||
curl -s http://127.0.0.1:8080/health
|
||
# Expected: {"status":"ok"}
|
||
|
||
curl -s http://127.0.0.1:8080/api/v1/stats
|
||
# Expected: {"announcements":...,"embeddings":0,"positions":...,"sources":6,"status":"ok","vector_ready":false}
|
||
```
|
||
|
||
Open `http://127.0.0.1:8080/` in your browser to inspect the Web Console.
|
||
|
||
---
|
||
|
||
## 🛠️ Usage
|
||
|
||
### Web Console
|
||
The built-in web console is served directly on `http://127.0.0.1:8080/`:
|
||
- **Feed Stream**: Paginated real-time recruitment notices with collapsible bodies, parsed position tables, and official attachment links.
|
||
- **Multi-Filter Search**: Search by keywords, administrative region code, employer type, and minimum education requirements.
|
||
- **Candidate White-Box Match**: Fill candidate resume attributes to calculate match scores and view clear transparent eligibility lists.
|
||
- **Daily Digest**: View generated markdown summaries of the past 24 hours.
|
||
- **Collector Monitor**: Health, authentication status, and last report timestamp for all edge collectors.
|
||
|
||
### CLI Commands
|
||
|
||
When running natively or using the provided wrapper `./bin/bangwen`:
|
||
|
||
```bash
|
||
# Ingest data via stdin or file
|
||
./bin/bangwen ingest --file ./sample.json
|
||
|
||
# Hybrid search recruitment posts
|
||
./bin/bangwen search "运维" --region 530400 --json
|
||
|
||
# Inspect notice or position details
|
||
./bin/bangwen show ann:1 --json
|
||
./bin/bangwen show pos:3 --json
|
||
|
||
# Run white-box resume matching
|
||
./bin/bangwen match --resume ./tests/fixtures/sample_resume.json --top 10
|
||
|
||
# Generate daily markdown digest
|
||
./bin/bangwen digest
|
||
|
||
# Inspect collection sources health
|
||
./bin/bangwen sources
|
||
```
|
||
|
||
### HTTP API Reference
|
||
|
||
| Endpoint | Method | Description | Auth Required |
|
||
|----------|--------|-------------|---------------|
|
||
| `/health` | `GET` | Service & database health check | No |
|
||
| `/api/v1/ingest` | `POST` | Ingest batched notices & positions | Bearer Token |
|
||
| `/api/v1/stats` | `GET` | Aggregation statistics and vector status | No |
|
||
| `/api/v1/sources` | `GET` | Source nodes health and last ingest report | No |
|
||
| `/api/v1/announcements` | `GET` | Paginated recruitment announcements list | No |
|
||
| `/api/v1/announcements/{id}` | `GET` | Detailed announcement with positions & attachments | No |
|
||
| `/api/v1/positions/{id}` | `GET` | Detailed structured job position | No |
|
||
| `/api/v1/search` | `GET` | Multi-dimensional hybrid keyword search | No |
|
||
| `/api/v1/match` | `POST` | Strict white-box resume qualification match | No |
|
||
| `/api/v1/digest` | `GET` | Daily markdown recruitment digest | No |
|
||
|
||
---
|
||
|
||
## ⚙️ Configuration
|
||
|
||
Environment variables configured in `.env`:
|
||
|
||
| Variable | Default | Required | Description |
|
||
|----------|---------|----------|-------------|
|
||
| `DATABASE_URL` | `postgres://bangwen:bangwen_dev@postgres:5432/bangwen` | Yes | PostgreSQL connection string with pgvector |
|
||
| `INGEST_TOKEN` | *None* | Recommended | Bearer token for authenticating `/api/v1/ingest` |
|
||
| `SERVER_HOST` | `0.0.0.0` | No | Host address for Rust HTTP gateway |
|
||
| `SERVER_PORT` | `8080` | No | Port for Rust HTTP gateway |
|
||
| `LOG_LEVEL` | `info` | No | Tracing log level (`error`, `warn`, `info`, `debug`) |
|
||
| `OLLAMA_BASE_URL` | `http://host.docker.internal:11434` | No | Optional Ollama instance for embedding generation |
|
||
| `EMBEDDING_MODEL` | `bge-m3` | No | Ollama embedding model name |
|
||
|
||
---
|
||
|
||
## 🔒 Security & Hardening
|
||
|
||
- **Non-Root Execution**: Both `bangwen-core` and `bangwen-collector` containers enforce explicit non-root user execution (`UID 10001`).
|
||
- **Strict Content Security Policy (CSP)**: `default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; frame-ancestors 'none';`. Inline scripts are completely eliminated.
|
||
- **Input Boundary Bounds**: Bounded validation for match requests (max 50 skills, 20 cities, 32-char education, 1940..=current birth year) and pagination limits (max offset 10000).
|
||
- **Anti-Deadlock Lock Ordering**: Ingestion pipeline enforces deterministic URL key sorting prior to row lock acquisition, eliminating PostgreSQL `40P01` deadlock conditions.
|
||
- **SSRF & Scheme Verification**: PoliteFetcher enforces private IP filtering, strict domain whitelists, and hop-by-hop validation during HTTP redirects.
|
||
|
||
---
|
||
|
||
## 🤝 Contributing
|
||
|
||
Contributions are welcome. Please ensure all modifications pass local validation:
|
||
```bash
|
||
# Rust core checks
|
||
cargo test --offline
|
||
cargo clippy --all-targets -- -D warnings
|
||
|
||
# Collectors checks
|
||
cd collectors && uv run python tests/test_content_and_security.py
|
||
```
|
||
|
||
---
|
||
|
||
## 📜 License
|
||
|
||
[MIT](LICENSE)
|
||
|
||
---
|
||
|
||
## 🤖 For AI Agents
|
||
|
||
See [`skill/SKILL.md`](./skill/SKILL.md) for operational instructions and security boundary guidelines.
|
||
|
||
<p align="right">(<a href="#readme-top">back to top</a>)</p>
|