bangwen/README.md

bangwen (榜文) 🚀

High-performance public job recruitment aggregation, hybrid retrieval, and white-box resume matching engine designed with VictoriaMetrics architectural principles.

---

📖 Table of Contents

• ✨ Features
• 🏛️ Architecture
• 🚀 Quickstart
  • Prerequisites
  • Installation & Deployment
  • Verification
• 🛠️ Usage
  • Web Console
  • CLI Commands
  • HTTP API Reference
• ⚙️ Configuration
• 🔒 Security & Hardening
• 🤝 Contributing
• 📜 License
• 🤖 For AI Agents

---

✨ Features

• ⚡ VictoriaMetrics-Inspired Simplicity: Single lightweight Rust binary acting as an ingestion gateway, hybrid search engine, and embedded web console without bloated microservices.
• 🛡️ Multi-Tier Deduplication Pipeline: Three-tier deduplication combining normalized URL canonicalization, 16-character content hashing, and 64-bit Hamming distance SimHash with short-text (<200 char) false-positive protection.
• 🎯 White-Box Resume Matching: Fully objective candidate scoring based on hard criteria (education levels mapped via standard dictionaries, age limits, region codes, and exact major/skill matches) with transparent audit reasons.
• 🔍 Hybrid Search Engine: Integrates exact SQL filtering, PostgreSQL pg_trgm GIN fuzzy matching, and pgvector HNSW cosine similarity with Reciprocal Rank Fusion (RRF).
• 🕸️ Dual-Mode Edge Collectors: Autonomous Python edge scrapers supporting declarative TOML site templates and specialized Python adapters with domain-level concurrency and polite throttles.
• 🔒 Hardened Security Surface:
  • Strict Content Security Policy (script-src 'self' with zero 'unsafe-inline').
  • Terminal sanitization stripping ANSI escape sequences, OSC 52, and control characters from all outputs.
  • Markdown injection prevention and SSRF hop-by-hop redirect verification.
  • Dedicated non-root container user execution (uid=10001).

---

🏛️ Architecture

                      +---------------------------------------+
                      |       Edge Public Recruitment Sites    |
                      +---------------------------------------+
                                          |
                      (HTTP Scraping & Polite Rate Limiting)
                                          v
                      +---------------------------------------+
                      |      bangwen-collector (Python)       |
                      |  - Declarative TOML (sources/)        |
                      |  - Custom Adapters (adapters/)        |
                      |  - SSRF-safe PoliteFetcher            |
                      +---------------------------------------+
                                          |
                              (HTTP POST /api/v1/ingest)
                           Bearer Token Authenticated
                                          v
                      +---------------------------------------+
                      |          bangwen-core (Rust)          |
                      |  - SimHash & URL Hash Deduplication   |
                      |  - Hybrid Search & White-box Matcher  |
                      |  - Embedded Web Console & Assets      |
                      +---------------------------------------+
                                          |
                               (SQL & Vector Storage)
                                          v
                      +---------------------------------------+
                      |         PostgreSQL 16 + pgvector      |
                      |  - announcement & position tables     |
                      |  - pg_trgm & HNSW vector indexes      |
                      +---------------------------------------+

---

🚀 Quickstart

Prerequisites

• Docker & Docker Compose v2+
• Optional for native development:
  • Rust (>= 1.80)
  • uv (>= 0.4.0) and Python 3.12+

Installation & Deployment

1. Clone the repository:

   git clone [email protected]:bitfennec/bangwen.git
   cd bangwen
   

2. Initialize environment credentials:

   cp .env.example .env
   # Generate a secure 64-character token
   sed -i "s/replace_with_your_secure_secret_token/$(openssl rand -hex 32)/" .env
   

3. Start services with Docker Compose:

   docker compose up -d --build
   

Verification

Check service health and initial ingestion metrics:

curl -s http://127.0.0.1:8080/health
# Expected: {"status":"ok"}

curl -s http://127.0.0.1:8080/api/v1/stats
# Expected: {"announcements":...,"embeddings":0,"positions":...,"sources":6,"status":"ok","vector_ready":false}

Open http://127.0.0.1:8080/ in your browser to inspect the Web Console.

---

🛠️ Usage

Web Console

The built-in web console is served directly on http://127.0.0.1:8080/:

• Feed Stream: Paginated real-time recruitment notices with collapsible bodies, parsed position tables, and official attachment links.
• Multi-Filter Search: Search by keywords, administrative region code, employer type, and minimum education requirements.
• Candidate White-Box Match: Fill candidate resume attributes to calculate match scores and view clear transparent eligibility lists.
• Daily Digest: View generated markdown summaries of the past 24 hours.
• Collector Monitor: Health, authentication status, and last report timestamp for all edge collectors.

CLI Commands

When running natively or using the provided wrapper ./bin/bangwen:

# Ingest data via stdin or file
./bin/bangwen ingest --file ./sample.json

# Hybrid search recruitment posts
./bin/bangwen search "运维" --region 530400 --json

# Inspect notice or position details
./bin/bangwen show ann:1 --json
./bin/bangwen show pos:3 --json

# Run white-box resume matching
./bin/bangwen match --resume ./tests/fixtures/sample_resume.json --top 10

# Generate daily markdown digest
./bin/bangwen digest

# Inspect collection sources health
./bin/bangwen sources

HTTP API Reference

Endpoint	Method	Description	Auth Required
/health	GET	Service & database health check	No
/api/v1/ingest	POST	Ingest batched notices & positions	Bearer Token
/api/v1/stats	GET	Aggregation statistics and vector status	No
/api/v1/sources	GET	Source nodes health and last ingest report	No
/api/v1/announcements	GET	Paginated recruitment announcements list	No
/api/v1/announcements/{id}	GET	Detailed announcement with positions & attachments	No
/api/v1/positions/{id}	GET	Detailed structured job position	No
/api/v1/search	GET	Multi-dimensional hybrid keyword search	No
/api/v1/match	POST	Strict white-box resume qualification match	No
/api/v1/digest	GET	Daily markdown recruitment digest	No

---

⚙️ Configuration

Environment variables configured in .env:

Variable	Default	Required	Description
DATABASE_URL	postgres://bangwen:bangwen_dev@postgres:5432/bangwen	Yes	PostgreSQL connection string with pgvector
INGEST_TOKEN	None	Recommended	Bearer token for authenticating /api/v1/ingest
SERVER_HOST	0.0.0.0	No	Host address for Rust HTTP gateway
SERVER_PORT	8080	No	Port for Rust HTTP gateway
LOG_LEVEL	info	No	Tracing log level (error, warn, info, debug)
OLLAMA_BASE_URL	http://host.docker.internal:11434	No	Optional Ollama instance for embedding generation
EMBEDDING_MODEL	bge-m3	No	Ollama embedding model name

---

🔒 Security & Hardening

• Non-Root Execution: Both bangwen-core and bangwen-collector containers enforce explicit non-root user execution (UID 10001).
• Strict Content Security Policy (CSP): default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; frame-ancestors 'none';. Inline scripts are completely eliminated.
• Input Boundary Bounds: Bounded validation for match requests (max 50 skills, 20 cities, 32-char education, 1940..=current birth year) and pagination limits (max offset 10000).
• Anti-Deadlock Lock Ordering: Ingestion pipeline enforces deterministic URL key sorting prior to row lock acquisition, eliminating PostgreSQL 40P01 deadlock conditions.
• SSRF & Scheme Verification: PoliteFetcher enforces private IP filtering, strict domain whitelists, and hop-by-hop validation during HTTP redirects.

---

🤝 Contributing

Contributions are welcome. Please ensure all modifications pass local validation:

# Rust core checks
cargo test --offline
cargo clippy --all-targets -- -D warnings

# Collectors checks
cd collectors && uv run python tests/test_content_and_security.py

---

📜 License

MIT

---

🤖 For AI Agents

See skill/SKILL.md for operational instructions and security boundary guidelines.

(back to top)