# bangwen (ζ¦ζ) π
> **High-performance public job recruitment aggregation, hybrid retrieval, and white-box resume matching engine designed with VictoriaMetrics architectural principles.**
---
## π Table of Contents
- [β¨ Features](#-features)
- [ποΈ Architecture](#οΈ-architecture)
- [π Quickstart](#-quickstart)
- [Prerequisites](#prerequisites)
- [Installation & Deployment](#installation--deployment)
- [Verification](#verification)
- [π οΈ Usage](#οΈ-usage)
- [Web Console](#web-console)
- [CLI Commands](#cli-commands)
- [HTTP API Reference](#http-api-reference)
- [βοΈ Configuration](#οΈ-configuration)
- [π Security & Hardening](#-security--hardening)
- [π€ Contributing](#-contributing)
- [π License](#-license)
- [π€ For AI Agents](#-for-ai-agents)
---
## β¨ Features
- **β‘ VictoriaMetrics-Inspired Simplicity**: Single lightweight Rust binary acting as an ingestion gateway, hybrid search engine, and embedded web console without bloated microservices.
- **π‘οΈ Multi-Tier Deduplication Pipeline**: Three-tier deduplication combining normalized URL canonicalization, 16-character content hashing, and 64-bit Hamming distance SimHash with short-text (<200 char) false-positive protection.
- **π― White-Box Resume Matching**: Fully objective candidate scoring based on hard criteria (education levels mapped via standard dictionaries, age limits, region codes, and exact major/skill matches) with transparent audit reasons.
- **π Hybrid Search Engine**: Integrates exact SQL filtering, PostgreSQL `pg_trgm` GIN fuzzy matching, and `pgvector` HNSW cosine similarity with Reciprocal Rank Fusion (RRF).
- **πΈοΈ Dual-Mode Edge Collectors**: Autonomous Python edge scrapers supporting declarative TOML site templates and specialized Python adapters with domain-level concurrency and polite throttles.
- **π Hardened Security Surface**:
- Strict Content Security Policy (`script-src 'self'` with zero `'unsafe-inline'`).
- Terminal sanitization stripping ANSI escape sequences, OSC 52, and control characters from all outputs.
- Markdown injection prevention and SSRF hop-by-hop redirect verification.
- Dedicated non-root container user execution (`uid=10001`).
---
## ποΈ Architecture
```
+---------------------------------------+
| Edge Public Recruitment Sites |
+---------------------------------------+
|
(HTTP Scraping & Polite Rate Limiting)
v
+---------------------------------------+
| bangwen-collector (Python) |
| - Declarative TOML (sources/) |
| - Custom Adapters (adapters/) |
| - SSRF-safe PoliteFetcher |
+---------------------------------------+
|
(HTTP POST /api/v1/ingest)
Bearer Token Authenticated
v
+---------------------------------------+
| bangwen-core (Rust) |
| - SimHash & URL Hash Deduplication |
| - Hybrid Search & White-box Matcher |
| - Embedded Web Console & Assets |
+---------------------------------------+
|
(SQL & Vector Storage)
v
+---------------------------------------+
| PostgreSQL 16 + pgvector |
| - announcement & position tables |
| - pg_trgm & HNSW vector indexes |
+---------------------------------------+
```
---
## π Quickstart
### Prerequisites
- [Docker](https://docs.docker.com/engine/install/) & [Docker Compose](https://docs.docker.com/compose/) v2+
- Optional for native development:
- [Rust](https://rustup.rs/) (>= 1.80)
- [uv](https://docs.astral.sh/uv/) (>= 0.4.0) and Python 3.12+
### Installation & Deployment
1. **Clone the repository**:
```bash
git clone git@gitea.o00o.men:bitfennec/bangwen.git
cd bangwen
```
2. **Initialize environment credentials**:
```bash
cp .env.example .env
# Generate a secure 64-character token
sed -i "s/replace_with_your_secure_secret_token/$(openssl rand -hex 32)/" .env
```
3. **Start services with Docker Compose**:
```bash
docker compose up -d --build
```
### Verification
Check service health and initial ingestion metrics:
```bash
curl -s http://127.0.0.1:8080/health
# Expected: {"status":"ok"}
curl -s http://127.0.0.1:8080/api/v1/stats
# Expected: {"announcements":...,"embeddings":0,"positions":...,"sources":6,"status":"ok","vector_ready":false}
```
Open `http://127.0.0.1:8080/` in your browser to inspect the Web Console.
---
## π οΈ Usage
### Web Console
The built-in web console is served directly on `http://127.0.0.1:8080/`:
- **Feed Stream**: Paginated real-time recruitment notices with collapsible bodies, parsed position tables, and official attachment links.
- **Multi-Filter Search**: Search by keywords, administrative region code, employer type, and minimum education requirements.
- **Candidate White-Box Match**: Fill candidate resume attributes to calculate match scores and view clear transparent eligibility lists.
- **Daily Digest**: View generated markdown summaries of the past 24 hours.
- **Collector Monitor**: Health, authentication status, and last report timestamp for all edge collectors.
### CLI Commands
When running natively or using the provided wrapper `./bin/bangwen`:
```bash
# Ingest data via stdin or file
./bin/bangwen ingest --file ./sample.json
# Hybrid search recruitment posts
./bin/bangwen search "θΏη»΄" --region 530400 --json
# Inspect notice or position details
./bin/bangwen show ann:1 --json
./bin/bangwen show pos:3 --json
# Run white-box resume matching
./bin/bangwen match --resume ./tests/fixtures/sample_resume.json --top 10
# Generate daily markdown digest
./bin/bangwen digest
# Inspect collection sources health
./bin/bangwen sources
```
### HTTP API Reference
| Endpoint | Method | Description | Auth Required |
|----------|--------|-------------|---------------|
| `/health` | `GET` | Service & database health check | No |
| `/api/v1/ingest` | `POST` | Ingest batched notices & positions | Bearer Token |
| `/api/v1/stats` | `GET` | Aggregation statistics and vector status | No |
| `/api/v1/sources` | `GET` | Source nodes health and last ingest report | No |
| `/api/v1/announcements` | `GET` | Paginated recruitment announcements list | No |
| `/api/v1/announcements/{id}` | `GET` | Detailed announcement with positions & attachments | No |
| `/api/v1/positions/{id}` | `GET` | Detailed structured job position | No |
| `/api/v1/search` | `GET` | Multi-dimensional hybrid keyword search | No |
| `/api/v1/match` | `POST` | Strict white-box resume qualification match | No |
| `/api/v1/digest` | `GET` | Daily markdown recruitment digest | No |
---
## βοΈ Configuration
Environment variables configured in `.env`:
| Variable | Default | Required | Description |
|----------|---------|----------|-------------|
| `DATABASE_URL` | `postgres://bangwen:bangwen_dev@postgres:5432/bangwen` | Yes | PostgreSQL connection string with pgvector |
| `INGEST_TOKEN` | *None* | Recommended | Bearer token for authenticating `/api/v1/ingest` |
| `SERVER_HOST` | `0.0.0.0` | No | Host address for Rust HTTP gateway |
| `SERVER_PORT` | `8080` | No | Port for Rust HTTP gateway |
| `LOG_LEVEL` | `info` | No | Tracing log level (`error`, `warn`, `info`, `debug`) |
| `OLLAMA_BASE_URL` | `http://host.docker.internal:11434` | No | Optional Ollama instance for embedding generation |
| `EMBEDDING_MODEL` | `bge-m3` | No | Ollama embedding model name |
---
## π Security & Hardening
- **Non-Root Execution**: Both `bangwen-core` and `bangwen-collector` containers enforce explicit non-root user execution (`UID 10001`).
- **Strict Content Security Policy (CSP)**: `default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'none'; base-uri 'self'; frame-ancestors 'none';`. Inline scripts are completely eliminated.
- **Input Boundary Bounds**: Bounded validation for match requests (max 50 skills, 20 cities, 32-char education, 1940..=current birth year) and pagination limits (max offset 10000).
- **Anti-Deadlock Lock Ordering**: Ingestion pipeline enforces deterministic URL key sorting prior to row lock acquisition, eliminating PostgreSQL `40P01` deadlock conditions.
- **SSRF & Scheme Verification**: PoliteFetcher enforces private IP filtering, strict domain whitelists, and hop-by-hop validation during HTTP redirects.
---
## π€ Contributing
Contributions are welcome. Please ensure all modifications pass local validation:
```bash
# Rust core checks
cargo test --offline
cargo clippy --all-targets -- -D warnings
# Collectors checks
cd collectors && uv run python tests/test_content_and_security.py
```
---
## π License
[MIT](LICENSE)
---
## π€ For AI Agents
See [`skill/SKILL.md`](./skill/SKILL.md) for operational instructions and security boundary guidelines.
(back to top)