feat: initialize bangwen recruitment aggregation and search engine

- Rust core engine with HTTP ingestion gateway, hybrid search, and white-box resume matching
- Python edge collectors supporting declarative TOML and specialized adapters
- PostgreSQL schema migrations with pgvector embedding support
- Hardened security controls: strict CSP, terminal sanitization, and input boundary validation
- Multi-service container orchestration with dedicated non-root execution
This commit is contained in:
bitfennec
2026-10-23 23:44:10 +08:00
commit 1059be1b62
61 changed files with 10489 additions and 0 deletions
+25
View File
@@ -0,0 +1,25 @@
# Bangwen Collectors
Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.
## Structure
- `sources/`: Declarative TOML configuration for crawling target sites.
- `adapters/`: Custom Python scrapers for complex or stateful collection logic.
- `common/`: Core fetcher, security validators, and ingestion protocol models.
- `runner.py`: Periodic crawler runner and domain-concurrency scheduler.
## Running
```bash
uv run python runner.py --list
uv run python runner.py --once
```
## Testing
Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:
```bash
cd collectors && uv run python tests/test_content_and_security.py
```