feat: initialize bangwen recruitment aggregation and search engine
- Rust core engine with HTTP ingestion gateway, hybrid search, and white-box resume matching - Python edge collectors supporting declarative TOML and specialized adapters - PostgreSQL schema migrations with pgvector embedding support - Hardened security controls: strict CSP, terminal sanitization, and input boundary validation - Multi-service container orchestration with dedicated non-root execution
This commit is contained in:
@@ -0,0 +1,25 @@
|
||||
# Bangwen Collectors
|
||||
|
||||
Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.
|
||||
|
||||
## Structure
|
||||
|
||||
- `sources/`: Declarative TOML configuration for crawling target sites.
|
||||
- `adapters/`: Custom Python scrapers for complex or stateful collection logic.
|
||||
- `common/`: Core fetcher, security validators, and ingestion protocol models.
|
||||
- `runner.py`: Periodic crawler runner and domain-concurrency scheduler.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
uv run python runner.py --list
|
||||
uv run python runner.py --once
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:
|
||||
|
||||
```bash
|
||||
cd collectors && uv run python tests/test_content_and_security.py
|
||||
```
|
||||
Reference in New Issue
Block a user