Files

T

bitfennec 1059be1b62 feat: initialize bangwen recruitment aggregation and search engine

- Rust core engine with HTTP ingestion gateway, hybrid search, and white-box resume matching
- Python edge collectors supporting declarative TOML and specialized adapters
- PostgreSQL schema migrations with pgvector embedding support
- Hardened security controls: strict CSP, terminal sanitization, and input boundary validation
- Multi-service container orchestration with dedicated non-root execution

2026-10-23 23:44:10 +08:00

adapters/sc

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

common

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

sources

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

src/bangwen_collectors

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

tests

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

.dockerignore

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

.python-version

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

config.example.toml

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

config.toml

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

Dockerfile

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

parse_positions.py

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

pyproject.toml

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

README.md

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

runner.py

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

uv.lock

feat: initialize bangwen recruitment aggregation and search engine

2026-10-23 23:44:10 +08:00

README.md

Bangwen Collectors

Lightweight declarative and custom python collectors for the Bangwen public job posting aggregation system.

Structure

sources/: Declarative TOML configuration for crawling target sites.
adapters/: Custom Python scrapers for complex or stateful collection logic.
common/: Core fetcher, security validators, and ingestion protocol models.
runner.py: Periodic crawler runner and domain-concurrency scheduler.

Running

uv run python runner.py --list
uv run python runner.py --once

Testing

Unit tests for collectors, SSRF guards, and content sanitizers are excluded from production Docker images and can be run directly on the host:

cd collectors && uv run python tests/test_content_and_security.py