Universal NLP Base¶

This project now uses a cross-industry, non-custom extraction baseline.

Components¶

1. Generic lexicons (file-based)
2. app/lexicons/common/overview_terms.txt
3. app/lexicons/common/requirement_terms.txt

4. app/lexicons/common/scoring_terms.txt

5. Signal engine

6. app/core/universal_nlp_base.py

7. Provides:

   • has_overview_signal(text, page_index)
   • has_requirement_signal(text)
   • has_scoring_signal(text)

8. Signal adapter

9. app/core/extraction_signals.py
10. Keeps old call sites stable, delegates to the universal engine.

Tokenizer backend¶

Backend order: 1. pkuseg (if installed and available) 2. jieba (if installed) 3. regex tokenizer fallback (always available)

You can force backend with env var:

NLP_TOKENIZER_BACKEND=pkuseg
NLP_TOKENIZER_BACKEND=jieba
NLP_TOKENIZER_BACKEND=auto

You can also override lexicon directory:

NLP_LEXICON_DIR=/absolute/path/to/lexicons

Optional ecosystem packages¶

These are common in industry for a generic Chinese NLP baseline:

• pkuseg
• jieba
• OpenHowNet
• fasttext-wheel

Current runtime only requires optional pkuseg/jieba; it degrades safely if missing.