Initial: integrated 2025 LawGraph (graphrag_pipeline) + 2026 kg_project
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(where WinRAR.exe)",
|
||||
"Read(//c/Program Files/WinRAR/**)",
|
||||
"Bash(pip install *)",
|
||||
"Bash(pip show *)",
|
||||
"Bash(where winget *)",
|
||||
"Bash(python -c \"import rarfile; print\\('rarfile OK'\\)\")"
|
||||
]
|
||||
}
|
||||
}
|
||||
+50
@@ -0,0 +1,50 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
.venv/
|
||||
venv/
|
||||
.python-version
|
||||
|
||||
# Secrets
|
||||
.env
|
||||
dofile/kg_project/config/api_keys.yaml
|
||||
|
||||
# LaTeX build artifacts
|
||||
officefile/latex/*.aux
|
||||
officefile/latex/*.log
|
||||
officefile/latex/*.out
|
||||
officefile/latex/*.synctex.gz
|
||||
officefile/latex/*.fls
|
||||
officefile/latex/*.fdb_latexmk
|
||||
officefile/latex/*.toc
|
||||
officefile/latex/*.lof
|
||||
officefile/latex/*.lot
|
||||
officefile/latex/*.bbl
|
||||
officefile/latex/*.blg
|
||||
|
||||
# LaTeX paper (independent Overleaf git repo — do NOT track in root)
|
||||
officefile/latex/
|
||||
|
||||
# Data files (large, tracked separately via archive/)
|
||||
data/
|
||||
|
||||
# Archive (frozen snapshots, not version-controlled)
|
||||
archive/
|
||||
|
||||
# Large archives
|
||||
*.rar
|
||||
*.zip
|
||||
dofile/**/output-*.zip
|
||||
|
||||
# Logs and runtime outputs
|
||||
dofile/*/logs/
|
||||
output/
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
@@ -0,0 +1,62 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Virtual Environment
|
||||
.venv/
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# Environment variables
|
||||
.env
|
||||
|
||||
# Output files
|
||||
output/
|
||||
*.json
|
||||
*.csv
|
||||
*.xlsx
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
logs/
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Project specific
|
||||
data/
|
||||
models/
|
||||
*.db
|
||||
*.sqlite
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
# 更新日志
|
||||
|
||||
## v0.1.0 (2025-01-XX)
|
||||
|
||||
### 新增功能
|
||||
|
||||
- ✅ 项目基础架构（uv项目配置、目录结构）
|
||||
- ✅ 文本预处理模块（HanLP集成、文档解析）
|
||||
- ✅ 本体模型定义（9种实体类型、8种关系类型）
|
||||
- ✅ Prompt模板系统（五个模块、NER/RE构建器）
|
||||
- ✅ LLM客户端封装（多提供商支持）
|
||||
- ✅ 知识抽取模块：
|
||||
- NER提取器（两阶段对话）
|
||||
- RE提取器（多模型并行、二次对话验证）
|
||||
- 三元组评估器（成对排序、多维度评分）
|
||||
- 实体关系分类器（RAG-based + Prompt-based）
|
||||
- ✅ 知识图谱构建（索引器、图谱管理、社区检测、摘要生成）
|
||||
- ✅ 分析和推理模块（法理结构提取、逻辑推理、演化分析框架）
|
||||
- ✅ 查询系统（全局搜索、局部搜索、DRIFT搜索）
|
||||
- ✅ 执行脚本（extract.py, build_kg.py, query_kg.py, complete_pipeline.py）
|
||||
|
||||
### 技术特性
|
||||
|
||||
- 支持多个LLM提供商（OpenAI, Anthropic, Qwen, Doubao, GLM）
|
||||
- 参考GraphRAG实现社区检测和摘要生成
|
||||
- 实现论文中描述的五个模块Prompt模板
|
||||
- 支持两阶段对话验证机制
|
||||
- LLM-as-a-Judge评估方法
|
||||
|
||||
### 已知限制
|
||||
|
||||
- 大规模数据处理需要进一步优化
|
||||
- 向量数据库使用简化实现，生产环境建议使用专业向量数据库
|
||||
- 社区检测在igraph未安装时使用替代方法
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
# 配置完成总结
|
||||
|
||||
## ✅ 已完成的配置
|
||||
|
||||
### 1. 硅基流动API配置
|
||||
- ✅ API密钥已配置：`sk-pvvtosiglncktlucwarxilvsypqcttqizgpcfdvodgcuaezn`
|
||||
- ✅ API端点：`https://api.siliconflow.cn/v1`
|
||||
- ✅ 环境变量已设置（`.env`文件）
|
||||
|
||||
### 2. 模型配置
|
||||
项目已配置为使用以下两个硅基流动模型：
|
||||
|
||||
#### NER（命名实体识别）
|
||||
- **模型**: `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`
|
||||
- **配置文件**: `src/extraction/ner.py`
|
||||
- **状态**: ✅ 已测试，可用
|
||||
|
||||
#### RE（关系抽取）- 多模型并行
|
||||
- **模型1**: `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`
|
||||
- **模型2**: `Qwen/Qwen2.5-7B-Instruct`
|
||||
- **配置文件**: `src/extraction/re.py`
|
||||
- **状态**: ✅ 已测试，两个模型都可用
|
||||
|
||||
#### 社区摘要生成
|
||||
- **模型**: `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`
|
||||
- **配置文件**: `src/kg_builder/indexer.py`
|
||||
- **状态**: ✅ 已配置
|
||||
|
||||
### 3. 环境修复
|
||||
- ✅ 虚拟环境已重新创建
|
||||
- ✅ 所有依赖包已安装（92个包）
|
||||
- ✅ 修复了conda环境路径干扰问题
|
||||
|
||||
## 📊 快速测试结果
|
||||
|
||||
测试了2个TextUnit的完整流程：
|
||||
|
||||
### 成功指标
|
||||
- ✅ 文档解析：成功
|
||||
- ✅ 实体识别（NER）：成功（使用deepseek-ai/DeepSeek-R1-Distill-Qwen-7B）
|
||||
- ✅ 关系抽取（RE）：成功（两个模型并行运行）
|
||||
- ✅ 知识图谱构建：成功
|
||||
- 节点数：67个
|
||||
- 边数：63条
|
||||
- 三元组数：63个
|
||||
- ✅ 社区检测：成功（11个社区）
|
||||
- ⚠️ 社区摘要生成：部分成功（需要更长时间，但已修复配置）
|
||||
|
||||
## 🚀 运行完整流程
|
||||
|
||||
完整流程正在后台运行，处理3个文档：
|
||||
```bash
|
||||
cd E:\Project\2025_LawGraph\dofile
|
||||
.venv\Scripts\python.exe scripts\complete_pipeline.py --data-dir ../data/1法律 --output-dir ./output --max-docs 3
|
||||
```
|
||||
|
||||
预计运行时间：由于需要处理62个TextUnit，每个都需要调用LLM API，可能需要10-30分钟。
|
||||
|
||||
## 📁 输出文件
|
||||
|
||||
流程完成后，将生成：
|
||||
- `output/knowledge_graph.json`: 知识图谱文件
|
||||
- `output/knowledge_graph_stats.json`: 统计信息
|
||||
|
||||
## 🔍 验证模型
|
||||
|
||||
运行以下命令验证模型配置：
|
||||
```bash
|
||||
.venv\Scripts\python.exe tests\test_my_models.py
|
||||
```
|
||||
|
||||
## 📝 下一步
|
||||
|
||||
1. 等待完整流程完成
|
||||
2. 查看生成的知识图谱文件
|
||||
3. 可以尝试查询功能：
|
||||
```bash
|
||||
.venv\Scripts\python.exe scripts/query_kg.py --kg output/knowledge_graph.json --query "乡村振兴" --mode global
|
||||
```
|
||||
|
||||
配置已完成，项目已就绪！🎉
|
||||
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# 环境变量配置说明
|
||||
|
||||
## 创建 .env 文件
|
||||
|
||||
在 `dofile` 目录下创建 `.env` 文件，添加以下配置：
|
||||
|
||||
```bash
|
||||
# LLM API Keys
|
||||
OPENAI_API_KEY=your_openai_api_key_here
|
||||
ANTHROPIC_API_KEY=your_anthropic_api_key_here
|
||||
DASHSCOPE_API_KEY=your_dashscope_api_key_here
|
||||
VOLCENGINE_ACCESS_KEY=your_volcengine_access_key_here
|
||||
VOLCENGINE_SECRET_KEY=your_volcengine_secret_key_here
|
||||
ZHIPUAI_API_KEY=your_zhipuai_api_key_here
|
||||
|
||||
# SiliconFlow API配置（硅基流动）
|
||||
SILICONFLOW_API_KEY=sk-pvvtosiglncktlucwarxilvsypqcttqizgpcfdvodgcuaezn
|
||||
SILICONFLOW_API_BASE=https://api.siliconflow.cn/v1
|
||||
|
||||
# 路径配置
|
||||
DATA_DIR=../data
|
||||
OUTPUT_DIR=./output
|
||||
|
||||
# 日志配置
|
||||
LOG_LEVEL=INFO
|
||||
|
||||
# HanLP配置（可选）
|
||||
HANLP_MODEL_PATH=
|
||||
|
||||
# LLM配置
|
||||
TEMPERATURE=0.4
|
||||
FREQUENCY_PENALTY=0.6
|
||||
PRESENCE_PENALTY=0.6
|
||||
|
||||
# 文本处理配置
|
||||
MAX_TEXTUNIT_LENGTH=500
|
||||
```
|
||||
|
||||
## 硅基流动配置
|
||||
|
||||
硅基流动的API已经配置完成，您可以使用以下方式调用：
|
||||
|
||||
```python
|
||||
from src.utils.llm_client import LLMClient, LLMProvider
|
||||
from src.utils.config import load_config
|
||||
|
||||
config = load_config()
|
||||
client = LLMClient(
|
||||
provider=LLMProvider.SILICONFLOW,
|
||||
model="你的模型名称", # 例如：Qwen/Qwen2.5-72B-Instruct
|
||||
config=config
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": "你好"}
|
||||
]
|
||||
response = client.chat(messages)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 支持的模型
|
||||
|
||||
硅基流动支持多种模型，常用的包括：
|
||||
- `Qwen/Qwen2.5-72B-Instruct`
|
||||
- `meta-llama/Llama-3.1-70B-Instruct`
|
||||
- `01-ai/Yi-1.5-34B-Chat`
|
||||
- 等等...
|
||||
|
||||
具体可用模型请查看硅基流动官网：https://siliconflow.cn/
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# 模型配置说明
|
||||
|
||||
## 当前使用的模型
|
||||
|
||||
项目已配置为使用以下两个硅基流动模型：
|
||||
|
||||
### 1. NER（命名实体识别）
|
||||
- **模型**: `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`
|
||||
- **用途**: 从法规文本中提取实体（目标要素、原则要素、空间要素等）
|
||||
- **特点**: DeepSeek R1系列，具有推理能力
|
||||
|
||||
### 2. RE（关系抽取）
|
||||
- **模型1**: `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`
|
||||
- **模型2**: `Qwen/Qwen2.5-7B-Instruct`
|
||||
- **用途**: 从文本中提取实体之间的关系
|
||||
- **特点**: 多模型并行，提高准确性和覆盖率
|
||||
|
||||
### 3. 社区摘要生成
|
||||
- **模型**: `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`
|
||||
- **用途**: 为知识图谱的社区生成摘要
|
||||
|
||||
## 配置位置
|
||||
|
||||
模型配置在以下文件中：
|
||||
- `src/extraction/ner.py`: NER模型配置
|
||||
- `src/extraction/re.py`: RE模型配置（多模型）
|
||||
- `src/kg_builder/indexer.py`: 社区摘要生成模型配置
|
||||
|
||||
## 如何修改模型
|
||||
|
||||
如果要修改使用的模型，请编辑上述文件中的模型名称字符串。
|
||||
|
||||
## 测试模型可用性
|
||||
|
||||
运行以下命令测试模型：
|
||||
```bash
|
||||
uv run python tests/test_model.py
|
||||
|
||||
# 测试指定的两个模型
|
||||
uv run python tests/test_my_models.py
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
# 项目实现状态
|
||||
|
||||
## 已完成工作 ✅
|
||||
|
||||
### 1. 项目基础架构
|
||||
- ✅ 使用uv进行项目管理和依赖管理
|
||||
- ✅ 完整的项目目录结构
|
||||
- ✅ 配置文件（pyproject.toml, .env.example）
|
||||
- ✅ README和项目文档
|
||||
|
||||
### 2. 文本预处理模块 (`src/preprocessing/`)
|
||||
- ✅ **TextProcessor**: HanLP集成，支持分词、依存句法、句法成分分析
|
||||
- ✅ **DocumentParser**: Word文档解析（.docx），支持章节结构识别
|
||||
- ✅ TextUnit切分功能（参考GraphRAG）
|
||||
|
||||
### 3. 本体模型定义 (`src/ontology/`)
|
||||
- ✅ **实体类型定义**: 9种实体类型（对应论文Table 1）
|
||||
- ✅ **关系类型定义**: 8种关系类型（对应论文Table 2）
|
||||
- ✅ **三维本体结构**: 法规层级—要素类型—关联关系
|
||||
- ✅ **四要素架构**: 目标—原则—要素—管控
|
||||
- ✅ **OntologySchema**: 本体模式管理和验证
|
||||
|
||||
### 4. Prompt模板系统 (`src/prompts/`)
|
||||
- ✅ **PromptTemplate**: 统一模板框架，实现五个模块：
|
||||
- 任务描述（系统消息+用户消息）
|
||||
- 候选目标
|
||||
- 任务示例（少样本学习）
|
||||
- 任务强调
|
||||
- 二次对话
|
||||
- ✅ **NERPromptBuilder**: NER任务Prompt构建
|
||||
- ✅ **REPromptBuilder**: RE任务Prompt构建
|
||||
|
||||
### 5. LLM客户端封装 (`src/utils/llm_client.py`)
|
||||
- ✅ 支持多个LLM提供商：
|
||||
- OpenAI (GPT-4o)
|
||||
- Anthropic (Claude)
|
||||
- 阿里云Qwen
|
||||
- 字节跳动Doubao
|
||||
- 智谱GLM
|
||||
- ✅ 统一API接口
|
||||
- ✅ 异步/同步支持
|
||||
- ✅ 重试机制（tenacity）
|
||||
|
||||
### 6. 知识抽取模块 (`src/extraction/`)
|
||||
- ✅ **NERExtractor**: 使用Qwen-Max进行实体识别，支持两阶段对话验证
|
||||
- ✅ **REExtractor**: 多模型并行关系抽取框架
|
||||
- ✅ **TripletEvaluator**: LLM-as-a-Judge评估框架
|
||||
- ✅ **EntityRelationClassifier**: RAG-based分类框架
|
||||
|
||||
### 7. 知识图谱构建 (`src/kg_builder/`)
|
||||
- ✅ **GraphIndexer**: 索引器框架（参考GraphRAG）
|
||||
- 文本切分
|
||||
- 实体提取
|
||||
- 图构建
|
||||
- 社区检测（待完善）
|
||||
- 摘要生成（待完善）
|
||||
- ✅ **KnowledgeGraph**: 图谱结构管理（NetworkX）
|
||||
|
||||
### 8. 分析和推理模块 (`src/analysis/`)
|
||||
- ✅ **LegalStructureExtractor**: 法理结构提取（四要素识别、跨法规引用）
|
||||
- ✅ **LegalReasoner**: 逻辑推理（路径查询、中心性分析）
|
||||
- ✅ **EvolutionAnalyzer**: 演化规律分析框架
|
||||
|
||||
### 9. 查询系统 (`src/query/`)
|
||||
- ✅ **GlobalSearcher**: 全局搜索框架（参考GraphRAG）
|
||||
- ✅ **LocalSearcher**: 局部搜索框架
|
||||
- ✅ **DriftSearcher**: DRIFT搜索框架
|
||||
|
||||
### 10. 工具脚本
|
||||
- ✅ `scripts/extract.py`: 知识抽取示例脚本
|
||||
|
||||
## 待完善功能 ⚠️
|
||||
|
||||
### 高优先级
|
||||
1. **三元组评估算法实现** (`src/extraction/evaluator.py`)
|
||||
- 成对排序逻辑
|
||||
- 五维度评分（语义准确性、一致性、事实性、准确性、可理解性）
|
||||
- 评分聚合和筛选
|
||||
|
||||
2. **RAG-based分类实现** (`src/extraction/classifier.py`)
|
||||
- 向量嵌入（使用Text-embedding-3-large）
|
||||
- 向量数据库（相似度检索）
|
||||
- 分类逻辑完善
|
||||
|
||||
3. **知识图谱构建完整实现** (`src/kg_builder/indexer.py`)
|
||||
- 完整的三元组提取流程
|
||||
- 社区检测（Leiden算法，需要igraph）
|
||||
- 社区摘要生成（使用LLM）
|
||||
|
||||
4. **RE提取器完善** (`src/extraction/re.py`)
|
||||
- 句法信息集成
|
||||
- 上下文信息利用
|
||||
- 省略成分补全逻辑
|
||||
|
||||
### 中优先级
|
||||
5. **查询系统实现** (`src/query/`)
|
||||
- 全局搜索：利用社区摘要
|
||||
- 局部搜索：邻居扩展算法
|
||||
- DRIFT搜索：社区上下文整合
|
||||
|
||||
6. **演化规律分析** (`src/analysis/evolution.py`)
|
||||
- 时间序列分析
|
||||
- 关系演化追踪
|
||||
- 网络演化分析
|
||||
|
||||
### 低优先级
|
||||
7. **共指消解完善** (`src/preprocessing/text_processor.py`)
|
||||
8. **性能优化和缓存机制**
|
||||
9. **可视化工具**
|
||||
10. **单元测试和集成测试**
|
||||
|
||||
## 使用说明
|
||||
|
||||
### 环境设置
|
||||
```bash
|
||||
cd dofile
|
||||
uv sync
|
||||
source .venv/bin/activate # 或 .venv\Scripts\activate (Windows)
|
||||
```
|
||||
|
||||
### 配置API密钥
|
||||
复制`.env.example`到`.env`，填入相应的API密钥。
|
||||
|
||||
### 运行示例
|
||||
```bash
|
||||
# 实体识别
|
||||
uv run python scripts/extract.py --input ../data/1法律/3-中华人民共和国城乡规划法.docx --output ./output/entities.json
|
||||
```
|
||||
|
||||
## 参考资源
|
||||
|
||||
- GraphRAG文档: https://msdocs.cn/graphrag/
|
||||
- 论文: `officefile/paper.tex`
|
||||
- 数据目录: `../data/`
|
||||
|
||||
## 技术债务
|
||||
|
||||
1. 错误处理和日志记录需要更完善
|
||||
2. 需要添加配置文件验证
|
||||
3. 异步处理可以进一步优化
|
||||
4. 需要添加单元测试
|
||||
5. 文档需要补充更多示例
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
# 项目完善总结
|
||||
|
||||
## 已完成的核心功能
|
||||
|
||||
### ✅ 1. 三元组评估器完整实现
|
||||
- **成对排序**: 比较不同模型生成的三元组组，确定最佳结果
|
||||
- **多维度评分**: 五维度评估（语义准确性、一致性、事实性、准确性、可理解性）
|
||||
- **Peer Examination**: 避免自我增强偏差，支持多评估者
|
||||
- **低质量筛选**: 基于评分阈值筛选三元组
|
||||
|
||||
### ✅ 2. RE提取器完善
|
||||
- **二次对话验证**: 完整实现两阶段对话优化
|
||||
- **实体格式化**: 改进实体列表的格式化处理
|
||||
- **句法信息集成**: 更好地利用依存句法和句法成分信息
|
||||
- **错误处理**: 完善的异常处理和结果解析
|
||||
|
||||
### ✅ 3. 知识图谱索引器完整实现
|
||||
- **完整索引流程**: 从TextUnit到知识图谱的完整流程
|
||||
- **图构建**: 实现三元组到图结构的转换
|
||||
- **社区检测**: Leiden算法实现（igraph）或连通分量替代
|
||||
- **社区摘要生成**: 使用LLM生成社区摘要
|
||||
- **元数据管理**: 存储图谱构建的元信息
|
||||
|
||||
### ✅ 4. RAG-based分类器实现
|
||||
- **向量嵌入**: 使用OpenAI text-embedding-3-large
|
||||
- **相似度计算**: 余弦相似度计算
|
||||
- **知识库构建**: 支持从已标注样本构建知识库
|
||||
- **Prompt-based分类**: 备选分类方法
|
||||
|
||||
### ✅ 5. 查询系统完整实现
|
||||
- **全局搜索**:
|
||||
- 社区相关性评估
|
||||
- LLM生成综合答案
|
||||
- 支持关键词匹配备选
|
||||
- **局部搜索**:
|
||||
- 实体邻居扩展
|
||||
- 深度控制
|
||||
- 中心性分析
|
||||
- 模糊匹配
|
||||
- **DRIFT搜索**:
|
||||
- 结合局部和全局信息
|
||||
- 社区上下文整合
|
||||
- 综合答案生成
|
||||
|
||||
### ✅ 6. 图谱管理功能
|
||||
- **保存/加载**: JSON和GraphML格式支持
|
||||
- **元数据保存**: 保存社区信息和统计信息
|
||||
|
||||
### ✅ 7. 执行脚本完善
|
||||
- `extract.py`: 实体识别脚本
|
||||
- `build_kg.py`: 知识图谱构建脚本
|
||||
- `query_kg.py`: 图谱查询脚本（支持三种模式）
|
||||
- `complete_pipeline.py`: 完整流程脚本
|
||||
|
||||
### ✅ 8. 文档和测试
|
||||
- 完整的README
|
||||
- 使用示例文档
|
||||
- 架构文档
|
||||
- 基础测试
|
||||
- 项目状态文档
|
||||
|
||||
## 项目文件结构
|
||||
|
||||
```
|
||||
dofile/
|
||||
├── pyproject.toml # uv项目配置 ✅
|
||||
├── README.md # 项目文档 ✅
|
||||
├── PROJECT_STATUS.md # 项目状态 ✅
|
||||
├── CHANGELOG.md # 更新日志 ✅
|
||||
├── USAGE_EXAMPLES.md # 使用示例 ✅
|
||||
├── .env.example # 环境变量模板 ✅
|
||||
├── .gitignore # Git忽略文件 ✅
|
||||
├── src/ # 源代码 ✅
|
||||
│ ├── preprocessing/ # 文本预处理 ✅
|
||||
│ ├── ontology/ # 本体模型 ✅
|
||||
│ ├── prompts/ # Prompt模板 ✅
|
||||
│ ├── extraction/ # 知识抽取 ✅
|
||||
│ ├── kg_builder/ # 图谱构建 ✅
|
||||
│ ├── analysis/ # 分析推理 ✅
|
||||
│ ├── query/ # 查询系统 ✅
|
||||
│ └── utils/ # 工具函数 ✅
|
||||
├── scripts/ # 执行脚本 ✅
|
||||
│ ├── extract.py
|
||||
│ ├── build_kg.py
|
||||
│ ├── query_kg.py
|
||||
│ └── complete_pipeline.py
|
||||
├── config/ # 配置文件 ✅
|
||||
│ ├── ontology.yaml
|
||||
│ └── prompts/
|
||||
├── docs/ # 文档 ✅
|
||||
│ └── ARCHITECTURE.md
|
||||
└── tests/ # 测试 ✅
|
||||
└── test_basic.py
|
||||
```
|
||||
|
||||
## 核心特性
|
||||
|
||||
1. **模块化设计**: 清晰的模块划分，易于扩展和维护
|
||||
2. **多LLM支持**: 支持5个主流LLM提供商
|
||||
3. **GraphRAG参考**: 参考GraphRAG实现社区检测和查询
|
||||
4. **论文方法实现**: 完整实现论文中描述的方法论
|
||||
5. **错误处理**: 完善的异常处理和日志记录
|
||||
6. **可配置性**: 通过配置文件和环境变量灵活配置
|
||||
|
||||
## 使用方法
|
||||
|
||||
### 快速开始
|
||||
```bash
|
||||
cd dofile
|
||||
uv sync
|
||||
cp .env.example .env
|
||||
# 编辑.env填入API密钥
|
||||
|
||||
# 完整流程
|
||||
uv run python scripts/complete_pipeline.py --data-dir ../data/1法律 --output-dir ./output --max-docs 5
|
||||
```
|
||||
|
||||
### 分步执行
|
||||
```bash
|
||||
# 1. 实体识别
|
||||
uv run python scripts/extract.py --input ../data/1法律/3-中华人民共和国城乡规划法.docx
|
||||
|
||||
# 2. 构建图谱
|
||||
uv run python scripts/build_kg.py --data-dir ../data/1法律 --output ./output/kg.json
|
||||
|
||||
# 3. 查询图谱
|
||||
uv run python scripts/query_kg.py --kg ./output/kg.json --query "城乡规划" --mode global
|
||||
```
|
||||
|
||||
## 技术亮点
|
||||
|
||||
1. **两阶段对话验证**: 提高知识抽取准确性
|
||||
2. **多模型并行**: 利用不同模型的优势
|
||||
3. **LLM-as-a-Judge**: 自动化质量评估
|
||||
4. **社区检测和摘要**: 参考GraphRAG的层次结构
|
||||
5. **RAG-based分类**: 利用向量检索提高分类准确率
|
||||
6. **DRIFT搜索**: 结合局部和全局信息的智能搜索
|
||||
|
||||
## 下一步建议
|
||||
|
||||
1. **性能优化**:
|
||||
- 批量处理优化
|
||||
- 缓存机制
|
||||
- 异步处理改进
|
||||
|
||||
2. **功能增强**:
|
||||
- 可视化工具
|
||||
- 更完善的演化分析
|
||||
- 增量更新支持
|
||||
|
||||
3. **生产化**:
|
||||
- 单元测试完善
|
||||
- 集成测试
|
||||
- 性能监控
|
||||
- 错误恢复机制
|
||||
|
||||
项目已基本完成论文方法论的实现，可以直接用于法规知识图谱的构建和分析。
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
# 国土空间规划法规知识图谱构建与分析系统
|
||||
|
||||
基于大语言模型的国土空间规划法规知识图谱构建与原理提取研究项目。
|
||||
|
||||
## 项目简介
|
||||
|
||||
本项目实现了论文《基于大模型的国土空间规划法规知识图谱构建与原理提取研究》中提出的方法论，包括：
|
||||
|
||||
- 法规文本预处理（HanLP）
|
||||
- 基于大模型的知识抽取（NER、RE、评估、分类）
|
||||
- 知识图谱构建（基于GraphRAG思路）
|
||||
- 法理结构提取
|
||||
- 法规逻辑推理与演化规律识别
|
||||
|
||||
## 技术栈
|
||||
|
||||
- Python 3.12+
|
||||
- uv: 虚拟环境管理和包管理
|
||||
- HanLP: 中文NLP处理
|
||||
- LLM APIs: OpenAI, Anthropic, Qwen, Doubao, GLM
|
||||
- NetworkX/iGraph: 图处理
|
||||
- PyYAML: 配置文件
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 1. 环境设置
|
||||
|
||||
使用uv创建虚拟环境并安装依赖：
|
||||
|
||||
```bash
|
||||
# 进入项目目录
|
||||
cd dofile
|
||||
|
||||
# 初始化uv项目并安装依赖
|
||||
uv sync
|
||||
|
||||
# 激活虚拟环境
|
||||
source .venv/bin/activate # Linux/Mac
|
||||
# 或
|
||||
.venv\Scripts\activate # Windows
|
||||
|
||||
# 如果uv未安装，先安装uv：
|
||||
# Windows: powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
|
||||
# Linux/Mac: curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
```
|
||||
|
||||
### 2. 配置环境变量
|
||||
|
||||
复制 `.env.example` 到 `.env` 并填入API密钥：
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
编辑 `.env` 文件，填入所需的API密钥：
|
||||
- OPENAI_API_KEY
|
||||
- ANTHROPIC_API_KEY
|
||||
- DASHSCOPE_API_KEY (Qwen)
|
||||
- VOLCENGINE_ACCESS_KEY / VOLCENGINE_SECRET_KEY (Doubao)
|
||||
- ZHIPUAI_API_KEY (GLM)
|
||||
- SILICONFLOW_API_KEY (硅基流动，已在示例中配置)
|
||||
- SILICONFLOW_API_BASE (可选，默认为 https://api.siliconflow.cn/v1)
|
||||
|
||||
### 3. 运行项目
|
||||
|
||||
```bash
|
||||
# 方式1: 完整流程（推荐）
|
||||
# 从文档解析到知识图谱构建
|
||||
uv run python scripts/complete_pipeline.py --data-dir ../data/1法律 --output-dir ./output --max-docs 5
|
||||
|
||||
# 方式2: 分步骤执行
|
||||
# 步骤1: 实体识别
|
||||
uv run python scripts/extract.py --input ../data/1法律/3-中华人民共和国城乡规划法.docx --output ./output/entities.json
|
||||
|
||||
# 步骤2: 构建知识图谱
|
||||
uv run python scripts/build_kg.py --data-dir ../data/1法律 --output ./output/kg.json --max-docs 5
|
||||
|
||||
# 步骤3: 查询知识图谱
|
||||
uv run python scripts/query_kg.py --kg ./output/kg.json --query "城乡规划" --mode global
|
||||
uv run python scripts/query_kg.py --kg ./output/kg.json --query "城市更新" --mode local
|
||||
uv run python scripts/query_kg.py --kg ./output/kg.json --query "土地管理" --mode drift
|
||||
```
|
||||
|
||||
### 4. 运行测试
|
||||
|
||||
```bash
|
||||
# 基础功能测试
|
||||
uv run python tests/test_basic.py
|
||||
|
||||
# 测试硅基流动API配置
|
||||
uv run python tests/test_siliconflow.py
|
||||
|
||||
# 测试文档解析
|
||||
uv run python tests/test_documents.py
|
||||
|
||||
# 测试模型可用性
|
||||
uv run python tests/test_model.py
|
||||
|
||||
# 测试指定模型
|
||||
uv run python tests/test_my_models.py
|
||||
|
||||
# 快速测试（小规模知识图谱构建）
|
||||
uv run python tests/test_quick.py
|
||||
```
|
||||
|
||||
### 5. 使用硅基流动API
|
||||
|
||||
硅基流动已经配置完成，您可以通过以下方式使用：
|
||||
|
||||
```python
|
||||
from src.utils.llm_client import LLMClient, LLMProvider
|
||||
from src.utils.config import load_config
|
||||
|
||||
config = load_config()
|
||||
client = LLMClient(
|
||||
provider=LLMProvider.SILICONFLOW,
|
||||
model="Qwen/Qwen2.5-72B-Instruct", # 请根据实际可用模型修改
|
||||
config=config
|
||||
)
|
||||
|
||||
messages = [{"role": "user", "content": "你好"}]
|
||||
response = client.chat(messages)
|
||||
```
|
||||
|
||||
更多使用示例请参考：
|
||||
- `examples/use_siliconflow.py` - 硅基流动使用示例
|
||||
- `ENV_SETUP.md` - 环境变量配置说明
|
||||
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
dofile/
|
||||
├── src/ # 源代码
|
||||
│ ├── preprocessing/ # 文本预处理
|
||||
│ ├── ontology/ # 本体模型定义
|
||||
│ ├── prompts/ # Prompt模板
|
||||
│ ├── extraction/ # 知识抽取
|
||||
│ ├── kg_builder/ # 知识图谱构建
|
||||
│ ├── analysis/ # 分析和推理
|
||||
│ ├── query/ # 查询系统
|
||||
│ └── utils/ # 工具函数
|
||||
├── scripts/ # 执行脚本
|
||||
├── config/ # 配置文件
|
||||
└── tests/ # 测试文件
|
||||
```
|
||||
|
||||
## 项目状态
|
||||
|
||||
### 已完成模块
|
||||
|
||||
✅ **项目基础结构**
|
||||
- uv项目配置（pyproject.toml）
|
||||
- 项目文档（README.md）
|
||||
- 目录结构
|
||||
|
||||
✅ **文本预处理模块**
|
||||
- HanLP文本处理器（分词、依存句法、句法成分分析）
|
||||
- 法规文档解析器（.docx文件解析）
|
||||
|
||||
✅ **本体模型定义**
|
||||
- 实体类型定义（9种类型）
|
||||
- 关系类型定义（8种类型）
|
||||
- 三维本体结构（法规层级—要素类型—关联关系）
|
||||
- 四要素架构（目标—原则—要素—管控）
|
||||
|
||||
✅ **Prompt模板系统**
|
||||
- 统一Prompt模板框架（五个模块）
|
||||
- NER Prompt构建器
|
||||
- RE Prompt构建器
|
||||
|
||||
✅ **LLM客户端封装**
|
||||
- 多提供商支持（OpenAI, Anthropic, Qwen, Doubao, GLM）
|
||||
- 统一API接口
|
||||
- 重试机制
|
||||
|
||||
✅ **知识抽取模块框架**
|
||||
- NER提取器（两阶段对话）
|
||||
- RE提取器（多模型并行）
|
||||
- 三元组评估器（LLM-as-a-Judge）
|
||||
- 实体关系分类器（RAG-based）
|
||||
|
||||
✅ **知识图谱构建框架**
|
||||
- 图谱索引器（参考GraphRAG）
|
||||
- 图谱结构管理（NetworkX）
|
||||
|
||||
✅ **分析和推理模块框架**
|
||||
- 法理结构提取器
|
||||
- 逻辑推理器
|
||||
- 演化规律分析器
|
||||
|
||||
✅ **查询系统框架**
|
||||
- 全局搜索（参考GraphRAG）
|
||||
- 局部搜索
|
||||
- DRIFT搜索
|
||||
|
||||
### 待完善模块
|
||||
|
||||
⚠️ **部分优化空间**：
|
||||
- 三元组评估的性能优化（批量评估）
|
||||
- RAG-based分类的向量数据库优化（使用专业向量数据库）
|
||||
- 大规模图谱的社区检测优化
|
||||
- 演化规律分析的时间序列处理完善
|
||||
- 可视化工具开发
|
||||
- 性能监控和日志分析
|
||||
|
||||
### 开发指南
|
||||
|
||||
各模块的详细文档：
|
||||
- `src/preprocessing/`: 文本预处理
|
||||
- `src/ontology/`: 本体模型定义
|
||||
- `src/prompts/`: Prompt模板系统
|
||||
- `src/extraction/`: 知识抽取
|
||||
- `src/kg_builder/`: 知识图谱构建
|
||||
- `src/analysis/`: 分析和推理
|
||||
- `src/query/`: 查询系统
|
||||
|
||||
## 许可证
|
||||
|
||||
[待定]
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
# 测试结果报告
|
||||
|
||||
## 测试目标
|
||||
测试运行项目，处理3个法规文档文件，构建知识图谱。
|
||||
|
||||
## 测试环境
|
||||
- 操作系统: Windows
|
||||
- Python版本: 3.12.7 (通过uv管理)
|
||||
- 虚拟环境: `.venv`
|
||||
- 测试时间: 2025-01-XX
|
||||
|
||||
## 已完成的工作
|
||||
|
||||
### ✅ 1. 项目依赖安装
|
||||
- 使用 `uv sync` 成功安装了所有依赖包
|
||||
- 包括：hanlp, openai, networkx, python-docx 等92个包
|
||||
|
||||
### ✅ 2. 硅基流动API配置
|
||||
- 成功添加硅基流动API支持
|
||||
- 配置了API密钥和端点
|
||||
- 更新了相关文档
|
||||
|
||||
### ✅ 3. 代码修复
|
||||
- 修复了 `pyproject.toml` 中的包配置问题
|
||||
- 修复了 `TextProcessor` 使其能在HanLP不可用时降级运行
|
||||
- 所有代码通过语法检查
|
||||
|
||||
## 遇到的问题
|
||||
|
||||
### ⚠️ 问题1: lxml DLL加载失败
|
||||
**错误信息**:
|
||||
```
|
||||
ImportError: DLL load failed while importing etree: 找不到指定的模块。
|
||||
```
|
||||
|
||||
**原因分析**:
|
||||
- 虚拟环境的Python在导入lxml时，误用了系统conda环境的lxml包
|
||||
- conda环境的lxml可能缺少必要的DLL文件或版本不兼容
|
||||
|
||||
**解决方案**:
|
||||
1. **方案A（推荐）**: 重新创建干净的虚拟环境
|
||||
```bash
|
||||
cd E:\Project\2025_LawGraph\dofile
|
||||
rm -rf .venv # 或手动删除.venv目录
|
||||
uv sync
|
||||
```
|
||||
|
||||
2. **方案B**: 在虚拟环境中强制重新安装lxml
|
||||
```bash
|
||||
.venv\Scripts\python.exe -m pip install --force-reinstall --no-deps lxml
|
||||
```
|
||||
|
||||
3. **方案C**: 使用conda环境（如果系统主要使用conda）
|
||||
```bash
|
||||
conda install lxml
|
||||
```
|
||||
|
||||
## 测试脚本说明
|
||||
|
||||
已创建以下测试脚本：
|
||||
|
||||
1. **`scripts/test_documents.py`** - 测试文档解析（不调用LLM）
|
||||
- 解析3个文档
|
||||
- 切分为TextUnit
|
||||
- 保存结果到JSON
|
||||
|
||||
2. **`scripts/test_siliconflow.py`** - 测试硅基流动API配置
|
||||
|
||||
3. **`scripts/complete_pipeline.py`** - 完整流程（包含LLM调用）
|
||||
|
||||
## 建议的测试步骤
|
||||
|
||||
### 步骤1: 修复环境问题
|
||||
```bash
|
||||
# 重新创建虚拟环境
|
||||
cd E:\Project\2025_LawGraph\dofile
|
||||
rm -rf .venv
|
||||
uv sync
|
||||
```
|
||||
|
||||
### 步骤2: 测试文档解析（不调用LLM）
|
||||
```bash
|
||||
uv run python scripts/test_documents.py
|
||||
```
|
||||
|
||||
### 步骤3: 测试完整流程（调用LLM，需要API密钥）
|
||||
```bash
|
||||
# 确保.env文件已配置API密钥
|
||||
uv run python scripts/complete_pipeline.py --data-dir ../data/1法律 --output-dir ./output --max-docs 3
|
||||
```
|
||||
|
||||
## 项目状态
|
||||
|
||||
✅ **已完成**:
|
||||
- 项目结构搭建
|
||||
- 所有核心模块实现
|
||||
- 硅基流动API集成
|
||||
- 文档完善
|
||||
|
||||
⚠️ **待解决**:
|
||||
- 虚拟环境配置问题（lxml DLL）
|
||||
- 完整的端到端测试
|
||||
|
||||
## 下一步
|
||||
|
||||
1. 修复虚拟环境问题后，可以成功运行测试
|
||||
2. 所有功能模块已实现，只需要解决环境配置即可正常使用
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
# 使用示例
|
||||
|
||||
## 示例1: 从单个文档提取实体
|
||||
|
||||
```python
|
||||
from src.preprocessing.document_parser import DocumentParser
|
||||
from src.extraction.ner import NERExtractor
|
||||
from src.utils.config import load_config
|
||||
|
||||
# 加载配置
|
||||
config = load_config()
|
||||
|
||||
# 解析文档
|
||||
parser = DocumentParser()
|
||||
doc = parser.parse_docx("../data/1法律/3-中华人民共和国城乡规划法.docx")
|
||||
|
||||
# 提取实体
|
||||
ner_extractor = NERExtractor(config=config)
|
||||
entities = ner_extractor.extract(doc["text"])
|
||||
|
||||
print(f"识别到 {sum(len(v) for v in entities.values())} 个实体")
|
||||
```
|
||||
|
||||
## 示例2: 构建知识图谱
|
||||
|
||||
```python
|
||||
from src.preprocessing.document_parser import DocumentParser
|
||||
from src.kg_builder.indexer import GraphIndexer
|
||||
from src.utils.config import load_config
|
||||
|
||||
# 加载配置
|
||||
config = load_config()
|
||||
|
||||
# 解析文档
|
||||
parser = DocumentParser()
|
||||
docs = parser.parse_directory("../data/1法律")
|
||||
|
||||
# 切分为TextUnit
|
||||
all_textunits = []
|
||||
for doc in docs[:3]: # 只处理前3个文档
|
||||
textunits = parser.split_into_textunits(doc, max_length=500)
|
||||
all_textunits.extend(textunits)
|
||||
|
||||
# 构建知识图谱
|
||||
indexer = GraphIndexer(config=config)
|
||||
kg = indexer.index(all_textunits)
|
||||
|
||||
print(f"图谱包含 {kg.number_of_nodes()} 个节点，{kg.number_of_edges()} 条边")
|
||||
```
|
||||
|
||||
## 示例3: 查询知识图谱
|
||||
|
||||
```python
|
||||
from src.kg_builder.graph import KnowledgeGraph
|
||||
from src.query.global_search import GlobalSearcher
|
||||
from src.query.local_search import LocalSearcher
|
||||
|
||||
# 加载图谱
|
||||
kg = KnowledgeGraph()
|
||||
kg.load("output/kg.json")
|
||||
|
||||
# 全局搜索
|
||||
global_searcher = GlobalSearcher(kg.graph, communities={})
|
||||
result = global_searcher.search("城市更新的法规要求")
|
||||
print(result["answer"])
|
||||
|
||||
# 局部搜索
|
||||
local_searcher = LocalSearcher(kg.graph)
|
||||
result = local_searcher.search("城乡规划", depth=2)
|
||||
print(f"找到 {result['subgraph']['num_nodes']} 个相关节点")
|
||||
```
|
||||
|
||||
## 示例4: 法理结构提取
|
||||
|
||||
```python
|
||||
from src.analysis.legal_structure import LegalStructureExtractor
|
||||
|
||||
extractor = LegalStructureExtractor()
|
||||
|
||||
# 提取四要素
|
||||
text = "为了促进城市可持续发展，应当坚持生态优先、绿色发展原则..."
|
||||
elements = extractor.extract_four_elements(text)
|
||||
print(elements)
|
||||
|
||||
# 提取跨法规引用
|
||||
references = extractor.extract_cross_references(text)
|
||||
print(references)
|
||||
```
|
||||
|
||||
## 示例5: 三元组评估
|
||||
|
||||
```python
|
||||
from src.extraction.evaluator import TripletEvaluator
|
||||
|
||||
evaluator = TripletEvaluator()
|
||||
|
||||
# 评估多个模型的结果
|
||||
peer_groups = {
|
||||
"model_1": [{"head": "A", "relation": "管控", "tail": "B"}],
|
||||
"model_2": [{"head": "A", "relation": "涉及", "tail": "B"}],
|
||||
}
|
||||
|
||||
result = evaluator.evaluate_peer_groups(
|
||||
peer_groups,
|
||||
source_text="原始文本..."
|
||||
)
|
||||
|
||||
print(f"最佳模型: {result['best_group']}")
|
||||
print(f"评分: {result['scores']}")
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
# 法规知识本体定义文件
|
||||
# 对应论文中的三维本体结构：法规层级—要素类型—关联关系
|
||||
|
||||
# 法规层级
|
||||
legal_levels:
|
||||
- 法律
|
||||
- 行政法规
|
||||
- 部门规章
|
||||
- 党中央国务院文件
|
||||
- 国家主管部门文件
|
||||
- 技术标准
|
||||
|
||||
# 实体类型（对应论文Table 1）
|
||||
entity_types:
|
||||
法规条文:
|
||||
definition: "法规文档中的具体条文，包括章、节、条、款、项等不同层级"
|
||||
category: "structural"
|
||||
目标要素:
|
||||
definition: "法规中明确表达的战略目标、发展愿景、总体目标等宏观性表述"
|
||||
category: "four_elements"
|
||||
element_type: "目标"
|
||||
原则要素:
|
||||
definition: "法规遵循的基本准则、指导方针、基本原则等"
|
||||
category: "four_elements"
|
||||
element_type: "原则"
|
||||
空间要素:
|
||||
definition: "法规涉及的地理空间实体，包括行政区划、功能区、保护区域等"
|
||||
category: "four_elements"
|
||||
element_type: "要素"
|
||||
功能要素:
|
||||
definition: "法规涉及的功能性要素，如土地利用类型、产业类型、基础设施等"
|
||||
category: "four_elements"
|
||||
element_type: "要素"
|
||||
环境要素:
|
||||
definition: "法规涉及的环境保护相关要素，如生态保护区、环境质量标准等"
|
||||
category: "four_elements"
|
||||
element_type: "要素"
|
||||
管控要素:
|
||||
definition: "法规对各类要素的管控要求，包括禁止、限制、引导等管控措施"
|
||||
category: "four_elements"
|
||||
element_type: "管控"
|
||||
行政区域:
|
||||
definition: "法规适用的行政区划，包括省、市、县、乡等不同层级"
|
||||
category: "spatial"
|
||||
时间要素:
|
||||
definition: "法规涉及的时间概念，如规划期限、实施阶段、有效期等"
|
||||
category: "temporal"
|
||||
|
||||
# 关系类型（对应论文Table 2）
|
||||
relation_types:
|
||||
引用:
|
||||
definition: "一条法规条文明确引用另一条法规条文"
|
||||
category: "legal"
|
||||
层级:
|
||||
definition: "上下级法规之间的隶属关系或法规条文内部的层级关系"
|
||||
category: "structural"
|
||||
包含:
|
||||
definition: "一个实体在逻辑上包含另一个实体"
|
||||
category: "logical"
|
||||
管控:
|
||||
definition: "管控要素对目标要素、原则要素或其他要素的约束关系"
|
||||
category: "four_elements"
|
||||
遵循:
|
||||
definition: "要素遵循某种原则或目标的关系"
|
||||
category: "four_elements"
|
||||
涉及:
|
||||
definition: "法规条文涉及某个空间、功能或环境要素"
|
||||
category: "semantic"
|
||||
适用:
|
||||
definition: "法规适用于某个行政区域"
|
||||
category: "spatial"
|
||||
演化:
|
||||
definition: "法规修订前后的变化关系或时间序列上的演化关系"
|
||||
category: "temporal"
|
||||
|
||||
# 四要素架构
|
||||
four_elements:
|
||||
目标:
|
||||
entities:
|
||||
- 目标要素
|
||||
原则:
|
||||
entities:
|
||||
- 原则要素
|
||||
要素:
|
||||
entities:
|
||||
- 空间要素
|
||||
- 功能要素
|
||||
- 环境要素
|
||||
管控:
|
||||
entities:
|
||||
- 管控要素
|
||||
|
||||
# 层级关系
|
||||
hierarchical_relations:
|
||||
- from: 目标
|
||||
to: 原则
|
||||
relation: 遵循
|
||||
- from: 原则
|
||||
to: 要素
|
||||
relation: 遵循
|
||||
- from: 要素
|
||||
to: 管控
|
||||
relation: 管控
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
# 系统架构文档
|
||||
|
||||
## 整体架构
|
||||
|
||||
本系统采用模块化设计，参考GraphRAG的实现思路，实现法规知识图谱的构建、分析和查询。
|
||||
|
||||
## 核心模块说明
|
||||
|
||||
### 1. 文本预处理模块 (`src/preprocessing/`)
|
||||
|
||||
**职责**：处理法规文档，为知识抽取做准备
|
||||
|
||||
- `TextProcessor`: 使用HanLP进行中文文本处理
|
||||
- 分词（粗粒度/细粒度）
|
||||
- 依存句法分析
|
||||
- 句法成分分析
|
||||
- 词性标注
|
||||
- 省略检测
|
||||
- 共指消解候选提取
|
||||
|
||||
- `DocumentParser`: 解析Word文档
|
||||
- .docx文件解析
|
||||
- 章节结构识别
|
||||
- TextUnit切分（参考GraphRAG）
|
||||
|
||||
### 2. 本体模型模块 (`src/ontology/`)
|
||||
|
||||
**职责**：定义知识图谱的语义结构
|
||||
|
||||
- `EntityType`: 9种实体类型枚举
|
||||
- `RelationType`: 8种关系类型枚举
|
||||
- `OntologySchema`: 三维本体结构管理
|
||||
- 法规层级维度
|
||||
- 要素类型维度
|
||||
- 关联关系维度
|
||||
- 四要素架构（目标—原则—要素—管控）
|
||||
|
||||
### 3. Prompt模板模块 (`src/prompts/`)
|
||||
|
||||
**职责**：构建统一的Prompt模板系统
|
||||
|
||||
- `PromptTemplate`: 统一模板框架
|
||||
- 任务描述模块
|
||||
- 候选目标模块
|
||||
- 任务示例模块（少样本学习）
|
||||
- 任务强调模块
|
||||
- 二次对话模块
|
||||
|
||||
- `NERPromptBuilder`: NER任务Prompt构建
|
||||
- `REPromptBuilder`: RE任务Prompt构建
|
||||
|
||||
### 4. 知识抽取模块 (`src/extraction/`)
|
||||
|
||||
**职责**：从文本中提取结构化知识
|
||||
|
||||
- `NERExtractor`: 命名实体识别
|
||||
- 使用Qwen-Max模型
|
||||
- 两阶段对话验证
|
||||
- JSON格式输出
|
||||
|
||||
- `REExtractor`: 关系抽取
|
||||
- 多模型并行（GPT-4o, Doubao-pro, GLM-4）
|
||||
- 二次对话验证
|
||||
- 句法信息集成
|
||||
- 上下文利用
|
||||
|
||||
- `TripletEvaluator`: 三元组评估
|
||||
- LLM-as-a-Judge方法
|
||||
- 成对排序
|
||||
- 五维度评分（语义准确性、一致性、事实性、准确性、可理解性）
|
||||
- 低质量三元组筛选
|
||||
|
||||
- `EntityRelationClassifier`: 实体关系分类
|
||||
- RAG-based分类（使用向量检索）
|
||||
- Prompt-based分类
|
||||
- 粗粒度和细粒度分类
|
||||
|
||||
### 5. 知识图谱构建模块 (`src/kg_builder/`)
|
||||
|
||||
**职责**：构建和管理知识图谱（参考GraphRAG）
|
||||
|
||||
- `GraphIndexer`: 图谱索引器
|
||||
- 文本切分为TextUnit
|
||||
- 实体和关系提取
|
||||
- 图构建
|
||||
- 社区检测（Leiden算法）
|
||||
- 社区摘要生成
|
||||
|
||||
- `KnowledgeGraph`: 图谱结构管理
|
||||
- 基于NetworkX
|
||||
- 三元组添加
|
||||
- 路径查询
|
||||
- 子图提取
|
||||
- 保存/加载
|
||||
|
||||
### 6. 分析和推理模块 (`src/analysis/`)
|
||||
|
||||
**职责**：分析法规体系和识别规律
|
||||
|
||||
- `LegalStructureExtractor`: 法理结构提取
|
||||
- 四要素提取（目标、原则、要素、管控）
|
||||
- 跨法规引用关系提取
|
||||
- 例外条款识别
|
||||
|
||||
- `LegalReasoner`: 逻辑推理
|
||||
- 路径查询（多跳关系）
|
||||
- 中心性分析
|
||||
- 子图分析
|
||||
|
||||
- `EvolutionAnalyzer`: 演化规律识别
|
||||
- 时间维度分析
|
||||
- 关系演化追踪
|
||||
- 网络演化分析
|
||||
|
||||
### 7. 查询系统模块 (`src/query/`)
|
||||
|
||||
**职责**：知识图谱查询和问答（参考GraphRAG）
|
||||
|
||||
- `GlobalSearcher`: 全局搜索
|
||||
- 利用社区摘要
|
||||
- 相关性评估
|
||||
- 综合答案生成
|
||||
|
||||
- `LocalSearcher`: 局部搜索
|
||||
- 实体邻居扩展
|
||||
- 深度控制
|
||||
- 子图提取
|
||||
|
||||
- `DriftSearcher`: DRIFT搜索
|
||||
- 结合局部和全局信息
|
||||
- 社区上下文整合
|
||||
- 综合答案生成
|
||||
|
||||
### 8. 工具模块 (`src/utils/`)
|
||||
|
||||
**职责**：提供通用工具函数
|
||||
|
||||
- `Config`: 配置管理
|
||||
- 环境变量加载
|
||||
- API密钥管理
|
||||
- 参数配置
|
||||
|
||||
- `LLMClient`: LLM客户端封装
|
||||
- 多提供商支持
|
||||
- 统一API接口
|
||||
- 异步/同步支持
|
||||
- 重试机制
|
||||
|
||||
- `file_utils`: 文件操作工具
|
||||
|
||||
## 数据流
|
||||
|
||||
```
|
||||
法规文档(.docx)
|
||||
↓
|
||||
DocumentParser (解析)
|
||||
↓
|
||||
TextUnit列表
|
||||
↓
|
||||
TextProcessor (预处理) + NERExtractor (实体识别)
|
||||
↓
|
||||
实体列表
|
||||
↓
|
||||
REExtractor (关系抽取) + TextProcessor (句法信息)
|
||||
↓
|
||||
三元组列表
|
||||
↓
|
||||
TripletEvaluator (评估筛选)
|
||||
↓
|
||||
高质量三元组
|
||||
↓
|
||||
GraphIndexer (图构建)
|
||||
↓
|
||||
KnowledgeGraph
|
||||
↓
|
||||
社区检测 + 摘要生成
|
||||
↓
|
||||
完整知识图谱 + 社区信息
|
||||
↓
|
||||
Query系统 (查询和问答)
|
||||
```
|
||||
|
||||
## 技术栈
|
||||
|
||||
- **语言**: Python 3.12+
|
||||
- **包管理**: uv
|
||||
- **NLP**: HanLP
|
||||
- **图处理**: NetworkX, python-igraph
|
||||
- **LLM**: OpenAI, Anthropic, Qwen, Doubao, GLM
|
||||
- **数据处理**: pandas, numpy
|
||||
|
||||
## 参考实现
|
||||
|
||||
- GraphRAG: https://msdocs.cn/graphrag/
|
||||
- 论文方法: `officefile/paper.tex`
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
"""
|
||||
使用硅基流动API的示例
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.utils.llm_client import LLMClient, LLMProvider
|
||||
from src.utils.config import load_config
|
||||
from src.extraction.ner import NERExtractor
|
||||
|
||||
|
||||
def example_basic_usage():
|
||||
"""基础使用示例"""
|
||||
print("示例1: 基础API调用")
|
||||
print("-" * 60)
|
||||
|
||||
config = load_config()
|
||||
client = LLMClient(
|
||||
provider=LLMProvider.SILICONFLOW,
|
||||
model="Qwen/Qwen2.5-72B-Instruct", # 请根据实际情况修改
|
||||
config=config
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "你是一个法规文本分析专家。"},
|
||||
{"role": "user", "content": "请分析以下文本中的实体：城市更新需要遵循国土空间规划的要求。"}
|
||||
]
|
||||
|
||||
response = client.chat(messages)
|
||||
print(f"响应: {response}\n")
|
||||
|
||||
|
||||
def example_ner_with_siliconflow():
|
||||
"""使用硅基流动进行实体识别"""
|
||||
print("示例2: 使用硅基流动进行实体识别")
|
||||
print("-" * 60)
|
||||
|
||||
config = load_config()
|
||||
|
||||
# 注意：NERExtractor默认使用Qwen，我们可以修改为使用硅基流动
|
||||
# 方法1: 直接创建NER提取器并替换LLM客户端
|
||||
ner_extractor = NERExtractor(model="Qwen/Qwen2.5-72B-Instruct", config=config)
|
||||
|
||||
# 替换为硅基流动客户端
|
||||
ner_extractor.llm_client = LLMClient(
|
||||
provider=LLMProvider.SILICONFLOW,
|
||||
model="Qwen/Qwen2.5-72B-Instruct", # 请根据实际情况修改
|
||||
config=config
|
||||
)
|
||||
|
||||
text = "城市更新应当遵循国土空间规划，坚持生态优先、绿色发展原则。"
|
||||
entities = ner_extractor.extract(text)
|
||||
|
||||
print(f"输入文本: {text}")
|
||||
print(f"识别到的实体: {entities}\n")
|
||||
|
||||
|
||||
def example_multi_provider_comparison():
|
||||
"""多提供商对比示例"""
|
||||
print("示例3: 对比不同LLM提供商的响应")
|
||||
print("-" * 60)
|
||||
|
||||
config = load_config()
|
||||
messages = [
|
||||
{"role": "user", "content": "什么是知识图谱？用一句话回答。"}
|
||||
]
|
||||
|
||||
providers = [
|
||||
(LLMProvider.SILICONFLOW, "Qwen/Qwen2.5-72B-Instruct"),
|
||||
# 可以添加其他提供商进行对比
|
||||
]
|
||||
|
||||
for provider, model in providers:
|
||||
try:
|
||||
client = LLMClient(provider=provider, model=model, config=config)
|
||||
response = client.chat(messages)
|
||||
print(f"{provider.value}: {response[:100]}...")
|
||||
except Exception as e:
|
||||
print(f"{provider.value}: 调用失败 - {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("=" * 60)
|
||||
print("硅基流动API使用示例")
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
# 运行示例
|
||||
try:
|
||||
example_basic_usage()
|
||||
except Exception as e:
|
||||
print(f"示例1执行失败: {e}\n")
|
||||
|
||||
try:
|
||||
example_ner_with_siliconflow()
|
||||
except Exception as e:
|
||||
print(f"示例2执行失败: {e}\n")
|
||||
|
||||
try:
|
||||
example_multi_provider_comparison()
|
||||
except Exception as e:
|
||||
print(f"示例3执行失败: {e}\n")
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
[project]
|
||||
name = "law-graph-kg"
|
||||
version = "0.1.0"
|
||||
description = "国土空间规划法规知识图谱构建与分析系统"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
"hanlp>=2.1.0",
|
||||
"openai>=1.0.0",
|
||||
"anthropic>=0.18.0",
|
||||
"dashscope>=1.17.0", # 阿里云Qwen API
|
||||
"volcengine-python-sdk>=1.0.0", # 字节跳动Doubao API
|
||||
"zhipuai>=2.0.0", # 智谱GLM API
|
||||
"networkx>=3.2.0",
|
||||
"python-igraph>=0.11.0",
|
||||
"pyyaml>=6.0",
|
||||
"pandas>=2.0.0",
|
||||
"python-dotenv>=1.0.0",
|
||||
"numpy>=1.24.0",
|
||||
"tqdm>=4.66.0",
|
||||
"aiohttp>=3.9.0",
|
||||
"tenacity>=8.2.0", # 重试机制
|
||||
"python-docx>=1.1.0", # Word文档解析
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.4.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
"black>=23.0.0",
|
||||
"ruff>=0.1.0",
|
||||
"mypy>=1.5.0",
|
||||
]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src"]
|
||||
|
||||
[tool.uv]
|
||||
dev-dependencies = [
|
||||
"pytest>=7.4.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
"black>=23.0.0",
|
||||
"ruff>=0.1.0",
|
||||
"mypy>=1.5.0",
|
||||
]
|
||||
|
||||
[tool.black]
|
||||
line-length = 100
|
||||
target-version = ['py312']
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
target-version = "py312"
|
||||
|
||||
[tool.mypy]
|
||||
python_version = "3.12"
|
||||
warn_return_any = true
|
||||
warn_unused_configs = true
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
知识图谱构建脚本
|
||||
示例用法：python scripts/build_kg.py --data-dir ../data/1法律 --output ./output/kg.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# 添加src到路径
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.preprocessing.document_parser import DocumentParser
|
||||
from src.kg_builder.indexer import GraphIndexer
|
||||
from src.kg_builder.graph import KnowledgeGraph
|
||||
from src.utils.config import Config, load_config
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="构建法规知识图谱")
|
||||
parser.add_argument(
|
||||
"--data-dir",
|
||||
type=str,
|
||||
required=True,
|
||||
help="法规数据目录路径"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=str,
|
||||
default="output/kg.json",
|
||||
help="输出文件路径"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-docs",
|
||||
type=int,
|
||||
default=None,
|
||||
help="最大处理文档数（用于测试）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-verification",
|
||||
action="store_true",
|
||||
help="不使用二次对话验证（加快速度）"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 加载配置
|
||||
config = load_config()
|
||||
|
||||
# 确保输出目录存在
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 解析文档
|
||||
logger.info(f"解析目录: {args.data_dir}")
|
||||
parser_obj = DocumentParser()
|
||||
docs = parser_obj.parse_directory(args.data_dir)
|
||||
|
||||
if args.max_docs:
|
||||
docs = docs[:args.max_docs]
|
||||
|
||||
logger.info(f"找到 {len(docs)} 个文档")
|
||||
|
||||
# 切分为TextUnit
|
||||
logger.info("切分文档为TextUnit...")
|
||||
all_textunits = []
|
||||
for doc in docs:
|
||||
textunits = parser_obj.split_into_textunits(
|
||||
doc,
|
||||
max_length=config.MAX_TEXTUNIT_LENGTH
|
||||
)
|
||||
for tu in textunits:
|
||||
tu["doc_id"] = doc.get("file_path", "")
|
||||
tu["id"] = f"{doc.get('file_path', '')}_{tu.get('paragraph_index', 0)}"
|
||||
all_textunits.extend(textunits)
|
||||
|
||||
logger.info(f"共生成 {len(all_textunits)} 个TextUnit")
|
||||
|
||||
# 构建知识图谱
|
||||
logger.info("开始构建知识图谱...")
|
||||
indexer = GraphIndexer(config=config)
|
||||
kg_graph = indexer.index(
|
||||
all_textunits,
|
||||
use_verification=not args.no_verification
|
||||
)
|
||||
|
||||
# 保存图谱
|
||||
kg_obj = KnowledgeGraph()
|
||||
kg_obj.graph = kg_graph
|
||||
kg_obj.save(str(output_path), format="json")
|
||||
|
||||
# 保存统计信息
|
||||
stats = {
|
||||
"num_docs": len(docs),
|
||||
"num_textunits": len(all_textunits),
|
||||
"num_nodes": kg_graph.number_of_nodes(),
|
||||
"num_edges": kg_graph.number_of_edges(),
|
||||
"metadata": kg_graph.graph.get("metadata", {})
|
||||
}
|
||||
|
||||
stats_path = output_path.parent / f"{output_path.stem}_stats.json"
|
||||
with open(stats_path, "w", encoding="utf-8") as f:
|
||||
json.dump(stats, f, ensure_ascii=False, indent=2)
|
||||
|
||||
logger.info(f"知识图谱构建完成！")
|
||||
logger.info(f" - 节点数: {stats['num_nodes']}")
|
||||
logger.info(f" - 边数: {stats['num_edges']}")
|
||||
logger.info(f" - 结果已保存到: {output_path}")
|
||||
logger.info(f" - 统计信息已保存到: {stats_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
完整流程脚本：从文档到知识图谱构建和查询
|
||||
示例用法：python scripts/complete_pipeline.py --data-dir ../data/1法律 --output-dir ./output
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# 修复：移除conda环境的路径，确保使用虚拟环境的包
|
||||
sys.path = [p for p in sys.path if 'anaconda3' not in p.lower() and 'conda' not in p.lower()]
|
||||
# 添加src到路径
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.preprocessing.document_parser import DocumentParser
|
||||
from src.kg_builder.indexer import GraphIndexer
|
||||
from src.kg_builder.graph import KnowledgeGraph
|
||||
from src.utils.config import Config, load_config
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="完整的知识图谱构建流程")
|
||||
parser.add_argument(
|
||||
"--data-dir",
|
||||
type=str,
|
||||
required=True,
|
||||
help="法规数据目录路径"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=str,
|
||||
default="output",
|
||||
help="输出目录"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-docs",
|
||||
type=int,
|
||||
default=None,
|
||||
help="最大处理文档数（用于测试）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-index",
|
||||
action="store_true",
|
||||
help="跳过索引构建（使用已有索引）"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 加载配置
|
||||
config = load_config()
|
||||
|
||||
# 创建输出目录
|
||||
output_dir = Path(args.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
kg_file = output_dir / "knowledge_graph.json"
|
||||
|
||||
# 步骤1: 构建知识图谱索引
|
||||
if not args.skip_index:
|
||||
logger.info("=" * 60)
|
||||
logger.info("步骤1: 解析文档并构建知识图谱")
|
||||
logger.info("=" * 60)
|
||||
|
||||
# 解析文档
|
||||
logger.info(f"解析目录: {args.data_dir}")
|
||||
parser_obj = DocumentParser()
|
||||
docs = parser_obj.parse_directory(args.data_dir)
|
||||
|
||||
if args.max_docs:
|
||||
docs = docs[:args.max_docs]
|
||||
|
||||
logger.info(f"找到 {len(docs)} 个文档")
|
||||
|
||||
# 切分为TextUnit
|
||||
logger.info("切分文档为TextUnit...")
|
||||
all_textunits = []
|
||||
for doc in docs:
|
||||
textunits = parser_obj.split_into_textunits(
|
||||
doc,
|
||||
max_length=config.MAX_TEXTUNIT_LENGTH
|
||||
)
|
||||
for tu in textunits:
|
||||
tu["doc_id"] = doc.get("file_path", "")
|
||||
tu["id"] = f"{doc.get('file_path', '')}_{tu.get('paragraph_index', 0)}"
|
||||
all_textunits.extend(textunits)
|
||||
|
||||
logger.info(f"共生成 {len(all_textunits)} 个TextUnit")
|
||||
|
||||
# 构建知识图谱
|
||||
logger.info("开始构建知识图谱...")
|
||||
indexer = GraphIndexer(config=config)
|
||||
kg_graph = indexer.index(all_textunits, use_verification=True)
|
||||
|
||||
# 保存图谱
|
||||
kg_obj = KnowledgeGraph()
|
||||
kg_obj.graph = kg_graph
|
||||
kg_obj.save(str(kg_file), format="json")
|
||||
|
||||
logger.info(f"知识图谱已保存到: {kg_file}")
|
||||
else:
|
||||
logger.info(f"跳过索引构建，从文件加载: {kg_file}")
|
||||
kg_obj = KnowledgeGraph()
|
||||
kg_obj.load(str(kg_file), format="json")
|
||||
|
||||
# 步骤2: 展示图谱统计信息
|
||||
logger.info("=" * 60)
|
||||
logger.info("步骤2: 知识图谱统计信息")
|
||||
logger.info("=" * 60)
|
||||
|
||||
if kg_obj.graph:
|
||||
metadata = kg_obj.graph.graph.get("metadata", {})
|
||||
logger.info(f"节点数: {kg_obj.graph.number_of_nodes()}")
|
||||
logger.info(f"边数: {kg_obj.graph.number_of_edges()}")
|
||||
logger.info(f"三元组数: {metadata.get('num_triplets', 0)}")
|
||||
logger.info(f"社区数: {metadata.get('num_communities', 0)}")
|
||||
|
||||
# 展示一些社区摘要
|
||||
summaries = metadata.get("summaries", {})
|
||||
if summaries:
|
||||
logger.info("\n前5个社区摘要：")
|
||||
for i, (comm_id, comm_data) in enumerate(list(summaries.items())[:5], 1):
|
||||
logger.info(f"{i}. {comm_id}: {comm_data.get('summary', '无摘要')[:100]}...")
|
||||
|
||||
logger.info("\n知识图谱构建流程完成！")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
知识抽取脚本
|
||||
示例用法：python scripts/extract.py --input data/sample.txt --output output/entities.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# 添加src到路径
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.preprocessing.document_parser import DocumentParser
|
||||
from src.extraction.ner import NERExtractor
|
||||
from src.utils.config import Config, load_config
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="法规知识抽取工具")
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
type=str,
|
||||
required=True,
|
||||
help="输入文件路径（.docx或.txt）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=str,
|
||||
default="output/entities.json",
|
||||
help="输出文件路径"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--use-verification",
|
||||
action="store_true",
|
||||
default=True,
|
||||
help="使用二次对话验证"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 加载配置
|
||||
config = load_config()
|
||||
|
||||
# 确保输出目录存在
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 解析文档
|
||||
input_path = Path(args.input)
|
||||
if input_path.suffix == ".docx":
|
||||
parser = DocumentParser()
|
||||
doc_data = parser.parse_docx(str(input_path))
|
||||
text = doc_data["text"]
|
||||
else:
|
||||
# 假设是纯文本文件
|
||||
with open(input_path, "r", encoding="utf-8") as f:
|
||||
text = f.read()
|
||||
|
||||
# 执行NER
|
||||
logger.info("开始实体识别...")
|
||||
ner_extractor = NERExtractor(config=config)
|
||||
entities = ner_extractor.extract(text, use_verification=args.use_verification)
|
||||
|
||||
# 保存结果
|
||||
result = {
|
||||
"input_file": str(input_path),
|
||||
"entities": entities,
|
||||
"statistics": {
|
||||
entity_type: len(entity_list)
|
||||
for entity_type, entity_list in entities.items()
|
||||
}
|
||||
}
|
||||
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
|
||||
logger.info(f"实体识别完成，结果已保存到: {output_path}")
|
||||
logger.info(f"统计信息: {result['statistics']}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
知识图谱查询脚本
|
||||
示例用法：python scripts/query_kg.py --kg output/kg.json --query "城市更新"
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# 添加src到路径
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.kg_builder.graph import KnowledgeGraph
|
||||
from src.query.global_search import GlobalSearcher
|
||||
from src.query.local_search import LocalSearcher
|
||||
from src.utils.config import load_config
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="查询知识图谱")
|
||||
parser.add_argument(
|
||||
"--kg",
|
||||
type=str,
|
||||
required=True,
|
||||
help="知识图谱文件路径（JSON格式）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--query",
|
||||
type=str,
|
||||
required=True,
|
||||
help="查询文本"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mode",
|
||||
type=str,
|
||||
choices=["global", "local", "drift"],
|
||||
default="global",
|
||||
help="查询模式"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=str,
|
||||
default=None,
|
||||
help="输出结果文件路径"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 加载知识图谱
|
||||
logger.info(f"加载知识图谱: {args.kg}")
|
||||
kg = KnowledgeGraph()
|
||||
kg.load(str(args.kg), format="json")
|
||||
|
||||
# 获取社区信息（从元数据）
|
||||
communities = {}
|
||||
if hasattr(kg, 'graph') and kg.graph:
|
||||
metadata = kg.graph.graph.get("metadata", {})
|
||||
communities = metadata.get("summaries", {})
|
||||
|
||||
# 执行查询
|
||||
logger.info(f"执行{args.mode}查询: {args.query}")
|
||||
|
||||
kg_graph = kg.graph if hasattr(kg, 'graph') and kg.graph else None
|
||||
|
||||
if args.mode == "global":
|
||||
searcher = GlobalSearcher(kg_graph, communities)
|
||||
result = searcher.search(args.query)
|
||||
elif args.mode == "local":
|
||||
searcher = LocalSearcher(kg_graph)
|
||||
# 尝试从查询中提取实体
|
||||
entity = args.query # 简化处理，可以将查询作为实体
|
||||
result = searcher.search(entity, depth=2)
|
||||
else:
|
||||
# DRIFT search
|
||||
from src.query.drift_search import DriftSearcher
|
||||
searcher = DriftSearcher(kg_graph, communities)
|
||||
result = searcher.search(args.query)
|
||||
|
||||
# 输出结果
|
||||
print("\n查询结果：")
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
|
||||
if args.output:
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
logger.info(f"结果已保存到: {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
"""
|
||||
国土空间规划法规知识图谱构建与分析系统
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
"""分析和推理模块"""
|
||||
|
||||
from .legal_structure import LegalStructureExtractor
|
||||
from .reasoning import LegalReasoner
|
||||
from .evolution import EvolutionAnalyzer
|
||||
|
||||
__all__ = ["LegalStructureExtractor", "LegalReasoner", "EvolutionAnalyzer"]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
"""
|
||||
演化规律识别模块
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import List, Dict, Any
|
||||
from datetime import datetime
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EvolutionAnalyzer:
|
||||
"""演化规律分析器"""
|
||||
|
||||
def analyze_temporal_changes(
|
||||
self,
|
||||
regulations: List[Dict[str, Any]],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
时间维度分析：识别不同时期法规内容的变化
|
||||
|
||||
Args:
|
||||
regulations: 法规列表，每个包含时间和内容
|
||||
|
||||
Returns:
|
||||
演化分析结果
|
||||
"""
|
||||
# TODO: 实现时间序列分析
|
||||
logger.info("执行时间维度演化分析...")
|
||||
return {}
|
||||
|
||||
def track_relation_evolution(
|
||||
self,
|
||||
entity: str,
|
||||
time_points: List[datetime],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
关系演化追踪
|
||||
|
||||
Args:
|
||||
entity: 实体名称
|
||||
time_points: 时间点列表
|
||||
|
||||
Returns:
|
||||
演化模式
|
||||
"""
|
||||
# TODO: 实现关系演化追踪
|
||||
logger.info(f"追踪实体 {entity} 的关系演化...")
|
||||
return {}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
"""
|
||||
法理结构提取模块
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import List, Dict, Any
|
||||
import re
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LegalStructureExtractor:
|
||||
"""法理结构提取器"""
|
||||
|
||||
def extract_four_elements(self, text: str) -> Dict[str, List[str]]:
|
||||
"""
|
||||
提取四要素：目标、原则、要素、管控
|
||||
|
||||
Args:
|
||||
text: 法规文本
|
||||
|
||||
Returns:
|
||||
四要素字典
|
||||
"""
|
||||
result = {
|
||||
"目标": [],
|
||||
"原则": [],
|
||||
"要素": [],
|
||||
"管控": []
|
||||
}
|
||||
|
||||
# 目标要素识别（"为了...""应当..."等）
|
||||
target_patterns = [
|
||||
r"为了[^，。]+",
|
||||
r"应当[^，。]+",
|
||||
r"旨在[^，。]+",
|
||||
]
|
||||
for pattern in target_patterns:
|
||||
matches = re.findall(pattern, text)
|
||||
result["目标"].extend(matches)
|
||||
|
||||
# 原则要素识别（"遵循...""坚持..."等）
|
||||
principle_patterns = [
|
||||
r"遵循[^，。]+",
|
||||
r"坚持[^，。]+",
|
||||
r"按照[^，。]+",
|
||||
]
|
||||
for pattern in principle_patterns:
|
||||
matches = re.findall(pattern, text)
|
||||
result["原则"].extend(matches)
|
||||
|
||||
# 管控要素识别（"禁止...""限制...""引导..."等）
|
||||
control_patterns = [
|
||||
r"禁止[^，。]+",
|
||||
r"限制[^，。]+",
|
||||
r"引导[^，。]+",
|
||||
r"不得[^，。]+",
|
||||
]
|
||||
for pattern in control_patterns:
|
||||
matches = re.findall(pattern, text)
|
||||
result["管控"].extend(matches)
|
||||
|
||||
return result
|
||||
|
||||
def extract_cross_references(self, text: str) -> List[Dict[str, str]]:
|
||||
"""
|
||||
提取跨法规引用关系
|
||||
|
||||
Args:
|
||||
text: 法规文本
|
||||
|
||||
Returns:
|
||||
引用关系列表
|
||||
"""
|
||||
references = []
|
||||
|
||||
# 引用模式："依据...""按照..."等
|
||||
patterns = [
|
||||
(r"依据《([^》]+)》", "依据"),
|
||||
(r"按照《([^》]+)》", "按照"),
|
||||
(r"参照《([^》]+)》", "参照"),
|
||||
]
|
||||
|
||||
for pattern, relation_type in patterns:
|
||||
matches = re.finditer(pattern, text)
|
||||
for match in matches:
|
||||
references.append({
|
||||
"cited_document": match.group(1),
|
||||
"relation": relation_type,
|
||||
"position": match.start()
|
||||
})
|
||||
|
||||
return references
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
"""
|
||||
逻辑推理模块
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import List, Dict, Any
|
||||
import networkx as nx
|
||||
|
||||
from ..kg_builder.graph import KnowledgeGraph
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LegalReasoner:
|
||||
"""法规逻辑推理器"""
|
||||
|
||||
def __init__(self, kg: KnowledgeGraph):
|
||||
"""
|
||||
初始化推理器
|
||||
|
||||
Args:
|
||||
kg: 知识图谱
|
||||
"""
|
||||
self.kg = kg
|
||||
|
||||
def path_query(self, source: str, target: str, max_hops: int = 3) -> List:
|
||||
"""
|
||||
路径查询：发现实体间的多跳关系
|
||||
|
||||
Args:
|
||||
source: 源实体
|
||||
target: 目标实体
|
||||
max_hops: 最大跳数
|
||||
|
||||
Returns:
|
||||
路径列表
|
||||
"""
|
||||
return self.kg.query_path(source, target, max_hops)
|
||||
|
||||
def centrality_analysis(self) -> Dict[str, Any]:
|
||||
"""
|
||||
中心性分析：识别关键节点
|
||||
|
||||
Returns:
|
||||
中心性分析结果
|
||||
"""
|
||||
graph = self.kg.graph
|
||||
degree_centrality = nx.degree_centrality(graph)
|
||||
betweenness_centrality = nx.betweenness_centrality(graph)
|
||||
|
||||
return {
|
||||
"degree": degree_centrality,
|
||||
"betweenness": betweenness_centrality
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
"""知识抽取模块"""
|
||||
|
||||
from .ner import NERExtractor
|
||||
from .re import REExtractor
|
||||
from .evaluator import TripletEvaluator
|
||||
from .classifier import EntityRelationClassifier
|
||||
|
||||
__all__ = [
|
||||
"NERExtractor",
|
||||
"REExtractor",
|
||||
"TripletEvaluator",
|
||||
"EntityRelationClassifier",
|
||||
]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,325 @@
|
||||
"""
|
||||
实体关系分类模块
|
||||
使用RAG-based方法进行分类
|
||||
"""
|
||||
|
||||
import logging
|
||||
import numpy as np
|
||||
from typing import List, Dict, Any, Optional
|
||||
from collections import defaultdict
|
||||
|
||||
from ..utils.config import Config
|
||||
from ..utils.llm_client import LLMClient, LLMProvider
|
||||
from ..ontology.entities import get_four_element_types
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EntityRelationClassifier:
|
||||
"""实体关系分类器"""
|
||||
|
||||
def __init__(self, config: Optional[Config] = None):
|
||||
"""
|
||||
初始化分类器
|
||||
|
||||
Args:
|
||||
config: 配置对象
|
||||
"""
|
||||
self.config = config or Config()
|
||||
self.embedding_model = None
|
||||
self.knowledge_base = {} # 向量数据库（简化版）
|
||||
self.four_elements = get_four_element_types()
|
||||
|
||||
def _get_embedding(self, text: str) -> Optional[np.ndarray]:
|
||||
"""
|
||||
获取文本嵌入向量
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
|
||||
Returns:
|
||||
嵌入向量
|
||||
"""
|
||||
try:
|
||||
# 使用OpenAI的embedding模型
|
||||
import openai
|
||||
client = openai.OpenAI(api_key=self.config.OPENAI_API_KEY)
|
||||
response = client.embeddings.create(
|
||||
model="text-embedding-3-large",
|
||||
input=text
|
||||
)
|
||||
return np.array(response.data[0].embedding)
|
||||
except Exception as e:
|
||||
logger.warning(f"获取嵌入向量失败: {e}")
|
||||
return None
|
||||
|
||||
def _cosine_similarity(self, vec_a: np.ndarray, vec_b: np.ndarray) -> float:
|
||||
"""计算余弦相似度"""
|
||||
dot_product = np.dot(vec_a, vec_b)
|
||||
norm_a = np.linalg.norm(vec_a)
|
||||
norm_b = np.linalg.norm(vec_b)
|
||||
if norm_a == 0 or norm_b == 0:
|
||||
return 0.0
|
||||
return dot_product / (norm_a * norm_b)
|
||||
|
||||
def build_knowledge_base(
|
||||
self,
|
||||
labeled_entities: List[Dict[str, Any]],
|
||||
):
|
||||
"""
|
||||
构建RAG知识库（使用已标注的实体样本）
|
||||
|
||||
Args:
|
||||
labeled_entities: 已标注实体列表，格式：[{"text": "实体", "label": "标签", ...}, ...]
|
||||
"""
|
||||
logger.info(f"构建知识库，包含 {len(labeled_entities)} 个样本...")
|
||||
|
||||
self.knowledge_base = defaultdict(list)
|
||||
|
||||
for entity in labeled_entities:
|
||||
text = entity.get("text", "")
|
||||
label = entity.get("label", "")
|
||||
|
||||
if not text or not label:
|
||||
continue
|
||||
|
||||
# 获取嵌入向量
|
||||
embedding = self._get_embedding(text)
|
||||
if embedding is not None:
|
||||
self.knowledge_base[label].append({
|
||||
"text": text,
|
||||
"embedding": embedding,
|
||||
"metadata": entity
|
||||
})
|
||||
|
||||
logger.info(f"知识库构建完成，包含 {len(self.knowledge_base)} 个类别")
|
||||
|
||||
def classify_entities_rag(
|
||||
self,
|
||||
entities: List[str],
|
||||
top_k: int = 5,
|
||||
) -> List[Dict[str, str]]:
|
||||
"""
|
||||
使用RAG方法对实体进行粗粒度分类
|
||||
|
||||
Args:
|
||||
entities: 实体文本列表
|
||||
top_k: 检索的top-k相似样本数
|
||||
|
||||
Returns:
|
||||
分类结果列表，格式：[{"entity": "...", "label": "...", "confidence": ...}, ...]
|
||||
"""
|
||||
if not self.knowledge_base:
|
||||
logger.warning("知识库为空，无法进行RAG分类")
|
||||
return [{"entity": e, "label": "未知", "confidence": 0.0} for e in entities]
|
||||
|
||||
results = []
|
||||
|
||||
for entity_text in entities:
|
||||
# 获取实体嵌入
|
||||
entity_embedding = self._get_embedding(entity_text)
|
||||
if entity_embedding is None:
|
||||
results.append({
|
||||
"entity": entity_text,
|
||||
"label": "未知",
|
||||
"confidence": 0.0
|
||||
})
|
||||
continue
|
||||
|
||||
# 在所有类别中查找最相似的样本
|
||||
best_match = None
|
||||
best_similarity = -1.0
|
||||
|
||||
for label, samples in self.knowledge_base.items():
|
||||
for sample in samples:
|
||||
similarity = self._cosine_similarity(
|
||||
entity_embedding,
|
||||
sample["embedding"]
|
||||
)
|
||||
if similarity > best_similarity:
|
||||
best_similarity = similarity
|
||||
best_match = label
|
||||
|
||||
results.append({
|
||||
"entity": entity_text,
|
||||
"label": best_match or "未知",
|
||||
"confidence": best_similarity
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
def classify_entities_prompt(
|
||||
self,
|
||||
entities: List[str],
|
||||
categories: List[str] = None,
|
||||
) -> List[Dict[str, str]]:
|
||||
"""
|
||||
使用Prompt方法对实体进行分类
|
||||
|
||||
Args:
|
||||
entities: 实体文本列表
|
||||
categories: 分类类别列表，默认使用四要素类型
|
||||
|
||||
Returns:
|
||||
分类结果列表
|
||||
"""
|
||||
if categories is None:
|
||||
categories = list(self.four_elements.keys())
|
||||
|
||||
try:
|
||||
llm_client = LLMClient(
|
||||
provider=LLMProvider.QWEN,
|
||||
model="qwen-max",
|
||||
config=self.config
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"LLM客户端初始化失败: {e}")
|
||||
return []
|
||||
|
||||
results = []
|
||||
|
||||
for entity_text in entities:
|
||||
prompt = f"""请将以下实体分类到以下类别之一：
|
||||
|
||||
类别：
|
||||
{chr(10).join([f'- {cat}' for cat in categories])}
|
||||
|
||||
实体：{entity_text}
|
||||
|
||||
请只回答类别名称："""
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "你是一个城市规划和法律文本分类专家。"},
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
|
||||
try:
|
||||
label = llm_client.chat(messages).strip()
|
||||
# 验证标签是否在有效类别中
|
||||
if label not in categories:
|
||||
label = "未知"
|
||||
|
||||
results.append({
|
||||
"entity": entity_text,
|
||||
"label": label,
|
||||
"confidence": 1.0
|
||||
})
|
||||
except Exception as e:
|
||||
logger.warning(f"实体分类失败 {entity_text}: {e}")
|
||||
results.append({
|
||||
"entity": entity_text,
|
||||
"label": "未知",
|
||||
"confidence": 0.0
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
def classify_entities(
|
||||
self,
|
||||
entities: List[str],
|
||||
method: str = "rag",
|
||||
) -> List[Dict[str, str]]:
|
||||
"""
|
||||
对实体进行粗粒度分类（目标、原则、要素、管控）
|
||||
|
||||
Args:
|
||||
entities: 实体文本列表
|
||||
method: 分类方法（"rag"或"prompt"）
|
||||
|
||||
Returns:
|
||||
分类结果列表
|
||||
"""
|
||||
logger.info(f"使用 {method} 方法对 {len(entities)} 个实体进行分类...")
|
||||
|
||||
if method == "rag":
|
||||
if not self.knowledge_base:
|
||||
logger.warning("RAG知识库为空，切换到prompt方法")
|
||||
method = "prompt"
|
||||
else:
|
||||
return self.classify_entities_rag(entities)
|
||||
|
||||
if method == "prompt":
|
||||
return self.classify_entities_prompt(entities)
|
||||
|
||||
return []
|
||||
|
||||
def classify_relations(
|
||||
self,
|
||||
relations: List[str],
|
||||
) -> List[Dict[str, str]]:
|
||||
"""
|
||||
对关系进行分类
|
||||
|
||||
Args:
|
||||
relations: 关系文本列表
|
||||
|
||||
Returns:
|
||||
分类结果列表
|
||||
"""
|
||||
logger.info(f"对 {len(relations)} 个关系进行分类...")
|
||||
|
||||
# 关系分类类别（对应论文Table 5）
|
||||
relation_categories = {
|
||||
"引用": "一条法规条文明确引用另一条法规条文",
|
||||
"层级": "上下级法规之间的隶属关系",
|
||||
"包含": "一个实体在逻辑上包含另一个实体",
|
||||
"管控": "管控要素对目标要素、原则要素或其他要素的约束关系",
|
||||
"遵循": "要素遵循某种原则或目标的关系",
|
||||
"涉及": "法规条文涉及某个空间、功能或环境要素",
|
||||
"适用": "法规适用于某个行政区域",
|
||||
"演化": "法规修订前后的变化关系",
|
||||
}
|
||||
|
||||
try:
|
||||
llm_client = LLMClient(
|
||||
provider=LLMProvider.QWEN,
|
||||
model="qwen-max",
|
||||
config=self.config
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"LLM客户端初始化失败: {e}")
|
||||
return []
|
||||
|
||||
results = []
|
||||
|
||||
for relation_text in relations:
|
||||
categories_desc = "\n".join([
|
||||
f"- {cat}: {desc}" for cat, desc in relation_categories.items()
|
||||
])
|
||||
|
||||
prompt = f"""请将以下关系分类到以下类别之一：
|
||||
|
||||
类别定义：
|
||||
{categories_desc}
|
||||
|
||||
关系：{relation_text}
|
||||
|
||||
请只回答类别名称："""
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "你是一个法规关系分类专家。"},
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
|
||||
try:
|
||||
label = llm_client.chat(messages).strip()
|
||||
# 验证标签
|
||||
if label not in relation_categories:
|
||||
label = "未知"
|
||||
|
||||
results.append({
|
||||
"relation": relation_text,
|
||||
"label": label,
|
||||
"confidence": 1.0
|
||||
})
|
||||
except Exception as e:
|
||||
logger.warning(f"关系分类失败 {relation_text}: {e}")
|
||||
results.append({
|
||||
"relation": relation_text,
|
||||
"label": "未知",
|
||||
"confidence": 0.0
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
|
||||
@@ -0,0 +1,322 @@
|
||||
"""
|
||||
三元组评估模块
|
||||
使用LLM-as-a-Judge方法评估三元组质量
|
||||
实现成对排序和多维度评分（五维度：语义准确性、一致性、事实性、准确性、可理解性）
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import random
|
||||
from typing import List, Dict, Any, Optional, Tuple
|
||||
|
||||
from ..utils.llm_client import LLMClient, LLMProvider
|
||||
from ..utils.config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 评估维度定义（对应论文Table 4）
|
||||
EVALUATION_DIMENSIONS = {
|
||||
"SEM": "语义准确性（Semantically）: 三元组是否准确表示或涵盖了源文本的语义？",
|
||||
"CON": "一致性（Consistency）: 三元组提供的信息是否与源文本的逻辑一致？",
|
||||
"FAC": "事实性（Factuality）: 三元组是否保留了源文本的事实陈述？",
|
||||
"ACC": "准确性（Accuracy）: 三元组中是否有不准确、遗漏或错误的信息？",
|
||||
"UND": "可理解性（Understandability）: 三元组是否对非专业人员来说可理解和可解释？",
|
||||
}
|
||||
|
||||
|
||||
class TripletEvaluator:
|
||||
"""三元组评估器（LLM-as-a-Judge）"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
judge_models: Optional[List[Tuple[str, str]]] = None,
|
||||
config: Optional[Config] = None,
|
||||
):
|
||||
"""
|
||||
初始化三元组评估器
|
||||
|
||||
Args:
|
||||
judge_models: 评估模型列表，格式[(provider, model), ...]
|
||||
config: 配置对象
|
||||
"""
|
||||
self.config = config or Config()
|
||||
|
||||
# 默认使用GPT-4o作为评估者
|
||||
if judge_models is None:
|
||||
judge_models = [
|
||||
("openai", "gpt-4o"),
|
||||
]
|
||||
|
||||
self.judge_clients = []
|
||||
for provider_str, model in judge_models:
|
||||
provider = LLMProvider(provider_str)
|
||||
self.judge_clients.append(LLMClient(provider, model, self.config))
|
||||
|
||||
def pairwise_ranking(
|
||||
self,
|
||||
triplet_group_a: List[Dict[str, str]],
|
||||
triplet_group_b: List[Dict[str, str]],
|
||||
source_text: str,
|
||||
judge_index: int = 0,
|
||||
) -> str:
|
||||
"""
|
||||
成对排序：比较两个三元组组，确定哪个更优
|
||||
|
||||
Args:
|
||||
triplet_group_a: 三元组组A
|
||||
triplet_group_b: 三元组组B
|
||||
source_text: 源文本
|
||||
judge_index: 使用的评估者索引
|
||||
|
||||
Returns:
|
||||
"A"或"B"，表示哪个组更优
|
||||
"""
|
||||
judge = self.judge_clients[judge_index]
|
||||
|
||||
# 构建评估提示
|
||||
prompt = f"""请比较两个三元组组，确定哪个更准确地反映了源文本的语义。
|
||||
|
||||
源文本：
|
||||
{source_text}
|
||||
|
||||
三元组组A：
|
||||
{json.dumps(triplet_group_a, ensure_ascii=False, indent=2)}
|
||||
|
||||
三元组组B：
|
||||
{json.dumps(triplet_group_b, ensure_ascii=False, indent=2)}
|
||||
|
||||
请从以下维度进行比较：
|
||||
1. 语义准确性：是否准确表示源文本的语义
|
||||
2. 完整性：是否涵盖了源文本中的关键信息
|
||||
3. 正确性：三元组之间的关系是否正确
|
||||
|
||||
请只回答"A"或"B"，表示哪个组更优。如果质量相当，请回答"TIE"（平局）。"""
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "你是一个知识图谱质量评估专家，擅长评估三元组的质量。"},
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
|
||||
try:
|
||||
result = judge.chat(messages)
|
||||
result = result.strip().upper()
|
||||
if result in ["A", "B", "TIE"]:
|
||||
return result
|
||||
else:
|
||||
# 尝试从结果中提取
|
||||
if "A" in result:
|
||||
return "A"
|
||||
elif "B" in result:
|
||||
return "B"
|
||||
else:
|
||||
return "TIE"
|
||||
except Exception as e:
|
||||
logger.error(f"成对排序失败: {e}")
|
||||
return "TIE"
|
||||
|
||||
def multi_dimensional_scoring(
|
||||
self,
|
||||
triplets: List[Dict[str, str]],
|
||||
source_text: str,
|
||||
judge_index: int = 0,
|
||||
) -> Dict[str, List[int]]:
|
||||
"""
|
||||
多维度评分：从五个维度对每个三元组进行评分（1-5分）
|
||||
|
||||
Args:
|
||||
triplets: 三元组列表
|
||||
source_text: 源文本
|
||||
judge_index: 使用的评估者索引
|
||||
|
||||
Returns:
|
||||
评分字典，格式：{维度: [分数列表]}
|
||||
"""
|
||||
judge = self.judge_clients[judge_index]
|
||||
|
||||
# 为每个三元组构建评估
|
||||
scores = {dim: [] for dim in EVALUATION_DIMENSIONS.keys()}
|
||||
|
||||
for triplet in triplets:
|
||||
triplet_str = f"<{triplet.get('head', '')}, {triplet.get('relation', '')}, {triplet.get('tail', '')}>"
|
||||
|
||||
prompt = f"""请从以下五个维度评估这个三元组的质量，每个维度给出1-5分的评分（5分最高，1分最低）。
|
||||
|
||||
源文本：
|
||||
{source_text}
|
||||
|
||||
三元组：
|
||||
{triplet_str}
|
||||
|
||||
评估维度：
|
||||
{chr(10).join([f'{dim}: {desc}' for dim, desc in EVALUATION_DIMENSIONS.items()])}
|
||||
|
||||
请以JSON格式输出评分：
|
||||
{{
|
||||
"SEM": <分数1-5>,
|
||||
"CON": <分数1-5>,
|
||||
"FAC": <分数1-5>,
|
||||
"ACC": <分数1-5>,
|
||||
"UND": <分数1-5>
|
||||
}}"""
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "你是一个知识图谱质量评估专家。"},
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
|
||||
try:
|
||||
result = judge.chat(messages)
|
||||
# 尝试解析JSON
|
||||
try:
|
||||
triplet_scores = json.loads(result)
|
||||
for dim in EVALUATION_DIMENSIONS.keys():
|
||||
score = triplet_scores.get(dim, 3) # 默认3分
|
||||
# 确保分数在1-5范围内
|
||||
score = max(1, min(5, int(score)))
|
||||
scores[dim].append(score)
|
||||
except json.JSONDecodeError:
|
||||
# 如果解析失败，使用默认分数
|
||||
logger.warning(f"无法解析评分结果，使用默认分数: {result}")
|
||||
for dim in EVALUATION_DIMENSIONS.keys():
|
||||
scores[dim].append(3)
|
||||
except Exception as e:
|
||||
logger.error(f"评分失败: {e}")
|
||||
for dim in EVALUATION_DIMENSIONS.keys():
|
||||
scores[dim].append(3)
|
||||
|
||||
return scores
|
||||
|
||||
def evaluate_peer_groups(
|
||||
self,
|
||||
peer_groups: Dict[str, List[Dict[str, str]]],
|
||||
source_text: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
评估多个模型生成的三元组组（Peer Examination）
|
||||
|
||||
Args:
|
||||
peer_groups: 多个模型的结果，格式：{"model_a": [triplets], "model_b": [triplets], ...}
|
||||
source_text: 源文本
|
||||
|
||||
Returns:
|
||||
评估结果，包含排序和评分
|
||||
"""
|
||||
model_names = list(peer_groups.keys())
|
||||
|
||||
# 随机打乱顺序以降低位置偏差
|
||||
shuffled_pairs = []
|
||||
for i, model_a in enumerate(model_names):
|
||||
for model_b in model_names[i+1:]:
|
||||
shuffled_pairs.append((model_a, model_b))
|
||||
random.shuffle(shuffled_pairs)
|
||||
|
||||
# 成对排序
|
||||
rankings = {}
|
||||
for model_a, model_b in shuffled_pairs:
|
||||
# 每个评估者只评估其他模型的结果，避免自我增强偏差
|
||||
for judge_idx in range(len(self.judge_clients)):
|
||||
result = self.pairwise_ranking(
|
||||
peer_groups[model_a],
|
||||
peer_groups[model_b],
|
||||
source_text,
|
||||
judge_idx
|
||||
)
|
||||
key = f"{model_a}_vs_{model_b}"
|
||||
if key not in rankings:
|
||||
rankings[key] = []
|
||||
rankings[key].append(result)
|
||||
|
||||
# 确定最佳组
|
||||
best_group = self._determine_best_group(rankings, model_names)
|
||||
|
||||
# 对最佳组进行多维度评分
|
||||
if best_group:
|
||||
scores = self.multi_dimensional_scoring(
|
||||
peer_groups[best_group],
|
||||
source_text
|
||||
)
|
||||
else:
|
||||
scores = {}
|
||||
|
||||
return {
|
||||
"rankings": rankings,
|
||||
"best_group": best_group,
|
||||
"scores": scores,
|
||||
"selected_triplets": peer_groups.get(best_group, [])
|
||||
}
|
||||
|
||||
def _determine_best_group(
|
||||
self,
|
||||
rankings: Dict[str, List[str]],
|
||||
model_names: List[str],
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
根据排序结果确定最佳组
|
||||
|
||||
Args:
|
||||
rankings: 排序结果
|
||||
model_names: 模型名称列表
|
||||
|
||||
Returns:
|
||||
最佳模型名称
|
||||
"""
|
||||
wins = {model: 0 for model in model_names}
|
||||
|
||||
for pair_key, results in rankings.items():
|
||||
# 统计每个结果
|
||||
counts = {"A": 0, "B": 0, "TIE": 0}
|
||||
for result in results:
|
||||
if result in counts:
|
||||
counts[result] += 1
|
||||
|
||||
# 解析模型名称
|
||||
parts = pair_key.split("_vs_")
|
||||
if len(parts) == 2:
|
||||
model_a, model_b = parts
|
||||
if counts["A"] > counts["B"]:
|
||||
wins[model_a] += 1
|
||||
elif counts["B"] > counts["A"]:
|
||||
wins[model_b] += 1
|
||||
|
||||
# 返回获胜次数最多的模型
|
||||
if wins:
|
||||
best_model = max(wins.items(), key=lambda x: x[1])
|
||||
return best_model[0] if best_model[1] > 0 else model_names[0]
|
||||
return model_names[0] if model_names else None
|
||||
|
||||
def filter_low_quality_triplets(
|
||||
self,
|
||||
triplets: List[Dict[str, str]],
|
||||
scores: Dict[str, List[int]],
|
||||
min_score: int = 2,
|
||||
) -> List[Dict[str, str]]:
|
||||
"""
|
||||
根据评分筛选低质量三元组
|
||||
|
||||
Args:
|
||||
triplets: 三元组列表
|
||||
scores: 评分字典
|
||||
min_score: 最低分数阈值
|
||||
|
||||
Returns:
|
||||
筛选后的三元组列表
|
||||
"""
|
||||
filtered = []
|
||||
for i, triplet in enumerate(triplets):
|
||||
# 计算平均分
|
||||
avg_score = 0
|
||||
count = 0
|
||||
for dim_scores in scores.values():
|
||||
if i < len(dim_scores):
|
||||
avg_score += dim_scores[i]
|
||||
count += 1
|
||||
if count > 0:
|
||||
avg_score /= count
|
||||
|
||||
# 只保留平均分 >= min_score的三元组
|
||||
if avg_score >= min_score:
|
||||
filtered.append(triplet)
|
||||
|
||||
return filtered
|
||||
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
"""
|
||||
命名实体识别模块
|
||||
使用Qwen-Max进行法规实体识别（两阶段对话）
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from ..prompts.ner_prompts import NERPromptBuilder
|
||||
from ..utils.llm_client import LLMClient, LLMProvider
|
||||
from ..utils.config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class NERExtractor:
|
||||
"""命名实体识别器"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model: Optional[str] = None,
|
||||
config: Optional[Config] = None,
|
||||
):
|
||||
"""
|
||||
初始化NER提取器
|
||||
|
||||
Args:
|
||||
model: 模型名称（如果为None，会根据配置自动选择）
|
||||
config: 配置对象
|
||||
"""
|
||||
self.config = config or Config()
|
||||
|
||||
# 优先使用硅基流动，如果API密钥未配置则尝试Qwen
|
||||
if self.config.SILICONFLOW_API_KEY:
|
||||
provider = LLMProvider.SILICONFLOW
|
||||
if model is None or model == "qwen-max": # 如果使用默认值，改为硅基流动模型
|
||||
# 使用指定的模型
|
||||
model = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
|
||||
elif self.config.DASHSCOPE_API_KEY:
|
||||
provider = LLMProvider.QWEN
|
||||
if model is None:
|
||||
model = "qwen-max"
|
||||
else:
|
||||
raise ValueError("请配置SILICONFLOW_API_KEY或DASHSCOPE_API_KEY")
|
||||
|
||||
self.llm_client = LLMClient(
|
||||
provider=provider,
|
||||
model=model,
|
||||
config=self.config
|
||||
)
|
||||
self.prompt_builder = NERPromptBuilder()
|
||||
|
||||
def extract(
|
||||
self,
|
||||
text: str,
|
||||
use_verification: bool = True,
|
||||
) -> Dict[str, List[str]]:
|
||||
"""
|
||||
从文本中提取实体（两阶段对话）
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
use_verification: 是否使用二次对话验证
|
||||
|
||||
Returns:
|
||||
实体字典，格式：{实体类型: [实体列表]}
|
||||
"""
|
||||
# 获取Prompt模板
|
||||
template = self.prompt_builder.build_template()
|
||||
|
||||
# 构建第一阶段对话
|
||||
system_msg = template.task_description["system"]
|
||||
user_msg = template.task_description["user"]
|
||||
|
||||
# 构建完整的第一阶段消息
|
||||
full_user_msg = user_msg
|
||||
|
||||
# 添加候选目标
|
||||
if template.candidate_targets:
|
||||
full_user_msg += "\n\n" + template.build_candidate_targets_section()
|
||||
|
||||
# 添加任务示例
|
||||
if template.task_examples:
|
||||
full_user_msg += "\n\n" + template.build_task_examples_section()
|
||||
|
||||
# 添加任务强调
|
||||
if template.task_emphasis:
|
||||
full_user_msg += "\n\n" + template.build_task_emphasis_section()
|
||||
|
||||
# 添加输入数据
|
||||
full_user_msg += f"\n\n输入数据：\n{text}"
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": system_msg},
|
||||
{"role": "user", "content": full_user_msg}
|
||||
]
|
||||
|
||||
# 第一阶段：初始提取
|
||||
logger.info("开始第一阶段实体识别...")
|
||||
first_result = self.llm_client.chat(messages)
|
||||
|
||||
# 解析第一阶段结果
|
||||
try:
|
||||
first_entities = json.loads(first_result)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("第一阶段结果不是有效JSON，尝试修复...")
|
||||
first_entities = self._try_parse_json(first_result)
|
||||
|
||||
# 第二阶段：验证和优化
|
||||
if use_verification:
|
||||
logger.info("开始第二阶段实体验证...")
|
||||
verification_prompt = self.prompt_builder.build_verification_prompt()
|
||||
|
||||
# 移除assistant消息，将第一阶段结果整合到user消息中（符合硅基流动API要求）
|
||||
second_messages = [
|
||||
{"role": "system", "content": "你需要验证和优化之前的结果。请仔细检查结果的正确性和完整性。"},
|
||||
{"role": "user", "content": f"{verification_prompt}\n\n原始输入：\n{text}\n\n第一阶段结果：\n{first_result}\n\n请验证并优化这个结果，确保输出为有效的JSON格式。"}
|
||||
]
|
||||
|
||||
second_result = self.llm_client.chat(second_messages)
|
||||
|
||||
try:
|
||||
final_entities = json.loads(second_result)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("第二阶段结果不是有效JSON，使用第一阶段结果")
|
||||
final_entities = first_entities
|
||||
else:
|
||||
final_entities = first_entities
|
||||
|
||||
return final_entities
|
||||
|
||||
def _try_parse_json(self, text: str) -> Dict[str, List[str]]:
|
||||
"""尝试从文本中提取JSON（支持markdown代码块格式）"""
|
||||
import re
|
||||
|
||||
# 方法1: 尝试从markdown代码块中提取（```json ... ```）
|
||||
json_block_match = re.search(r'```(?:json)?\s*(\{.*?\})\s*```', text, re.DOTALL)
|
||||
if json_block_match:
|
||||
try:
|
||||
json_str = json_block_match.group(1).strip()
|
||||
return json.loads(json_str)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# 方法2: 尝试找到完整的JSON对象（处理嵌套花括号，从后往前找最后一个完整的JSON）
|
||||
# 从后往前找到最后一个完整的JSON对象
|
||||
brace_count = 0
|
||||
end_idx = -1
|
||||
for i in range(len(text) - 1, -1, -1):
|
||||
char = text[i]
|
||||
if char == '}':
|
||||
if end_idx == -1:
|
||||
end_idx = i
|
||||
brace_count += 1
|
||||
elif char == '{':
|
||||
brace_count -= 1
|
||||
if brace_count == 0 and end_idx != -1:
|
||||
try:
|
||||
json_str = text[i:end_idx+1]
|
||||
return json.loads(json_str)
|
||||
except json.JSONDecodeError:
|
||||
end_idx = -1
|
||||
continue
|
||||
|
||||
# 方法3: 简单的正则匹配（作为fallback）
|
||||
json_match = re.search(r'\{[^{}]*(?:\{[^{}]*\}[^{}]*)*\}', text, re.DOTALL)
|
||||
if json_match:
|
||||
try:
|
||||
return json.loads(json_match.group())
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# 如果都失败，返回空字典
|
||||
logger.error(f"无法解析JSON结果。原始输出前200字符: {text[:200]}")
|
||||
return {}
|
||||
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
"""
|
||||
关系抽取模块
|
||||
使用多个LLM（GPT-4o, Doubao-pro, GLM-4）并行进行关系抽取
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from ..prompts.re_prompts import REPromptBuilder
|
||||
from ..utils.llm_client import LLMClient, LLMProvider
|
||||
from ..utils.config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class REExtractor:
|
||||
"""关系抽取器"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
models: Optional[List[str]] = None,
|
||||
config: Optional[Config] = None,
|
||||
):
|
||||
"""
|
||||
初始化RE提取器
|
||||
|
||||
Args:
|
||||
models: 使用的模型列表，默认使用多个模型并行
|
||||
config: 配置对象
|
||||
"""
|
||||
self.config = config or Config()
|
||||
|
||||
# 默认使用多个模型（优先使用硅基流动）
|
||||
if models is None:
|
||||
models = []
|
||||
# 优先使用硅基流动（使用两个模型）
|
||||
if self.config.SILICONFLOW_API_KEY:
|
||||
models.append(("siliconflow", "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"))
|
||||
models.append(("siliconflow", "Qwen/Qwen2.5-7B-Instruct"))
|
||||
# 如果有其他API密钥，也添加
|
||||
if self.config.OPENAI_API_KEY:
|
||||
models.append(("openai", "gpt-4o"))
|
||||
if self.config.VOLCENGINE_ACCESS_KEY and self.config.VOLCENGINE_SECRET_KEY:
|
||||
models.append(("doubao", "doubao-pro-32k"))
|
||||
if self.config.ZHIPUAI_API_KEY:
|
||||
models.append(("glm", "glm-4-airx"))
|
||||
|
||||
# 如果都没有配置，至少使用硅基流动（即使没有密钥也会报错）
|
||||
if not models:
|
||||
models = [
|
||||
("siliconflow", "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"),
|
||||
("siliconflow", "Qwen/Qwen2.5-7B-Instruct")
|
||||
]
|
||||
|
||||
self.clients = []
|
||||
for provider_str, model in models:
|
||||
try:
|
||||
provider = LLMProvider(provider_str)
|
||||
self.clients.append(LLMClient(provider, model, self.config))
|
||||
except Exception as e:
|
||||
logger.warning(f"初始化{provider_str}客户端失败: {e}，跳过该模型")
|
||||
|
||||
self.prompt_builder = REPromptBuilder()
|
||||
|
||||
def extract(
|
||||
self,
|
||||
sentence: str,
|
||||
entities: List[Dict[str, str]],
|
||||
syntax_info: Dict = None,
|
||||
context: str = None,
|
||||
use_verification: bool = True,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
从句子中提取关系（多模型并行，支持二次对话验证）
|
||||
|
||||
Args:
|
||||
sentence: 输入句子
|
||||
entities: 已识别的实体列表，格式：[{"text": "实体文本", "type": "实体类型"}, ...]
|
||||
syntax_info: 句法分析结果
|
||||
context: 上下文段落
|
||||
use_verification: 是否使用二次对话验证
|
||||
|
||||
Returns:
|
||||
结果列表，每个包含模型名称和三元组列表
|
||||
"""
|
||||
# 格式化实体列表（用于Prompt）
|
||||
formatted_entities = self._format_entities_for_prompt(entities)
|
||||
|
||||
# 格式化句法信息
|
||||
formatted_syntax = self._format_syntax_info(syntax_info) if syntax_info else "无句法信息"
|
||||
|
||||
# 构建Prompt模板
|
||||
template = self.prompt_builder.build_template(
|
||||
entities=entities,
|
||||
syntax_info=formatted_syntax,
|
||||
context=context or ""
|
||||
)
|
||||
|
||||
# 构建第一阶段消息
|
||||
system_msg = template.task_description["system"]
|
||||
user_msg = template.task_description["user"]
|
||||
|
||||
# 构建完整的用户消息
|
||||
input_data = f"""已识别实体：
|
||||
{formatted_entities}
|
||||
|
||||
句子文本：
|
||||
{sentence}
|
||||
|
||||
句法分析结果：
|
||||
{formatted_syntax}
|
||||
|
||||
上下文段落：
|
||||
{context or "无上下文"}"""
|
||||
|
||||
full_user_msg = user_msg
|
||||
if template.candidate_targets:
|
||||
full_user_msg += "\n\n" + template.build_candidate_targets_section()
|
||||
if template.task_examples:
|
||||
full_user_msg += "\n\n" + template.build_task_examples_section()
|
||||
if template.task_emphasis:
|
||||
full_user_msg += "\n\n" + template.build_task_emphasis_section()
|
||||
full_user_msg += f"\n\n输入数据：\n{input_data}"
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": system_msg},
|
||||
{"role": "user", "content": full_user_msg}
|
||||
]
|
||||
|
||||
# 并行调用多个模型
|
||||
results = []
|
||||
for i, client in enumerate(self.clients):
|
||||
try:
|
||||
model_name = f"model_{i+1}"
|
||||
logger.info(f"使用模型 {model_name} 进行关系抽取...")
|
||||
|
||||
# 第一阶段：初始提取
|
||||
first_result = client.chat(messages)
|
||||
|
||||
# 解析第一阶段结果
|
||||
first_triplets = self._parse_triplets(first_result)
|
||||
|
||||
# 第二阶段：验证和优化
|
||||
if use_verification and first_triplets:
|
||||
verification_prompt = self.prompt_builder.build_verification_prompt()
|
||||
|
||||
# 移除assistant消息，将第一阶段结果整合到user消息中（符合硅基流动API要求）
|
||||
second_messages = [
|
||||
{"role": "system", "content": "你需要验证和优化之前提取的三元组结果。请特别关注语义逻辑的正确性、遗漏关系的补全，以及利用上下文补全省略的成分。"},
|
||||
{"role": "user", "content": f"{verification_prompt}\n\n原始输入：\n{input_data}\n\n第一阶段结果：\n{first_result}\n\n请验证并优化这个结果，确保输出为有效的JSON格式。"}
|
||||
]
|
||||
|
||||
second_result = client.chat(second_messages)
|
||||
final_triplets = self._parse_triplets(second_result)
|
||||
else:
|
||||
final_triplets = first_triplets
|
||||
|
||||
results.append({
|
||||
"model": model_name,
|
||||
"triplets": final_triplets,
|
||||
"first_triplets": first_triplets,
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"模型 {i+1} 调用失败: {e}")
|
||||
results.append({
|
||||
"model": f"model_{i+1}",
|
||||
"triplets": [],
|
||||
"error": str(e)
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
def _format_entities_for_prompt(self, entities: List[Dict[str, str]]) -> str:
|
||||
"""格式化实体列表用于Prompt"""
|
||||
if not entities:
|
||||
return "无识别实体"
|
||||
|
||||
lines = []
|
||||
for entity in entities:
|
||||
entity_type = entity.get("type", "未知类型")
|
||||
entity_text = entity.get("text", "")
|
||||
lines.append(f"- {entity_type}: {entity_text}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def _format_syntax_info(self, syntax_info: Dict) -> str:
|
||||
"""格式化句法信息"""
|
||||
if not syntax_info:
|
||||
return "无句法信息"
|
||||
|
||||
formatted = []
|
||||
if "dependency" in syntax_info:
|
||||
deps = syntax_info["dependency"]
|
||||
formatted.append("依存关系：")
|
||||
for dep in deps[:5]: # 只显示前5个
|
||||
formatted.append(f" {dep.get('word', '')} <-{dep.get('deprel', '')}- {dep.get('head', '')}")
|
||||
|
||||
return "\n".join(formatted)
|
||||
|
||||
def _parse_triplets(self, result: str) -> List[Dict[str, str]]:
|
||||
"""解析三元组结果"""
|
||||
# 尝试解析JSON
|
||||
try:
|
||||
triplets = json.loads(result)
|
||||
if isinstance(triplets, list):
|
||||
# 验证三元组格式
|
||||
valid_triplets = []
|
||||
for t in triplets:
|
||||
if isinstance(t, dict) and "head" in t and "relation" in t and "tail" in t:
|
||||
valid_triplets.append({
|
||||
"head": str(t.get("head", "")),
|
||||
"relation": str(t.get("relation", "")),
|
||||
"tail": str(t.get("tail", ""))
|
||||
})
|
||||
return valid_triplets
|
||||
elif isinstance(triplets, dict):
|
||||
# 可能是嵌套结构，尝试提取
|
||||
return []
|
||||
return []
|
||||
except json.JSONDecodeError:
|
||||
# 尝试从文本中提取JSON
|
||||
import re
|
||||
json_match = re.search(r'\[.*\]', result, re.DOTALL)
|
||||
if json_match:
|
||||
try:
|
||||
triplets = json.loads(json_match.group())
|
||||
if isinstance(triplets, list):
|
||||
return triplets
|
||||
except:
|
||||
pass
|
||||
logger.warning(f"无法解析三元组结果: {result[:100]}")
|
||||
return []
|
||||
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
"""知识图谱构建模块（参考GraphRAG）"""
|
||||
|
||||
from .indexer import GraphIndexer
|
||||
from .graph import KnowledgeGraph
|
||||
|
||||
__all__ = ["GraphIndexer", "KnowledgeGraph"]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
"""
|
||||
知识图谱结构管理
|
||||
"""
|
||||
|
||||
import networkx as nx
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional
|
||||
import json
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class KnowledgeGraph:
|
||||
"""知识图谱类"""
|
||||
|
||||
def __init__(self):
|
||||
"""初始化知识图谱"""
|
||||
self.graph = nx.DiGraph()
|
||||
|
||||
def add_triplet(
|
||||
self,
|
||||
head: str,
|
||||
relation: str,
|
||||
tail: str,
|
||||
metadata: Optional[Dict] = None,
|
||||
):
|
||||
"""
|
||||
添加三元组到图谱
|
||||
|
||||
Args:
|
||||
head: 头实体
|
||||
relation: 关系
|
||||
tail: 尾实体
|
||||
metadata: 元数据
|
||||
"""
|
||||
if not self.graph.has_node(head):
|
||||
self.graph.add_node(head, type="entity")
|
||||
if not self.graph.has_node(tail):
|
||||
self.graph.add_node(tail, type="entity")
|
||||
|
||||
self.graph.add_edge(head, tail, relation=relation, **(metadata or {}))
|
||||
|
||||
def query_path(self, source: str, target: str, max_hops: int = 3) -> List:
|
||||
"""查询路径"""
|
||||
try:
|
||||
paths = list(nx.all_simple_paths(
|
||||
self.graph, source, target, cutoff=max_hops
|
||||
))
|
||||
return paths
|
||||
except nx.NodeNotFound:
|
||||
return []
|
||||
|
||||
def get_subgraph(self, nodes: List[str]) -> nx.DiGraph:
|
||||
"""获取子图"""
|
||||
return self.graph.subgraph(nodes)
|
||||
|
||||
def save(self, filepath: str, format: str = "json"):
|
||||
"""
|
||||
保存图谱
|
||||
|
||||
Args:
|
||||
filepath: 文件路径
|
||||
format: 保存格式（"json"或"graphml"）
|
||||
"""
|
||||
if format == "json":
|
||||
data = {
|
||||
"nodes": [
|
||||
{"id": str(node), **{k: v for k, v in attrs.items()}}
|
||||
for node, attrs in self.graph.nodes(data=True)
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"source": str(u),
|
||||
"target": str(v),
|
||||
**{k: v for k, v in attrs.items()}
|
||||
}
|
||||
for u, v, attrs in self.graph.edges(data=True)
|
||||
],
|
||||
"metadata": self.graph.graph.get("metadata", {})
|
||||
}
|
||||
with open(filepath, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, ensure_ascii=False, indent=2)
|
||||
elif format == "graphml":
|
||||
try:
|
||||
nx.write_graphml(self.graph, filepath)
|
||||
except Exception as e:
|
||||
logger.error(f"保存GraphML失败: {e}")
|
||||
|
||||
def load(self, filepath: str, format: str = "json"):
|
||||
"""
|
||||
从文件加载图谱
|
||||
|
||||
Args:
|
||||
filepath: 文件路径
|
||||
format: 文件格式（"json"或"graphml"）
|
||||
"""
|
||||
if format == "json":
|
||||
with open(filepath, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
self.graph.clear()
|
||||
|
||||
# 加载节点
|
||||
for node_data in data.get("nodes", []):
|
||||
node_id = node_data.pop("id", "")
|
||||
if node_id:
|
||||
self.graph.add_node(node_id, **node_data)
|
||||
|
||||
# 加载边
|
||||
for edge_data in data.get("edges", []):
|
||||
source = edge_data.pop("source", "")
|
||||
target = edge_data.pop("target", "")
|
||||
if source and target:
|
||||
self.graph.add_edge(source, target, **edge_data)
|
||||
|
||||
# 加载元数据
|
||||
if "metadata" in data:
|
||||
self.graph.graph["metadata"] = data["metadata"]
|
||||
|
||||
logger.info(f"图谱加载完成：{self.graph.number_of_nodes()} 个节点，{self.graph.number_of_edges()} 条边")
|
||||
|
||||
elif format == "graphml":
|
||||
try:
|
||||
self.graph = nx.read_graphml(filepath)
|
||||
except Exception as e:
|
||||
logger.error(f"加载GraphML失败: {e}")
|
||||
raise
|
||||
|
||||
|
||||
@@ -0,0 +1,361 @@
|
||||
"""
|
||||
知识图谱索引器（参考GraphRAG）
|
||||
实现索引阶段：文本切分、实体提取、图构建、社区分析、摘要生成
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional
|
||||
import networkx as nx
|
||||
|
||||
from ..extraction.ner import NERExtractor
|
||||
from ..extraction.re import REExtractor
|
||||
from ..utils.config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GraphIndexer:
|
||||
"""
|
||||
知识图谱索引器
|
||||
参考GraphRAG的实现思路
|
||||
"""
|
||||
|
||||
def __init__(self, config: Optional[Config] = None):
|
||||
"""
|
||||
初始化索引器
|
||||
|
||||
Args:
|
||||
config: 配置对象
|
||||
"""
|
||||
self.config = config or Config()
|
||||
self.ner_extractor = NERExtractor(config=config)
|
||||
self.re_extractor = REExtractor(config=config)
|
||||
self.graph = nx.DiGraph()
|
||||
|
||||
def index(
|
||||
self,
|
||||
textunits: List[Dict[str, Any]],
|
||||
use_verification: bool = True,
|
||||
) -> nx.DiGraph:
|
||||
"""
|
||||
构建知识图谱索引（参考GraphRAG流程）
|
||||
|
||||
Args:
|
||||
textunits: TextUnit列表
|
||||
use_verification: 是否使用二次对话验证
|
||||
|
||||
Returns:
|
||||
构建的知识图谱
|
||||
"""
|
||||
logger.info(f"开始索引 {len(textunits)} 个TextUnit...")
|
||||
|
||||
# 步骤1: 提取实体和关系（从每个TextUnit）
|
||||
all_triplets = []
|
||||
all_entities = {}
|
||||
|
||||
from ..preprocessing.text_processor import TextProcessor
|
||||
text_processor = TextProcessor(self.config.HANLP_MODEL_PATH)
|
||||
|
||||
for i, textunit in enumerate(textunits):
|
||||
text = textunit.get("text", "")
|
||||
textunit_id = textunit.get("id", f"textunit_{i}")
|
||||
|
||||
if not text.strip():
|
||||
continue
|
||||
|
||||
logger.info(f"处理TextUnit {i+1}/{len(textunits)}: {textunit_id}")
|
||||
|
||||
# 1.1 NER：提取实体
|
||||
entities_result = self.ner_extractor.extract(text, use_verification=use_verification)
|
||||
|
||||
# 转换实体格式为RE需要的格式
|
||||
entity_list = []
|
||||
for entity_type, entity_texts in entities_result.items():
|
||||
for entity_text in entity_texts:
|
||||
if entity_text: # 过滤空字符串
|
||||
entity_list.append({
|
||||
"text": entity_text,
|
||||
"type": entity_type
|
||||
})
|
||||
|
||||
all_entities[textunit_id] = entity_list
|
||||
|
||||
# 1.2 文本预处理：获取句法信息
|
||||
sentence_analysis = text_processor.process_sentence(text)
|
||||
syntax_info = {
|
||||
"dependency": sentence_analysis.get("dependency", []),
|
||||
"constituency": sentence_analysis.get("constituency", {})
|
||||
}
|
||||
|
||||
# 1.3 RE：提取关系（多模型并行）
|
||||
if entity_list:
|
||||
re_results = self.re_extractor.extract(
|
||||
sentence=text,
|
||||
entities=entity_list,
|
||||
syntax_info=syntax_info,
|
||||
context=textunit.get("context", ""),
|
||||
use_verification=use_verification
|
||||
)
|
||||
|
||||
# 收集所有模型的三元组
|
||||
for re_result in re_results:
|
||||
triplets = re_result.get("triplets", [])
|
||||
for triplet in triplets:
|
||||
triplet["textunit_id"] = textunit_id
|
||||
triplet["source"] = text
|
||||
all_triplets.append(triplet)
|
||||
|
||||
logger.info(f"提取完成：{len(all_triplets)} 个三元组，{sum(len(ents) for ents in all_entities.values())} 个实体")
|
||||
|
||||
# 步骤2: 三元组评估和筛选
|
||||
evaluated_triplets = all_triplets
|
||||
|
||||
# 注意：完整的三元组评估需要大量LLM调用，可能会很慢
|
||||
# 这里可以根据需要启用评估，或者使用简化的筛选策略
|
||||
if False and all_triplets: # 默认关闭，避免大量API调用
|
||||
from ..extraction.evaluator import TripletEvaluator
|
||||
evaluator = TripletEvaluator(config=self.config)
|
||||
|
||||
# 按TextUnit分组进行评估
|
||||
textunit_groups = {}
|
||||
for triplet in all_triplets:
|
||||
textunit_id = triplet.get("textunit_id")
|
||||
source = triplet.get("source", "")
|
||||
if textunit_id not in textunit_groups:
|
||||
textunit_groups[textunit_id] = {
|
||||
"triplets": [],
|
||||
"source": source
|
||||
}
|
||||
textunit_groups[textunit_id]["triplets"].append(triplet)
|
||||
|
||||
# 对每个TextUnit的三元组进行评估（可选）
|
||||
evaluated_triplets = []
|
||||
for textunit_id, group_data in list(textunit_groups.items())[:5]: # 限制评估数量
|
||||
triplets = group_data["triplets"]
|
||||
source = group_data["source"]
|
||||
evaluated_triplets.extend(triplets)
|
||||
|
||||
# 简单去重
|
||||
seen = set()
|
||||
unique_triplets = []
|
||||
for triplet in evaluated_triplets:
|
||||
key = (triplet.get("head", ""), triplet.get("relation", ""), triplet.get("tail", ""))
|
||||
if key and key not in seen:
|
||||
seen.add(key)
|
||||
unique_triplets.append(triplet)
|
||||
|
||||
evaluated_triplets = unique_triplets
|
||||
|
||||
# 步骤3: 构建图
|
||||
self._build_graph(evaluated_triplets if all_triplets else [])
|
||||
|
||||
# 步骤4: 社区层次结构（使用Leiden算法）
|
||||
communities = self._detect_communities()
|
||||
|
||||
# 步骤5: 生成社区摘要
|
||||
summaries = self._generate_community_summaries(communities)
|
||||
|
||||
# 存储元数据
|
||||
self.graph.graph["metadata"] = {
|
||||
"num_textunits": len(textunits),
|
||||
"num_triplets": len(all_triplets),
|
||||
"num_entities": sum(len(ents) for ents in all_entities.values()),
|
||||
"num_communities": len(communities),
|
||||
"communities": communities,
|
||||
"summaries": summaries
|
||||
}
|
||||
|
||||
return self.graph
|
||||
|
||||
def _build_graph(self, triplets: List[Dict[str, str]]):
|
||||
"""
|
||||
构建图结构
|
||||
|
||||
Args:
|
||||
triplets: 三元组列表
|
||||
"""
|
||||
logger.info(f"构建图结构，包含 {len(triplets)} 个三元组...")
|
||||
|
||||
self.graph.clear()
|
||||
|
||||
for triplet in triplets:
|
||||
head = triplet.get("head", "")
|
||||
tail = triplet.get("tail", "")
|
||||
relation = triplet.get("relation", "")
|
||||
|
||||
if not head or not tail or not relation:
|
||||
continue
|
||||
|
||||
# 添加节点（如果不存在）
|
||||
if not self.graph.has_node(head):
|
||||
self.graph.add_node(head, type="entity")
|
||||
if not self.graph.has_node(tail):
|
||||
self.graph.add_node(tail, type="entity")
|
||||
|
||||
# 添加边
|
||||
if self.graph.has_edge(head, tail):
|
||||
# 如果边已存在，更新关系列表
|
||||
edge_data = self.graph[head][tail]
|
||||
if "relations" not in edge_data:
|
||||
edge_data["relations"] = [edge_data.get("relation", "")]
|
||||
if relation not in edge_data["relations"]:
|
||||
edge_data["relations"].append(relation)
|
||||
else:
|
||||
# 添加新边
|
||||
self.graph.add_edge(
|
||||
head,
|
||||
tail,
|
||||
relation=relation,
|
||||
textunit_id=triplet.get("textunit_id", ""),
|
||||
source=triplet.get("source", "")
|
||||
)
|
||||
|
||||
logger.info(f"图构建完成：{self.graph.number_of_nodes()} 个节点，{self.graph.number_of_edges()} 条边")
|
||||
|
||||
def _detect_communities(self):
|
||||
"""
|
||||
检测社区（使用Leiden算法）
|
||||
|
||||
Returns:
|
||||
社区列表，每个社区包含节点列表
|
||||
"""
|
||||
if self.graph.number_of_nodes() == 0:
|
||||
return []
|
||||
|
||||
try:
|
||||
import igraph as ig
|
||||
logger.info("使用Leiden算法检测社区...")
|
||||
|
||||
# 将NetworkX图转换为igraph图
|
||||
# 创建节点映射
|
||||
node_list = list(self.graph.nodes())
|
||||
node_to_idx = {node: i for i, node in enumerate(node_list)}
|
||||
|
||||
# 创建边列表（使用节点索引）
|
||||
edges = [(node_to_idx[u], node_to_idx[v]) for u, v in self.graph.edges()]
|
||||
|
||||
# 创建无向图
|
||||
g_ig = ig.Graph(edges, directed=False)
|
||||
|
||||
# 设置节点名称
|
||||
g_ig.vs["name"] = node_list
|
||||
|
||||
# 运行Leiden算法
|
||||
communities_result = g_ig.community_leiden(
|
||||
objective_function="modularity",
|
||||
resolution_parameter=1.0
|
||||
)
|
||||
|
||||
# 转换为节点列表
|
||||
communities = []
|
||||
for community in communities_result:
|
||||
nodes = [g_ig.vs[i]["name"] for i in community]
|
||||
communities.append(nodes)
|
||||
|
||||
logger.info(f"检测到 {len(communities)} 个社区")
|
||||
return communities
|
||||
|
||||
except ImportError:
|
||||
logger.warning("igraph未安装，无法使用Leiden算法，使用简单连通分量代替")
|
||||
# 使用NetworkX的连通分量作为替代
|
||||
if not self.graph.is_directed():
|
||||
components = list(nx.connected_components(self.graph.to_undirected()))
|
||||
else:
|
||||
# 对于有向图，转换为无向图
|
||||
undirected = self.graph.to_undirected()
|
||||
components = list(nx.connected_components(undirected))
|
||||
return [list(comp) for comp in components]
|
||||
except Exception as e:
|
||||
logger.error(f"社区检测失败: {e}")
|
||||
return []
|
||||
|
||||
def _generate_community_summaries(self, communities: List):
|
||||
"""
|
||||
生成社区摘要（使用LLM）
|
||||
|
||||
Args:
|
||||
communities: 社区列表
|
||||
|
||||
Returns:
|
||||
社区摘要字典
|
||||
"""
|
||||
if not communities:
|
||||
return {}
|
||||
|
||||
logger.info(f"为 {len(communities)} 个社区生成摘要...")
|
||||
|
||||
summaries = {}
|
||||
|
||||
# 使用第一个可用的LLM客户端生成摘要
|
||||
try:
|
||||
from ..utils.llm_client import LLMClient, LLMProvider
|
||||
|
||||
# 优先使用硅基流动
|
||||
if self.config.SILICONFLOW_API_KEY:
|
||||
summarizer = LLMClient(
|
||||
provider=LLMProvider.SILICONFLOW,
|
||||
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
|
||||
config=self.config
|
||||
)
|
||||
elif self.config.DASHSCOPE_API_KEY:
|
||||
summarizer = LLMClient(
|
||||
provider=LLMProvider.QWEN,
|
||||
model="qwen-max",
|
||||
config=self.config
|
||||
)
|
||||
else:
|
||||
raise ValueError("请配置SILICONFLOW_API_KEY或DASHSCOPE_API_KEY")
|
||||
|
||||
for i, community_nodes in enumerate(communities[:20]): # 限制前20个社区
|
||||
# 获取社区中的节点和关系信息
|
||||
subgraph = self.graph.subgraph(community_nodes)
|
||||
|
||||
# 构建摘要提示
|
||||
nodes_info = "\n".join([f"- {node}" for node in list(community_nodes)[:10]])
|
||||
edges_info = "\n".join([
|
||||
f"- {u} --{self.graph[u][v].get('relation', '')}--> {v}"
|
||||
for u, v in list(subgraph.edges())[:10]
|
||||
])
|
||||
|
||||
prompt = f"""请为以下知识图谱社区生成一个简洁的摘要（1-2句话），概括该社区的主要主题和内容。
|
||||
|
||||
社区节点：
|
||||
{nodes_info}
|
||||
|
||||
社区关系：
|
||||
{edges_info}
|
||||
|
||||
摘要："""
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "你是一个知识图谱分析专家，擅长生成简洁准确的社区摘要。"},
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
|
||||
try:
|
||||
summary = summarizer.chat(messages)
|
||||
summaries[f"community_{i+1}"] = {
|
||||
"nodes": list(community_nodes),
|
||||
"summary": summary.strip(),
|
||||
"size": len(community_nodes)
|
||||
}
|
||||
except Exception as e:
|
||||
logger.warning(f"社区 {i+1} 摘要生成失败: {e}")
|
||||
summaries[f"community_{i+1}"] = {
|
||||
"nodes": list(community_nodes),
|
||||
"summary": "摘要生成失败",
|
||||
"size": len(community_nodes)
|
||||
}
|
||||
except Exception as e:
|
||||
logger.warning(f"摘要生成器初始化失败: {e}")
|
||||
# 生成简单摘要
|
||||
for i, community_nodes in enumerate(communities):
|
||||
summaries[f"community_{i+1}"] = {
|
||||
"nodes": list(community_nodes),
|
||||
"summary": f"包含 {len(community_nodes)} 个实体的社区",
|
||||
"size": len(community_nodes)
|
||||
}
|
||||
|
||||
return summaries
|
||||
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
"""本体模型定义模块"""
|
||||
|
||||
from .entities import EntityType, get_entity_definitions
|
||||
from .relations import RelationType, get_relation_definitions
|
||||
from .schema import OntologySchema
|
||||
|
||||
__all__ = [
|
||||
"EntityType",
|
||||
"get_entity_definitions",
|
||||
"RelationType",
|
||||
"get_relation_definitions",
|
||||
"OntologySchema",
|
||||
]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
"""
|
||||
实体类型定义
|
||||
对应论文中的9种实体类型
|
||||
"""
|
||||
|
||||
from enum import Enum
|
||||
from typing import Dict
|
||||
|
||||
|
||||
class EntityType(str, Enum):
|
||||
"""实体类型枚举"""
|
||||
|
||||
# 法规条文
|
||||
LEGAL_PROVISION = "法规条文"
|
||||
|
||||
# 目标要素
|
||||
TARGET_ELEMENT = "目标要素"
|
||||
|
||||
# 原则要素
|
||||
PRINCIPLE_ELEMENT = "原则要素"
|
||||
|
||||
# 空间要素
|
||||
SPATIAL_ELEMENT = "空间要素"
|
||||
|
||||
# 功能要素
|
||||
FUNCTIONAL_ELEMENT = "功能要素"
|
||||
|
||||
# 环境要素
|
||||
ENVIRONMENTAL_ELEMENT = "环境要素"
|
||||
|
||||
# 管控要素
|
||||
CONTROL_ELEMENT = "管控要素"
|
||||
|
||||
# 行政区域
|
||||
ADMINISTRATIVE_REGION = "行政区域"
|
||||
|
||||
# 时间要素
|
||||
TEMPORAL_ELEMENT = "时间要素"
|
||||
|
||||
|
||||
def get_entity_definitions() -> Dict[str, str]:
|
||||
"""
|
||||
获取实体类型定义（对应论文Table 1）
|
||||
|
||||
Returns:
|
||||
实体类型名称到定义的映射
|
||||
"""
|
||||
return {
|
||||
EntityType.LEGAL_PROVISION.value: "法规文档中的具体条文，包括章、节、条、款、项等不同层级",
|
||||
EntityType.TARGET_ELEMENT.value: "法规中明确表达的战略目标、发展愿景、总体目标等宏观性表述",
|
||||
EntityType.PRINCIPLE_ELEMENT.value: "法规遵循的基本准则、指导方针、基本原则等",
|
||||
EntityType.SPATIAL_ELEMENT.value: "法规涉及的地理空间实体，包括行政区划、功能区、保护区域等",
|
||||
EntityType.FUNCTIONAL_ELEMENT.value: "法规涉及的功能性要素，如土地利用类型、产业类型、基础设施等",
|
||||
EntityType.ENVIRONMENTAL_ELEMENT.value: "法规涉及的环境保护相关要素，如生态保护区、环境质量标准等",
|
||||
EntityType.CONTROL_ELEMENT.value: "法规对各类要素的管控要求，包括禁止、限制、引导等管控措施",
|
||||
EntityType.ADMINISTRATIVE_REGION.value: "法规适用的行政区划，包括省、市、县、乡等不同层级",
|
||||
EntityType.TEMPORAL_ELEMENT.value: "法规涉及的时间概念，如规划期限、实施阶段、有效期等",
|
||||
}
|
||||
|
||||
|
||||
def get_four_element_types() -> Dict[str, str]:
|
||||
"""
|
||||
获取四要素类型（目标、原则、要素、管控）
|
||||
|
||||
Returns:
|
||||
四要素类型定义
|
||||
"""
|
||||
return {
|
||||
"目标": EntityType.TARGET_ELEMENT.value,
|
||||
"原则": EntityType.PRINCIPLE_ELEMENT.value,
|
||||
"要素": [
|
||||
EntityType.SPATIAL_ELEMENT.value,
|
||||
EntityType.FUNCTIONAL_ELEMENT.value,
|
||||
EntityType.ENVIRONMENTAL_ELEMENT.value,
|
||||
],
|
||||
"管控": EntityType.CONTROL_ELEMENT.value,
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
"""
|
||||
关系类型定义
|
||||
对应论文中的8种关系类型
|
||||
"""
|
||||
|
||||
from enum import Enum
|
||||
from typing import Dict
|
||||
|
||||
|
||||
class RelationType(str, Enum):
|
||||
"""关系类型枚举"""
|
||||
|
||||
# 引用关系
|
||||
CITES = "引用"
|
||||
|
||||
# 层级关系
|
||||
HIERARCHY = "层级"
|
||||
|
||||
# 包含关系
|
||||
CONTAINS = "包含"
|
||||
|
||||
# 管控关系
|
||||
CONTROLS = "管控"
|
||||
|
||||
# 遵循关系
|
||||
FOLLOWS = "遵循"
|
||||
|
||||
# 涉及关系
|
||||
INVOLVES = "涉及"
|
||||
|
||||
# 适用关系
|
||||
APPLIES_TO = "适用"
|
||||
|
||||
# 演化关系
|
||||
EVOLVES = "演化"
|
||||
|
||||
|
||||
def get_relation_definitions() -> Dict[str, str]:
|
||||
"""
|
||||
获取关系类型定义（对应论文Table 2）
|
||||
|
||||
Returns:
|
||||
关系类型名称到定义的映射
|
||||
"""
|
||||
return {
|
||||
RelationType.CITES.value: "一条法规条文明确引用另一条法规条文",
|
||||
RelationType.HIERARCHY.value: "上下级法规之间的隶属关系或法规条文内部的层级关系",
|
||||
RelationType.CONTAINS.value: "一个实体在逻辑上包含另一个实体",
|
||||
RelationType.CONTROLS.value: "管控要素对目标要素、原则要素或其他要素的约束关系",
|
||||
RelationType.FOLLOWS.value: "要素遵循某种原则或目标的关系",
|
||||
RelationType.INVOLVES.value: "法规条文涉及某个空间、功能或环境要素",
|
||||
RelationType.APPLIES_TO.value: "法规适用于某个行政区域",
|
||||
RelationType.EVOLVES.value: "法规修订前后的变化关系或时间序列上的演化关系",
|
||||
}
|
||||
|
||||
|
||||
def get_hierarchical_relations() -> Dict[str, str]:
|
||||
"""
|
||||
获取层级化关系（目标—原则—要素—管控）
|
||||
|
||||
Returns:
|
||||
层级关系定义
|
||||
"""
|
||||
return {
|
||||
"目标→原则": RelationType.FOLLOWS.value,
|
||||
"原则→要素": RelationType.FOLLOWS.value,
|
||||
"要素→管控": RelationType.CONTROLS.value,
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
"""
|
||||
本体模式定义
|
||||
实现三维本体结构：法规层级—要素类型—关联关系
|
||||
"""
|
||||
|
||||
from typing import Dict, List, Optional
|
||||
from .entities import EntityType, get_entity_definitions, get_four_element_types
|
||||
from .relations import RelationType, get_relation_definitions, get_hierarchical_relations
|
||||
|
||||
|
||||
class OntologySchema:
|
||||
"""
|
||||
法规知识本体模式
|
||||
三维结构：法规层级—要素类型—关联关系
|
||||
"""
|
||||
|
||||
# 法规层级
|
||||
LEGAL_LEVELS = [
|
||||
"法律",
|
||||
"行政法规",
|
||||
"部门规章",
|
||||
"党中央国务院文件",
|
||||
"国家主管部门文件",
|
||||
"技术标准",
|
||||
]
|
||||
|
||||
def __init__(self):
|
||||
"""初始化本体模式"""
|
||||
self.entity_types = get_entity_definitions()
|
||||
self.relation_types = get_relation_definitions()
|
||||
self.four_elements = get_four_element_types()
|
||||
self.hierarchical_relations = get_hierarchical_relations()
|
||||
|
||||
def get_entity_type(self, entity_name: str) -> Optional[str]:
|
||||
"""
|
||||
根据实体名称获取实体类型
|
||||
|
||||
Args:
|
||||
entity_name: 实体名称
|
||||
|
||||
Returns:
|
||||
实体类型，如果未找到返回None
|
||||
"""
|
||||
for entity_type, definition in self.entity_types.items():
|
||||
if entity_name in definition:
|
||||
return entity_type
|
||||
return None
|
||||
|
||||
def get_relation_type(self, relation_name: str) -> Optional[str]:
|
||||
"""
|
||||
根据关系名称获取关系类型
|
||||
|
||||
Args:
|
||||
relation_name: 关系名称
|
||||
|
||||
Returns:
|
||||
关系类型，如果未找到返回None
|
||||
"""
|
||||
return self.relation_types.get(relation_name)
|
||||
|
||||
def is_valid_entity_relation_pair(
|
||||
self,
|
||||
head_entity_type: str,
|
||||
relation_type: str,
|
||||
tail_entity_type: str
|
||||
) -> bool:
|
||||
"""
|
||||
检查实体-关系-实体三元组是否有效
|
||||
|
||||
Args:
|
||||
head_entity_type: 头实体类型
|
||||
relation_type: 关系类型
|
||||
tail_entity_type: 尾实体类型
|
||||
|
||||
Returns:
|
||||
是否为有效的三元组
|
||||
"""
|
||||
# 检查层级关系：目标→原则→要素→管控
|
||||
if relation_type == RelationType.FOLLOWS.value:
|
||||
if (head_entity_type == EntityType.TARGET_ELEMENT.value and
|
||||
tail_entity_type == EntityType.PRINCIPLE_ELEMENT.value):
|
||||
return True
|
||||
if (head_entity_type == EntityType.PRINCIPLE_ELEMENT.value and
|
||||
tail_entity_type in [
|
||||
EntityType.SPATIAL_ELEMENT.value,
|
||||
EntityType.FUNCTIONAL_ELEMENT.value,
|
||||
EntityType.ENVIRONMENTAL_ELEMENT.value,
|
||||
]):
|
||||
return True
|
||||
|
||||
if relation_type == RelationType.CONTROLS.value:
|
||||
if (head_entity_type == EntityType.CONTROL_ELEMENT.value and
|
||||
tail_entity_type in [
|
||||
EntityType.TARGET_ELEMENT.value,
|
||||
EntityType.PRINCIPLE_ELEMENT.value,
|
||||
EntityType.SPATIAL_ELEMENT.value,
|
||||
EntityType.FUNCTIONAL_ELEMENT.value,
|
||||
EntityType.ENVIRONMENTAL_ELEMENT.value,
|
||||
]):
|
||||
return True
|
||||
|
||||
# 其他关系类型的基本检查
|
||||
return (head_entity_type in self.entity_types and
|
||||
tail_entity_type in self.entity_types and
|
||||
relation_type in self.relation_types)
|
||||
|
||||
def get_schema_summary(self) -> Dict:
|
||||
"""
|
||||
获取本体模式摘要
|
||||
|
||||
Returns:
|
||||
包含实体类型数、关系类型数、法规层级数等的字典
|
||||
"""
|
||||
return {
|
||||
"legal_levels": len(self.LEGAL_LEVELS),
|
||||
"entity_types": len(self.entity_types),
|
||||
"relation_types": len(self.relation_types),
|
||||
"four_elements": list(self.four_elements.keys()),
|
||||
"entity_definitions": self.entity_types,
|
||||
"relation_definitions": self.relation_types,
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
"""文本预处理模块"""
|
||||
|
||||
from .text_processor import TextProcessor
|
||||
from .document_parser import DocumentParser
|
||||
|
||||
__all__ = ["TextProcessor", "DocumentParser"]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,242 @@
|
||||
"""
|
||||
法规文档解析模块
|
||||
解析Word文档（.docx）格式的法规文件，提取文本并进行切分
|
||||
"""
|
||||
|
||||
import os
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Optional
|
||||
from docx import Document
|
||||
from docx.shared import Inches
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DocumentParser:
|
||||
"""法规文档解析器"""
|
||||
|
||||
def __init__(self):
|
||||
"""初始化文档解析器"""
|
||||
pass
|
||||
|
||||
def parse_docx(self, file_path: str) -> Dict:
|
||||
"""
|
||||
解析.docx文档
|
||||
|
||||
Args:
|
||||
file_path: 文档路径
|
||||
|
||||
Returns:
|
||||
包含文档信息的字典：
|
||||
- title: 文档标题
|
||||
- text: 完整文本
|
||||
- paragraphs: 段落列表
|
||||
- structure: 文档结构（章节信息）
|
||||
"""
|
||||
try:
|
||||
doc = Document(file_path)
|
||||
|
||||
# 提取标题（通常在第一段）
|
||||
title = ""
|
||||
if doc.paragraphs:
|
||||
title = doc.paragraphs[0].text.strip()
|
||||
|
||||
# 提取所有段落
|
||||
paragraphs = []
|
||||
structure = []
|
||||
current_chapter = None
|
||||
current_section = None
|
||||
|
||||
for para in doc.paragraphs:
|
||||
text = para.text.strip()
|
||||
if not text:
|
||||
continue
|
||||
|
||||
# 识别章节结构
|
||||
if self._is_chapter_heading(text):
|
||||
current_chapter = text
|
||||
current_section = None
|
||||
structure.append({
|
||||
'type': 'chapter',
|
||||
'title': text,
|
||||
'level': self._get_heading_level(para)
|
||||
})
|
||||
elif self._is_section_heading(text):
|
||||
current_section = text
|
||||
structure.append({
|
||||
'type': 'section',
|
||||
'title': text,
|
||||
'chapter': current_chapter,
|
||||
'level': self._get_heading_level(para)
|
||||
})
|
||||
|
||||
paragraphs.append({
|
||||
'text': text,
|
||||
'chapter': current_chapter,
|
||||
'section': current_section,
|
||||
'style': para.style.name if para.style else None
|
||||
})
|
||||
|
||||
# 合并所有文本
|
||||
full_text = '\n'.join([p['text'] for p in paragraphs])
|
||||
|
||||
return {
|
||||
'file_path': file_path,
|
||||
'title': title,
|
||||
'text': full_text,
|
||||
'paragraphs': paragraphs,
|
||||
'structure': structure
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"解析文档失败 {file_path}: {e}")
|
||||
raise
|
||||
|
||||
def _is_chapter_heading(self, text: str) -> bool:
|
||||
"""判断是否为章节标题"""
|
||||
# 匹配模式：第一章、第一章、第一编等
|
||||
patterns = [
|
||||
r'^第[一二三四五六七八九十]+章',
|
||||
r'^第[一二三四五六七八九十]+编',
|
||||
r'^第\d+章',
|
||||
r'^第\d+编'
|
||||
]
|
||||
import re
|
||||
for pattern in patterns:
|
||||
if re.match(pattern, text):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _is_section_heading(self, text: str) -> bool:
|
||||
"""判断是否为节标题"""
|
||||
patterns = [
|
||||
r'^第[一二三四五六七八九十]+节',
|
||||
r'^第\d+节'
|
||||
]
|
||||
import re
|
||||
for pattern in patterns:
|
||||
if re.match(pattern, text):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _get_heading_level(self, para) -> int:
|
||||
"""获取标题层级"""
|
||||
style_name = para.style.name if para.style else ""
|
||||
if 'Heading 1' in style_name or '标题 1' in style_name:
|
||||
return 1
|
||||
elif 'Heading 2' in style_name or '标题 2' in style_name:
|
||||
return 2
|
||||
elif 'Heading 3' in style_name or '标题 3' in style_name:
|
||||
return 3
|
||||
else:
|
||||
return 0
|
||||
|
||||
def split_into_textunits(
|
||||
self,
|
||||
doc_data: Dict,
|
||||
max_length: int = 500
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
将文档切分为TextUnit（参考GraphRAG）
|
||||
|
||||
Args:
|
||||
doc_data: 解析后的文档数据
|
||||
max_length: 每个TextUnit的最大字符数
|
||||
|
||||
Returns:
|
||||
TextUnit列表，每个包含：
|
||||
- text: 文本内容
|
||||
- chapter: 所属章节
|
||||
- section: 所属节
|
||||
- paragraph_index: 段落索引
|
||||
- start_char: 起始字符位置
|
||||
- end_char: 结束字符位置
|
||||
"""
|
||||
textunits = []
|
||||
current_text = ""
|
||||
current_unit = {
|
||||
'chapter': None,
|
||||
'section': None,
|
||||
'paragraph_index': None,
|
||||
'start_char': 0
|
||||
}
|
||||
char_offset = 0
|
||||
|
||||
for i, para in enumerate(doc_data['paragraphs']):
|
||||
para_text = para['text']
|
||||
|
||||
# 如果当前累积文本加上新段落超过限制，保存当前TextUnit
|
||||
if len(current_text) + len(para_text) > max_length and current_text:
|
||||
current_unit.update({
|
||||
'text': current_text.strip(),
|
||||
'end_char': char_offset,
|
||||
'file_path': doc_data['file_path']
|
||||
})
|
||||
textunits.append(current_unit.copy())
|
||||
|
||||
# 开始新的TextUnit
|
||||
current_text = para_text
|
||||
current_unit = {
|
||||
'chapter': para.get('chapter'),
|
||||
'section': para.get('section'),
|
||||
'paragraph_index': i,
|
||||
'start_char': char_offset
|
||||
}
|
||||
else:
|
||||
# 累积文本
|
||||
if current_text:
|
||||
current_text += "\n" + para_text
|
||||
else:
|
||||
current_text = para_text
|
||||
current_unit.update({
|
||||
'chapter': para.get('chapter'),
|
||||
'section': para.get('section'),
|
||||
'paragraph_index': i,
|
||||
'start_char': char_offset
|
||||
})
|
||||
|
||||
char_offset += len(para_text) + 1 # +1 for newline
|
||||
|
||||
# 保存最后一个TextUnit
|
||||
if current_text:
|
||||
current_unit.update({
|
||||
'text': current_text.strip(),
|
||||
'end_char': char_offset,
|
||||
'file_path': doc_data['file_path']
|
||||
})
|
||||
textunits.append(current_unit)
|
||||
|
||||
return textunits
|
||||
|
||||
def parse_directory(
|
||||
self,
|
||||
directory: str,
|
||||
file_pattern: str = "*.docx"
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
解析目录中的所有文档
|
||||
|
||||
Args:
|
||||
directory: 目录路径
|
||||
file_pattern: 文件匹配模式
|
||||
|
||||
Returns:
|
||||
所有文档的解析结果列表
|
||||
"""
|
||||
docs = []
|
||||
dir_path = Path(directory)
|
||||
|
||||
for file_path in dir_path.rglob(file_pattern):
|
||||
try:
|
||||
doc_data = self.parse_docx(str(file_path))
|
||||
docs.append(doc_data)
|
||||
logger.info(f"成功解析文档: {file_path}")
|
||||
except Exception as e:
|
||||
logger.warning(f"跳过文档 {file_path}: {e}")
|
||||
|
||||
return docs
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,271 @@
|
||||
"""
|
||||
HanLP文本处理模块
|
||||
处理中文法规文本的分词、依存句法分析和句法成分分析
|
||||
"""
|
||||
|
||||
from typing import List, Dict, Optional, Tuple
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 延迟导入hanlp，避免在导入时失败
|
||||
try:
|
||||
import hanlp
|
||||
HANLP_AVAILABLE = True
|
||||
except ImportError:
|
||||
HANLP_AVAILABLE = False
|
||||
logger.warning("HanLP未安装，将使用简化处理")
|
||||
|
||||
|
||||
class TextProcessor:
|
||||
"""文本处理器，使用HanLP进行中文文本处理"""
|
||||
|
||||
def __init__(self, model_path: Optional[str] = None):
|
||||
"""
|
||||
初始化文本处理器
|
||||
|
||||
Args:
|
||||
model_path: HanLP模型路径，如果为None则使用默认模型
|
||||
"""
|
||||
self.hanlp = None
|
||||
|
||||
if not HANLP_AVAILABLE:
|
||||
logger.warning("HanLP未安装，将使用简化处理。安装命令: pip install 'hanlp[full]' -U")
|
||||
return
|
||||
|
||||
try:
|
||||
if model_path:
|
||||
logger.info(f"加载HanLP模型: {model_path}")
|
||||
self.hanlp = hanlp.load(model_path)
|
||||
logger.info("HanLP模型加载成功")
|
||||
else:
|
||||
# 尝试加载HanLP默认模型（较小的中文模型）
|
||||
logger.info("尝试加载HanLP默认模型...")
|
||||
try:
|
||||
# 方法1: 尝试使用预定义的小型中文模型
|
||||
import hanlp.pretrained
|
||||
model_name = hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH
|
||||
logger.info(f"使用模型: {model_name}")
|
||||
self.hanlp = hanlp.load(model_name)
|
||||
logger.info("HanLP默认模型加载成功")
|
||||
except (AttributeError, ImportError) as e1:
|
||||
logger.debug(f"方法1失败: {e1}，尝试方法2...")
|
||||
try:
|
||||
# 方法2: 使用pipeline方式
|
||||
self.hanlp = hanlp.pipeline('tok/coarse')
|
||||
logger.info("HanLP pipeline加载成功")
|
||||
except Exception as e2:
|
||||
logger.debug(f"方法2失败: {e2}，尝试方法3...")
|
||||
# 方法3: 尝试最简单的加载方式
|
||||
self.hanlp = hanlp.load()
|
||||
logger.info("HanLP默认加载成功")
|
||||
except Exception as e:
|
||||
logger.warning(f"HanLP模型加载失败: {e}，将使用简化处理（仅文档解析和基本分词）")
|
||||
logger.info("提示: 首次加载会自动下载模型，可能需要一些时间")
|
||||
logger.info("如需使用完整功能，请安装完整版: pip install 'hanlp[full]' -U")
|
||||
self.hanlp = None
|
||||
|
||||
def tokenize(self, text: str, coarse: bool = True) -> List[str]:
|
||||
"""
|
||||
文本分词
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
coarse: 是否使用粗粒度分词（默认True，保持术语完整性）
|
||||
|
||||
Returns:
|
||||
分词结果列表
|
||||
"""
|
||||
if self.hanlp is None:
|
||||
# 简化处理：按标点符号和空格简单切分
|
||||
import re
|
||||
tokens = re.split(r'[，。；：！？、\s]+', text)
|
||||
return [t.strip() for t in tokens if t.strip()]
|
||||
|
||||
if coarse:
|
||||
# 使用粗粒度分词器，保持术语完整性（如"开敞空间"）
|
||||
result = self.hanlp(text, tasks='tok/coarse')
|
||||
else:
|
||||
result = self.hanlp(text, tasks='tok/fine')
|
||||
return result.get('tok', [])
|
||||
|
||||
def dependency_parse(self, text: str) -> List[Dict]:
|
||||
"""
|
||||
依存句法分析
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
|
||||
Returns:
|
||||
依存关系列表，每个元素包含：head（依存头）、deprel（依存关系类型）
|
||||
"""
|
||||
if self.hanlp is None:
|
||||
# 返回空列表，简化处理
|
||||
return []
|
||||
|
||||
# 分别获取分词和依存句法，因为HanLP MTL模型不支持同时指定多个任务
|
||||
# 先获取分词结果
|
||||
tok_result = self.hanlp(text, tasks='tok/coarse')
|
||||
tokens = tok_result.get('tok/coarse', [])
|
||||
|
||||
# 再获取依存句法分析（会自动使用已分词的token）
|
||||
dep_result = self.hanlp(text, tasks='dep')
|
||||
dependencies = dep_result.get('dep', [])
|
||||
|
||||
parsed = []
|
||||
for i, dep in enumerate(dependencies):
|
||||
# HanLP返回的依赖关系格式：(head_index, deprel)
|
||||
word = tokens[i] if i < len(tokens) else ''
|
||||
|
||||
if isinstance(dep, tuple):
|
||||
# tuple格式：(head_index, deprel)
|
||||
head_idx = dep[0] if len(dep) > 0 else -1
|
||||
deprel = dep[1] if len(dep) > 1 else ''
|
||||
head_word = tokens[head_idx] if 0 <= head_idx < len(tokens) else '' if head_idx >= 0 else 'ROOT'
|
||||
|
||||
parsed.append({
|
||||
'word': word,
|
||||
'head': head_idx,
|
||||
'head_word': head_word,
|
||||
'deprel': deprel
|
||||
})
|
||||
elif hasattr(dep, 'form'):
|
||||
# 如果是对象格式，有form属性
|
||||
parsed.append({
|
||||
'word': dep.form,
|
||||
'head': dep.head,
|
||||
'head_word': tokens[dep.head] if 0 <= dep.head < len(tokens) else '' if dep.head >= 0 else 'ROOT',
|
||||
'deprel': dep.deprel
|
||||
})
|
||||
elif isinstance(dep, dict):
|
||||
# 如果是字典格式
|
||||
head_idx = dep.get('head', -1)
|
||||
parsed.append({
|
||||
'word': dep.get('form', dep.get('word', word)),
|
||||
'head': head_idx,
|
||||
'head_word': tokens[head_idx] if 0 <= head_idx < len(tokens) else '' if head_idx >= 0 else 'ROOT',
|
||||
'deprel': dep.get('deprel', '')
|
||||
})
|
||||
else:
|
||||
# 其他格式
|
||||
logger.warning(f"未知的依存关系格式: {type(dep)}, {dep}")
|
||||
parsed.append({
|
||||
'word': word,
|
||||
'head': -1,
|
||||
'head_word': 'ROOT',
|
||||
'deprel': ''
|
||||
})
|
||||
return parsed
|
||||
|
||||
def constituency_parse(self, text: str) -> Dict:
|
||||
"""
|
||||
句法成分分析
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
|
||||
Returns:
|
||||
句法成分树（嵌套字典结构）
|
||||
"""
|
||||
if self.hanlp is None:
|
||||
# 返回空字典，简化处理
|
||||
return {}
|
||||
|
||||
result = self.hanlp(text, tasks='con')
|
||||
return result.get('con', {})
|
||||
|
||||
def pos_tag(self, text: str) -> List[Tuple[str, str]]:
|
||||
"""
|
||||
词性标注
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
|
||||
Returns:
|
||||
(词, 词性)元组列表
|
||||
"""
|
||||
if self.hanlp is None:
|
||||
# 简化处理：返回分词结果和默认词性
|
||||
tokens = self.tokenize(text)
|
||||
return [(token, 'UNKNOWN') for token in tokens]
|
||||
|
||||
result = self.hanlp(text, tasks='pos')
|
||||
tokens = result.get('tok', [])
|
||||
pos_tags = result.get('pos', [])
|
||||
return list(zip(tokens, pos_tags))
|
||||
|
||||
def process_sentence(
|
||||
self,
|
||||
sentence: str,
|
||||
include_context: bool = True
|
||||
) -> Dict:
|
||||
"""
|
||||
处理单个句子，返回所有分析结果
|
||||
|
||||
Args:
|
||||
sentence: 输入句子
|
||||
include_context: 是否包含上下文信息
|
||||
|
||||
Returns:
|
||||
包含分词、依存、句法成分等信息的字典
|
||||
"""
|
||||
result = {
|
||||
'text': sentence,
|
||||
'tokens': self.tokenize(sentence, coarse=True),
|
||||
'dependency': self.dependency_parse(sentence),
|
||||
'constituency': self.constituency_parse(sentence),
|
||||
'pos_tags': self.pos_tag(sentence)
|
||||
}
|
||||
return result
|
||||
|
||||
def has_omitted_subject(self, dependencies: List[Dict]) -> bool:
|
||||
"""
|
||||
检查句子是否省略了主语
|
||||
|
||||
Args:
|
||||
dependencies: 依存关系列表
|
||||
|
||||
Returns:
|
||||
是否存在省略的主语
|
||||
"""
|
||||
# 检查是否有主语相关的依存关系（nsubj等）
|
||||
subject_deprels = ['nsubj', 'nsubjpass', 'csubj', 'csubjpass']
|
||||
has_subject = any(
|
||||
dep['deprel'] in subject_deprels
|
||||
for dep in dependencies
|
||||
)
|
||||
return not has_subject
|
||||
|
||||
def extract_coreference_candidates(
|
||||
self,
|
||||
text: str,
|
||||
pronouns: List[str] = None
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
提取共指消解候选
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
pronouns: 代词列表，默认为["其", "该", "这", "那"]
|
||||
|
||||
Returns:
|
||||
共指候选列表
|
||||
"""
|
||||
if pronouns is None:
|
||||
pronouns = ["其", "该", "这", "那"]
|
||||
|
||||
candidates = []
|
||||
tokens = self.tokenize(text)
|
||||
pos_tags = self.pos_tag(text)
|
||||
|
||||
for i, (token, pos) in enumerate(zip(tokens, pos_tags)):
|
||||
if token in pronouns or pos[1] == 'PN': # PN表示代词
|
||||
candidates.append({
|
||||
'pronoun': token,
|
||||
'position': i,
|
||||
'pos': pos[1]
|
||||
})
|
||||
|
||||
return candidates
|
||||
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
"""Prompt模板模块"""
|
||||
|
||||
from .template import PromptTemplate, PromptModule
|
||||
from .ner_prompts import NERPromptBuilder
|
||||
from .re_prompts import REPromptBuilder
|
||||
|
||||
__all__ = [
|
||||
"PromptTemplate",
|
||||
"PromptModule",
|
||||
"NERPromptBuilder",
|
||||
"REPromptBuilder",
|
||||
]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
"""
|
||||
NER (命名实体识别) Prompt构建器
|
||||
"""
|
||||
|
||||
from typing import List, Dict, Any
|
||||
from .template import PromptTemplate
|
||||
from ..ontology.entities import EntityType, get_entity_definitions
|
||||
|
||||
|
||||
class NERPromptBuilder:
|
||||
"""NER Prompt构建器"""
|
||||
|
||||
def __init__(self):
|
||||
"""初始化NER Prompt构建器"""
|
||||
self.entity_definitions = get_entity_definitions()
|
||||
|
||||
def build_template(self) -> PromptTemplate:
|
||||
"""
|
||||
构建NER Prompt模板
|
||||
|
||||
Returns:
|
||||
PromptTemplate对象
|
||||
"""
|
||||
# 任务描述
|
||||
system_message = "你是一个经验丰富的国土空间规划法规分析专家，擅长从法规文本中识别各类知识实体。"
|
||||
user_message = """请从给定的法规文本中识别并分类所有实体。
|
||||
|
||||
任务要求：
|
||||
1. 仔细阅读输入的法规文本
|
||||
2. 识别所有属于预定义实体类型的实体
|
||||
3. 对每个实体进行正确分类
|
||||
4. 确保识别结果的完整性和准确性
|
||||
|
||||
输出格式要求：请以JSON格式输出，格式如下：
|
||||
{
|
||||
"法规条文": ["实体1", "实体2", ...],
|
||||
"目标要素": ["实体1", "实体2", ...],
|
||||
"原则要素": ["实体1", "实体2", ...],
|
||||
...
|
||||
}
|
||||
|
||||
每个实体类型对应一个列表，包含该类型的所有实体。如果某个类型没有实体，请使用空列表[]。"""
|
||||
|
||||
# 候选目标
|
||||
candidate_targets = self.entity_definitions
|
||||
|
||||
# 任务示例
|
||||
task_examples = [
|
||||
{
|
||||
"input": "优先保障先进制造业、战略性新兴产业、都市产业等发展空间，促进价值创新园区建设。",
|
||||
"output": '{"法规条文": [], "目标要素": [], "原则要素": [], "空间要素": [], "功能要素": ["先进制造业", "战略性新兴产业", "都市产业", "价值创新园区"], "环境要素": [], "管控要素": ["优先保障", "促进"], "行政区域": [], "时间要素": []}'
|
||||
},
|
||||
{
|
||||
"input": "广州市应当坚持生态优先、绿色发展，建设宜居花城、活力全球城市。",
|
||||
"output": '{"法规条文": [], "目标要素": ["宜居花城", "活力全球城市"], "原则要素": ["生态优先", "绿色发展"], "空间要素": [], "功能要素": [], "环境要素": [], "管控要素": ["坚持", "建设"], "行政区域": ["广州市"], "时间要素": []}'
|
||||
}
|
||||
]
|
||||
|
||||
# 任务强调
|
||||
task_emphasis = """严格按要求：
|
||||
1. **仅输出JSON格式结果**，不要添加任何解释性文字、markdown代码块标记或其他内容
|
||||
2. 直接输出JSON对象，不要使用```json```等markdown标记
|
||||
3. 不要添加"好的"、"让我分析"等开场白或解释
|
||||
4. 确保所有实体都被正确识别和分类
|
||||
5. 保持实体文本的完整性，不要截断
|
||||
|
||||
重要：请直接输出JSON，格式如下（不要添加任何其他文字）：
|
||||
{"法规条文": [], "目标要素": [], "原则要素": [], ...}"""
|
||||
|
||||
return PromptTemplate(
|
||||
task_description={
|
||||
"system": system_message,
|
||||
"user": user_message
|
||||
},
|
||||
candidate_targets=candidate_targets,
|
||||
task_examples=task_examples,
|
||||
task_emphasis=task_emphasis,
|
||||
)
|
||||
|
||||
def build_verification_prompt(self) -> str:
|
||||
"""
|
||||
构建验证提示（用于二次对话）
|
||||
|
||||
Returns:
|
||||
验证提示文本
|
||||
"""
|
||||
return """你之前提取的地理规划知识实体可能不完整或不准确。请结合之前的提取内容和经验，重新提取并验证结果。
|
||||
|
||||
请特别关注：
|
||||
1. 是否有遗漏的实体
|
||||
2. 实体的分类是否正确
|
||||
3. 实体文本是否完整"""
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
"""
|
||||
RE (关系抽取) Prompt构建器
|
||||
"""
|
||||
|
||||
from typing import List, Dict, Any
|
||||
from .template import PromptTemplate
|
||||
from ..ontology.relations import RelationType, get_relation_definitions
|
||||
|
||||
|
||||
class REPromptBuilder:
|
||||
"""RE Prompt构建器"""
|
||||
|
||||
def __init__(self):
|
||||
"""初始化RE Prompt构建器"""
|
||||
self.relation_definitions = get_relation_definitions()
|
||||
|
||||
def build_template(
|
||||
self,
|
||||
entities: List[Dict[str, str]],
|
||||
syntax_info: Dict = None,
|
||||
context: str = None
|
||||
) -> PromptTemplate:
|
||||
"""
|
||||
构建RE Prompt模板
|
||||
|
||||
Args:
|
||||
entities: 已识别的实体列表
|
||||
syntax_info: 句法分析结果
|
||||
context: 上下文段落信息
|
||||
|
||||
Returns:
|
||||
PromptTemplate对象
|
||||
"""
|
||||
# 任务描述
|
||||
system_message = "你是一个自然语言处理专家，擅长在城市规划领域处理关系抽取任务。"
|
||||
|
||||
user_message = """请从给定的法规文本中识别实体之间的关系，并构建知识三元组。
|
||||
|
||||
任务要求：
|
||||
1. 基于已识别的实体列表，找出实体之间的语义关系
|
||||
2. 对于省略主语的句子，需要利用上下文信息补全
|
||||
3. 构建<head entity, relation, tail entity>形式的三元组
|
||||
4. 确保三元组的语义逻辑正确
|
||||
|
||||
输出格式要求：请以JSON格式输出三元组列表，格式如下：
|
||||
[
|
||||
{"head": "头实体", "relation": "关系类型", "tail": "尾实体"},
|
||||
...
|
||||
]
|
||||
|
||||
如果句子中无法提取完整的三元组，缺失的部分请标记为null。"""
|
||||
|
||||
# 候选目标（关系类型）
|
||||
candidate_targets = self.relation_definitions
|
||||
|
||||
# 构建输入数据说明
|
||||
input_data_desc = f"""已识别实体：
|
||||
{self._format_entities(entities)}
|
||||
|
||||
句子文本：
|
||||
[待填充]
|
||||
|
||||
句法分析结果：
|
||||
{syntax_info or "待填充"}
|
||||
|
||||
上下文段落：
|
||||
{context or "待填充"}"""
|
||||
|
||||
# 任务示例
|
||||
task_examples = [
|
||||
{
|
||||
"input": "实体：['广州市', '生态优先']，句子：'广州市应当坚持生态优先、绿色发展。'",
|
||||
"output": '[{"head": "广州市", "relation": "遵循", "tail": "生态优先"}]'
|
||||
}
|
||||
]
|
||||
|
||||
# 任务强调
|
||||
task_emphasis = """严格按要求：
|
||||
1. 仅输出JSON格式的三元组数组，不要添加任何解释
|
||||
2. 确保三元组中的实体必须是已识别实体列表中的实体
|
||||
3. 关系类型必须是预定义关系类型之一
|
||||
4. 对于省略成分，利用上下文补全"""
|
||||
|
||||
template = PromptTemplate(
|
||||
task_description={
|
||||
"system": system_message,
|
||||
"user": user_message
|
||||
},
|
||||
candidate_targets=candidate_targets,
|
||||
task_examples=task_examples,
|
||||
task_emphasis=task_emphasis,
|
||||
)
|
||||
|
||||
# 存储额外信息
|
||||
template.input_data_description = input_data_desc
|
||||
return template
|
||||
|
||||
def _format_entities(self, entities: List[Dict[str, str]]) -> str:
|
||||
"""格式化实体列表"""
|
||||
if not entities:
|
||||
return "无"
|
||||
|
||||
lines = []
|
||||
for entity in entities:
|
||||
entity_type = entity.get("type", "未知类型")
|
||||
entity_text = entity.get("text", "")
|
||||
lines.append(f"- {entity_type}: {entity_text}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def build_verification_prompt(self) -> str:
|
||||
"""
|
||||
构建验证提示（用于二次对话）
|
||||
|
||||
Returns:
|
||||
验证提示文本
|
||||
"""
|
||||
return """请验证和优化之前提取的三元组结果。
|
||||
|
||||
请特别关注：
|
||||
1. 三元组的语义逻辑是否正确
|
||||
2. 是否遗漏了重要的关系
|
||||
3. 对于省略的成分，是否利用上下文正确补全
|
||||
4. 三元组是否准确反映了源文本的语义"""
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
"""
|
||||
统一Prompt模板框架
|
||||
实现五个核心模块：任务描述、候选目标、任务示例、任务强调、二次对话
|
||||
"""
|
||||
|
||||
from typing import Dict, List, Optional, Any
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class PromptModule(str, Enum):
|
||||
"""Prompt模块类型"""
|
||||
TASK_DESCRIPTION = "任务描述"
|
||||
CANDIDATE_TARGETS = "候选目标"
|
||||
TASK_EXAMPLES = "任务示例"
|
||||
TASK_EMPHASIS = "任务强调"
|
||||
SECOND_CONVERSATION = "二次对话"
|
||||
|
||||
|
||||
class PromptTemplate:
|
||||
"""
|
||||
统一Prompt模板类
|
||||
实现论文中描述的五个模块设计
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
task_description: Optional[str] = None,
|
||||
candidate_targets: Optional[Dict[str, str]] = None,
|
||||
task_examples: Optional[List[Dict[str, Any]]] = None,
|
||||
task_emphasis: Optional[str] = None,
|
||||
second_conversation: Optional[str] = None,
|
||||
):
|
||||
"""
|
||||
初始化Prompt模板
|
||||
|
||||
Args:
|
||||
task_description: 任务描述（系统消息和用户消息）
|
||||
candidate_targets: 候选目标（实体类型或关系类型定义）
|
||||
task_examples: 任务示例（少样本学习）
|
||||
task_emphasis: 任务强调（输出格式要求）
|
||||
second_conversation: 二次对话提示
|
||||
"""
|
||||
self.task_description = task_description or {}
|
||||
self.candidate_targets = candidate_targets or {}
|
||||
self.task_examples = task_examples or []
|
||||
self.task_emphasis = task_emphasis or ""
|
||||
self.second_conversation = second_conversation or ""
|
||||
|
||||
def build_system_message(self, role: str, capability: str, task: str) -> str:
|
||||
"""
|
||||
构建系统消息（任务描述模块的一部分）
|
||||
|
||||
Args:
|
||||
role: 模型角色（如"经验丰富的国土空间规划法规分析专家"）
|
||||
capability: 模型能力
|
||||
task: 任务描述
|
||||
|
||||
Returns:
|
||||
系统消息字符串
|
||||
"""
|
||||
return f"""你是一个{role}，{capability}。你的任务是{task}。"""
|
||||
|
||||
def build_user_message(
|
||||
self,
|
||||
task_steps: List[str],
|
||||
input_format: str,
|
||||
output_format: str,
|
||||
) -> str:
|
||||
"""
|
||||
构建用户消息（任务描述模块的一部分）
|
||||
|
||||
Args:
|
||||
task_steps: 任务步骤列表
|
||||
input_format: 输入格式说明
|
||||
output_format: 输出格式说明
|
||||
|
||||
Returns:
|
||||
用户消息字符串
|
||||
"""
|
||||
steps_str = "\n".join([f"{i+1}. {step}" for i, step in enumerate(task_steps)])
|
||||
return f"""任务步骤：
|
||||
{steps_str}
|
||||
|
||||
输入格式：
|
||||
{input_format}
|
||||
|
||||
输出格式：
|
||||
{output_format}"""
|
||||
|
||||
def build_candidate_targets_section(self) -> str:
|
||||
"""
|
||||
构建候选目标部分
|
||||
|
||||
Returns:
|
||||
候选目标文本
|
||||
"""
|
||||
if not self.candidate_targets:
|
||||
return ""
|
||||
|
||||
lines = ["候选目标及其定义："]
|
||||
for target, definition in self.candidate_targets.items():
|
||||
lines.append(f"- {target}: {definition}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def build_task_examples_section(self) -> str:
|
||||
"""
|
||||
构建任务示例部分（少样本学习）
|
||||
|
||||
Returns:
|
||||
任务示例文本
|
||||
"""
|
||||
if not self.task_examples:
|
||||
return ""
|
||||
|
||||
examples_text = "任务示例：\n\n"
|
||||
for i, example in enumerate(self.task_examples, 1):
|
||||
examples_text += f"示例 {i}：\n"
|
||||
examples_text += f"输入：{example.get('input', '')}\n"
|
||||
examples_text += f"输出：{example.get('output', '')}\n\n"
|
||||
|
||||
return examples_text
|
||||
|
||||
def build_task_emphasis_section(self) -> str:
|
||||
"""
|
||||
构建任务强调部分
|
||||
|
||||
Returns:
|
||||
任务强调文本
|
||||
"""
|
||||
if not self.task_emphasis:
|
||||
return ""
|
||||
|
||||
return f"""重要提示：
|
||||
{self.task_emphasis}
|
||||
|
||||
请严格按照输出格式要求，仅输出结果，避免任何解释性内容。"""
|
||||
|
||||
def build_first_conversation(
|
||||
self,
|
||||
input_data: str,
|
||||
system_message: str,
|
||||
user_message: str,
|
||||
) -> List[Dict[str, str]]:
|
||||
"""
|
||||
构建第一阶段对话消息
|
||||
|
||||
Args:
|
||||
input_data: 输入数据
|
||||
system_message: 系统消息
|
||||
user_message: 用户消息
|
||||
|
||||
Returns:
|
||||
对话消息列表
|
||||
"""
|
||||
# 构建完整的用户消息
|
||||
full_user_message = user_message
|
||||
|
||||
# 添加候选目标
|
||||
if self.candidate_targets:
|
||||
full_user_message += "\n\n" + self.build_candidate_targets_section()
|
||||
|
||||
# 添加任务示例
|
||||
if self.task_examples:
|
||||
full_user_message += "\n\n" + self.build_task_examples_section()
|
||||
|
||||
# 添加任务强调
|
||||
if self.task_emphasis:
|
||||
full_user_message += "\n\n" + self.build_task_emphasis_section()
|
||||
|
||||
# 添加输入数据
|
||||
full_user_message += f"\n\n输入数据：\n{input_data}"
|
||||
|
||||
return [
|
||||
{"role": "system", "content": system_message},
|
||||
{"role": "user", "content": full_user_message}
|
||||
]
|
||||
|
||||
def build_second_conversation(
|
||||
self,
|
||||
original_input: str,
|
||||
first_result: str,
|
||||
verification_instruction: str,
|
||||
) -> List[Dict[str, str]]:
|
||||
"""
|
||||
构建第二阶段对话消息（验证和优化）
|
||||
|
||||
Args:
|
||||
original_input: 原始输入数据
|
||||
first_result: 第一阶段的结果
|
||||
verification_instruction: 验证指令
|
||||
|
||||
Returns:
|
||||
对话消息列表
|
||||
"""
|
||||
system_message = """你需要验证和优化之前的结果。请仔细检查结果的正确性和完整性，根据上下文信息补全缺失的内容，并修正任何错误。"""
|
||||
|
||||
user_message = f"""{verification_instruction}
|
||||
|
||||
原始输入：
|
||||
{original_input}
|
||||
|
||||
之前的结果：
|
||||
{first_result}
|
||||
|
||||
请对结果进行验证、补全和优化。"""
|
||||
|
||||
return [
|
||||
{"role": "system", "content": system_message},
|
||||
{"role": "assistant", "content": first_result},
|
||||
{"role": "user", "content": user_message}
|
||||
]
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""将模板转换为字典"""
|
||||
return {
|
||||
"task_description": self.task_description,
|
||||
"candidate_targets": self.candidate_targets,
|
||||
"task_examples": self.task_examples,
|
||||
"task_emphasis": self.task_emphasis,
|
||||
"second_conversation": self.second_conversation,
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
"""查询模块（参考GraphRAG）"""
|
||||
|
||||
from .global_search import GlobalSearcher
|
||||
from .local_search import LocalSearcher
|
||||
from .drift_search import DriftSearcher
|
||||
|
||||
__all__ = ["GlobalSearcher", "LocalSearcher", "DriftSearcher"]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
"""
|
||||
DRIFT搜索（参考GraphRAG）
|
||||
结合社区信息的上下文搜索（Community-aware search）
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional, Set
|
||||
import networkx as nx
|
||||
|
||||
from .local_search import LocalSearcher
|
||||
from .global_search import GlobalSearcher
|
||||
from ..utils.config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DriftSearcher:
|
||||
"""DRIFT搜索器（结合社区信息的局部搜索）"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
kg: Optional[nx.DiGraph] = None,
|
||||
communities: Dict = None,
|
||||
config: Optional[Config] = None,
|
||||
):
|
||||
"""
|
||||
初始化DRIFT搜索器
|
||||
|
||||
Args:
|
||||
kg: 知识图谱
|
||||
communities: 社区信息（包含摘要）
|
||||
config: 配置对象
|
||||
"""
|
||||
self.kg = kg
|
||||
self.communities = communities or {}
|
||||
self.config = config or Config()
|
||||
|
||||
# 使用局部搜索器作为基础
|
||||
self.local_searcher = LocalSearcher(kg, config)
|
||||
self.global_searcher = GlobalSearcher(kg, communities, config)
|
||||
|
||||
def search(self, query: str, entity: str = None, depth: int = 2) -> Dict[str, Any]:
|
||||
"""
|
||||
DRIFT搜索：结合社区上下文信息的局部搜索
|
||||
|
||||
Args:
|
||||
query: 查询文本
|
||||
entity: 实体名称（如果提供，则进行局部搜索）
|
||||
depth: 搜索深度
|
||||
|
||||
Returns:
|
||||
搜索结果，结合了局部搜索和社区信息
|
||||
"""
|
||||
logger.info(f"执行DRIFT搜索: query={query}, entity={entity}")
|
||||
|
||||
result = {
|
||||
"query": query,
|
||||
"entity": entity,
|
||||
}
|
||||
|
||||
# 1. 如果有实体，先进行局部搜索
|
||||
local_result = None
|
||||
if entity:
|
||||
local_result = self.local_searcher.search(entity, depth=depth)
|
||||
result["local_search"] = local_result
|
||||
|
||||
# 2. 执行全局搜索获取相关社区
|
||||
global_result = self.global_searcher.search(query, top_k=3)
|
||||
result["global_context"] = global_result
|
||||
|
||||
# 3. 结合社区信息扩展局部搜索结果
|
||||
if local_result and "subgraph" in local_result:
|
||||
subgraph_nodes = set(local_result["subgraph"]["nodes"])
|
||||
|
||||
# 从相关社区中获取额外的上下文节点
|
||||
context_nodes = set()
|
||||
for comm in global_result.get("relevant_communities", []):
|
||||
context_nodes.update(comm.get("nodes", []))
|
||||
|
||||
# 合并节点
|
||||
expanded_nodes = subgraph_nodes | context_nodes
|
||||
expanded_nodes = list(expanded_nodes)[:100] # 限制大小
|
||||
|
||||
# 构建扩展子图
|
||||
if self.kg:
|
||||
expanded_subgraph = self.kg.subgraph(expanded_nodes)
|
||||
|
||||
# 提取社区上下文关系
|
||||
community_relations = []
|
||||
for comm in global_result.get("relevant_communities", []):
|
||||
comm_nodes = set(comm.get("nodes", []))
|
||||
# 找到子图中的社区节点之间的连接
|
||||
for u in comm_nodes:
|
||||
if u in expanded_subgraph:
|
||||
for v in expanded_subgraph.successors(u):
|
||||
if v in comm_nodes:
|
||||
edge_data = expanded_subgraph[u][v]
|
||||
community_relations.append({
|
||||
"source": u,
|
||||
"target": v,
|
||||
"relation": edge_data.get("relation", ""),
|
||||
"community": comm.get("community_id", ""),
|
||||
"summary": comm.get("summary", "")
|
||||
})
|
||||
|
||||
result["expanded_subgraph"] = {
|
||||
"nodes": expanded_nodes,
|
||||
"num_nodes": len(expanded_nodes),
|
||||
"num_edges": expanded_subgraph.number_of_edges(),
|
||||
"community_relations": community_relations
|
||||
}
|
||||
|
||||
# 4. 生成综合答案（结合局部和全局信息）
|
||||
if self.global_searcher.llm_client:
|
||||
answer = self._generate_combined_answer(
|
||||
query,
|
||||
local_result,
|
||||
global_result
|
||||
)
|
||||
result["answer"] = answer
|
||||
|
||||
return result
|
||||
|
||||
def _generate_combined_answer(
|
||||
self,
|
||||
query: str,
|
||||
local_result: Optional[Dict],
|
||||
global_result: Dict,
|
||||
) -> str:
|
||||
"""
|
||||
生成结合局部和全局信息的综合答案
|
||||
|
||||
Args:
|
||||
query: 查询文本
|
||||
local_result: 局部搜索结果
|
||||
global_result: 全局搜索结果
|
||||
|
||||
Returns:
|
||||
综合答案
|
||||
"""
|
||||
llm_client = self.global_searcher.llm_client
|
||||
if not llm_client:
|
||||
return "答案生成功能不可用"
|
||||
|
||||
# 构建上下文信息
|
||||
context_parts = []
|
||||
|
||||
# 添加局部搜索信息
|
||||
if local_result and "subgraph" in local_result:
|
||||
subgraph = local_result["subgraph"]
|
||||
context_parts.append(f"""局部实体关系网络：
|
||||
- 实体节点数：{subgraph['num_nodes']}
|
||||
- 关系边数：{subgraph['num_edges']}
|
||||
- 主要关系：{', '.join([r['relation'] for r in subgraph.get('relations', [])[:5]])}""")
|
||||
|
||||
# 添加全局社区信息
|
||||
communities_info = "\n\n".join([
|
||||
f"社区 {i+1}：{comm['summary']}"
|
||||
for i, comm in enumerate(global_result.get("relevant_communities", [])[:3])
|
||||
])
|
||||
if communities_info:
|
||||
context_parts.append(f"相关社区摘要：\n{communities_info}")
|
||||
|
||||
context = "\n\n".join(context_parts)
|
||||
|
||||
prompt = f"""基于以下知识图谱的局部和全局信息，回答用户的问题。
|
||||
|
||||
查询：{query}
|
||||
|
||||
知识图谱上下文：
|
||||
{context}
|
||||
|
||||
请生成一个准确、完整的答案，充分利用局部实体关系和全局社区信息："""
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "你是一个法规知识问答专家，擅长结合知识图谱的局部和全局信息回答复杂问题。"},
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
|
||||
try:
|
||||
answer = llm_client.chat(messages)
|
||||
return answer.strip()
|
||||
except Exception as e:
|
||||
logger.error(f"综合答案生成失败: {e}")
|
||||
return "答案生成失败"
|
||||
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
"""
|
||||
全局搜索（参考GraphRAG）
|
||||
利用社区摘要进行整体推理
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional
|
||||
import networkx as nx
|
||||
|
||||
from ..utils.llm_client import LLMClient, LLMProvider
|
||||
from ..utils.config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GlobalSearcher:
|
||||
"""全局搜索器"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
kg: Optional[nx.DiGraph] = None,
|
||||
communities: Dict = None,
|
||||
config: Optional[Config] = None,
|
||||
):
|
||||
"""
|
||||
初始化全局搜索器
|
||||
|
||||
Args:
|
||||
kg: 知识图谱
|
||||
communities: 社区信息（包含摘要）
|
||||
config: 配置对象
|
||||
"""
|
||||
self.kg = kg
|
||||
self.communities = communities or {}
|
||||
self.config = config or Config()
|
||||
|
||||
try:
|
||||
self.llm_client = LLMClient(
|
||||
provider=LLMProvider.QWEN,
|
||||
model="qwen-max",
|
||||
config=self.config
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"LLM客户端初始化失败: {e}")
|
||||
self.llm_client = None
|
||||
|
||||
def search(self, query: str, top_k: int = 5) -> Dict[str, Any]:
|
||||
"""
|
||||
全局搜索：利用社区摘要进行整体推理
|
||||
|
||||
Args:
|
||||
query: 查询文本
|
||||
top_k: 返回top-k个最相关的社区
|
||||
|
||||
Returns:
|
||||
搜索结果，包含相关社区及其摘要
|
||||
"""
|
||||
logger.info(f"执行全局搜索: {query}")
|
||||
|
||||
if not self.communities:
|
||||
logger.warning("社区信息为空，无法执行全局搜索")
|
||||
return {
|
||||
"query": query,
|
||||
"results": [],
|
||||
"message": "社区信息不可用"
|
||||
}
|
||||
|
||||
# 使用LLM评估查询与社区摘要的相关性
|
||||
relevant_communities = []
|
||||
|
||||
if self.llm_client:
|
||||
# 使用LLM进行相关性评估
|
||||
for comm_id, comm_data in self.communities.items():
|
||||
summary = comm_data.get("summary", "")
|
||||
nodes = comm_data.get("nodes", [])
|
||||
|
||||
if not summary:
|
||||
continue
|
||||
|
||||
# 评估相关性
|
||||
relevance_score = self._evaluate_relevance(query, summary)
|
||||
|
||||
relevant_communities.append({
|
||||
"community_id": comm_id,
|
||||
"summary": summary,
|
||||
"nodes": nodes,
|
||||
"size": len(nodes),
|
||||
"relevance_score": relevance_score
|
||||
})
|
||||
else:
|
||||
# 简单的关键词匹配
|
||||
query_lower = query.lower()
|
||||
for comm_id, comm_data in self.communities.items():
|
||||
summary = comm_data.get("summary", "")
|
||||
if query_lower in summary.lower():
|
||||
relevant_communities.append({
|
||||
"community_id": comm_id,
|
||||
"summary": summary,
|
||||
"nodes": comm_data.get("nodes", []),
|
||||
"size": len(comm_data.get("nodes", [])),
|
||||
"relevance_score": 0.5
|
||||
})
|
||||
|
||||
# 按相关性排序
|
||||
relevant_communities.sort(key=lambda x: x.get("relevance_score", 0), reverse=True)
|
||||
|
||||
# 取top-k
|
||||
top_communities = relevant_communities[:top_k]
|
||||
|
||||
# 使用LLM生成综合答案
|
||||
answer = self._generate_answer(query, top_communities)
|
||||
|
||||
return {
|
||||
"query": query,
|
||||
"answer": answer,
|
||||
"relevant_communities": top_communities,
|
||||
"total_communities": len(self.communities)
|
||||
}
|
||||
|
||||
def _evaluate_relevance(self, query: str, summary: str) -> float:
|
||||
"""
|
||||
评估查询与社区摘要的相关性
|
||||
|
||||
Args:
|
||||
query: 查询文本
|
||||
summary: 社区摘要
|
||||
|
||||
Returns:
|
||||
相关性分数（0-1）
|
||||
"""
|
||||
if not self.llm_client:
|
||||
return 0.5
|
||||
|
||||
prompt = f"""请评估以下查询与社区摘要的相关性，给出0-1之间的分数（1表示高度相关，0表示不相关）。
|
||||
|
||||
查询：{query}
|
||||
|
||||
社区摘要：{summary}
|
||||
|
||||
请只回答一个0-1之间的浮点数："""
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "你是一个相关性评估专家。"},
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
|
||||
try:
|
||||
result = self.llm_client.chat(messages).strip()
|
||||
score = float(result)
|
||||
return max(0.0, min(1.0, score))
|
||||
except:
|
||||
return 0.5
|
||||
|
||||
def _generate_answer(self, query: str, communities: List[Dict]) -> str:
|
||||
"""
|
||||
基于相关社区生成综合答案
|
||||
|
||||
Args:
|
||||
query: 查询文本
|
||||
communities: 相关社区列表
|
||||
|
||||
Returns:
|
||||
生成的答案
|
||||
"""
|
||||
if not self.llm_client:
|
||||
return "答案生成功能不可用（LLM未初始化）"
|
||||
|
||||
communities_info = "\n\n".join([
|
||||
f"社区 {i+1}（包含{comm['size']}个实体）：\n{comm['summary']}"
|
||||
for i, comm in enumerate(communities)
|
||||
])
|
||||
|
||||
prompt = f"""基于以下知识图谱社区摘要，回答用户的问题。
|
||||
|
||||
查询：{query}
|
||||
|
||||
相关社区摘要：
|
||||
{communities_info}
|
||||
|
||||
请生成一个准确、完整的答案："""
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "你是一个法规知识问答专家，擅长基于知识图谱信息回答法规相关问题。"},
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
|
||||
try:
|
||||
answer = self.llm_client.chat(messages)
|
||||
return answer.strip()
|
||||
except Exception as e:
|
||||
logger.error(f"答案生成失败: {e}")
|
||||
return "答案生成失败"
|
||||
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
"""
|
||||
局部搜索（参考GraphRAG）
|
||||
扩展到邻居和相关概念
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional, Set
|
||||
import networkx as nx
|
||||
|
||||
from ..utils.config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LocalSearcher:
|
||||
"""局部搜索器"""
|
||||
|
||||
def __init__(self, kg: Optional[nx.DiGraph] = None, config: Optional[Config] = None):
|
||||
"""
|
||||
初始化局部搜索器
|
||||
|
||||
Args:
|
||||
kg: 知识图谱
|
||||
config: 配置对象
|
||||
"""
|
||||
self.kg = kg
|
||||
self.config = config or Config()
|
||||
|
||||
def search(self, entity: str, depth: int = 2, max_nodes: int = 50) -> Dict[str, Any]:
|
||||
"""
|
||||
局部搜索：从指定实体扩展到邻居和相关概念
|
||||
|
||||
Args:
|
||||
entity: 实体名称
|
||||
depth: 搜索深度（跳数）
|
||||
max_nodes: 最大节点数
|
||||
|
||||
Returns:
|
||||
搜索结果，包含子图和相关信息
|
||||
"""
|
||||
logger.info(f"执行局部搜索: {entity}, depth={depth}")
|
||||
|
||||
if not self.kg:
|
||||
return {
|
||||
"entity": entity,
|
||||
"error": "知识图谱未加载"
|
||||
}
|
||||
|
||||
# 查找实体节点（支持模糊匹配）
|
||||
target_nodes = self._find_entity_nodes(entity)
|
||||
|
||||
if not target_nodes:
|
||||
return {
|
||||
"entity": entity,
|
||||
"error": f"未找到实体: {entity}",
|
||||
"suggestions": list(self.kg.nodes())[:10] # 提供一些建议
|
||||
}
|
||||
|
||||
# 扩展搜索：收集所有相关的节点
|
||||
explored_nodes: Set[str] = set()
|
||||
nodes_to_explore = list(target_nodes)
|
||||
|
||||
for current_depth in range(depth + 1):
|
||||
if not nodes_to_explore:
|
||||
break
|
||||
|
||||
current_level_nodes = list(nodes_to_explore)
|
||||
nodes_to_explore = []
|
||||
|
||||
for node in current_level_nodes:
|
||||
if node in explored_nodes:
|
||||
continue
|
||||
|
||||
explored_nodes.add(node)
|
||||
|
||||
# 添加到下一层探索的节点
|
||||
if current_depth < depth:
|
||||
# 前驱节点
|
||||
predecessors = list(self.kg.predecessors(node))
|
||||
# 后继节点
|
||||
successors = list(self.kg.successors(node))
|
||||
|
||||
for neighbor in predecessors + successors:
|
||||
if neighbor not in explored_nodes:
|
||||
nodes_to_explore.append(neighbor)
|
||||
|
||||
if len(explored_nodes) >= max_nodes:
|
||||
break
|
||||
|
||||
# 构建子图
|
||||
subgraph_nodes = list(explored_nodes)
|
||||
subgraph = self.kg.subgraph(subgraph_nodes)
|
||||
|
||||
# 提取关系信息
|
||||
relations = []
|
||||
for u, v, data in subgraph.edges(data=True):
|
||||
relations.append({
|
||||
"source": u,
|
||||
"target": v,
|
||||
"relation": data.get("relation", ""),
|
||||
"metadata": {k: v for k, v in data.items() if k != "relation"}
|
||||
})
|
||||
|
||||
# 计算中心性（在子图中）
|
||||
centrality = {}
|
||||
if subgraph.number_of_nodes() > 0:
|
||||
try:
|
||||
degree_centrality = nx.degree_centrality(subgraph)
|
||||
centrality = {
|
||||
node: {
|
||||
"degree": degree_centrality.get(node, 0),
|
||||
"neighbors": len(list(subgraph.neighbors(node)))
|
||||
}
|
||||
for node in target_nodes
|
||||
}
|
||||
except:
|
||||
pass
|
||||
|
||||
return {
|
||||
"entity": entity,
|
||||
"target_nodes": list(target_nodes),
|
||||
"subgraph": {
|
||||
"nodes": subgraph_nodes,
|
||||
"num_nodes": len(subgraph_nodes),
|
||||
"num_edges": subgraph.number_of_edges(),
|
||||
"relations": relations
|
||||
},
|
||||
"centrality": centrality,
|
||||
"depth": depth
|
||||
}
|
||||
|
||||
def _find_entity_nodes(self, entity: str) -> List[str]:
|
||||
"""
|
||||
在图中查找实体节点（支持模糊匹配）
|
||||
|
||||
Args:
|
||||
entity: 实体名称
|
||||
|
||||
Returns:
|
||||
匹配的节点列表
|
||||
"""
|
||||
if not self.kg:
|
||||
return []
|
||||
|
||||
entity_lower = entity.lower()
|
||||
exact_matches = []
|
||||
partial_matches = []
|
||||
|
||||
for node in self.kg.nodes():
|
||||
node_str = str(node).lower()
|
||||
if node_str == entity_lower:
|
||||
exact_matches.append(node)
|
||||
elif entity_lower in node_str or node_str in entity_lower:
|
||||
partial_matches.append(node)
|
||||
|
||||
# 优先返回精确匹配
|
||||
if exact_matches:
|
||||
return exact_matches
|
||||
else:
|
||||
return partial_matches[:5] # 最多返回5个部分匹配
|
||||
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
"""工具函数模块"""
|
||||
|
||||
from .config import Config, load_config
|
||||
from .llm_client import LLMClient, LLMProvider
|
||||
from .file_utils import save_json, load_json, save_text
|
||||
|
||||
__all__ = [
|
||||
"Config",
|
||||
"load_config",
|
||||
"LLMClient",
|
||||
"LLMProvider",
|
||||
"save_json",
|
||||
"load_json",
|
||||
"save_text",
|
||||
]
|
||||
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
"""
|
||||
配置管理模块
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# 加载.env文件
|
||||
env_path = Path(__file__).parent.parent.parent / ".env"
|
||||
if env_path.exists():
|
||||
load_dotenv(env_path)
|
||||
|
||||
|
||||
class Config:
|
||||
"""配置类"""
|
||||
|
||||
# API Keys
|
||||
OPENAI_API_KEY: Optional[str] = os.getenv("OPENAI_API_KEY")
|
||||
ANTHROPIC_API_KEY: Optional[str] = os.getenv("ANTHROPIC_API_KEY")
|
||||
DASHSCOPE_API_KEY: Optional[str] = os.getenv("DASHSCOPE_API_KEY")
|
||||
VOLCENGINE_ACCESS_KEY: Optional[str] = os.getenv("VOLCENGINE_ACCESS_KEY")
|
||||
VOLCENGINE_SECRET_KEY: Optional[str] = os.getenv("VOLCENGINE_SECRET_KEY")
|
||||
ZHIPUAI_API_KEY: Optional[str] = os.getenv("ZHIPUAI_API_KEY")
|
||||
SILICONFLOW_API_KEY: Optional[str] = os.getenv("SILICONFLOW_API_KEY")
|
||||
SILICONFLOW_API_BASE: Optional[str] = os.getenv("SILICONFLOW_API_BASE", "https://api.siliconflow.cn/v1")
|
||||
|
||||
# 路径配置
|
||||
DATA_DIR: str = os.getenv("DATA_DIR", "../data")
|
||||
OUTPUT_DIR: str = os.getenv("OUTPUT_DIR", "./output")
|
||||
|
||||
# 日志配置
|
||||
LOG_LEVEL: str = os.getenv("LOG_LEVEL", "INFO")
|
||||
|
||||
# HanLP配置
|
||||
HANLP_MODEL_PATH: Optional[str] = os.getenv("HANLP_MODEL_PATH")
|
||||
|
||||
# LLM配置
|
||||
TEMPERATURE: float = float(os.getenv("TEMPERATURE", "0.4"))
|
||||
FREQUENCY_PENALTY: float = float(os.getenv("FREQUENCY_PENALTY", "0.6"))
|
||||
PRESENCE_PENALTY: float = float(os.getenv("PRESENCE_PENALTY", "0.6"))
|
||||
|
||||
# 文本处理配置
|
||||
MAX_TEXTUNIT_LENGTH: int = int(os.getenv("MAX_TEXTUNIT_LENGTH", "500"))
|
||||
|
||||
|
||||
def load_config() -> Config:
|
||||
"""加载配置"""
|
||||
return Config()
|
||||
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
"""
|
||||
文件工具函数
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def save_json(data: Any, filepath: str, indent: int = 2, ensure_ascii: bool = False):
|
||||
"""
|
||||
保存数据为JSON文件
|
||||
|
||||
Args:
|
||||
data: 要保存的数据
|
||||
filepath: 文件路径
|
||||
indent: JSON缩进
|
||||
ensure_ascii: 是否确保ASCII编码
|
||||
"""
|
||||
path = Path(filepath)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, ensure_ascii=ensure_ascii, indent=indent)
|
||||
|
||||
logger.info(f"数据已保存到: {filepath}")
|
||||
|
||||
|
||||
def load_json(filepath: str) -> Any:
|
||||
"""
|
||||
从JSON文件加载数据
|
||||
|
||||
Args:
|
||||
filepath: 文件路径
|
||||
|
||||
Returns:
|
||||
加载的数据
|
||||
"""
|
||||
with open(filepath, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def save_text(text: str, filepath: str):
|
||||
"""保存文本到文件"""
|
||||
path = Path(filepath)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
f.write(text)
|
||||
|
||||
logger.info(f"文本已保存到: {filepath}")
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
"""
|
||||
LLM客户端封装
|
||||
支持多种LLM提供商：OpenAI, Anthropic, Qwen, Doubao, GLM, SiliconFlow
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import List, Dict, Optional, Any
|
||||
from enum import Enum
|
||||
from tenacity import retry, stop_after_attempt, wait_exponential
|
||||
|
||||
from .config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LLMProvider(str, Enum):
|
||||
"""LLM提供商枚举"""
|
||||
OPENAI = "openai"
|
||||
ANTHROPIC = "anthropic"
|
||||
QWEN = "qwen"
|
||||
DOUBAO = "doubao"
|
||||
GLM = "glm"
|
||||
SILICONFLOW = "siliconflow"
|
||||
|
||||
|
||||
class LLMClient:
|
||||
"""LLM客户端，统一封装不同提供商的API"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
provider: LLMProvider,
|
||||
model: str,
|
||||
config: Optional[Config] = None,
|
||||
):
|
||||
"""
|
||||
初始化LLM客户端
|
||||
|
||||
Args:
|
||||
provider: LLM提供商
|
||||
model: 模型名称
|
||||
config: 配置对象
|
||||
"""
|
||||
self.provider = provider
|
||||
self.model = model
|
||||
self.config = config or Config()
|
||||
|
||||
# 初始化对应的客户端
|
||||
self._init_client()
|
||||
|
||||
def _init_client(self):
|
||||
"""初始化对应提供商的客户端"""
|
||||
if self.provider == LLMProvider.OPENAI:
|
||||
try:
|
||||
import openai
|
||||
self.client = openai.OpenAI(api_key=self.config.OPENAI_API_KEY)
|
||||
except ImportError:
|
||||
raise ImportError("请安装openai包: pip install openai")
|
||||
|
||||
elif self.provider == LLMProvider.ANTHROPIC:
|
||||
try:
|
||||
import anthropic
|
||||
self.client = anthropic.Anthropic(api_key=self.config.ANTHROPIC_API_KEY)
|
||||
except ImportError:
|
||||
raise ImportError("请安装anthropic包: pip install anthropic")
|
||||
|
||||
elif self.provider == LLMProvider.QWEN:
|
||||
try:
|
||||
import dashscope
|
||||
dashscope.api_key = self.config.DASHSCOPE_API_KEY
|
||||
self.client = dashscope
|
||||
except ImportError:
|
||||
raise ImportError("请安装dashscope包: pip install dashscope")
|
||||
|
||||
elif self.provider == LLMProvider.DOUBAO:
|
||||
try:
|
||||
from volcengine.maas import MaasService, MaasException, ChatRole
|
||||
self.client = MaasService(
|
||||
"cn-beijing",
|
||||
access_key=self.config.VOLCENGINE_ACCESS_KEY,
|
||||
secret_key=self.config.VOLCENGINE_SECRET_KEY,
|
||||
)
|
||||
except ImportError:
|
||||
raise ImportError("请安装volcengine-python-sdk包")
|
||||
|
||||
elif self.provider == LLMProvider.GLM:
|
||||
try:
|
||||
import zhipuai
|
||||
zhipuai.api_key = self.config.ZHIPUAI_API_KEY
|
||||
self.client = zhipuai
|
||||
except ImportError:
|
||||
raise ImportError("请安装zhipuai包: pip install zhipuai")
|
||||
|
||||
elif self.provider == LLMProvider.SILICONFLOW:
|
||||
try:
|
||||
import openai
|
||||
# 硅基流动兼容OpenAI API格式
|
||||
self.client = openai.OpenAI(
|
||||
api_key=self.config.SILICONFLOW_API_KEY,
|
||||
base_url=self.config.SILICONFLOW_API_BASE or "https://api.siliconflow.cn/v1"
|
||||
)
|
||||
except ImportError:
|
||||
raise ImportError("请安装openai包: pip install openai")
|
||||
|
||||
else:
|
||||
raise ValueError(f"不支持的LLM提供商: {self.provider}")
|
||||
|
||||
@retry(
|
||||
stop=stop_after_attempt(3),
|
||||
wait=wait_exponential(multiplier=1, min=4, max=10)
|
||||
)
|
||||
async def chat_async(
|
||||
self,
|
||||
messages: List[Dict[str, str]],
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
) -> str:
|
||||
"""
|
||||
异步聊天接口
|
||||
|
||||
Args:
|
||||
messages: 消息列表
|
||||
temperature: 温度参数
|
||||
max_tokens: 最大token数
|
||||
|
||||
Returns:
|
||||
模型响应文本
|
||||
"""
|
||||
temperature = temperature or self.config.TEMPERATURE
|
||||
|
||||
try:
|
||||
if self.provider == LLMProvider.OPENAI:
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.model,
|
||||
messages=messages,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
return response.choices[0].message.content
|
||||
|
||||
elif self.provider == LLMProvider.ANTHROPIC:
|
||||
response = await self.client.messages.create(
|
||||
model=self.model,
|
||||
messages=messages,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens or 4096,
|
||||
)
|
||||
return response.content[0].text
|
||||
|
||||
elif self.provider == LLMProvider.QWEN:
|
||||
response = self.client.Generation.call(
|
||||
model=self.model,
|
||||
messages=messages,
|
||||
temperature=temperature,
|
||||
)
|
||||
return response.output.text
|
||||
|
||||
elif self.provider == LLMProvider.DOUBAO:
|
||||
# 字节跳动API同步调用
|
||||
response = self.client.chat(
|
||||
model=self.model,
|
||||
messages=messages,
|
||||
parameters={
|
||||
"temperature": temperature,
|
||||
"max_tokens": max_tokens or 4096,
|
||||
}
|
||||
)
|
||||
return response.get("choices", [{}])[0].get("message", {}).get("content", "")
|
||||
|
||||
elif self.provider == LLMProvider.GLM:
|
||||
response = self.client.model_api.invoke(
|
||||
model=self.model,
|
||||
messages=messages,
|
||||
temperature=temperature,
|
||||
)
|
||||
return response.get("data", {}).get("choices", [{}])[0].get("message", {}).get("content", "")
|
||||
|
||||
elif self.provider == LLMProvider.SILICONFLOW:
|
||||
# 硅基流动兼容OpenAI API格式
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.model,
|
||||
messages=messages,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
return response.choices[0].message.content
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"LLM调用失败 ({self.provider}): {e}")
|
||||
raise
|
||||
|
||||
def chat(
|
||||
self,
|
||||
messages: List[Dict[str, str]],
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
) -> str:
|
||||
"""
|
||||
同步聊天接口（内部调用异步接口）
|
||||
|
||||
Args:
|
||||
messages: 消息列表
|
||||
temperature: 温度参数
|
||||
max_tokens: 最大token数
|
||||
|
||||
Returns:
|
||||
模型响应文本
|
||||
"""
|
||||
try:
|
||||
loop = asyncio.get_event_loop()
|
||||
except RuntimeError:
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
|
||||
return loop.run_until_complete(
|
||||
self.chat_async(messages, temperature, max_tokens)
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
# 测试文件说明
|
||||
|
||||
本目录包含所有测试脚本。
|
||||
|
||||
## 测试文件列表
|
||||
|
||||
### 1. `test_basic.py`
|
||||
基础功能测试，包括：
|
||||
- 模块导入测试
|
||||
- 本体模型测试
|
||||
- Prompt模板测试
|
||||
|
||||
运行方式：
|
||||
```bash
|
||||
uv run python tests/test_basic.py
|
||||
```
|
||||
|
||||
### 2. `test_siliconflow.py`
|
||||
测试硅基流动API配置，验证API密钥和连接是否正常。
|
||||
|
||||
运行方式：
|
||||
```bash
|
||||
uv run python tests/test_siliconflow.py
|
||||
```
|
||||
|
||||
### 3. `test_documents.py`
|
||||
测试文档解析功能，包括：
|
||||
- 文档解析（.docx文件）
|
||||
- TextUnit切分
|
||||
- 不调用LLM，仅测试解析逻辑
|
||||
|
||||
运行方式：
|
||||
```bash
|
||||
uv run python tests/test_documents.py
|
||||
```
|
||||
|
||||
### 4. `test_model.py`
|
||||
测试硅基流动平台上的多个模型，找出可用模型。
|
||||
|
||||
运行方式：
|
||||
```bash
|
||||
uv run python tests/test_model.py
|
||||
```
|
||||
|
||||
### 5. `test_my_models.py`
|
||||
测试项目配置使用的两个指定模型：
|
||||
- `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`
|
||||
- `Qwen/Qwen2.5-7B-Instruct`
|
||||
|
||||
运行方式：
|
||||
```bash
|
||||
uv run python tests/test_my_models.py
|
||||
```
|
||||
|
||||
### 6. `test_quick.py`
|
||||
快速测试完整流程，处理少量TextUnit（1个文档的前2个），验证：
|
||||
- 文档解析
|
||||
- 实体识别（NER）
|
||||
- 关系抽取（RE）
|
||||
- 知识图谱构建
|
||||
- 社区检测
|
||||
|
||||
运行方式：
|
||||
```bash
|
||||
uv run python tests/test_quick.py
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 测试脚本需要使用配置的API密钥（`.env`文件）
|
||||
- 部分测试会调用LLM API，可能需要一些时间
|
||||
- 确保已安装所有依赖包：`uv sync`
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
"""
|
||||
基础功能测试
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
def test_imports():
|
||||
"""测试模块导入"""
|
||||
try:
|
||||
from src.preprocessing.text_processor import TextProcessor
|
||||
from src.preprocessing.document_parser import DocumentParser
|
||||
from src.ontology.schema import OntologySchema
|
||||
from src.prompts.ner_prompts import NERPromptBuilder
|
||||
from src.extraction.ner import NERExtractor
|
||||
from src.kg_builder.graph import KnowledgeGraph
|
||||
print("✅ 所有模块导入成功")
|
||||
return True
|
||||
except ImportError as e:
|
||||
print(f"❌ 模块导入失败: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_ontology():
|
||||
"""测试本体模型"""
|
||||
try:
|
||||
from src.ontology.schema import OntologySchema
|
||||
|
||||
schema = OntologySchema()
|
||||
summary = schema.get_schema_summary()
|
||||
|
||||
assert summary["entity_types"] == 9
|
||||
assert summary["relation_types"] == 8
|
||||
print("✅ 本体模型测试通过")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"❌ 本体模型测试失败: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_prompt_template():
|
||||
"""测试Prompt模板"""
|
||||
try:
|
||||
from src.prompts.template import PromptTemplate
|
||||
|
||||
template = PromptTemplate(
|
||||
task_description={"system": "test", "user": "test"},
|
||||
candidate_targets={"entity": "definition"},
|
||||
task_examples=[{"input": "test", "output": "test"}],
|
||||
task_emphasis="test",
|
||||
)
|
||||
|
||||
assert template.task_description is not None
|
||||
print("✅ Prompt模板测试通过")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"❌ Prompt模板测试失败: {e}")
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("开始运行基础测试...")
|
||||
test_imports()
|
||||
test_ontology()
|
||||
test_prompt_template()
|
||||
print("测试完成！")
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
测试文档解析（仅测试3个文件，不调用LLM）
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
# 修复：移除conda环境的路径，确保使用虚拟环境的包
|
||||
sys.path = [p for p in sys.path if 'anaconda3' not in p.lower() and 'conda' not in p.lower()]
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.preprocessing.document_parser import DocumentParser
|
||||
from src.utils.config import load_config
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def main():
|
||||
"""测试文档解析"""
|
||||
logger.info("=" * 60)
|
||||
logger.info("测试文档解析（3个文件）")
|
||||
logger.info("=" * 60)
|
||||
|
||||
# 配置
|
||||
config = load_config()
|
||||
data_dir = Path("../data/1法律")
|
||||
output_dir = Path("./output")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 解析文档
|
||||
logger.info(f"解析目录: {data_dir}")
|
||||
parser = DocumentParser()
|
||||
docs = parser.parse_directory(str(data_dir))
|
||||
|
||||
# 只处理前3个
|
||||
docs = docs[:3]
|
||||
logger.info(f"处理 {len(docs)} 个文档")
|
||||
|
||||
# 切分为TextUnit
|
||||
logger.info("切分文档为TextUnit...")
|
||||
all_textunits = []
|
||||
for i, doc in enumerate(docs, 1):
|
||||
logger.info(f"处理文档 {i}/{len(docs)}: {Path(doc.get('file_path', '')).name}")
|
||||
textunits = parser.split_into_textunits(
|
||||
doc,
|
||||
max_length=config.MAX_TEXTUNIT_LENGTH
|
||||
)
|
||||
for tu in textunits:
|
||||
tu["doc_id"] = doc.get("file_path", "")
|
||||
tu["id"] = f"{doc.get('file_path', '')}_{tu.get('paragraph_index', 0)}"
|
||||
all_textunits.extend(textunits)
|
||||
logger.info(f" - 生成 {len(textunits)} 个TextUnit")
|
||||
|
||||
logger.info(f"共生成 {len(all_textunits)} 个TextUnit")
|
||||
|
||||
# 保存结果
|
||||
output_file = output_dir / "test_textunits.json"
|
||||
result = {
|
||||
"num_docs": len(docs),
|
||||
"num_textunits": len(all_textunits),
|
||||
"docs": [
|
||||
{
|
||||
"file_path": doc.get("file_path", ""),
|
||||
"title": doc.get("title", ""),
|
||||
"text_length": len(doc.get("text", "")),
|
||||
}
|
||||
for doc in docs
|
||||
],
|
||||
"sample_textunits": [
|
||||
{
|
||||
"id": tu.get("id", ""),
|
||||
"text": tu.get("text", "")[:200] + "..." if len(tu.get("text", "")) > 200 else tu.get("text", ""),
|
||||
"paragraph_index": tu.get("paragraph_index", 0),
|
||||
}
|
||||
for tu in all_textunits[:10] # 只保存前10个作为示例
|
||||
]
|
||||
}
|
||||
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
json.dump(result, f, ensure_ascii=False, indent=2)
|
||||
|
||||
logger.info("=" * 60)
|
||||
logger.info("测试完成！")
|
||||
logger.info(f"结果已保存到: {output_file}")
|
||||
logger.info(f"统计信息:")
|
||||
logger.info(f" - 文档数: {result['num_docs']}")
|
||||
logger.info(f" - TextUnit数: {result['num_textunits']}")
|
||||
logger.info("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
测试硅基流动API并找到可用的模型
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
sys.path = [p for p in sys.path if 'anaconda3' not in p.lower() and 'conda' not in p.lower()]
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.utils.config import load_config
|
||||
from src.utils.llm_client import LLMClient, LLMProvider
|
||||
|
||||
# 常见的硅基流动模型名称
|
||||
TEST_MODELS = [
|
||||
"deepseek-ai/DeepSeek-V2.5",
|
||||
"deepseek-ai/DeepSeek-V2",
|
||||
"Qwen/Qwen2.5-72B-Instruct",
|
||||
"Qwen/Qwen2.5-72B-Chat",
|
||||
"meta-llama/Llama-3.1-70B-Instruct",
|
||||
"01-ai/Yi-1.5-34B-Chat",
|
||||
"mistralai/Mistral-7B-Instruct-v0.2",
|
||||
]
|
||||
|
||||
def test_model(provider, model_name):
|
||||
"""测试单个模型"""
|
||||
try:
|
||||
config = load_config()
|
||||
client = LLMClient(provider=provider, model=model_name, config=config)
|
||||
messages = [{"role": "user", "content": "你好"}]
|
||||
response = client.chat(messages)
|
||||
print(f"✅ 模型 {model_name} 可用")
|
||||
print(f" 响应: {response[:50]}...")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"❌ 模型 {model_name} 不可用: {str(e)[:100]}")
|
||||
return False
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("=" * 60)
|
||||
print("测试硅基流动模型")
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
config = load_config()
|
||||
if not config.SILICONFLOW_API_KEY:
|
||||
print("❌ SILICONFLOW_API_KEY 未配置")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"API密钥: {config.SILICONFLOW_API_KEY[:20]}...")
|
||||
print(f"API Base: {config.SILICONFLOW_API_BASE}")
|
||||
print()
|
||||
|
||||
# 测试所有模型
|
||||
available_models = []
|
||||
for model in TEST_MODELS:
|
||||
if test_model(LLMProvider.SILICONFLOW, model):
|
||||
available_models.append(model)
|
||||
print()
|
||||
|
||||
print("=" * 60)
|
||||
if available_models:
|
||||
print(f"✅ 找到 {len(available_models)} 个可用模型:")
|
||||
for model in available_models:
|
||||
print(f" - {model}")
|
||||
print(f"\n建议使用的模型: {available_models[0]}")
|
||||
else:
|
||||
print("❌ 没有找到可用的模型，请检查API密钥或模型名称")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
测试用户指定的两个模型
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path = [p for p in sys.path if 'anaconda3' not in p.lower() and 'conda' not in p.lower()]
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.utils.llm_client import LLMClient, LLMProvider
|
||||
from src.utils.config import load_config
|
||||
|
||||
def test_model(provider, model_name):
|
||||
"""测试单个模型"""
|
||||
try:
|
||||
config = load_config()
|
||||
client = LLMClient(provider=provider, model=model_name, config=config)
|
||||
messages = [{"role": "user", "content": "你好"}]
|
||||
response = client.chat(messages)
|
||||
print(f"✅ 模型 {model_name} 可用")
|
||||
print(f" 响应: {response[:50]}...")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"❌ 模型 {model_name} 不可用: {str(e)[:100]}")
|
||||
return False
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("=" * 60)
|
||||
print("测试指定的两个模型")
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
config = load_config()
|
||||
if not config.SILICONFLOW_API_KEY:
|
||||
print("❌ SILICONFLOW_API_KEY 未配置")
|
||||
sys.exit(1)
|
||||
|
||||
# 测试用户指定的两个模型
|
||||
models_to_test = [
|
||||
"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
|
||||
"Qwen/Qwen2.5-7B-Instruct",
|
||||
]
|
||||
|
||||
available = []
|
||||
for model in models_to_test:
|
||||
if test_model(LLMProvider.SILICONFLOW, model):
|
||||
available.append(model)
|
||||
print()
|
||||
|
||||
print("=" * 60)
|
||||
if len(available) == len(models_to_test):
|
||||
print(f"✅ 所有模型都可用！")
|
||||
elif available:
|
||||
print(f"⚠️ 部分模型可用: {len(available)}/{len(models_to_test)}")
|
||||
else:
|
||||
print("❌ 没有可用模型")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
快速测试：只处理1个文档的前2个TextUnit
|
||||
"""
|
||||
|
||||
import sys
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
sys.path = [p for p in sys.path if 'anaconda3' not in p.lower() and 'conda' not in p.lower()]
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.preprocessing.document_parser import DocumentParser
|
||||
from src.kg_builder.indexer import GraphIndexer
|
||||
from src.utils.config import load_config
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
def main():
|
||||
logger.info("=" * 60)
|
||||
logger.info("快速测试：处理1个文档的前2个TextUnit")
|
||||
logger.info("=" * 60)
|
||||
|
||||
config = load_config()
|
||||
|
||||
# 解析1个文档
|
||||
parser = DocumentParser()
|
||||
doc_path = "../data/1法律/10-中华人民共和国乡村振兴促进法.docx"
|
||||
doc = parser.parse_docx(doc_path)
|
||||
|
||||
# 切分为TextUnit，只取前2个
|
||||
textunits = parser.split_into_textunits(doc, max_length=config.MAX_TEXTUNIT_LENGTH)[:2]
|
||||
for tu in textunits:
|
||||
tu["doc_id"] = doc.get("file_path", "")
|
||||
tu["id"] = f"{doc.get('file_path', '')}_{tu.get('paragraph_index', 0)}"
|
||||
|
||||
logger.info(f"准备处理 {len(textunits)} 个TextUnit")
|
||||
|
||||
# 构建知识图谱（只处理这2个）
|
||||
logger.info("开始构建知识图谱...")
|
||||
indexer = GraphIndexer(config=config)
|
||||
kg_graph = indexer.index(textunits, use_verification=True)
|
||||
|
||||
logger.info("=" * 60)
|
||||
logger.info("快速测试完成！")
|
||||
logger.info(f"节点数: {kg_graph.number_of_nodes()}")
|
||||
logger.info(f"边数: {kg_graph.number_of_edges()}")
|
||||
logger.info("=" * 60)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
测试硅基流动API配置
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.utils.llm_client import LLMClient, LLMProvider
|
||||
from src.utils.config import load_config
|
||||
|
||||
|
||||
def test_siliconflow():
|
||||
"""测试硅基流动API"""
|
||||
print("=" * 60)
|
||||
print("测试硅基流动API配置")
|
||||
print("=" * 60)
|
||||
|
||||
# 加载配置
|
||||
config = load_config()
|
||||
|
||||
# 检查API密钥
|
||||
if not config.SILICONFLOW_API_KEY:
|
||||
print("❌ 错误: SILICONFLOW_API_KEY 未配置")
|
||||
print("请在 .env 文件中设置 SILICONFLOW_API_KEY")
|
||||
return False
|
||||
|
||||
print(f"✅ API密钥已配置: {config.SILICONFLOW_API_KEY[:20]}...")
|
||||
print(f"✅ API Base URL: {config.SILICONFLOW_API_BASE}")
|
||||
|
||||
# 创建客户端（使用一个常见的模型，用户需要根据实际可用模型调整）
|
||||
print("\n尝试创建客户端...")
|
||||
try:
|
||||
# 注意：这里使用的模型名称需要根据硅基流动平台实际可用模型调整
|
||||
# 常用的模型包括：Qwen/Qwen2.5-72B-Instruct, meta-llama/Llama-3.1-70B-Instruct 等
|
||||
client = LLMClient(
|
||||
provider=LLMProvider.SILICONFLOW,
|
||||
model="Qwen/Qwen2.5-72B-Instruct", # 请根据实际情况修改模型名称
|
||||
config=config
|
||||
)
|
||||
print("✅ 客户端创建成功")
|
||||
except Exception as e:
|
||||
print(f"❌ 客户端创建失败: {e}")
|
||||
return False
|
||||
|
||||
# 测试简单对话
|
||||
print("\n测试API调用...")
|
||||
messages = [
|
||||
{"role": "user", "content": "请用一句话介绍知识图谱。"}
|
||||
]
|
||||
|
||||
try:
|
||||
response = client.chat(messages)
|
||||
print(f"✅ API调用成功")
|
||||
print(f"\n响应内容：\n{response}")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"❌ API调用失败: {e}")
|
||||
print("\n提示：")
|
||||
print("1. 请检查API密钥是否正确")
|
||||
print("2. 请检查网络连接")
|
||||
print("3. 请确认模型名称是否在硅基流动平台可用")
|
||||
print("4. 可以访问 https://siliconflow.cn/ 查看可用模型列表")
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = test_siliconflow()
|
||||
if success:
|
||||
print("\n" + "=" * 60)
|
||||
print("✅ 硅基流动API配置测试通过！")
|
||||
print("=" * 60)
|
||||
else:
|
||||
print("\n" + "=" * 60)
|
||||
print("❌ 硅基流动API配置测试失败，请检查配置")
|
||||
print("=" * 60)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
Generated
+2379
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,10 @@
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
config/api_keys.yaml
|
||||
output/
|
||||
logs/
|
||||
*.log
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
@@ -0,0 +1,122 @@
|
||||
llm:
|
||||
model: "deepseek-chat"
|
||||
temperature: 0.0
|
||||
max_tokens: 8192
|
||||
max_retries: 3
|
||||
retry_delay: 2
|
||||
request_timeout: 120
|
||||
batch_timeout: 600
|
||||
|
||||
batch_processing:
|
||||
batch_size: 10
|
||||
concurrent_requests: 5
|
||||
test_batch_size: 5
|
||||
incremental_save: true
|
||||
|
||||
deduplication:
|
||||
similarity_threshold: 0.85
|
||||
enable_fuzzy_matching: true
|
||||
text_normalization: true
|
||||
|
||||
validation:
|
||||
enable_entity_validation: true
|
||||
enable_relationship_validation: true
|
||||
min_confidence_score: 0.7
|
||||
strict_mode: false
|
||||
filter_generic_concepts: true
|
||||
|
||||
output:
|
||||
nodes_file: "output/nodes_llm_v2.csv"
|
||||
relationships_file: "output/rels_llm_v2.csv"
|
||||
report_file: "output/extraction_report_v2.md"
|
||||
encoding: "utf-8-sig"
|
||||
|
||||
progress:
|
||||
progress_file: "output/progress_v2.json"
|
||||
checkpoint_interval: 10
|
||||
|
||||
logging:
|
||||
log_file: "logs/deep_extraction_v2.log"
|
||||
level: "INFO"
|
||||
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
|
||||
# ============================================================
|
||||
# 第二轮本体：城市更新法规政策工具箱 (21实体 + 34关系)
|
||||
# ============================================================
|
||||
|
||||
entity_type_prompts:
|
||||
Agency: "主体机构：发布主体、实施主体、监管主体、责任主体，包括政府部门（全国人大常委会、国务院、自然资源部、住建部、生态环境部、发改委、地方政府）、实施机构、平台公司、社区组织、第三方机构等。属性：level（国家/省级/市级/县级/乡镇街道/社区）、agency_type（立法机关/行政机关/主管部门/实施主体/监管主体/责任主体/市场主体/社区组织/第三方机构）"
|
||||
LegalObject: "法规事项：法规政策所规制的事项，回答'管什么事项'。如规划编制、土地利用、建设管理、历史文化保护、生态保护、财政支持、金融支持、产权登记、公众参与、行政审批、监督管理、城市更新、公共服务、基础设施、安全韧性等。属性：category"
|
||||
SpatialObject: "空间对象：城市更新中的具体空间对象，回答'作用于什么空间'。如老旧小区、城中村、低效用地、工业遗存、历史文化街区、历史建筑、公共空间、公共服务设施、市政基础设施、存量建筑、建设用地、城市更新单元、生态保护红线、永久基本农田、城镇开发边界等。属性：spatial_type"
|
||||
RenewalScene: "更新场景：城市更新应用场景，回答'用于什么更新场景'。如老旧小区改造、城中村改造、低效用地再开发、工业遗存更新、历史文化保护更新、完整社区建设、公共空间提升、基础设施补短板、城市安全韧性提升、城市生态修复等。属性：scene_type、derivation_method（原文提取/语义归纳/人工标注/模型推理）"
|
||||
PolicyTool: "具体政策工具：城市更新中用于实现治理目标的具体政策、制度、措施或治理手段。属性：tool_type（命令型/激励型/劝诫型/能力型/系统变革型）、mandatory_level（强制/引导/鼓励/支持/禁止/限制）、source_text"
|
||||
ToolCategory: "政策工具类别：归类政策工具的分类节点。属性：classification_system（一般政策工具分类/城市更新工具分类）、urban_renewal_category（规划引导/土地配置/开发激励/财政金融/权益协调/实施管理/监督反馈）"
|
||||
Procedure: "程序：审批、备案、评估、论证、公示、听证、入库、验收、监督、征收、补偿等行政或治理程序。属性：procedure_type、time_limit"
|
||||
Obligation: "权利义务：法规规定的权利、义务、禁止或责任。属性：obligation_type（权利/义务/禁止/职责/鼓励/支持）、modality（应当/可以/不得/禁止/鼓励/支持/负责/依法）"
|
||||
Condition: "适用条件：政策工具或程序适用时的前置条件。如'经批准后''符合XX条件'。属性：condition_type（规划条件/审批条件/权属条件/安全条件/环保条件/公众参与条件/程序条件/资金条件）"
|
||||
Constraint: "约束条件：对行为设置的边界性、底线性或限制性要求。如'不得突破''严守''控制线'。属性：constraint_type（用途管制/空间边界/强制标准/历史保护/生态保护/权益保护/安全底线/公共利益）"
|
||||
Penalty: "法律责任：违反法规后的处罚。属性：penalty_type（行政处罚/刑事处罚/民事责任/行政处分/信用惩戒/整改责任）、amount_or_measure"
|
||||
TimePoint: "时间节点：生效、修订、废止、截止、过渡期等时间信息。属性：time_type（发布日期/生效日期/修订日期/废止日期/截止日期/过渡期）"
|
||||
Region: "适用区域：全国、省、市、县、特定区域等。属性：level（全国/区域/省/市/县/乡镇街道/社区/特定区域）"
|
||||
|
||||
relationship_type_prompts:
|
||||
# 文档层次
|
||||
has_chapter: "文档包含章"
|
||||
has_section: "章包含节"
|
||||
has_article: "文件/章/节包含条文"
|
||||
has_clause: "条文包含款"
|
||||
article_in_document: "条文或款属于某一法规文件"
|
||||
# 法条依据
|
||||
cites: "引用其他法规：如'根据《城乡规划法》第X条'、'依据《土地管理法》规定'、'按照国务院有关规定'"
|
||||
implements: "实施上位法：如'为实施《城乡规划法》制定本办法'、'根据《土地管理法》授权'"
|
||||
amends: "修正/修订：如'对《XX法》作如下修改'、'将第X条修改为'"
|
||||
replaces: "替代/废止：如'本法自施行之日起，XX法同时废止'"
|
||||
supplements: "补充规定：如'在XX基础上补充规定'"
|
||||
basis_for_planning: "法规或条文作为规划编制、审批、实施或政策工具设立的依据"
|
||||
# 主体关系
|
||||
issued_by: "法规文件由某主体发布或制定"
|
||||
implemented_by: "条文、政策工具、程序或义务由某主体负责实施"
|
||||
assigned_to: "监管职责、管理职责、实施任务分配给某主体"
|
||||
# 语义规制
|
||||
regulates: "条文对某法规事项进行规制（准入、禁止、限制、鼓励、保护等）"
|
||||
defines_object: "条文对法规事项、空间对象或更新场景进行定义或界定"
|
||||
applies_to: "条文、程序或政策工具适用于某法规事项、空间对象、更新场景或区域"
|
||||
sets_obligation: "条文设定权利、义务、禁止、职责或支持性要求"
|
||||
sets_condition: "条文、政策工具或程序设置适用条件（前置条件）"
|
||||
sets_constraint: "条文、政策工具或程序设置边界性、底线性或限制性要求"
|
||||
prescribes_penalty: "条文规定了违法行为的法律后果"
|
||||
# 政策工具
|
||||
extracts_tool: "从条文中识别出具体政策工具"
|
||||
belongs_to_category: "具体政策工具属于某类政策工具类别"
|
||||
applies_to_scene: "政策工具适用于某类城市更新场景"
|
||||
targets_object: "政策工具作用于某一法规事项或空间对象"
|
||||
coordinates_with: "两个政策工具之间存在并列协同或组合使用关系"
|
||||
supports_tool: "一个政策工具对另一个形成前置支撑或条件支撑"
|
||||
# 程序
|
||||
requires_procedure: "条文、政策工具或法规事项要求履行某类程序"
|
||||
procedure_for: "某一程序对应某类法规事项、空间对象、更新场景或政策工具"
|
||||
precedes_procedure: "一个程序在时序上先于另一个程序"
|
||||
# 辅助
|
||||
governs_region: "法规政策或条文适用于某一区域"
|
||||
effective_timeline: "法规文件关联时间节点（发布/生效/修订/废止）"
|
||||
standard_for: "标准、规范或条文适用于某一法规事项、空间对象、程序或政策工具"
|
||||
article_effective_timeline: "条文关联时间节点（生效/截止/过渡期）"
|
||||
|
||||
extraction_tips:
|
||||
citation_patterns: "注意识别法条引用模式：'根据《XX法》'、'依照第X条规定'、'按照国务院XX规定'、'参照XX标准执行'"
|
||||
article_classification: "法条类型判断：管理性（应当/必须）、禁止性（不得/禁止）、授权性（可以/有权）、程序性（申请/审批/备案）、处罚性（罚款/责令/没收）"
|
||||
cross_reference: "特别注意跨文档引用：当条文引用其他法规时，创建cites关系并记录引用上下文"
|
||||
entity_boundary_rules: |
|
||||
实体边界规则（严格遵守）：
|
||||
- LegalObject：法规事项，回答"管什么事项"，如规划编制、土地利用、财政支持
|
||||
- SpatialObject：空间对象，回答"作用于什么空间"，如老旧小区、城中村、低效用地
|
||||
- RenewalScene：更新场景，回答"用于什么更新场景"，如老旧小区改造、城中村改造
|
||||
- Procedure：程序，强调流程步骤（审批/备案/评估/公示/验收/入库）
|
||||
- PolicyTool：政策工具，强调治理手段、政策措施、制度安排或支持机制
|
||||
- Obligation：权利义务，强调主体应当/可以/不得做什么
|
||||
- Condition：适用条件，强调前置条件（经批准/符合/满足/具备）
|
||||
- Constraint：约束条件，强调底线限制（不得突破/严守/控制线/强制性标准）
|
||||
do_not_extract: |
|
||||
- 不要新增本体之外的实体类型或关系类型
|
||||
- 文本中没有依据时不要强行抽取
|
||||
- conflicts_with和has_gap不纳入本阶段抽取
|
||||
@@ -0,0 +1,86 @@
|
||||
project_name: "urban_planning_law_kg"
|
||||
project_description: "城市规划法律法规知识图谱构建"
|
||||
|
||||
data:
|
||||
source_dir: "E:\\Project\\SI\\2026_KG_PlanningLaw\\data\\城市规划法律法规\\城市更新法规数据库"
|
||||
total_documents: 296
|
||||
categories:
|
||||
- {id: 0, name: "0知识图谱目录", type: "Index"}
|
||||
- {id: 1, name: "1法律", type: "Law"}
|
||||
- {id: 2, name: "2行政法规", type: "AdministrativeRegulation"}
|
||||
- {id: 3, name: "3部门规章", type: "DepartmentalRule"}
|
||||
- {id: 4, name: "4党中央国务院文件", type: "PolicyDocument"}
|
||||
- {id: 5, name: "5国家主管部门文件", type: "PolicyDocument"}
|
||||
- {id: 6, name: "6主要技术标准", type: "PolicyDocument"}
|
||||
min_text_length: 50
|
||||
|
||||
api:
|
||||
config_file: "config/api_keys.yaml"
|
||||
model: "deepseek-chat"
|
||||
max_tokens: 8192
|
||||
temperature: 0.0
|
||||
|
||||
processing:
|
||||
concurrent_requests: 5
|
||||
max_retries: 3
|
||||
retry_delay: 2
|
||||
batch_size: 10
|
||||
|
||||
paths:
|
||||
config_dir: "config"
|
||||
data_dir: "data"
|
||||
output_dir: "output"
|
||||
logs_dir: "logs"
|
||||
ontology_dir: "ontology"
|
||||
scripts_dir: "scripts"
|
||||
src_dir: "src"
|
||||
|
||||
ontology:
|
||||
ontology_file: "ontology/legal_ontology.json"
|
||||
entity_ontology_file: "config/legal_ontology.yaml"
|
||||
|
||||
neo4j:
|
||||
uri: "bolt://localhost:7687"
|
||||
username: "neo4j"
|
||||
password: "password"
|
||||
database: "neo4j"
|
||||
|
||||
extraction:
|
||||
entity_types:
|
||||
- Law
|
||||
- AdministrativeRegulation
|
||||
- DepartmentalRule
|
||||
- PolicyDocument
|
||||
- Chapter
|
||||
- Section
|
||||
- Article
|
||||
- GovernmentBody
|
||||
- LegalSubject
|
||||
- SpatialConcept
|
||||
- AdministrativeProcedure
|
||||
- Obligation
|
||||
- Penalty
|
||||
- TimePoint
|
||||
- Region
|
||||
relation_types:
|
||||
- cites
|
||||
- implements
|
||||
- amends
|
||||
- replaces
|
||||
- supplements
|
||||
- issued_by
|
||||
- implemented_by
|
||||
- regulates
|
||||
- defines_spatial
|
||||
- requires_procedure
|
||||
- sets_obligation
|
||||
- prescribes_penalty
|
||||
- applies_to
|
||||
- governs_region
|
||||
confidence_threshold:
|
||||
entity: 0.8
|
||||
relation: 0.7
|
||||
|
||||
logging:
|
||||
level: "INFO"
|
||||
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
@@ -0,0 +1,246 @@
|
||||
entity_types:
|
||||
Law:
|
||||
prefix: "LAW"
|
||||
description: "全国人大及其常委会制定的法律"
|
||||
required_attributes: [name]
|
||||
optional_attributes:
|
||||
- full_title
|
||||
- promulgation_date
|
||||
- effective_date
|
||||
- status
|
||||
- amendment_dates
|
||||
|
||||
AdministrativeRegulation:
|
||||
prefix: "REG"
|
||||
description: "国务院制定的行政法规"
|
||||
required_attributes: [name]
|
||||
optional_attributes:
|
||||
- full_title
|
||||
- promulgation_date
|
||||
- effective_date
|
||||
- status
|
||||
- order_number
|
||||
|
||||
DepartmentalRule:
|
||||
prefix: "RULE"
|
||||
description: "各部委制定的部门规章"
|
||||
required_attributes: [name]
|
||||
optional_attributes:
|
||||
- full_title
|
||||
- promulgation_date
|
||||
- effective_date
|
||||
- status
|
||||
- order_number
|
||||
|
||||
PolicyDocument:
|
||||
prefix: "DOC"
|
||||
description: "党中央/国务院/主管部门文件、技术标准"
|
||||
required_attributes: [name]
|
||||
optional_attributes:
|
||||
- full_title
|
||||
- issue_date
|
||||
- document_number
|
||||
- document_type
|
||||
- status
|
||||
|
||||
Chapter:
|
||||
prefix: "CH"
|
||||
description: "法律文件的章"
|
||||
required_attributes: [name, number]
|
||||
optional_attributes:
|
||||
- title
|
||||
|
||||
Section:
|
||||
prefix: "SEC"
|
||||
description: "法律文件的节"
|
||||
required_attributes: [name, number]
|
||||
optional_attributes:
|
||||
- title
|
||||
|
||||
Article:
|
||||
prefix: "ART"
|
||||
description: "法律文件的具体条文"
|
||||
required_attributes: [number, text]
|
||||
optional_attributes:
|
||||
- article_type
|
||||
|
||||
GovernmentBody:
|
||||
prefix: "GOV"
|
||||
description: "立法/行政/主管部门"
|
||||
required_attributes: [name]
|
||||
optional_attributes:
|
||||
- full_name
|
||||
- level
|
||||
- type
|
||||
|
||||
LegalSubject:
|
||||
prefix: "SUBJ"
|
||||
description: "法律规制的事项或主体"
|
||||
required_attributes: [name]
|
||||
optional_attributes:
|
||||
- category
|
||||
- description
|
||||
|
||||
SpatialConcept:
|
||||
prefix: "SPAT"
|
||||
description: "国土空间规划中的空间概念"
|
||||
required_attributes: [name]
|
||||
optional_attributes:
|
||||
- type
|
||||
- definition
|
||||
|
||||
AdministrativeProcedure:
|
||||
prefix: "PROC"
|
||||
description: "规划/用地/建设的行政程序"
|
||||
required_attributes: [name]
|
||||
optional_attributes:
|
||||
- type
|
||||
- time_limit
|
||||
- applicable_subjects
|
||||
|
||||
Obligation:
|
||||
prefix: "OBL"
|
||||
description: "法律条文规定的权利、义务或禁止"
|
||||
required_attributes: [text]
|
||||
optional_attributes:
|
||||
- type
|
||||
|
||||
Penalty:
|
||||
prefix: "PEN"
|
||||
description: "违反法律规定的处罚"
|
||||
required_attributes: [type, description]
|
||||
optional_attributes:
|
||||
- applicable_violation
|
||||
- legal_basis
|
||||
|
||||
TimePoint:
|
||||
prefix: "TIME"
|
||||
description: "法规中提到的具体时间"
|
||||
required_attributes: [description]
|
||||
optional_attributes:
|
||||
- year
|
||||
- time_type
|
||||
|
||||
Region:
|
||||
prefix: "REGION"
|
||||
description: "法规适用的区域"
|
||||
required_attributes: [name]
|
||||
optional_attributes:
|
||||
- level
|
||||
- feature
|
||||
|
||||
relationship_types:
|
||||
has_chapter:
|
||||
description: "包含章"
|
||||
source_types: ["Law", "AdministrativeRegulation", "DepartmentalRule", "PolicyDocument"]
|
||||
target_types: ["Chapter"]
|
||||
|
||||
has_section:
|
||||
description: "包含节"
|
||||
source_types: ["Chapter"]
|
||||
target_types: ["Section"]
|
||||
|
||||
contains_article:
|
||||
description: "包含条"
|
||||
source_types: ["Chapter", "Section"]
|
||||
target_types: ["Article"]
|
||||
|
||||
article_in_document:
|
||||
description: "条属于文件"
|
||||
source_types: ["Article"]
|
||||
target_types: ["Law", "AdministrativeRegulation", "DepartmentalRule", "PolicyDocument"]
|
||||
|
||||
cites:
|
||||
description: "引用其他法规"
|
||||
source_types: ["Article", "PolicyDocument"]
|
||||
target_types: ["Law", "AdministrativeRegulation", "DepartmentalRule"]
|
||||
|
||||
implements:
|
||||
description: "实施/落实上位法"
|
||||
source_types: ["PolicyDocument", "DepartmentalRule"]
|
||||
target_types: ["Law", "AdministrativeRegulation"]
|
||||
|
||||
amends:
|
||||
description: "修正/修订"
|
||||
source_types: ["Law", "AdministrativeRegulation", "DepartmentalRule"]
|
||||
target_types: ["Law", "AdministrativeRegulation", "DepartmentalRule"]
|
||||
|
||||
replaces:
|
||||
description: "替代/废止"
|
||||
source_types: ["Law", "AdministrativeRegulation", "DepartmentalRule"]
|
||||
target_types: ["Law", "AdministrativeRegulation", "DepartmentalRule"]
|
||||
|
||||
supplements:
|
||||
description: "补充"
|
||||
source_types: ["PolicyDocument"]
|
||||
target_types: ["Law", "AdministrativeRegulation", "DepartmentalRule"]
|
||||
|
||||
issued_by:
|
||||
description: "发布机关"
|
||||
source_types: ["Law", "AdministrativeRegulation", "DepartmentalRule", "PolicyDocument"]
|
||||
target_types: ["GovernmentBody"]
|
||||
|
||||
implemented_by:
|
||||
description: "实施机关"
|
||||
source_types: ["Article", "AdministrativeProcedure"]
|
||||
target_types: ["GovernmentBody"]
|
||||
|
||||
regulates:
|
||||
description: "规制事项"
|
||||
source_types: ["Article", "Obligation"]
|
||||
target_types: ["LegalSubject"]
|
||||
|
||||
defines_spatial:
|
||||
description: "界定空间概念"
|
||||
source_types: ["Article"]
|
||||
target_types: ["SpatialConcept"]
|
||||
|
||||
applies_to:
|
||||
description: "适用于"
|
||||
source_types: ["Article", "AdministrativeProcedure"]
|
||||
target_types: ["LegalSubject"]
|
||||
|
||||
sets_obligation:
|
||||
description: "定义务"
|
||||
source_types: ["Article"]
|
||||
target_types: ["Obligation"]
|
||||
|
||||
prescribes_penalty:
|
||||
description: "规定处罚"
|
||||
source_types: ["Article", "Obligation"]
|
||||
target_types: ["Penalty"]
|
||||
|
||||
requires_procedure:
|
||||
description: "要求程序"
|
||||
source_types: ["Article"]
|
||||
target_types: ["AdministrativeProcedure"]
|
||||
|
||||
procedure_for:
|
||||
description: "程序对应事项"
|
||||
source_types: ["AdministrativeProcedure"]
|
||||
target_types: ["LegalSubject"]
|
||||
|
||||
governs_region:
|
||||
description: "管辖区域"
|
||||
source_types: ["Law", "AdministrativeRegulation", "DepartmentalRule"]
|
||||
target_types: ["Region"]
|
||||
|
||||
effective_timeline:
|
||||
description: "生效时间线"
|
||||
source_types: ["Law", "AdministrativeRegulation", "DepartmentalRule"]
|
||||
target_types: ["TimePoint"]
|
||||
|
||||
related_to:
|
||||
description: "相关"
|
||||
source_types: ["Any"]
|
||||
target_types: ["Any"]
|
||||
|
||||
standard_for:
|
||||
description: "标准适用于"
|
||||
source_types: ["PolicyDocument"]
|
||||
target_types: ["LegalSubject"]
|
||||
|
||||
category_index:
|
||||
description: "分类索引"
|
||||
source_types: ["Law", "AdministrativeRegulation", "DepartmentalRule", "PolicyDocument"]
|
||||
target_types: ["LegalSubject"]
|
||||
@@ -0,0 +1,139 @@
|
||||
"""
|
||||
Neo4j数据导入脚本 - 从CSV导入法规知识图谱数据
|
||||
"""
|
||||
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
from neo4j import GraphDatabase
|
||||
import json
|
||||
import logging
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Neo4jImporter:
|
||||
"""Neo4j数据导入器"""
|
||||
|
||||
def __init__(self, uri: str, username: str, password: str, database: str = 'neo4j'):
|
||||
self.driver = GraphDatabase.driver(uri, auth=(username, password))
|
||||
self.database = database
|
||||
logger.info(f"连接Neo4j: {uri}")
|
||||
|
||||
def close(self):
|
||||
self.driver.close()
|
||||
|
||||
def _run_query(self, query, parameters=None):
|
||||
with self.driver.session(database=self.database) as session:
|
||||
result = session.run(query, parameters or {})
|
||||
return [record.data() for record in result]
|
||||
|
||||
def create_constraints(self):
|
||||
"""创建约束和索引"""
|
||||
schema_file = Path(__file__).parent / 'schema.cypher'
|
||||
if schema_file.exists():
|
||||
with open(schema_file, 'r', encoding='utf-8') as f:
|
||||
content = f.read()
|
||||
|
||||
for line in content.split('\n'):
|
||||
line = line.strip()
|
||||
if line.startswith('CREATE ') and not line.startswith('//'):
|
||||
try:
|
||||
self._run_query(line)
|
||||
logger.info(f"执行: {line[:60]}...")
|
||||
except Exception as e:
|
||||
if 'already exists' not in str(e):
|
||||
logger.warning(f"约束跳过: {e}")
|
||||
|
||||
def import_nodes(self, nodes_csv: str):
|
||||
"""导入节点"""
|
||||
nodes_df = pd.read_csv(nodes_csv, encoding='utf-8-sig')
|
||||
logger.info(f"导入 {len(nodes_df)} 个节点")
|
||||
|
||||
# 按类型分批导入
|
||||
for node_type in nodes_df['type'].unique():
|
||||
type_df = nodes_df[nodes_df['type'] == node_type]
|
||||
label = node_type
|
||||
|
||||
for _, row in type_df.iterrows():
|
||||
props = {}
|
||||
try:
|
||||
props = json.loads(row.get('properties', '{}'))
|
||||
except:
|
||||
pass
|
||||
props['id'] = row['id']
|
||||
props['name'] = row['label']
|
||||
|
||||
query = f"MERGE (n:{label} {{id: $id}}) SET n += $props"
|
||||
self._run_query(query, {'id': row['id'], 'props': props})
|
||||
|
||||
logger.info(f" {label}: {len(type_df)} 个")
|
||||
|
||||
def import_relationships(self, rels_csv: str):
|
||||
"""导入关系"""
|
||||
rels_df = pd.read_csv(rels_csv, encoding='utf-8-sig')
|
||||
logger.info(f"导入 {len(rels_df)} 个关系")
|
||||
|
||||
imported = 0
|
||||
for _, row in rels_df.iterrows():
|
||||
rel_type = row['type']
|
||||
props = {}
|
||||
try:
|
||||
props = json.loads(row.get('properties', '{}'))
|
||||
except:
|
||||
pass
|
||||
|
||||
query = f"""
|
||||
MATCH (a {{id: $source}})
|
||||
MATCH (b {{id: $target}})
|
||||
MERGE (a)-[r:{rel_type}]->(b)
|
||||
SET r += $props
|
||||
"""
|
||||
try:
|
||||
self._run_query(query, {
|
||||
'source': row['source'],
|
||||
'target': row['target'],
|
||||
'props': props
|
||||
})
|
||||
imported += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"关系导入失败: {row['source']} -> {row['target']} ({rel_type}): {e}")
|
||||
|
||||
logger.info(f"成功导入 {imported}/{len(rels_df)} 个关系")
|
||||
|
||||
def import_all(self, nodes_csv: str, rels_csv: str):
|
||||
"""完整导入流程"""
|
||||
logger.info("开始导入...")
|
||||
self.create_constraints()
|
||||
self.import_nodes(nodes_csv)
|
||||
self.import_relationships(rels_csv)
|
||||
logger.info("导入完成！")
|
||||
|
||||
|
||||
def main():
|
||||
import yaml
|
||||
|
||||
config_file = Path(__file__).parent.parent / 'config' / 'legal_config.yaml'
|
||||
with open(config_file, 'r', encoding='utf-8') as f:
|
||||
config = yaml.safe_load(f)
|
||||
|
||||
neo4j_config = config['neo4j']
|
||||
importer = Neo4jImporter(
|
||||
neo4j_config['uri'],
|
||||
neo4j_config['username'],
|
||||
neo4j_config['password'],
|
||||
neo4j_config['database']
|
||||
)
|
||||
|
||||
output_dir = Path(__file__).parent.parent / 'output'
|
||||
nodes_csv = str(output_dir / 'nodes_merged.csv')
|
||||
rels_csv = str(output_dir / 'rels_merged.csv')
|
||||
|
||||
try:
|
||||
importer.import_all(nodes_csv, rels_csv)
|
||||
finally:
|
||||
importer.close()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,62 @@
|
||||
// Neo4j数据模型 - 城市规划法律法规知识图谱
|
||||
|
||||
// ============================================================
|
||||
// 约束和索引
|
||||
// ============================================================
|
||||
|
||||
CREATE CONSTRAINT law_id IF NOT EXISTS FOR (l:Law) REQUIRE l.law_id IS UNIQUE;
|
||||
CREATE CONSTRAINT reg_id IF NOT EXISTS FOR (r:AdministrativeRegulation) REQUIRE r.reg_id IS UNIQUE;
|
||||
CREATE CONSTRAINT rule_id IF NOT EXISTS FOR (r:DepartmentalRule) REQUIRE r.rule_id IS UNIQUE;
|
||||
CREATE CONSTRAINT doc_id IF NOT EXISTS FOR (d:PolicyDocument) REQUIRE d.doc_id IS UNIQUE;
|
||||
CREATE CONSTRAINT chapter_id IF NOT EXISTS FOR (c:Chapter) REQUIRE c.chapter_id IS UNIQUE;
|
||||
CREATE CONSTRAINT section_id IF NOT EXISTS FOR (s:Section) REQUIRE s.section_id IS UNIQUE;
|
||||
CREATE CONSTRAINT article_id IF NOT EXISTS FOR (a:Article) REQUIRE a.article_id IS UNIQUE;
|
||||
CREATE CONSTRAINT body_id IF NOT EXISTS FOR (g:GovernmentBody) REQUIRE g.body_id IS UNIQUE;
|
||||
CREATE CONSTRAINT subject_id IF NOT EXISTS FOR (s:LegalSubject) REQUIRE s.subject_id IS UNIQUE;
|
||||
CREATE CONSTRAINT concept_id IF NOT EXISTS FOR (s:SpatialConcept) REQUIRE s.concept_id IS UNIQUE;
|
||||
CREATE CONSTRAINT proc_id IF NOT EXISTS FOR (p:AdministrativeProcedure) REQUIRE p.proc_id IS UNIQUE;
|
||||
|
||||
// 索引
|
||||
CREATE INDEX law_name IF NOT EXISTS FOR (l:Law) ON (l.name);
|
||||
CREATE INDEX law_status IF NOT EXISTS FOR (l:Law) ON (l.status);
|
||||
CREATE INDEX article_type IF NOT EXISTS FOR (a:Article) ON (a.article_type);
|
||||
CREATE INDEX body_name IF NOT EXISTS FOR (g:GovernmentBody) ON (g.name);
|
||||
CREATE INDEX subject_category IF NOT EXISTS FOR (s:LegalSubject) ON (s.category);
|
||||
|
||||
// 全文搜索
|
||||
CREATE FULLTEXT INDEX law_search IF NOT EXISTS FOR (l:Law) ON EACH [l.name, l.full_title];
|
||||
CREATE FULLTEXT INDEX article_search IF NOT EXISTS FOR (a:Article) ON EACH [a.text];
|
||||
|
||||
// ============================================================
|
||||
// 导入命令示例
|
||||
// ============================================================
|
||||
|
||||
// LOAD CSV WITH HEADERS FROM 'file:///nodes.csv' AS row
|
||||
// WITH row WHERE row.type = 'Law'
|
||||
// CREATE (:Law {law_id: row.id, name: row.label, full_title: apoc.text.replace(row.label, '"', '')});
|
||||
//
|
||||
// LOAD CSV WITH HEADERS FROM 'file:///rels.csv' AS row
|
||||
// MATCH (a {id: row.source})
|
||||
// MATCH (b {id: row.target})
|
||||
// CALL apoc.create.relationship(a, row.type, {}, b) YIELD rel
|
||||
// RETURN rel;
|
||||
|
||||
// ============================================================
|
||||
// 常用查询
|
||||
// ============================================================
|
||||
|
||||
// 查询某法律的所有条文
|
||||
// MATCH (l:Law {name: '城乡规划法'})-[:HAS_CHAPTER]->(ch)-[:CONTAINS_ARTICLE]->(art:Article)
|
||||
// RETURN ch.title, art.number, art.text;
|
||||
|
||||
// 查询引用某法律的法规
|
||||
// MATCH (d)-[r:CITES]->(l:Law {name: '城乡规划法'})
|
||||
// RETURN d.name, type(r), r.context;
|
||||
|
||||
// 查询涉及某主题的所有条文
|
||||
// MATCH (art:Article)-[:REGULATES]->(s:LegalSubject {name: '建设用地'})
|
||||
// RETURN art.text;
|
||||
|
||||
// 查询某机关发布的所有文件
|
||||
// MATCH (d)-[:ISSUED_BY]->(g:GovernmentBody {name: '自然资源部'})
|
||||
// RETURN d.name, labels(d);
|
||||
@@ -0,0 +1,213 @@
|
||||
{
|
||||
"ontology": {
|
||||
"name": "Urban Planning Law Knowledge Graph Ontology",
|
||||
"version": "1.0",
|
||||
"description": "城市规划法律法规知识图谱本体模型",
|
||||
"created_date": "2026-05-13",
|
||||
"concepts": [
|
||||
{
|
||||
"id": "Law",
|
||||
"name": "法律",
|
||||
"description": "全国人大及其常委会制定的法律",
|
||||
"properties": [
|
||||
{"name": "law_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "full_title", "type": "string"},
|
||||
{"name": "promulgation_date", "type": "string"},
|
||||
{"name": "effective_date", "type": "string"},
|
||||
{"name": "amendment_dates", "type": "array<string>"},
|
||||
{"name": "status", "type": "string", "enum": ["现行有效", "修订", "废止", "草案"]},
|
||||
{"name": "legal_hierarchy", "type": "string", "default": "法律"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "AdministrativeRegulation",
|
||||
"name": "行政法规",
|
||||
"description": "国务院制定的行政法规",
|
||||
"properties": [
|
||||
{"name": "reg_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "full_title", "type": "string"},
|
||||
{"name": "promulgation_date", "type": "string"},
|
||||
{"name": "effective_date", "type": "string"},
|
||||
{"name": "status", "type": "string"},
|
||||
{"name": "order_number", "type": "string"},
|
||||
{"name": "legal_hierarchy", "type": "string", "default": "行政法规"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "DepartmentalRule",
|
||||
"name": "部门规章",
|
||||
"description": "各部委制定的部门规章",
|
||||
"properties": [
|
||||
{"name": "rule_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "full_title", "type": "string"},
|
||||
{"name": "promulgation_date", "type": "string"},
|
||||
{"name": "effective_date", "type": "string"},
|
||||
{"name": "status", "type": "string"},
|
||||
{"name": "order_number", "type": "string"},
|
||||
{"name": "legal_hierarchy", "type": "string", "default": "部门规章"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "PolicyDocument",
|
||||
"name": "政策文件",
|
||||
"description": "党中央/国务院/主管部门文件、技术标准",
|
||||
"properties": [
|
||||
{"name": "doc_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "full_title", "type": "string"},
|
||||
{"name": "issue_date", "type": "string"},
|
||||
{"name": "document_number", "type": "string"},
|
||||
{"name": "document_type", "type": "string", "enum": ["意见", "通知", "纲要", "指南", "标准", "规范", "办法", "规定", "决定", "批复", "函", "其他"]},
|
||||
{"name": "status", "type": "string"},
|
||||
{"name": "legal_hierarchy", "type": "string", "default": "政策文件"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Chapter",
|
||||
"name": "章",
|
||||
"description": "法律文件的章",
|
||||
"properties": [
|
||||
{"name": "chapter_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "number", "type": "string"},
|
||||
{"name": "title", "type": "string"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Section",
|
||||
"name": "节",
|
||||
"description": "法律文件的节",
|
||||
"properties": [
|
||||
{"name": "section_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "number", "type": "string"},
|
||||
{"name": "title", "type": "string"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Article",
|
||||
"name": "条",
|
||||
"description": "法律文件的具体条文",
|
||||
"properties": [
|
||||
{"name": "article_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "number", "type": "string"},
|
||||
{"name": "text", "type": "string"},
|
||||
{"name": "article_type", "type": "string", "enum": ["管理性", "禁止性", "授权性", "程序性", "处罚性", "定义性"]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "GovernmentBody",
|
||||
"name": "政府机构",
|
||||
"description": "立法/行政/主管部门",
|
||||
"properties": [
|
||||
{"name": "body_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "full_name", "type": "string"},
|
||||
{"name": "level", "type": "string", "enum": ["国家", "省级", "市级", "县级"]},
|
||||
{"name": "type", "type": "string", "enum": ["立法机关", "行政机关", "主管部门", "地方机关"]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "LegalSubject",
|
||||
"name": "法定事项",
|
||||
"description": "法律规制的事项或主体",
|
||||
"properties": [
|
||||
{"name": "subject_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "category", "type": "string", "enum": ["土地", "规划", "建设", "环境", "文物", "交通", "防灾", "生态", "基础设施", "更新", "保护", "测绘", "登记", "处罚", "住房", "市政", "园林", "水资源", "矿业"]},
|
||||
{"name": "description", "type": "string"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "SpatialConcept",
|
||||
"name": "空间概念",
|
||||
"description": "国土空间规划中的空间概念",
|
||||
"properties": [
|
||||
{"name": "concept_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "type", "type": "string", "enum": ["国土空间", "生态红线", "永久基本农田", "城镇开发边界", "规划区", "建设用地", "农用地", "未利用地", "历史文化保护区", "控制线", "城市更新单元"]},
|
||||
{"name": "definition", "type": "string"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "AdministrativeProcedure",
|
||||
"name": "行政程序",
|
||||
"description": "规划/用地/建设的行政程序",
|
||||
"properties": [
|
||||
{"name": "proc_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "type", "type": "string", "enum": ["审批", "核准", "备案", "登记", "许可", "验收", "监督", "听证", "征收", "拆迁", "出让", "转让"]},
|
||||
{"name": "time_limit", "type": "string"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Obligation",
|
||||
"name": "权利义务",
|
||||
"description": "法律条文规定的权利、义务或禁止",
|
||||
"properties": [
|
||||
{"name": "oblig_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "text", "type": "string"},
|
||||
{"name": "type", "type": "string", "enum": ["权利", "义务", "禁止", "限制", "鼓励"]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Penalty",
|
||||
"name": "法律责任",
|
||||
"description": "违反法律规定的处罚",
|
||||
"properties": [
|
||||
{"name": "penalty_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "type", "type": "string", "enum": ["行政处罚", "刑事处罚", "民事责任", "行政处分"]},
|
||||
{"name": "description", "type": "string"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "TimePoint",
|
||||
"name": "时间节点",
|
||||
"description": "法规中提到的具体时间",
|
||||
"properties": [
|
||||
{"name": "time_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "year", "type": "integer"},
|
||||
{"name": "description", "type": "string"},
|
||||
{"name": "time_type", "type": "string", "enum": ["生效日期", "废止日期", "修订日期", "过渡期", "截止日期"]}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Region",
|
||||
"name": "区域",
|
||||
"description": "法规适用的区域",
|
||||
"properties": [
|
||||
{"name": "region_id", "type": "string", "required": true, "unique": true},
|
||||
{"name": "name", "type": "string", "required": true},
|
||||
{"name": "level", "type": "string", "enum": ["全国", "区域", "省", "市", "县", "特定区域"]}
|
||||
]
|
||||
}
|
||||
],
|
||||
"relationships": [
|
||||
{"name": "has_chapter", "description": "包含章", "from": "Law/AdministrativeRegulation/DepartmentalRule/PolicyDocument", "to": "Chapter", "properties": [{"name": "order_number", "type": "integer"}]},
|
||||
{"name": "has_section", "description": "包含节", "from": "Chapter", "to": "Section", "properties": [{"name": "order_number", "type": "integer"}]},
|
||||
{"name": "contains_article", "description": "包含条", "from": "Chapter/Section", "to": "Article", "properties": [{"name": "order_number", "type": "integer"}]},
|
||||
{"name": "article_in_document", "description": "条属于文件", "from": "Article", "to": "Law/AdministrativeRegulation/DepartmentalRule/PolicyDocument", "properties": []},
|
||||
{"name": "cites", "description": "引用", "from": "Article/PolicyDocument", "to": "Law/AdministrativeRegulation/DepartmentalRule", "properties": [{"name": "context", "type": "string"}]},
|
||||
{"name": "implements", "description": "实施/落实", "from": "PolicyDocument/DepartmentalRule", "to": "Law/AdministrativeRegulation", "properties": [{"name": "scope", "type": "string"}]},
|
||||
{"name": "amends", "description": "修正/修订", "from": "Law/Regulation/Rule", "to": "Law/Regulation/Rule", "properties": [{"name": "amendment_date", "type": "string"}]},
|
||||
{"name": "replaces", "description": "替代/废止", "from": "Law/Regulation/Rule", "to": "Law/Regulation/Rule", "properties": [{"name": "replacement_date", "type": "string"}]},
|
||||
{"name": "supplements", "description": "补充", "from": "PolicyDocument", "to": "Law/Regulation/Rule", "properties": [{"name": "scope", "type": "string"}]},
|
||||
{"name": "issued_by", "description": "发布机关", "from": "Law/Regulation/Rule/PolicyDocument", "to": "GovernmentBody", "properties": [{"name": "role", "type": "string"}]},
|
||||
{"name": "implemented_by", "description": "实施机关", "from": "Article/AdministrativeProcedure", "to": "GovernmentBody", "properties": [{"name": "role", "type": "string"}]},
|
||||
{"name": "regulates", "description": "规制", "from": "Article/Obligation", "to": "LegalSubject", "properties": [{"name": "regulation_type", "type": "string"}]},
|
||||
{"name": "defines_spatial", "description": "界定空间", "from": "Article", "to": "SpatialConcept", "properties": [{"name": "scope", "type": "string"}]},
|
||||
{"name": "applies_to", "description": "适用于", "from": "Article/AdministrativeProcedure", "to": "LegalSubject", "properties": [{"name": "context", "type": "string"}]},
|
||||
{"name": "sets_obligation", "description": "定义务", "from": "Article", "to": "Obligation", "properties": [{"name": "obligation_type", "type": "string"}]},
|
||||
{"name": "prescribes_penalty", "description": "规定处罚", "from": "Article/Obligation", "to": "Penalty", "properties": [{"name": "conditions", "type": "string"}]},
|
||||
{"name": "requires_procedure", "description": "要求程序", "from": "Article", "to": "AdministrativeProcedure", "properties": [{"name": "mandatory", "type": "string"}]},
|
||||
{"name": "procedure_for", "description": "程序对应事项", "from": "AdministrativeProcedure", "to": "LegalSubject", "properties": [{"name": "purpose", "type": "string"}]},
|
||||
{"name": "procedure_step", "description": "程序步骤", "from": "AdministrativeProcedure", "to": "AdministrativeProcedure", "properties": [{"name": "step_order", "type": "integer"}]},
|
||||
{"name": "governs_region", "description": "管辖区域", "from": "Law/Regulation/Rule", "to": "Region", "properties": [{"name": "scope", "type": "string"}]},
|
||||
{"name": "effective_timeline", "description": "生效时间线", "from": "Law/Regulation/Rule", "to": "TimePoint", "properties": [{"name": "timeline_type", "type": "string"}]},
|
||||
{"name": "related_to", "description": "相关", "from": "Any", "to": "Any", "properties": [{"name": "description", "type": "string"}]},
|
||||
{"name": "standard_for", "description": "标准适用于", "from": "PolicyDocument", "to": "LegalSubject", "properties": [{"name": "standard_type", "type": "string"}]},
|
||||
{"name": "category_index", "description": "分类索引", "from": "Law/Regulation/Rule/PolicyDocument", "to": "LegalSubject", "properties": [{"name": "source", "type": "string"}]}
|
||||
]
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,29 @@
|
||||
# 城市规划法律法规知识图谱构建 - 依赖包
|
||||
|
||||
# LLM相关
|
||||
langchain-deepseek>=0.1.0
|
||||
langchain-core>=0.1.0
|
||||
|
||||
# 文档读取
|
||||
python-docx>=1.1.0
|
||||
|
||||
# Neo4j
|
||||
neo4j>=5.15.0
|
||||
|
||||
# 数据处理
|
||||
pandas>=2.0.0
|
||||
numpy>=1.24.0
|
||||
|
||||
# 文本相似度
|
||||
python-Levenshtein>=0.23.0
|
||||
|
||||
# 可视化
|
||||
networkx>=3.1
|
||||
matplotlib>=3.7.0
|
||||
|
||||
# 配置
|
||||
pyyaml>=6.0
|
||||
python-dotenv>=1.0.0
|
||||
|
||||
# 工具
|
||||
tqdm>=4.65.0
|
||||
@@ -0,0 +1,221 @@
|
||||
"""
|
||||
Word文档读取器 - 读取城市更新法规数据库中所有.docx文件
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import List, Dict, Any
|
||||
|
||||
try:
|
||||
from docx import Document
|
||||
except ImportError:
|
||||
print("错误: 请先安装 python-docx")
|
||||
print("运行: pip install python-docx")
|
||||
raise
|
||||
|
||||
|
||||
class DocxReader:
|
||||
"""Word文档读取器"""
|
||||
|
||||
# 目录类别映射
|
||||
CATEGORY_MAP = {
|
||||
'0知识图谱目录': {'id': 0, 'type': 'Index', 'name': '知识图谱目录'},
|
||||
'1法律': {'id': 1, 'type': 'Law', 'name': '法律'},
|
||||
'2行政法规': {'id': 2, 'type': 'AdministrativeRegulation', 'name': '行政法规'},
|
||||
'3部门规章': {'id': 3, 'type': 'DepartmentalRule', 'name': '部门规章'},
|
||||
'4党中央国务院文件': {'id': 4, 'type': 'PolicyDocument', 'name': '党中央国务院文件'},
|
||||
'5国家主管部门文件': {'id': 5, 'type': 'PolicyDocument', 'name': '国家主管部门文件'},
|
||||
'6主要技术标准': {'id': 6, 'type': 'PolicyDocument', 'name': '主要技术标准'},
|
||||
}
|
||||
|
||||
def __init__(self, source_dir: str):
|
||||
self.source_dir = Path(source_dir)
|
||||
self.logger = self._setup_logger()
|
||||
self.documents = []
|
||||
|
||||
def _setup_logger(self):
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
return logging.getLogger(__name__)
|
||||
|
||||
def _extract_text_from_docx(self, filepath: Path) -> str:
|
||||
"""从Word文档提取纯文本"""
|
||||
doc = Document(str(filepath))
|
||||
paragraphs = []
|
||||
for para in doc.paragraphs:
|
||||
text = para.text.strip()
|
||||
if text:
|
||||
paragraphs.append(text)
|
||||
|
||||
# 也提取表格中的文本
|
||||
for table in doc.tables:
|
||||
for row in table.rows:
|
||||
row_text = []
|
||||
for cell in row.cells:
|
||||
cell_text = cell.text.strip()
|
||||
if cell_text:
|
||||
row_text.append(cell_text)
|
||||
if row_text:
|
||||
paragraphs.append(' | '.join(row_text))
|
||||
|
||||
return '\n'.join(paragraphs)
|
||||
|
||||
def _parse_filename_info(self, filename: str) -> Dict[str, Any]:
|
||||
"""从文件名解析基本信息"""
|
||||
info = {
|
||||
'original_filename': filename,
|
||||
'document_number': None,
|
||||
'document_name': None,
|
||||
'is_draft': False,
|
||||
}
|
||||
|
||||
# 去掉扩展名
|
||||
name_without_ext = Path(filename).stem
|
||||
|
||||
# 检测是否为草案/征求意见稿
|
||||
if '草案' in name_without_ext or '征求意见稿' in name_without_ext:
|
||||
info['is_draft'] = True
|
||||
|
||||
# 尝试提取编号前缀 (如 "3-中华人民共和国城乡规划法")
|
||||
match = re.match(r'^(\d+)[-—]\s*(.+)$', name_without_ext)
|
||||
if match:
|
||||
info['file_number'] = int(match.group(1))
|
||||
name_part = match.group(2)
|
||||
else:
|
||||
name_part = name_without_ext
|
||||
|
||||
# 提取书名号中的名称
|
||||
title_match = re.search(r'《(.+?)》', name_part)
|
||||
if title_match:
|
||||
info['document_name'] = title_match.group(1)
|
||||
else:
|
||||
# 去掉常见前缀
|
||||
cleaned = re.sub(r'^(中华人民共和国|国务院|国土资源部|建设部|住房城乡建设部|自然资源部)\s*', '', name_part)
|
||||
info['document_name'] = cleaned if cleaned else name_part
|
||||
|
||||
return info
|
||||
|
||||
def read_all_documents(self) -> List[Dict[str, Any]]:
|
||||
"""读取所有Word文档"""
|
||||
self.logger.info(f"开始读取文档目录: {self.source_dir}")
|
||||
self.documents = []
|
||||
errors = []
|
||||
|
||||
for subdir_name, category_info in self.CATEGORY_MAP.items():
|
||||
subdir = self.source_dir / subdir_name
|
||||
if not subdir.exists():
|
||||
self.logger.warning(f"子目录不存在: {subdir}")
|
||||
continue
|
||||
|
||||
docx_files = sorted(subdir.glob('*.docx'))
|
||||
self.logger.info(f" {subdir_name}: 发现 {len(docx_files)} 个文件")
|
||||
|
||||
for docx_file in docx_files:
|
||||
try:
|
||||
text = self._extract_text_from_docx(docx_file)
|
||||
filename_info = self._parse_filename_info(docx_file.name)
|
||||
|
||||
doc_record = {
|
||||
'doc_id': f"{category_info['type'][:3].upper()}-{len(self.documents):04d}",
|
||||
'filename': docx_file.name,
|
||||
'category_id': category_info['id'],
|
||||
'category_name': category_info['name'],
|
||||
'document_type': category_info['type'],
|
||||
'subdirectory': subdir_name,
|
||||
'raw_text': text,
|
||||
'text_length': len(text),
|
||||
'conversion_timestamp': datetime.now().isoformat(),
|
||||
**filename_info,
|
||||
}
|
||||
|
||||
self.documents.append(doc_record)
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"读取失败: {docx_file.name} - {str(e)}"
|
||||
self.logger.error(error_msg)
|
||||
errors.append({
|
||||
'filename': docx_file.name,
|
||||
'subdirectory': subdir_name,
|
||||
'error': str(e)
|
||||
})
|
||||
|
||||
self.logger.info(f"读取完成: {len(self.documents)} 个文档, {len(errors)} 个错误")
|
||||
return self.documents
|
||||
|
||||
def save_to_json(self, output_path: str):
|
||||
"""保存到JSON文件"""
|
||||
output = Path(output_path)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
result = {
|
||||
'metadata': {
|
||||
'total_documents': len(self.documents),
|
||||
'source_dir': str(self.source_dir),
|
||||
'created_at': datetime.now().isoformat(),
|
||||
'category_distribution': {},
|
||||
},
|
||||
'documents': self.documents,
|
||||
}
|
||||
|
||||
# 统计分类分布
|
||||
for doc in self.documents:
|
||||
cat = doc['category_name']
|
||||
result['metadata']['category_distribution'][cat] = \
|
||||
result['metadata']['category_distribution'].get(cat, 0) + 1
|
||||
|
||||
with open(output, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, indent=2, ensure_ascii=False)
|
||||
|
||||
self.logger.info(f"已保存到: {output}")
|
||||
return result
|
||||
|
||||
def print_summary(self):
|
||||
"""打印摘要"""
|
||||
print("\n" + "=" * 60)
|
||||
print("文档读取摘要")
|
||||
print("=" * 60)
|
||||
print(f"总文档数: {len(self.documents)}")
|
||||
|
||||
# 按类别统计
|
||||
category_counts = {}
|
||||
total_chars = 0
|
||||
for doc in self.documents:
|
||||
cat = doc['category_name']
|
||||
category_counts[cat] = category_counts.get(cat, 0) + 1
|
||||
total_chars += doc['text_length']
|
||||
|
||||
print("\n类别分布:")
|
||||
for cat, count in sorted(category_counts.items()):
|
||||
print(f" {cat}: {count} 个")
|
||||
|
||||
print(f"\n总字符数: {total_chars:,}")
|
||||
print(f"平均字符数: {total_chars // max(len(self.documents), 1):,}")
|
||||
|
||||
# 文本长度分布
|
||||
lengths = [doc['text_length'] for doc in self.documents]
|
||||
if lengths:
|
||||
print(f"\n文本长度:")
|
||||
print(f" 最短: {min(lengths):,} 字符")
|
||||
print(f" 最长: {max(lengths):,} 字符")
|
||||
print(f" 中位数: {sorted(lengths)[len(lengths)//2]:,} 字符")
|
||||
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
def main():
|
||||
source_dir = r"E:\Project\SI\2026_KG_PlanningLaw\data\城市规划法律法规\城市更新法规数据库"
|
||||
output_path = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output\all_documents.json"
|
||||
|
||||
reader = DocxReader(source_dir)
|
||||
reader.read_all_documents()
|
||||
reader.save_to_json(output_path)
|
||||
reader.print_summary()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,314 @@
|
||||
"""
|
||||
从结构化法规数据生成知识图谱CSV文件（nodes.csv + rels.csv）
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Any, Tuple
|
||||
import logging
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# 节点类型到ID前缀的映射
|
||||
TYPE_PREFIX = {
|
||||
'Law': 'LAW',
|
||||
'AdministrativeRegulation': 'REG',
|
||||
'DepartmentalRule': 'RULE',
|
||||
'PolicyDocument': 'DOC',
|
||||
'Index': 'DOC',
|
||||
'Chapter': 'CH',
|
||||
'Section': 'SEC',
|
||||
'Article': 'ART',
|
||||
'GovernmentBody': 'GOV',
|
||||
}
|
||||
|
||||
|
||||
def generate_id(prefix: str, text: str) -> str:
|
||||
"""根据文本哈希生成ID"""
|
||||
return f"{prefix}-{abs(hash(text)) % 100000:05d}"
|
||||
|
||||
|
||||
def build_document_nodes(doc: Dict[str, Any], doc_index: int) -> Tuple[List[Dict], List[Dict]]:
|
||||
"""从单个文档构建节点和关系"""
|
||||
nodes = []
|
||||
rels = []
|
||||
rel_id = 0
|
||||
|
||||
# 文档类型映射
|
||||
doc_type = doc['document_type']
|
||||
if doc_type == 'Index':
|
||||
doc_type = 'PolicyDocument'
|
||||
|
||||
# 1. 创建文档节点
|
||||
doc_id = generate_id(TYPE_PREFIX.get(doc_type, 'DOC'), doc['title'])
|
||||
|
||||
doc_node = {
|
||||
'id': doc_id,
|
||||
'label': doc['title'],
|
||||
'type': doc_type,
|
||||
'properties': json.dumps({
|
||||
'full_title': doc.get('document_name', ''),
|
||||
'promulgation_date': doc.get('promulgation_date'),
|
||||
'effective_date': doc.get('effective_date'),
|
||||
'document_number': doc.get('document_number'),
|
||||
'is_draft': doc.get('is_draft', False),
|
||||
'category': doc.get('category_name', ''),
|
||||
'filename': doc.get('filename', ''),
|
||||
'text_length': doc.get('text_length', 0),
|
||||
'article_count': doc.get('article_count', 0),
|
||||
}, ensure_ascii=False)
|
||||
}
|
||||
nodes.append(doc_node)
|
||||
|
||||
# 2. 创建发布机关节点和关系
|
||||
if doc.get('issuing_body'):
|
||||
body_name = doc['issuing_body']
|
||||
body_id = generate_id('GOV', body_name)
|
||||
|
||||
# 检查是否已存在
|
||||
body_node = {
|
||||
'id': body_id,
|
||||
'label': body_name,
|
||||
'type': 'GovernmentBody',
|
||||
'properties': json.dumps({}, ensure_ascii=False)
|
||||
}
|
||||
nodes.append(body_node)
|
||||
|
||||
rel_id += 1
|
||||
rels.append({
|
||||
'source': doc_id,
|
||||
'target': body_id,
|
||||
'type': 'ISSUED_BY',
|
||||
'properties': json.dumps({}, ensure_ascii=False)
|
||||
})
|
||||
|
||||
# 3. 创建章节条文节点和层次关系
|
||||
for ch_idx, chapter in enumerate(doc.get('chapters', [])):
|
||||
ch_title = chapter.get('title', f'第{ch_idx+1}章')
|
||||
ch_id = generate_id('CH', f"{doc_id}_{ch_title}")
|
||||
|
||||
ch_node = {
|
||||
'id': ch_id,
|
||||
'label': ch_title,
|
||||
'type': 'Chapter',
|
||||
'properties': json.dumps({
|
||||
'name': chapter.get('name', ''),
|
||||
'number': str(ch_idx + 1),
|
||||
}, ensure_ascii=False)
|
||||
}
|
||||
nodes.append(ch_node)
|
||||
|
||||
# 文档→章
|
||||
rel_id += 1
|
||||
rels.append({
|
||||
'source': doc_id,
|
||||
'target': ch_id,
|
||||
'type': 'HAS_CHAPTER',
|
||||
'properties': json.dumps({'order_number': ch_idx + 1}, ensure_ascii=False)
|
||||
})
|
||||
|
||||
# 章下的条文
|
||||
for art_idx, article in enumerate(chapter.get('articles', [])):
|
||||
art_number = article.get('number', str(art_idx + 1))
|
||||
art_text = article.get('text', '')
|
||||
art_id = generate_id('ART', f"{doc_id}_{art_number}")
|
||||
|
||||
art_node = {
|
||||
'id': art_id,
|
||||
'label': f"第{art_number}条",
|
||||
'type': 'Article',
|
||||
'properties': json.dumps({
|
||||
'number': art_number,
|
||||
'text': art_text[:500],
|
||||
'article_type': article.get('article_type', ''),
|
||||
}, ensure_ascii=False)
|
||||
}
|
||||
nodes.append(art_node)
|
||||
|
||||
# 章→条
|
||||
rel_id += 1
|
||||
rels.append({
|
||||
'source': ch_id,
|
||||
'target': art_id,
|
||||
'type': 'CONTAINS_ARTICLE',
|
||||
'properties': json.dumps({'order_number': art_idx + 1}, ensure_ascii=False)
|
||||
})
|
||||
|
||||
# 条→文档
|
||||
rel_id += 1
|
||||
rels.append({
|
||||
'source': art_id,
|
||||
'target': doc_id,
|
||||
'type': 'ARTICLE_IN_DOCUMENT',
|
||||
'properties': json.dumps({}, ensure_ascii=False)
|
||||
})
|
||||
|
||||
# 节
|
||||
for sec_idx, section in enumerate(chapter.get('sections', [])):
|
||||
sec_title = section.get('title', f'第{sec_idx+1}节')
|
||||
sec_id = generate_id('SEC', f"{doc_id}_{ch_title}_{sec_title}")
|
||||
|
||||
sec_node = {
|
||||
'id': sec_id,
|
||||
'label': sec_title,
|
||||
'type': 'Section',
|
||||
'properties': json.dumps({
|
||||
'name': section.get('name', ''),
|
||||
'number': str(sec_idx + 1),
|
||||
}, ensure_ascii=False)
|
||||
}
|
||||
nodes.append(sec_node)
|
||||
|
||||
# 章→节
|
||||
rel_id += 1
|
||||
rels.append({
|
||||
'source': ch_id,
|
||||
'target': sec_id,
|
||||
'type': 'HAS_SECTION',
|
||||
'properties': json.dumps({'order_number': sec_idx + 1}, ensure_ascii=False)
|
||||
})
|
||||
|
||||
# 节下的条文
|
||||
for art_idx, article in enumerate(section.get('articles', [])):
|
||||
art_number = article.get('number', str(art_idx + 1))
|
||||
art_text = article.get('text', '')
|
||||
art_id = generate_id('ART', f"{doc_id}_{art_number}")
|
||||
|
||||
art_node = {
|
||||
'id': art_id,
|
||||
'label': f"第{art_number}条",
|
||||
'type': 'Article',
|
||||
'properties': json.dumps({
|
||||
'number': art_number,
|
||||
'text': art_text[:500],
|
||||
'article_type': article.get('article_type', ''),
|
||||
}, ensure_ascii=False)
|
||||
}
|
||||
nodes.append(art_node)
|
||||
|
||||
# 节→条
|
||||
rel_id += 1
|
||||
rels.append({
|
||||
'source': sec_id,
|
||||
'target': art_id,
|
||||
'type': 'CONTAINS_ARTICLE',
|
||||
'properties': json.dumps({'order_number': art_idx + 1}, ensure_ascii=False)
|
||||
})
|
||||
|
||||
# 条→文档
|
||||
rel_id += 1
|
||||
rels.append({
|
||||
'source': art_id,
|
||||
'target': doc_id,
|
||||
'type': 'ARTICLE_IN_DOCUMENT',
|
||||
'properties': json.dumps({}, ensure_ascii=False)
|
||||
})
|
||||
|
||||
return nodes, rels
|
||||
|
||||
|
||||
def deduplicate_nodes(nodes: List[Dict]) -> List[Dict]:
|
||||
"""去重节点（基于id）"""
|
||||
seen = {}
|
||||
for node in nodes:
|
||||
if node['id'] not in seen:
|
||||
seen[node['id']] = node
|
||||
else:
|
||||
# 合并properties
|
||||
existing = seen[node['id']]
|
||||
if existing['properties'] == '{}' and node['properties'] != '{}':
|
||||
existing['properties'] = node['properties']
|
||||
return list(seen.values())
|
||||
|
||||
|
||||
def generate_report(nodes_df: pd.DataFrame, rels_df: pd.DataFrame, output_dir: Path):
|
||||
"""生成统计报告"""
|
||||
lines = ["# 结构化CSV提取报告\n"]
|
||||
lines.append(f"生成时间: {pd.Timestamp.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n")
|
||||
|
||||
# 节点统计
|
||||
lines.append("## 节点统计\n\n")
|
||||
lines.append(f"**节点总数**: {len(nodes_df)}\n\n")
|
||||
|
||||
node_type_counts = nodes_df['type'].value_counts().sort_index()
|
||||
lines.append("| 节点类型 | 数量 | 占比 |\n")
|
||||
lines.append("|---------|------|------|\n")
|
||||
for nt, count in node_type_counts.items():
|
||||
lines.append(f"| {nt} | {count} | {count/len(nodes_df)*100:.1f}% |\n")
|
||||
|
||||
# 关系统计
|
||||
lines.append("\n## 关系统计\n\n")
|
||||
lines.append(f"**关系总数**: {len(rels_df)}\n\n")
|
||||
|
||||
rel_type_counts = rels_df['type'].value_counts().sort_index()
|
||||
lines.append("| 关系类型 | 数量 | 占比 |\n")
|
||||
lines.append("|---------|------|------|\n")
|
||||
for rt, count in rel_type_counts.items():
|
||||
lines.append(f"| {rt} | {count} | {count/len(rels_df)*100:.1f}% |\n")
|
||||
|
||||
report_path = output_dir / 'extraction_report_structured.md'
|
||||
with open(report_path, 'w', encoding='utf-8') as f:
|
||||
f.writelines(lines)
|
||||
|
||||
print(f"\n报告已保存: {report_path}")
|
||||
|
||||
|
||||
def main():
|
||||
input_path = Path(r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output\structured_documents.json")
|
||||
output_dir = Path(r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"读取结构化文档: {input_path}")
|
||||
with open(input_path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
|
||||
documents = data['documents']
|
||||
logger.info(f"共 {len(documents)} 个文档")
|
||||
|
||||
all_nodes = []
|
||||
all_rels = []
|
||||
|
||||
for i, doc in enumerate(documents):
|
||||
nodes, rels = build_document_nodes(doc, i)
|
||||
all_nodes.extend(nodes)
|
||||
all_rels.extend(rels)
|
||||
|
||||
if (i + 1) % 50 == 0:
|
||||
logger.info(f" 已处理 {i + 1}/{len(documents)}")
|
||||
|
||||
# 去重
|
||||
all_nodes = deduplicate_nodes(all_nodes)
|
||||
|
||||
# 保存
|
||||
nodes_df = pd.DataFrame(all_nodes)
|
||||
rels_df = pd.DataFrame(all_rels)
|
||||
|
||||
nodes_path = output_dir / 'nodes_structured.csv'
|
||||
rels_path = output_dir / 'rels_structured.csv'
|
||||
|
||||
nodes_df.to_csv(nodes_path, index=False, encoding='utf-8-sig')
|
||||
rels_df.to_csv(rels_path, index=False, encoding='utf-8-sig')
|
||||
|
||||
logger.info(f"节点已保存: {nodes_path} ({len(nodes_df)} 行)")
|
||||
logger.info(f"关系已保存: {rels_path} ({len(rels_df)} 行)")
|
||||
|
||||
# 统计
|
||||
print(f"\n{'='*60}")
|
||||
print("结构化CSV提取完成")
|
||||
print(f"{'='*60}")
|
||||
print(f"节点总数: {len(nodes_df)}")
|
||||
for nt, count in nodes_df['type'].value_counts().sort_index().items():
|
||||
print(f" - {nt}: {count}")
|
||||
print(f"关系总数: {len(rels_df)}")
|
||||
for rt, count in rels_df['type'].value_counts().sort_index().items():
|
||||
print(f" - {rt}: {count}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
generate_report(nodes_df, rels_df, output_dir)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,354 @@
|
||||
"""
|
||||
法规元数据解析器 - 从文档文本中解析章节结构、发布机关、日期等
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Any, Optional
|
||||
|
||||
|
||||
class LegalMetadataExtractor:
|
||||
"""法规元数据解析器"""
|
||||
|
||||
def __init__(self):
|
||||
self.logger = self._setup_logger()
|
||||
|
||||
def _setup_logger(self):
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
return logging.getLogger(__name__)
|
||||
|
||||
def _extract_document_title(self, text: str, filename: str) -> str:
|
||||
"""提取文档标题"""
|
||||
# 尝试从书名号中提取
|
||||
match = re.search(r'《(.+?)》', text[:2000])
|
||||
if match:
|
||||
return match.group(1)
|
||||
|
||||
# 尝试从前几行中提取
|
||||
lines = text[:500].split('\n')
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
# 过滤掉编号前缀
|
||||
cleaned = re.sub(r'^\d+[-—]\s*', '', line)
|
||||
if len(cleaned) > 4 and len(cleaned) < 100:
|
||||
return cleaned
|
||||
|
||||
# 从文件名提取
|
||||
name = Path(filename).stem
|
||||
name = re.sub(r'^\d+[-—]\s*', '', name)
|
||||
return name
|
||||
|
||||
def _extract_issuing_body(self, text: str) -> Optional[str]:
|
||||
"""提取发布机关"""
|
||||
# 常见模式：在文末签名块
|
||||
patterns = [
|
||||
r'(?:发布|公布|印发)[^\n]*?[机关部院会委局厅处]\s*[：:]\s*(.+?)(?:\n|$)',
|
||||
r'^[\s]*(.{4,20}(?:部|委员会|院|局|厅|处|办公室|小组))\s*$',
|
||||
]
|
||||
|
||||
# 从后往前搜索（发布机关通常在文末）
|
||||
text_end = text[-3000:] if len(text) > 3000 else text
|
||||
lines = text_end.split('\n')
|
||||
|
||||
for line in reversed(lines):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
# 匹配机关名称
|
||||
if re.match(r'^.{2,15}(?:部|委员会|院|局|厅|处|办公室|小组|政府|大会|人大常委会)$', line):
|
||||
return line
|
||||
|
||||
# 从标题区域搜索
|
||||
text_start = text[:1000]
|
||||
body_patterns = [
|
||||
r'((?:全国人民代表大会(?:常务委员会)?|国务院|.{2,10}部|.{2,10}委员会|.{2,10}局|.{2,10}厅))\s*(?:令|公告|通知|制定)',
|
||||
r'(.{2,10}(?:部|委员会|局|厅))\s*(?:令|公告|通知|印发)',
|
||||
]
|
||||
for pattern in body_patterns:
|
||||
match = re.search(pattern, text_start)
|
||||
if match:
|
||||
return match.group(1)
|
||||
|
||||
return None
|
||||
|
||||
def _extract_dates(self, text: str) -> Dict[str, Optional[str]]:
|
||||
"""提取日期信息"""
|
||||
dates = {'promulgation_date': None, 'effective_date': None}
|
||||
|
||||
# 搜索发布日期
|
||||
date_patterns = [
|
||||
r'(\d{4})\s*年\s*(\d{1,2})\s*月\s*(\d{1,2})\s*日\s*(?:起?\s*)?(?:施行|实施|生效|执行)',
|
||||
r'自\s*(\d{4})\s*年\s*(\d{1,2})\s*月\s*(\d{1,2})\s*日\s*起?\s*(?:施行|实施|生效|执行)',
|
||||
]
|
||||
for pattern in date_patterns:
|
||||
match = re.search(pattern, text)
|
||||
if match:
|
||||
dates['effective_date'] = f"{match.group(1)}-{match.group(2).zfill(2)}-{match.group(3).zfill(2)}"
|
||||
break
|
||||
|
||||
# 搜索公布日期（通常在文末）
|
||||
pub_patterns = [
|
||||
r'(\d{4})\s*年\s*(\d{1,2})\s*月\s*(\d{1,2})\s*日\s*(?:公布|发布|印发)',
|
||||
r'(?:公布|发布|印发)\s*[：:]*\s*(\d{4})\s*年\s*(\d{1,2})\s*月\s*(\d{1,2})\s*日',
|
||||
]
|
||||
text_end = text[-2000:] if len(text) > 2000 else text
|
||||
for pattern in pub_patterns:
|
||||
match = re.search(pattern, text_end)
|
||||
if match:
|
||||
dates['promulgation_date'] = f"{match.group(1)}-{match.group(2).zfill(2)}-{match.group(3).zfill(2)}"
|
||||
break
|
||||
|
||||
# 如果没有明确日期，尝试从文末找任意日期
|
||||
if not dates['promulgation_date']:
|
||||
match = re.findall(r'(\d{4})\s*年\s*(\d{1,2})\s*月\s*(\d{1,2})\s*日', text_end)
|
||||
if match:
|
||||
last_date = match[-1]
|
||||
dates['promulgation_date'] = f"{last_date[0]}-{last_date[1].zfill(2)}-{last_date[2].zfill(2)}"
|
||||
|
||||
return dates
|
||||
|
||||
def _extract_document_number(self, text: str) -> Optional[str]:
|
||||
"""提取文号"""
|
||||
patterns = [
|
||||
r'[（(]\s*(\d{4})\s*[）)]\s*[^\s]*?\s*第?\s*(\d+)\s*号',
|
||||
r'第?\s*(\d+)\s*号\s*[令公告通知]',
|
||||
r'([A-Za-z一-鿿]+[〔(]\d{4}[）)][^\s]*?号)',
|
||||
r'([国发|国办发|建发|自然资发|国土资发|建城|建规|住建|建住房|建村|办发|发改委][〔(]\d{4}[）)][^\s]*?号)',
|
||||
]
|
||||
for pattern in patterns:
|
||||
match = re.search(pattern, text[:2000])
|
||||
if match:
|
||||
return match.group(0).strip()
|
||||
return None
|
||||
|
||||
def _parse_chapter_structure(self, text: str) -> List[Dict[str, Any]]:
|
||||
"""解析章节结构"""
|
||||
chapters = []
|
||||
current_chapter = None
|
||||
current_section = None
|
||||
current_articles = []
|
||||
|
||||
# 按行处理
|
||||
lines = text.split('\n')
|
||||
|
||||
def flush_articles():
|
||||
nonlocal current_articles
|
||||
result = current_articles
|
||||
current_articles = []
|
||||
return result
|
||||
|
||||
for line in lines:
|
||||
line_stripped = line.strip()
|
||||
if not line_stripped:
|
||||
continue
|
||||
|
||||
# 检测章标题
|
||||
ch_match = re.match(r'^第[一二三四五六七八九十百]+[章节部分]\s*(.*)$', line_stripped)
|
||||
if ch_match:
|
||||
# 先保存之前的章/节的文章
|
||||
if current_section:
|
||||
current_section['articles'] = flush_articles()
|
||||
elif current_chapter:
|
||||
current_chapter['articles'].extend(flush_articles())
|
||||
|
||||
current_chapter = {
|
||||
'title': line_stripped,
|
||||
'name': ch_match.group(1).strip() if ch_match.group(1) else line_stripped,
|
||||
'sections': [],
|
||||
'articles': [],
|
||||
}
|
||||
chapters.append(current_chapter)
|
||||
current_section = None
|
||||
continue
|
||||
|
||||
# 检测节标题
|
||||
sec_match = re.match(r'^第[一二三四五六七八九十百]+节\s*(.*)$', line_stripped)
|
||||
if sec_match and current_chapter:
|
||||
if current_section:
|
||||
current_section['articles'] = flush_articles()
|
||||
|
||||
current_section = {
|
||||
'title': line_stripped,
|
||||
'name': sec_match.group(1).strip() if sec_match.group(1) else line_stripped,
|
||||
'articles': [],
|
||||
}
|
||||
current_chapter['sections'].append(current_section)
|
||||
continue
|
||||
|
||||
# 检测条文
|
||||
art_match = re.match(r'^第[一二三四五六七八九十百零千]+条\s*(.*)$', line_stripped)
|
||||
if art_match:
|
||||
article = {
|
||||
'number': line_stripped.split('条')[0].replace('第', ''),
|
||||
'text': art_match.group(1).strip(),
|
||||
'full_text': line_stripped,
|
||||
}
|
||||
current_articles.append(article)
|
||||
continue
|
||||
|
||||
# 续接上一条
|
||||
if current_articles and line_stripped:
|
||||
current_articles[-1]['text'] += ' ' + line_stripped
|
||||
current_articles[-1]['full_text'] += ' ' + line_stripped
|
||||
|
||||
# 刷新最后的文章
|
||||
if current_section:
|
||||
current_section['articles'].extend(flush_articles())
|
||||
elif current_chapter:
|
||||
current_chapter['articles'].extend(flush_articles())
|
||||
|
||||
return chapters
|
||||
|
||||
def _classify_article_type(self, text: str) -> str:
|
||||
"""分类条文类型"""
|
||||
if re.search(r'不得|禁止|严禁', text):
|
||||
return '禁止性'
|
||||
elif re.search(r'可以|有权|依法享有', text):
|
||||
return '授权性'
|
||||
elif re.search(r'应当|必须|须|应当依法', text):
|
||||
return '管理性'
|
||||
elif re.search(r'申请|审批|备案|登记|许可|核准', text):
|
||||
return '程序性'
|
||||
elif re.search(r'罚款|责令|没收|吊销|刑事|处分', text):
|
||||
return '处罚性'
|
||||
else:
|
||||
return '定义性'
|
||||
|
||||
def process_document(self, doc: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""处理单个文档"""
|
||||
text = doc.get('raw_text', '')
|
||||
filename = doc.get('filename', '')
|
||||
|
||||
# 提取元数据
|
||||
title = self._extract_document_title(text, filename)
|
||||
issuing_body = self._extract_issuing_body(text)
|
||||
dates = self._extract_dates(text)
|
||||
doc_number = self._extract_document_number(text)
|
||||
|
||||
# 解析章节结构
|
||||
chapters = self._parse_chapter_structure(text)
|
||||
|
||||
# 统计条文数
|
||||
total_articles = 0
|
||||
for ch in chapters:
|
||||
total_articles += len(ch.get('articles', []))
|
||||
for sec in ch.get('sections', []):
|
||||
total_articles += len(sec.get('articles', []))
|
||||
|
||||
# 分类条文
|
||||
for ch in chapters:
|
||||
for art in ch.get('articles', []):
|
||||
art['article_type'] = self._classify_article_type(art['text'])
|
||||
for sec in ch.get('sections', []):
|
||||
for art in sec.get('articles', []):
|
||||
art['article_type'] = self._classify_article_type(art['text'])
|
||||
|
||||
result = {
|
||||
'doc_id': doc['doc_id'],
|
||||
'document_type': doc['document_type'],
|
||||
'category_name': doc['category_name'],
|
||||
'title': title,
|
||||
'document_name': doc.get('document_name', title),
|
||||
'issuing_body': issuing_body,
|
||||
'promulgation_date': dates['promulgation_date'],
|
||||
'effective_date': dates['effective_date'],
|
||||
'document_number': doc_number,
|
||||
'is_draft': doc.get('is_draft', False),
|
||||
'text_length': doc['text_length'],
|
||||
'filename': filename,
|
||||
'chapter_count': len(chapters),
|
||||
'article_count': total_articles,
|
||||
'chapters': chapters,
|
||||
}
|
||||
|
||||
return result
|
||||
|
||||
def process_all(self, input_path: str, output_path: str) -> List[Dict[str, Any]]:
|
||||
"""处理所有文档"""
|
||||
self.logger.info(f"读取文档数据: {input_path}")
|
||||
|
||||
with open(input_path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
|
||||
documents = data['documents']
|
||||
self.logger.info(f"共 {len(documents)} 个文档待处理")
|
||||
|
||||
results = []
|
||||
for i, doc in enumerate(documents):
|
||||
if doc['text_length'] == 0:
|
||||
self.logger.warning(f"跳过空文档: {doc['filename']}")
|
||||
continue
|
||||
|
||||
try:
|
||||
result = self.process_document(doc)
|
||||
results.append(result)
|
||||
|
||||
if (i + 1) % 50 == 0:
|
||||
self.logger.info(f" 已处理 {i + 1}/{len(documents)}")
|
||||
except Exception as e:
|
||||
self.logger.error(f"处理失败: {doc['filename']} - {str(e)}")
|
||||
|
||||
# 保存
|
||||
output = Path(output_path)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
save_data = {
|
||||
'metadata': {
|
||||
'total_documents': len(results),
|
||||
'processed_at': datetime.now().isoformat(),
|
||||
},
|
||||
'documents': results,
|
||||
}
|
||||
|
||||
with open(output, 'w', encoding='utf-8') as f:
|
||||
json.dump(save_data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
self.logger.info(f"已保存到: {output}")
|
||||
self._print_summary(results)
|
||||
|
||||
return results
|
||||
|
||||
def _print_summary(self, results: List[Dict]):
|
||||
"""打印摘要"""
|
||||
total_articles = sum(r['article_count'] for r in results)
|
||||
total_chapters = sum(r['chapter_count'] for r in results)
|
||||
docs_with_body = sum(1 for r in results if r['issuing_body'])
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print("法规元数据解析摘要")
|
||||
print(f"{'='*60}")
|
||||
print(f"处理文档数: {len(results)}")
|
||||
print(f"解析出章数: {total_chapters}")
|
||||
print(f"解析出条文数: {total_articles}")
|
||||
print(f"提取发布机关: {docs_with_body}/{len(results)} ({docs_with_body/max(len(results),1)*100:.1f}%)")
|
||||
|
||||
# 按类型统计
|
||||
type_counts = {}
|
||||
for r in results:
|
||||
t = r['document_type']
|
||||
type_counts[t] = type_counts.get(t, 0) + 1
|
||||
print("\n按类型统计:")
|
||||
for t, c in sorted(type_counts.items()):
|
||||
print(f" {t}: {c} 个")
|
||||
|
||||
print(f"{'='*60}")
|
||||
|
||||
|
||||
def main():
|
||||
input_path = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output\all_documents.json"
|
||||
output_path = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output\structured_documents.json"
|
||||
|
||||
extractor = LegalMetadataExtractor()
|
||||
extractor.process_all(input_path, output_path)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,239 @@
|
||||
"""
|
||||
法规知识图谱可视化工具
|
||||
使用networkx和matplotlib绘制知识图谱
|
||||
"""
|
||||
|
||||
import pandas as pd
|
||||
import networkx as nx
|
||||
import matplotlib.pyplot as plt
|
||||
import matplotlib.patches as mpatches
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import json
|
||||
|
||||
|
||||
def setup_chinese_font():
|
||||
"""设置中文字体"""
|
||||
for font in ['Microsoft YaHei', 'SimHei', 'SimSun', 'KaiTi']:
|
||||
try:
|
||||
plt.rcParams['font.sans-serif'] = [font]
|
||||
plt.rcParams['axes.unicode_minus'] = False
|
||||
return
|
||||
except:
|
||||
continue
|
||||
|
||||
|
||||
# 节点类型颜色
|
||||
NODE_COLORS = {
|
||||
'Law': '#E74C3C',
|
||||
'AdministrativeRegulation': '#E67E22',
|
||||
'DepartmentalRule': '#F1C40F',
|
||||
'PolicyDocument': '#2ECC71',
|
||||
'Chapter': '#3498DB',
|
||||
'Section': '#9B59B6',
|
||||
'Article': '#1ABC9C',
|
||||
'GovernmentBody': '#8E44AD',
|
||||
'LegalSubject': '#E91E63',
|
||||
'SpatialConcept': '#00BCD4',
|
||||
'AdministrativeProcedure': '#FF9800',
|
||||
'Obligation': '#F44336',
|
||||
'Penalty': '#795548',
|
||||
'TimePoint': '#607D8B',
|
||||
'Region': '#4CAF50',
|
||||
}
|
||||
|
||||
# 节点类型大小
|
||||
NODE_SIZES = {
|
||||
'Law': 600,
|
||||
'AdministrativeRegulation': 500,
|
||||
'DepartmentalRule': 400,
|
||||
'PolicyDocument': 350,
|
||||
'Chapter': 250,
|
||||
'Section': 200,
|
||||
'Article': 100,
|
||||
'GovernmentBody': 400,
|
||||
'LegalSubject': 300,
|
||||
'SpatialConcept': 250,
|
||||
'AdministrativeProcedure': 200,
|
||||
'Obligation': 150,
|
||||
'Penalty': 150,
|
||||
'TimePoint': 100,
|
||||
'Region': 150,
|
||||
}
|
||||
|
||||
|
||||
def load_data(nodes_csv, rels_csv):
|
||||
"""加载数据"""
|
||||
nodes_df = pd.read_csv(nodes_csv, encoding='utf-8-sig')
|
||||
rels_df = pd.read_csv(rels_csv, encoding='utf-8-sig')
|
||||
print(f"加载节点: {len(nodes_df)}, 关系: {len(rels_df)}")
|
||||
return nodes_df, rels_df
|
||||
|
||||
|
||||
def build_graph(nodes_df, rels_df):
|
||||
"""构建NetworkX图"""
|
||||
G = nx.DiGraph()
|
||||
|
||||
for _, row in nodes_df.iterrows():
|
||||
node_id = row['id']
|
||||
label = str(row['label'])
|
||||
node_type = row['type']
|
||||
|
||||
display_label = label[:12] + '...' if len(label) > 12 else label
|
||||
|
||||
G.add_node(node_id,
|
||||
label=display_label,
|
||||
full_label=label,
|
||||
node_type=node_type,
|
||||
color=NODE_COLORS.get(node_type, '#CCCCCC'),
|
||||
size=NODE_SIZES.get(node_type, 150))
|
||||
|
||||
for _, row in rels_df.iterrows():
|
||||
source = row['source']
|
||||
target = row['target']
|
||||
rel_type = row['type']
|
||||
if source in G.nodes() and target in G.nodes():
|
||||
G.add_edge(source, target, rel_type=rel_type)
|
||||
|
||||
print(f"图: {G.number_of_nodes()} 节点, {G.number_of_edges()} 边")
|
||||
return G
|
||||
|
||||
|
||||
def draw_graph(G, output_path, title="法规知识图谱", max_nodes=300):
|
||||
"""绘制图谱"""
|
||||
if G.number_of_nodes() > max_nodes:
|
||||
degrees = dict(G.degree())
|
||||
top_nodes = sorted(degrees, key=degrees.get, reverse=True)[:max_nodes]
|
||||
G = G.subgraph(top_nodes).copy()
|
||||
|
||||
plt.figure(figsize=(24, 18))
|
||||
pos = nx.spring_layout(G, k=2.5, iterations=50, seed=42)
|
||||
|
||||
# 边
|
||||
nx.draw_networkx_edges(G, pos, alpha=0.2, width=0.5, edge_color='gray',
|
||||
arrows=True, arrowsize=8)
|
||||
|
||||
# 按类型绘制节点
|
||||
node_types = {}
|
||||
for node, data in G.nodes(data=True):
|
||||
nt = data.get('node_type', 'Unknown')
|
||||
node_types.setdefault(nt, []).append(node)
|
||||
|
||||
for nt, nodes in node_types.items():
|
||||
color = NODE_COLORS.get(nt, '#CCCCCC')
|
||||
size = NODE_SIZES.get(nt, 150)
|
||||
nx.draw_networkx_nodes(G, pos, nodelist=nodes, node_color=color,
|
||||
node_size=size, alpha=0.8, edgecolors='white', linewidths=1)
|
||||
|
||||
# 标签（只显示非Article类型）
|
||||
labels = {n: d['label'] for n, d in G.nodes(data=True)
|
||||
if d.get('node_type') != 'Article'}
|
||||
if len(labels) <= 150:
|
||||
nx.draw_networkx_labels(G, pos, labels=labels, font_size=7, font_weight='bold')
|
||||
|
||||
# 图例
|
||||
patches = [mpatches.Patch(color=NODE_COLORS[nt], label=nt)
|
||||
for nt in node_types if nt in NODE_COLORS]
|
||||
plt.legend(handles=patches, loc='upper right', fontsize=10, framealpha=0.9)
|
||||
|
||||
plt.title(title, fontsize=16, fontweight='bold', pad=20)
|
||||
plt.axis('off')
|
||||
plt.tight_layout()
|
||||
plt.savefig(output_path, dpi=150, bbox_inches='tight')
|
||||
plt.close()
|
||||
print(f"保存: {output_path}")
|
||||
|
||||
|
||||
def draw_subgraphs(G, output_dir):
|
||||
"""绘制子图"""
|
||||
output_dir = Path(output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 1. 文档层次图
|
||||
doc_types = ['Law', 'AdministrativeRegulation', 'DepartmentalRule', 'PolicyDocument', 'Chapter']
|
||||
G1_nodes = [n for n, d in G.nodes(data=True) if d.get('node_type') in doc_types]
|
||||
if G1_nodes:
|
||||
G1 = G.subgraph(G1_nodes).copy()
|
||||
draw_graph(G1, output_dir / 'kg_document_hierarchy.png', '法规文档层次结构')
|
||||
|
||||
# 2. 引用网络
|
||||
ref_rels = ['CITES', 'IMPLEMENTS', 'AMENDS', 'REPLACES', 'SUPPLEMENTS']
|
||||
ref_edges = [(u, v) for u, v, d in G.edges(data=True) if d.get('rel_type') in ref_rels]
|
||||
if ref_edges:
|
||||
G2 = G.edge_subgraph(ref_edges).copy()
|
||||
draw_graph(G2, output_dir / 'kg_citation_network.png', '法规引用网络')
|
||||
|
||||
# 3. 主题规制图
|
||||
subject_types = ['LegalSubject', 'SpatialConcept']
|
||||
subject_nodes = [n for n, d in G.nodes(data=True) if d.get('node_type') in subject_types]
|
||||
if subject_nodes:
|
||||
neighbors = set(subject_nodes)
|
||||
for node in subject_nodes:
|
||||
neighbors.update(G.predecessors(node))
|
||||
neighbors.update(G.successors(node))
|
||||
G3 = G.subgraph(neighbors).copy()
|
||||
draw_graph(G3, output_dir / 'kg_subject_regulation.png', '法规主题规制关系')
|
||||
|
||||
# 4. 机构关系图
|
||||
gov_nodes = [n for n, d in G.nodes(data=True) if d.get('node_type') == 'GovernmentBody']
|
||||
if gov_nodes:
|
||||
neighbors = set(gov_nodes)
|
||||
for node in gov_nodes:
|
||||
neighbors.update(G.predecessors(node))
|
||||
neighbors.update(G.successors(node))
|
||||
G4 = G.subgraph(neighbors).copy()
|
||||
draw_graph(G4, output_dir / 'kg_government_body.png', '政府机构关系')
|
||||
|
||||
|
||||
def print_statistics(G):
|
||||
"""打印统计"""
|
||||
print(f"\n{'='*60}")
|
||||
print("图谱统计")
|
||||
print(f"{'='*60}")
|
||||
print(f"节点: {G.number_of_nodes()}, 边: {G.number_of_edges()}")
|
||||
|
||||
type_counts = {}
|
||||
for _, d in G.nodes(data=True):
|
||||
nt = d.get('node_type', 'Unknown')
|
||||
type_counts[nt] = type_counts.get(nt, 0) + 1
|
||||
|
||||
print("\n节点类型:")
|
||||
for nt, c in sorted(type_counts.items()):
|
||||
print(f" {nt}: {c} ({c/G.number_of_nodes()*100:.1f}%)")
|
||||
|
||||
rel_counts = {}
|
||||
for _, _, d in G.edges(data=True):
|
||||
rt = d.get('rel_type', 'Unknown')
|
||||
rel_counts[rt] = rel_counts.get(rt, 0) + 1
|
||||
|
||||
print("\n关系类型:")
|
||||
for rt, c in sorted(rel_counts.items()):
|
||||
print(f" {rt}: {c}")
|
||||
|
||||
degrees = [d for _, d in G.degree()]
|
||||
print(f"\n平均连接度: {np.mean(degrees):.2f}")
|
||||
top = sorted(G.degree(), key=lambda x: x[1], reverse=True)[:10]
|
||||
print("连接度最高:")
|
||||
for node, deg in top:
|
||||
data = G.nodes[node]
|
||||
print(f" [{data.get('node_type')}] {data.get('full_label', node)}: {deg}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
|
||||
def main():
|
||||
setup_chinese_font()
|
||||
|
||||
base = Path(r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output")
|
||||
|
||||
# 优先使用合并后的文件
|
||||
nodes_csv = base / 'nodes_merged.csv' if (base / 'nodes_merged.csv').exists() else base / 'nodes_structured.csv'
|
||||
rels_csv = base / 'rels_merged.csv' if (base / 'rels_merged.csv').exists() else base / 'rels_structured.csv'
|
||||
|
||||
nodes_df, rels_df = load_data(nodes_csv, rels_csv)
|
||||
G = build_graph(nodes_df, rels_df)
|
||||
print_statistics(G)
|
||||
draw_subgraphs(G, base / 'visualizations')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,176 @@
|
||||
"""
|
||||
实体规范化器 - 去重、ID生成和规范化
|
||||
"""
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
import logging
|
||||
from typing import Dict, List, Any, Optional
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
yaml = None
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LegalEntityNormalizer:
|
||||
"""法规实体规范化器"""
|
||||
|
||||
# 法律领域常见同义词映射
|
||||
SYNONYM_MAP = {
|
||||
'城乡规划': '城市规划',
|
||||
'住建部': '住房和城乡建设部',
|
||||
'建设部': '住房和城乡建设部',
|
||||
'自然资源部': '自然资源部',
|
||||
'国土部': '自然资源部',
|
||||
'国土资源部': '自然资源部',
|
||||
'环保部': '生态环境部',
|
||||
'环境保护部': '生态环境部',
|
||||
'国家发改委': '国家发展和改革委员会',
|
||||
'发改委': '国家发展和改革委员会',
|
||||
'国务院': '国务院',
|
||||
# 第一轮旧名称 → 第二轮新名称
|
||||
'GovernmentBody': 'Agency',
|
||||
'LegalSubject': 'LegalObject',
|
||||
'SpatialConcept': 'SpatialObject',
|
||||
'AdministrativeProcedure': 'Procedure',
|
||||
# 城市更新领域同义词
|
||||
'老旧小区': '老旧小区',
|
||||
'城中村': '城中村',
|
||||
'棚户区': '棚户区',
|
||||
'旧城改造': '城市更新',
|
||||
'旧区改造': '城市更新',
|
||||
'三旧改造': '城市更新',
|
||||
}
|
||||
|
||||
def __init__(self, ontology_file: str = None):
|
||||
self.entity_registry = {}
|
||||
self.text_to_id_map = {}
|
||||
self.type_counters = {}
|
||||
|
||||
if ontology_file:
|
||||
self._load_ontology(ontology_file)
|
||||
|
||||
logger.info("LegalEntityNormalizer 初始化完成")
|
||||
|
||||
def _load_ontology(self, ontology_file: str):
|
||||
if yaml is None:
|
||||
return
|
||||
path = Path(ontology_file)
|
||||
if path.exists():
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
self.ontology = yaml.safe_load(f)
|
||||
|
||||
def normalize_text(self, text: str) -> str:
|
||||
"""标准化文本"""
|
||||
if not text:
|
||||
return ""
|
||||
text = text.strip()
|
||||
text = text.replace(' ', ' ').replace('\xa0', ' ')
|
||||
return text
|
||||
|
||||
def generate_entity_id(self, entity_type: str, entity_text: str) -> str:
|
||||
"""生成实体ID"""
|
||||
type_prefixes = {
|
||||
# 第二轮本体：城市更新法规政策工具箱
|
||||
'Agency': 'AGENCY',
|
||||
'LegalObject': 'OBJ',
|
||||
'SpatialObject': 'SPAT',
|
||||
'RenewalScene': 'SCENE',
|
||||
'PolicyTool': 'TOOL',
|
||||
'ToolCategory': 'TCAT',
|
||||
'Procedure': 'PROC',
|
||||
'Obligation': 'OBL',
|
||||
'Condition': 'COND',
|
||||
'Constraint': 'CONST',
|
||||
'Penalty': 'PEN',
|
||||
'TimePoint': 'TIME',
|
||||
'Region': 'REGION',
|
||||
# 兼容第一轮旧名称
|
||||
'GovernmentBody': 'AGENCY',
|
||||
'LegalSubject': 'OBJ',
|
||||
'SpatialConcept': 'SPAT',
|
||||
'AdministrativeProcedure': 'PROC',
|
||||
}
|
||||
prefix = type_prefixes.get(entity_type, 'ENT')
|
||||
|
||||
if entity_type not in self.type_counters:
|
||||
self.type_counters[entity_type] = 0
|
||||
self.type_counters[entity_type] += 1
|
||||
|
||||
text_hash = abs(hash(entity_text)) % 100000
|
||||
counter = self.type_counters[entity_type]
|
||||
return f"{prefix}-{text_hash:05d}-{counter}"
|
||||
|
||||
def normalize_entity(self, raw_entity: Dict[str, Any]) -> Optional[str]:
|
||||
"""规范化单个实体"""
|
||||
entity_text = raw_entity.get('text', '')
|
||||
entity_type = raw_entity.get('type', '')
|
||||
attributes = raw_entity.get('attributes', {})
|
||||
|
||||
if not entity_text or not entity_type:
|
||||
return None
|
||||
|
||||
# 标准化文本
|
||||
normalized_text = self.normalize_text(entity_text)
|
||||
|
||||
# 同义词映射
|
||||
if normalized_text in self.SYNONYM_MAP:
|
||||
normalized_text = self.SYNONYM_MAP[normalized_text]
|
||||
|
||||
# 检查是否已存在
|
||||
key = f"{entity_type}:{normalized_text}"
|
||||
if key in self.text_to_id_map:
|
||||
return self.text_to_id_map[key]
|
||||
|
||||
# 生成新ID
|
||||
entity_id = self.generate_entity_id(entity_type, normalized_text)
|
||||
|
||||
# 注册
|
||||
self.entity_registry[entity_id] = {
|
||||
'canonical_name': normalized_text,
|
||||
'display_name': entity_text,
|
||||
'type': entity_type,
|
||||
'attributes': attributes,
|
||||
}
|
||||
self.text_to_id_map[key] = entity_id
|
||||
|
||||
return entity_id
|
||||
|
||||
def normalize_batch(self, extraction_results: List[Dict]) -> Dict[str, str]:
|
||||
"""批量规范化"""
|
||||
for result in extraction_results:
|
||||
if not result or 'entities' not in result:
|
||||
continue
|
||||
for entity in result.get('entities', []):
|
||||
self.normalize_entity(entity)
|
||||
|
||||
logger.info(f"规范化完成: {len(self.entity_registry)} 个唯一实体")
|
||||
return self.text_to_id_map.copy()
|
||||
|
||||
def get_entity_nodes(self) -> List[Dict[str, Any]]:
|
||||
"""获取所有实体节点"""
|
||||
nodes = []
|
||||
for entity_id, entity_data in self.entity_registry.items():
|
||||
node = {
|
||||
'id': entity_id,
|
||||
'label': entity_data['display_name'],
|
||||
'type': entity_data['type'],
|
||||
'properties': json.dumps(entity_data['attributes'], ensure_ascii=False)
|
||||
}
|
||||
nodes.append(node)
|
||||
return nodes
|
||||
|
||||
def get_statistics(self) -> Dict[str, Any]:
|
||||
"""获取统计信息"""
|
||||
stats = {
|
||||
'total_entities': len(self.entity_registry),
|
||||
'entities_by_type': {},
|
||||
}
|
||||
for entity_data in self.entity_registry.values():
|
||||
et = entity_data['type']
|
||||
stats['entities_by_type'][et] = stats['entities_by_type'].get(et, 0) + 1
|
||||
return stats
|
||||
@@ -0,0 +1,189 @@
|
||||
"""
|
||||
关系构建器 - 从LLM抽取结果构建关系
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Dict, List, Any, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RelationshipBuilder:
|
||||
"""关系构建器"""
|
||||
|
||||
# LLM关系类型到大写关系类型的映射
|
||||
REL_TYPE_MAP = {
|
||||
# 文档层次
|
||||
'has_chapter': 'HAS_CHAPTER',
|
||||
'has_section': 'HAS_SECTION',
|
||||
'has_article': 'HAS_ARTICLE',
|
||||
'has_clause': 'HAS_CLAUSE',
|
||||
'article_in_document': 'ARTICLE_IN_DOCUMENT',
|
||||
# 法条依据
|
||||
'cites': 'CITES',
|
||||
'implements': 'IMPLEMENTS',
|
||||
'amends': 'AMENDS',
|
||||
'replaces': 'REPLACES',
|
||||
'supplements': 'SUPPLEMENTS',
|
||||
'basis_for_planning': 'BASIS_FOR_PLANNING',
|
||||
# 主体关系
|
||||
'issued_by': 'ISSUED_BY',
|
||||
'implemented_by': 'IMPLEMENTED_BY',
|
||||
'assigned_to': 'ASSIGNED_TO',
|
||||
# 语义规制
|
||||
'regulates': 'REGULATES',
|
||||
'defines_object': 'DEFINES_OBJECT',
|
||||
'applies_to': 'APPLIES_TO',
|
||||
'sets_obligation': 'SETS_OBLIGATION',
|
||||
'sets_condition': 'SETS_CONDITION',
|
||||
'sets_constraint': 'SETS_CONSTRAINT',
|
||||
'prescribes_penalty': 'PRESCRIBES_PENALTY',
|
||||
# 政策工具
|
||||
'extracts_tool': 'EXTRACTS_TOOL',
|
||||
'belongs_to_category': 'BELONGS_TO_CATEGORY',
|
||||
'applies_to_scene': 'APPLIES_TO_SCENE',
|
||||
'targets_object': 'TARGETS_OBJECT',
|
||||
'coordinates_with': 'COORDINATES_WITH',
|
||||
'supports_tool': 'SUPPORTS_TOOL',
|
||||
# 程序
|
||||
'requires_procedure': 'REQUIRES_PROCEDURE',
|
||||
'procedure_for': 'PROCEDURE_FOR',
|
||||
'precedes_procedure': 'PRECEDES_PROCEDURE',
|
||||
# 辅助
|
||||
'governs_region': 'GOVERNS_REGION',
|
||||
'effective_timeline': 'EFFECTIVE_TIMELINE',
|
||||
'standard_for': 'STANDARD_FOR',
|
||||
'article_effective_timeline': 'ARTICLE_EFFECTIVE_TIMELINE',
|
||||
# 兼容第一轮旧名称
|
||||
'defines_spatial': 'DEFINES_OBJECT',
|
||||
}
|
||||
|
||||
def __init__(self):
|
||||
self.relationships = []
|
||||
|
||||
def build_from_extraction(
|
||||
self,
|
||||
extraction_results: List[Dict],
|
||||
entity_id_map: Dict[str, str],
|
||||
source_doc_id: str = None
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
从LLM抽取结果构建关系
|
||||
|
||||
Args:
|
||||
extraction_results: LLM抽取结果列表
|
||||
entity_id_map: 文本到ID的映射
|
||||
source_doc_id: 源文档ID
|
||||
"""
|
||||
relationships = []
|
||||
rel_counter = 0
|
||||
|
||||
for result in extraction_results:
|
||||
if not result or 'relationships' not in result:
|
||||
continue
|
||||
|
||||
# 建立实体文本到ID的快速查找
|
||||
local_entity_map = {}
|
||||
if 'entities' in result:
|
||||
for entity in result['entities']:
|
||||
text = entity.get('text', '')
|
||||
etype = entity.get('type', '')
|
||||
key = f"{etype}:{text}"
|
||||
if key in entity_id_map:
|
||||
local_entity_map[text] = entity_id_map[key]
|
||||
|
||||
for rel in result.get('relationships', []):
|
||||
from_text = rel.get('from_entity', '')
|
||||
to_text = rel.get('to_entity', '')
|
||||
rel_type = rel.get('type', '')
|
||||
context = rel.get('context', '')
|
||||
confidence = rel.get('confidence', 0.8)
|
||||
|
||||
# 查找实体ID
|
||||
from_id = local_entity_map.get(from_text)
|
||||
to_id = local_entity_map.get(to_text)
|
||||
|
||||
if not from_id or not to_id:
|
||||
continue
|
||||
|
||||
rel_type_upper = self.REL_TYPE_MAP.get(rel_type, rel_type.upper())
|
||||
|
||||
rel_counter += 1
|
||||
relationships.append({
|
||||
'source': from_id,
|
||||
'target': to_id,
|
||||
'type': rel_type_upper,
|
||||
'properties': json.dumps({
|
||||
'context': context,
|
||||
'confidence': confidence,
|
||||
'source_doc': source_doc_id,
|
||||
}, ensure_ascii=False)
|
||||
})
|
||||
|
||||
self.relationships.extend(relationships)
|
||||
logger.info(f"构建了 {len(relationships)} 个关系")
|
||||
return relationships
|
||||
|
||||
def build_reference_relationships(
|
||||
self,
|
||||
resolved_refs: List[Dict],
|
||||
doc_id_map: Dict[str, str]
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
从引用解析结果构建引用关系
|
||||
|
||||
Args:
|
||||
resolved_refs: CitationResolver解析的引用列表
|
||||
doc_id_map: 文档标题到ID的映射
|
||||
"""
|
||||
relationships = []
|
||||
|
||||
for ref in resolved_refs:
|
||||
source_title = ref.get('source_title', '')
|
||||
target_id = ref.get('target_id', '')
|
||||
rel_type = ref.get('relation_type', 'CITES')
|
||||
context = ref.get('context', '')
|
||||
|
||||
source_id = doc_id_map.get(source_title)
|
||||
if not source_id:
|
||||
continue
|
||||
|
||||
relationships.append({
|
||||
'source': source_id,
|
||||
'target': target_id,
|
||||
'type': rel_type,
|
||||
'properties': json.dumps({'context': context}, ensure_ascii=False)
|
||||
})
|
||||
|
||||
logger.info(f"构建了 {len(relationships)} 个引用关系")
|
||||
return relationships
|
||||
|
||||
def validate_relationships(
|
||||
self,
|
||||
relationships: List[Dict],
|
||||
valid_node_ids: set
|
||||
) -> Dict[str, Any]:
|
||||
"""验证关系数据"""
|
||||
valid = 0
|
||||
invalid = 0
|
||||
broken_links = []
|
||||
|
||||
for rel in relationships:
|
||||
source = rel.get('source', '')
|
||||
target = rel.get('target', '')
|
||||
|
||||
if source in valid_node_ids and target in valid_node_ids:
|
||||
valid += 1
|
||||
else:
|
||||
invalid += 1
|
||||
if source not in valid_node_ids:
|
||||
broken_links.append(f"source not found: {source}")
|
||||
if target not in valid_node_ids:
|
||||
broken_links.append(f"target not found: {target}")
|
||||
|
||||
return {
|
||||
'valid_relationships': valid,
|
||||
'invalid_relationships': invalid,
|
||||
'broken_links': broken_links[:50],
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
"""
|
||||
深度法规实体抽取管道
|
||||
协调LLM抽取、实体规范化、关系构建
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import pandas as pd
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Any
|
||||
|
||||
import sys
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
from knowledge_extraction.llm_legal_extractor import LegalLLMExtractor
|
||||
from knowledge_extraction.citation_resolver import CitationResolver
|
||||
from data_processing.entity_normalizer import LegalEntityNormalizer
|
||||
from data_processing.relationship_builder import RelationshipBuilder
|
||||
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DeepExtractionPipeline:
|
||||
"""深度法规实体抽取管道"""
|
||||
|
||||
def __init__(self, config_file: str):
|
||||
self.config_file = config_file
|
||||
self.extractor = LegalLLMExtractor(config_file)
|
||||
self.config = self.extractor.config
|
||||
self.normalizer = LegalEntityNormalizer()
|
||||
self.builder = RelationshipBuilder()
|
||||
self.project_root = Path(__file__).parent.parent
|
||||
|
||||
def load_structured_documents(self, input_file: str) -> List[Dict]:
|
||||
"""加载结构化文档数据"""
|
||||
with open(input_file, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
return data['documents']
|
||||
|
||||
def _prepare_sections(self, documents: List[Dict]) -> List[Dict]:
|
||||
"""准备章节级别的抽取单元"""
|
||||
sections = []
|
||||
|
||||
for doc in documents:
|
||||
doc_title = doc.get('title', '')
|
||||
chapters = doc.get('chapters', [])
|
||||
|
||||
for chapter in chapters:
|
||||
# 将章节下所有条文合并为一个抽取单元
|
||||
texts = []
|
||||
for art in chapter.get('articles', []):
|
||||
art_text = art.get('full_text', art.get('text', ''))
|
||||
if art_text:
|
||||
texts.append(art_text)
|
||||
|
||||
for section in chapter.get('sections', []):
|
||||
for art in section.get('articles', []):
|
||||
art_text = art.get('full_text', art.get('text', ''))
|
||||
if art_text:
|
||||
texts.append(art_text)
|
||||
|
||||
combined_text = '\n'.join(texts)
|
||||
|
||||
if len(combined_text.strip()) > 20:
|
||||
sections.append({
|
||||
'text': combined_text,
|
||||
'document_title': doc_title,
|
||||
'section_title': chapter.get('title', ''),
|
||||
'doc_id': doc.get('doc_id', ''),
|
||||
})
|
||||
|
||||
logger.info(f"准备了 {len(sections)} 个章节抽取单元")
|
||||
return sections
|
||||
|
||||
def _prepare_documents_for_references(self, documents: List[Dict]) -> List[Dict]:
|
||||
"""准备文档级引用抽取"""
|
||||
doc_list = []
|
||||
for doc in documents:
|
||||
doc_list.append({
|
||||
'title': doc.get('title', ''),
|
||||
'text': doc.get('text_preview', ''), # 需要原始文本
|
||||
'doc_id': doc.get('doc_id', ''),
|
||||
})
|
||||
return doc_list
|
||||
|
||||
async def run(
|
||||
self,
|
||||
structured_input: str,
|
||||
raw_input: str,
|
||||
output_dir: str,
|
||||
max_sections: int = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
执行完整抽取流程
|
||||
|
||||
Args:
|
||||
structured_input: 结构化文档JSON路径
|
||||
raw_input: 原始文档JSON路径（用于引用抽取）
|
||||
output_dir: 输出目录
|
||||
max_sections: 最大处理章节数（测试用）
|
||||
"""
|
||||
start_time = datetime.now()
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 1. 加载数据
|
||||
logger.info("加载结构化文档...")
|
||||
documents = self.load_structured_documents(structured_input)
|
||||
logger.info(f"加载 {len(documents)} 个文档")
|
||||
|
||||
# 2. 准备章节抽取单元
|
||||
sections = self._prepare_sections(documents)
|
||||
if max_sections:
|
||||
sections = sections[:max_sections]
|
||||
logger.info(f"限制处理: {max_sections} 个章节")
|
||||
|
||||
# 3. 加载已有的结构化节点（用于引用解析）
|
||||
nodes_csv = output_path / 'nodes_structured.csv'
|
||||
known_docs = []
|
||||
if nodes_csv.exists():
|
||||
nodes_df = pd.read_csv(nodes_csv, encoding='utf-8-sig')
|
||||
known_docs = [
|
||||
{'id': row['id'], 'label': row['label'], 'type': row['type']}
|
||||
for _, row in nodes_df.iterrows()
|
||||
if row['type'] in ['Law', 'AdministrativeRegulation', 'DepartmentalRule', 'PolicyDocument']
|
||||
]
|
||||
logger.info(f"已知文档节点: {len(known_docs)}")
|
||||
|
||||
# 4. LLM章节级抽取
|
||||
logger.info("开始LLM章节级抽取...")
|
||||
extraction_results = await self.extractor.batch_extract_sections(
|
||||
sections,
|
||||
concurrent=self.extractor.config.get('batch_processing', {}).get('concurrent_requests', 5)
|
||||
)
|
||||
|
||||
# 5. 实体规范化
|
||||
logger.info("规范化实体...")
|
||||
entity_id_map = self.normalizer.normalize_batch(extraction_results)
|
||||
entity_nodes = self.normalizer.get_entity_nodes()
|
||||
stats = self.normalizer.get_statistics()
|
||||
logger.info(f"规范化后实体: {stats['total_entities']}")
|
||||
|
||||
# 6. 关系构建
|
||||
logger.info("构建关系...")
|
||||
relationships = self.builder.build_from_extraction(
|
||||
extraction_results, entity_id_map
|
||||
)
|
||||
|
||||
# 7. 文档级引用抽取
|
||||
logger.info("开始文档级引用抽取...")
|
||||
# 加载原始文本用于引用抽取
|
||||
with open(raw_input, 'r', encoding='utf-8') as f:
|
||||
raw_data = json.load(f)
|
||||
|
||||
doc_id_map = {d['label']: d['id'] for d in known_docs}
|
||||
# 补充从文档数据中获取原始文本
|
||||
docs_for_ref = []
|
||||
raw_docs_map = {d['filename']: d for d in raw_data['documents']}
|
||||
|
||||
for doc in documents:
|
||||
filename = doc.get('filename', '')
|
||||
raw_doc = raw_docs_map.get(filename, {})
|
||||
raw_text = raw_doc.get('raw_text', '')
|
||||
if raw_text and len(raw_text) > 50:
|
||||
docs_for_ref.append({
|
||||
'title': doc['title'],
|
||||
'text': raw_text[:5000], # 截取前5000字符
|
||||
})
|
||||
|
||||
ref_results = await self.extractor.batch_extract_references(docs_for_ref, concurrent=3)
|
||||
|
||||
# 8. 引用解析
|
||||
logger.info("解析法规引用...")
|
||||
resolver = CitationResolver(known_docs)
|
||||
|
||||
all_references = []
|
||||
for result in ref_results:
|
||||
if not result or 'references' not in result:
|
||||
continue
|
||||
doc_title = result.get('metadata', {}).get('document_title', '')
|
||||
for ref in result['references']:
|
||||
ref['document_title'] = doc_title
|
||||
all_references.append(ref)
|
||||
|
||||
resolved_refs, unresolved_refs = resolver.resolve_batch(all_references)
|
||||
ref_relationships = self.builder.build_reference_relationships(resolved_refs, doc_id_map)
|
||||
relationships.extend(ref_relationships)
|
||||
|
||||
# 9. 保存结果
|
||||
logger.info("保存结果...")
|
||||
|
||||
# 从配置获取输出文件名
|
||||
output_cfg = self.config.get('output', {})
|
||||
llm_nodes_name = output_cfg.get('nodes_file', 'output/nodes_llm_v2.csv').split('/')[-1]
|
||||
llm_rels_name = output_cfg.get('relationships_file', 'output/rels_llm_v2.csv').split('/')[-1]
|
||||
report_name = output_cfg.get('report_file', 'output/extraction_report_v2.md').split('/')[-1]
|
||||
|
||||
# LLM节点
|
||||
llm_nodes_df = pd.DataFrame(entity_nodes)
|
||||
llm_nodes_path = output_path / llm_nodes_name
|
||||
llm_nodes_df.to_csv(llm_nodes_path, index=False, encoding='utf-8-sig')
|
||||
|
||||
# LLM关系
|
||||
llm_rels_df = pd.DataFrame(relationships)
|
||||
llm_rels_path = output_path / llm_rels_name
|
||||
llm_rels_df.to_csv(llm_rels_path, index=False, encoding='utf-8-sig')
|
||||
|
||||
# 10. 合并
|
||||
logger.info("合并结构化和LLM结果...")
|
||||
if nodes_csv.exists():
|
||||
structured_nodes = pd.read_csv(nodes_csv, encoding='utf-8-sig')
|
||||
merged_nodes = pd.concat([structured_nodes, llm_nodes_df], ignore_index=True)
|
||||
merged_nodes = merged_nodes.drop_duplicates(subset=['id'], keep='first')
|
||||
else:
|
||||
merged_nodes = llm_nodes_df
|
||||
|
||||
rels_csv = output_path / 'rels_structured.csv'
|
||||
if rels_csv.exists():
|
||||
structured_rels = pd.read_csv(rels_csv, encoding='utf-8-sig')
|
||||
merged_rels = pd.concat([structured_rels, llm_rels_df], ignore_index=True)
|
||||
else:
|
||||
merged_rels = llm_rels_df
|
||||
|
||||
merged_nodes_path = output_path / 'nodes_merged_v2.csv'
|
||||
merged_rels_path = output_path / 'rels_merged_v2.csv'
|
||||
merged_nodes.to_csv(merged_nodes_path, index=False, encoding='utf-8-sig')
|
||||
merged_rels.to_csv(merged_rels_path, index=False, encoding='utf-8-sig')
|
||||
|
||||
# 验证
|
||||
valid_ids = set(merged_nodes['id'].tolist())
|
||||
validation = self.builder.validate_relationships(
|
||||
merged_rels.to_dict('records'), valid_ids
|
||||
)
|
||||
|
||||
# 生成报告
|
||||
duration = (datetime.now() - start_time).total_seconds()
|
||||
self._generate_report(
|
||||
output_path, sections, extraction_results, entity_nodes,
|
||||
relationships, resolved_refs, unresolved_refs, validation, duration,
|
||||
report_name=report_name
|
||||
)
|
||||
|
||||
result = {
|
||||
'total_sections': len(sections),
|
||||
'total_entities': len(entity_nodes),
|
||||
'total_relationships': len(relationships),
|
||||
'total_references_resolved': len(resolved_refs),
|
||||
'total_references_unresolved': len(unresolved_refs),
|
||||
'duration_seconds': duration,
|
||||
}
|
||||
|
||||
logger.info(f"\n抽取完成！耗时: {duration:.1f}s")
|
||||
logger.info(f"章节: {result['total_sections']}, 实体: {result['total_entities']}, "
|
||||
f"关系: {result['total_relationships']}, 引用: {result['total_references_resolved']}")
|
||||
|
||||
return result
|
||||
|
||||
def _generate_report(self, output_path, sections, extraction_results,
|
||||
entity_nodes, relationships, resolved_refs,
|
||||
unresolved_refs, validation, duration,
|
||||
report_name='extraction_report_v2.md'):
|
||||
"""生成抽取报告"""
|
||||
lines = ["# 第二轮深度抽取报告（城市更新政策工具箱本体）\n"]
|
||||
lines.append(f"**生成时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
|
||||
lines.append(f"**耗时**: {duration:.1f}秒\n\n")
|
||||
|
||||
lines.append("## 统计\n\n")
|
||||
lines.append(f"- 处理章节: {len(sections)}\n")
|
||||
lines.append(f"- 抽取实体: {len(entity_nodes)}\n")
|
||||
lines.append(f"- 构建关系: {len(relationships)}\n")
|
||||
lines.append(f"- 解析引用: {len(resolved_refs)} 成功, {len(unresolved_refs)} 未匹配\n")
|
||||
|
||||
# 实体类型分布
|
||||
lines.append("\n## 实体类型分布\n\n")
|
||||
type_counts = {}
|
||||
for node in entity_nodes:
|
||||
t = node['type']
|
||||
type_counts[t] = type_counts.get(t, 0) + 1
|
||||
lines.append("| 类型 | 数量 |\n|------|------|\n")
|
||||
for t, c in sorted(type_counts.items()):
|
||||
lines.append(f"| {t} | {c} |\n")
|
||||
|
||||
# 关系类型分布
|
||||
lines.append("\n## 关系类型分布\n\n")
|
||||
rel_type_counts = {}
|
||||
for rel in relationships:
|
||||
t = rel['type']
|
||||
rel_type_counts[t] = rel_type_counts.get(t, 0) + 1
|
||||
lines.append("| 类型 | 数量 |\n|------|------|\n")
|
||||
for t, c in sorted(rel_type_counts.items()):
|
||||
lines.append(f"| {t} | {c} |\n")
|
||||
|
||||
# 数据质量
|
||||
lines.append("\n## 数据质量\n\n")
|
||||
lines.append(f"- 有效关系: {validation['valid_relationships']}\n")
|
||||
lines.append(f"- 无效关系: {validation['invalid_relationships']}\n")
|
||||
|
||||
report_path = output_path / report_name
|
||||
with open(report_path, 'w', encoding='utf-8') as f:
|
||||
f.writelines(lines)
|
||||
logger.info(f"报告已保存: {report_path}")
|
||||
|
||||
|
||||
async def main():
|
||||
config_file = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\config\deep_extraction_config.yaml"
|
||||
structured_input = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output\structured_documents.json"
|
||||
raw_input = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output\all_documents.json"
|
||||
output_dir = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output"
|
||||
|
||||
pipeline = DeepExtractionPipeline(config_file)
|
||||
|
||||
# 测试模式：只处理前20个章节
|
||||
import sys
|
||||
max_sections = 20
|
||||
if '--full' in sys.argv:
|
||||
max_sections = None
|
||||
|
||||
results = await pipeline.run(
|
||||
structured_input, raw_input, output_dir,
|
||||
max_sections=max_sections
|
||||
)
|
||||
|
||||
print("\n结果:")
|
||||
print(json.dumps(results, indent=2, ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,134 @@
|
||||
"""
|
||||
法规引用解析器 - 将文本中的法规引用映射到已知文档节点
|
||||
"""
|
||||
|
||||
import re
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
from difflib import SequenceMatcher
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CitationResolver:
|
||||
"""法规引用解析器"""
|
||||
|
||||
def __init__(self, known_documents: List[Dict]):
|
||||
"""
|
||||
Args:
|
||||
known_documents: 已知文档列表，每个包含 id, label, type
|
||||
"""
|
||||
self.known_docs = {}
|
||||
self.name_variants = {}
|
||||
|
||||
for doc in known_documents:
|
||||
name = doc['label']
|
||||
doc_id = doc['id']
|
||||
self.known_docs[name] = doc_id
|
||||
|
||||
# 生成名称变体
|
||||
variants = self._generate_variants(name)
|
||||
for variant in variants:
|
||||
self.name_variants[variant] = doc_id
|
||||
|
||||
logger.info(f"CitationResolver 初始化: {len(self.known_docs)} 个已知文档")
|
||||
|
||||
def _generate_variants(self, name: str) -> List[str]:
|
||||
"""生成法规名称变体"""
|
||||
variants = [name]
|
||||
|
||||
# 去掉书名号
|
||||
cleaned = name.replace('《', '').replace('》', '')
|
||||
if cleaned != name:
|
||||
variants.append(cleaned)
|
||||
|
||||
# 去掉"中华人民共和国"前缀
|
||||
if name.startswith('中华人民共和国'):
|
||||
short = name.replace('中华人民共和国', '')
|
||||
variants.append(short)
|
||||
|
||||
# 简称
|
||||
if '实施条例' in name:
|
||||
variants.append(name.replace('实施条例', '条例'))
|
||||
if '实施办法' in name:
|
||||
variants.append(name.replace('实施办法', '办法'))
|
||||
|
||||
return list(set(variants))
|
||||
|
||||
def resolve(self, referenced_name: str, threshold: float = 0.7) -> Optional[str]:
|
||||
"""
|
||||
解析引用名称到文档ID
|
||||
|
||||
Args:
|
||||
referenced_name: 被引用的法规名称
|
||||
threshold: 相似度阈值
|
||||
|
||||
Returns:
|
||||
文档ID，未找到返回None
|
||||
"""
|
||||
if not referenced_name:
|
||||
return None
|
||||
|
||||
# 清理名称
|
||||
cleaned = referenced_name.strip()
|
||||
cleaned = re.sub(r'[《》]', '', cleaned)
|
||||
|
||||
# 精确匹配
|
||||
if cleaned in self.known_docs:
|
||||
return self.known_docs[cleaned]
|
||||
if cleaned in self.name_variants:
|
||||
return self.name_variants[cleaned]
|
||||
|
||||
# 模糊匹配
|
||||
best_match = None
|
||||
best_score = 0
|
||||
|
||||
for known_name, doc_id in self.known_docs.items():
|
||||
score = SequenceMatcher(None, cleaned, known_name).ratio()
|
||||
if score > best_score and score >= threshold:
|
||||
best_score = score
|
||||
best_match = doc_id
|
||||
|
||||
if best_match:
|
||||
logger.info(f"模糊匹配: '{referenced_name}' -> {best_match} (score={best_score:.2f})")
|
||||
|
||||
return best_match
|
||||
|
||||
def resolve_batch(self, references: List[Dict]) -> List[Tuple[str, str]]:
|
||||
"""
|
||||
批量解析引用
|
||||
|
||||
Returns:
|
||||
List of (source_doc_title, target_doc_id) pairs
|
||||
"""
|
||||
resolved = []
|
||||
unresolved = []
|
||||
|
||||
for ref in references:
|
||||
doc_title = ref.get('document_title', '')
|
||||
referenced_law = ref.get('referenced_law', '')
|
||||
relation_type = ref.get('relation_type', 'cites')
|
||||
context = ref.get('context', '')
|
||||
|
||||
target_id = self.resolve(referenced_law)
|
||||
|
||||
if target_id:
|
||||
resolved.append({
|
||||
'source_title': doc_title,
|
||||
'target_id': target_id,
|
||||
'target_name': referenced_law,
|
||||
'relation_type': relation_type.upper(),
|
||||
'context': context,
|
||||
})
|
||||
else:
|
||||
unresolved.append({
|
||||
'source_title': doc_title,
|
||||
'target_name': referenced_law,
|
||||
'relation_type': relation_type,
|
||||
})
|
||||
|
||||
logger.info(f"引用解析完成: {len(resolved)} 成功, {len(unresolved)} 未匹配")
|
||||
|
||||
return resolved, unresolved
|
||||
@@ -0,0 +1,303 @@
|
||||
"""
|
||||
基于DeepSeek的法规知识抽取器
|
||||
从法条文本中抽取法律实体和关系
|
||||
使用OpenAI SDK直接调用DeepSeek API
|
||||
"""
|
||||
|
||||
import os
|
||||
# 解决Windows上httpx代理检测导致的连接失败
|
||||
os.environ['NO_PROXY'] = '*'
|
||||
|
||||
import yaml
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
try:
|
||||
from openai import OpenAI
|
||||
except ImportError:
|
||||
raise ImportError("请安装: pip install openai")
|
||||
|
||||
|
||||
class LegalLLMExtractor:
|
||||
"""基于DeepSeek的法规知识抽取器"""
|
||||
|
||||
def __init__(self, config_file: str):
|
||||
self.config = self._load_config(config_file)
|
||||
self.api_key = self._load_api_key()
|
||||
|
||||
self.client = OpenAI(
|
||||
api_key=self.api_key,
|
||||
base_url='https://api.deepseek.com',
|
||||
)
|
||||
|
||||
self.logger = logging.getLogger(__name__)
|
||||
|
||||
def _load_config(self, config_file: str) -> Dict:
|
||||
config_path = Path(config_file)
|
||||
if not config_path.exists():
|
||||
config_path = Path(__file__).parent.parent.parent / 'config' / 'deep_extraction_config.yaml'
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
return yaml.safe_load(f)
|
||||
|
||||
def _load_api_key(self) -> str:
|
||||
api_file = Path(__file__).parent.parent.parent / 'config' / 'api_keys.yaml'
|
||||
if api_file.exists():
|
||||
with open(api_file, 'r', encoding='utf-8') as f:
|
||||
api_config = yaml.safe_load(f)
|
||||
key = api_config.get('deepseek_api_key', '')
|
||||
if key and key != 'YOUR_API_KEY_HERE':
|
||||
return key
|
||||
key = os.environ.get('DEEPSEEK_API_KEY', '')
|
||||
if key:
|
||||
return key
|
||||
raise ValueError("未找到DeepSeek API密钥，请配置 config/api_keys.yaml 或设置 DEEPSEEK_API_KEY 环境变量")
|
||||
|
||||
def _extract_json(self, response_text: str) -> str:
|
||||
"""从响应中提取JSON"""
|
||||
if '```json' in response_text:
|
||||
start = response_text.find('```json') + 7
|
||||
end = response_text.find('```', start)
|
||||
if end > start:
|
||||
return response_text[start:end].strip()
|
||||
elif '```' in response_text:
|
||||
start = response_text.find('```') + 3
|
||||
end = response_text.find('```', start)
|
||||
if end > start:
|
||||
content = response_text[start:end].strip()
|
||||
if content.startswith('json'):
|
||||
content = content[4:].strip()
|
||||
return content
|
||||
return response_text.strip()
|
||||
|
||||
def _call_api(self, prompt: str, max_retries: int = 3) -> Optional[str]:
|
||||
"""同步调用API"""
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.config['llm']['model'],
|
||||
messages=[{'role': 'user', 'content': prompt}],
|
||||
temperature=self.config['llm']['temperature'],
|
||||
max_tokens=self.config['llm']['max_tokens'],
|
||||
)
|
||||
return response.choices[0].message.content
|
||||
except Exception as e:
|
||||
self.logger.error(f"API调用失败 (attempt {attempt+1}/{max_retries}): {e}")
|
||||
if attempt == max_retries - 1:
|
||||
return None
|
||||
import time
|
||||
time.sleep(self.config['llm'].get('retry_delay', 2))
|
||||
|
||||
async def _call_api_async(self, prompt: str, max_retries: int = 3) -> Optional[str]:
|
||||
"""异步调用API"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, lambda: self._call_api(prompt, max_retries))
|
||||
|
||||
async def extract_from_section(
|
||||
self,
|
||||
section_text: str,
|
||||
document_title: str,
|
||||
section_title: str,
|
||||
max_retries: int = 3
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""从章节文本中抽取实体和关系（基于动态配置的本体）"""
|
||||
if len(section_text) > 5000:
|
||||
section_text = section_text[:5000]
|
||||
|
||||
# 从配置动态构建实体类型列表
|
||||
entity_prompts = self.config.get('entity_type_prompts', {})
|
||||
entity_types_desc = '\n'.join(
|
||||
f"- {etype}：{desc}"
|
||||
for etype, desc in entity_prompts.items()
|
||||
)
|
||||
|
||||
# 从配置动态构建关系类型列表
|
||||
rel_prompts = self.config.get('relationship_type_prompts', {})
|
||||
rel_types_desc = '\n'.join(
|
||||
f"- {rtype}：{desc}"
|
||||
for rtype, desc in rel_prompts.items()
|
||||
)
|
||||
|
||||
# 边界规则
|
||||
boundary_rules = self.config.get('extraction_tips', {}).get('entity_boundary_rules', '')
|
||||
do_not = self.config.get('extraction_tips', {}).get('do_not_extract', '')
|
||||
|
||||
prompt = f"""你是一位中国城市更新法规政策分析专家。请从以下法规章节中识别实体和关系。
|
||||
|
||||
所属法规：{document_title}
|
||||
章节名称：{section_title}
|
||||
|
||||
文本内容：
|
||||
{section_text}
|
||||
|
||||
## 需要识别的实体类型
|
||||
|
||||
{entity_types_desc}
|
||||
|
||||
## 需要识别的关系类型
|
||||
|
||||
{rel_types_desc}
|
||||
|
||||
{boundary_rules}
|
||||
|
||||
{do_not}
|
||||
|
||||
## 输出JSON格式
|
||||
|
||||
严格输出以下JSON格式，不要输出其他内容：
|
||||
{{
|
||||
"entities": [
|
||||
{{
|
||||
"text": "实体原文表述",
|
||||
"type": "实体类型（必须是上述类型之一）",
|
||||
"attributes": {{
|
||||
"description": "实体简要描述"
|
||||
}}
|
||||
}}
|
||||
],
|
||||
"relationships": [
|
||||
{{
|
||||
"from_entity": "源实体文本",
|
||||
"from_type": "源实体类型",
|
||||
"to_entity": "目标实体文本",
|
||||
"to_type": "目标实体类型",
|
||||
"type": "关系类型（必须是上述类型之一）",
|
||||
"context": "关系上下文原文",
|
||||
"confidence": 0.9
|
||||
}}
|
||||
]
|
||||
}}
|
||||
|
||||
## 注意事项
|
||||
- 只抽取文本中明确提到的实体和关系，不要推测
|
||||
- 关系的from_entity和to_entity必须是entities中出现的实体
|
||||
- 保持原文表述，不要改写实体名称
|
||||
- confidence范围0.0-1.0，表示抽取的可信度
|
||||
- 重点关注PolicyTool（政策工具）的识别，这是本次抽取的核心目标
|
||||
- 确保输出有效JSON，不要输出JSON以外的内容"""
|
||||
|
||||
response_text = await self._call_api_async(prompt, max_retries)
|
||||
if not response_text:
|
||||
return None
|
||||
|
||||
try:
|
||||
json_text = self._extract_json(response_text)
|
||||
result = json.loads(json_text)
|
||||
result['metadata'] = {
|
||||
'document_title': document_title,
|
||||
'section_title': section_title,
|
||||
'extraction_time': datetime.now().isoformat(),
|
||||
}
|
||||
return result
|
||||
except json.JSONDecodeError as e:
|
||||
self.logger.warning(f"JSON解析失败: {e}")
|
||||
return None
|
||||
|
||||
async def extract_document_references(
|
||||
self,
|
||||
document_title: str,
|
||||
full_text: str,
|
||||
max_retries: int = 3
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""从文档全文中抽取跨文档引用关系"""
|
||||
text_sample = full_text[:3000]
|
||||
if len(full_text) > 5000:
|
||||
text_sample += '\n...\n' + full_text[-2000:]
|
||||
|
||||
prompt = f"""你是一位中国法律专家。请从以下法规文本中识别引用的其他法规文件。
|
||||
|
||||
法规名称：{document_title}
|
||||
|
||||
文本片段：
|
||||
{text_sample}
|
||||
|
||||
请识别以下引用关系：
|
||||
- cites（引用）：文本中提到的其他法规
|
||||
- implements（实施）：文本声明为实施某上位法而制定
|
||||
- amends（修正）：文本对其他法规的修改
|
||||
- replaces（替代）：文本声明替代或废止其他法规
|
||||
|
||||
输出JSON格式：
|
||||
{{
|
||||
"references": [
|
||||
{{
|
||||
"referenced_law": "被引用的法规名称",
|
||||
"relation_type": "cites/implements/amends/replaces",
|
||||
"context": "引用上下文原文"
|
||||
}}
|
||||
]
|
||||
}}
|
||||
|
||||
注意：只识别文本中明确提到的法规名称（书名号《》内的名称），不要推测。"""
|
||||
|
||||
response_text = await self._call_api_async(prompt, max_retries)
|
||||
if not response_text:
|
||||
return None
|
||||
|
||||
try:
|
||||
json_text = self._extract_json(response_text)
|
||||
result = json.loads(json_text)
|
||||
result['metadata'] = {
|
||||
'document_title': document_title,
|
||||
'extraction_time': datetime.now().isoformat(),
|
||||
}
|
||||
return result
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
async def batch_extract_sections(
|
||||
self,
|
||||
sections: List[Dict[str, str]],
|
||||
concurrent: int = 5,
|
||||
progress_callback=None
|
||||
) -> List[Optional[Dict]]:
|
||||
"""批量抽取章节实体"""
|
||||
results = []
|
||||
batch_size = concurrent
|
||||
|
||||
for i in range(0, len(sections), batch_size):
|
||||
batch = sections[i:i + batch_size]
|
||||
batch_num = i // batch_size + 1
|
||||
total_batches = (len(sections) + batch_size - 1) // batch_size
|
||||
|
||||
self.logger.info(f"处理章节批次 {batch_num}/{total_batches}: {len(batch)} 个")
|
||||
|
||||
tasks = [
|
||||
self.extract_from_section(
|
||||
s['text'], s['document_title'], s['section_title']
|
||||
)
|
||||
for s in batch
|
||||
]
|
||||
batch_results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
results.extend(batch_results)
|
||||
|
||||
if progress_callback:
|
||||
await progress_callback(batch_num, total_batches, batch_results, batch)
|
||||
|
||||
return results
|
||||
|
||||
async def batch_extract_references(
|
||||
self,
|
||||
documents: List[Dict[str, str]],
|
||||
concurrent: int = 3
|
||||
) -> List[Optional[Dict]]:
|
||||
"""批量抽取文档引用"""
|
||||
results = []
|
||||
batch_size = concurrent
|
||||
|
||||
for i in range(0, len(documents), batch_size):
|
||||
batch = documents[i:i + batch_size]
|
||||
|
||||
tasks = [
|
||||
self.extract_document_references(d['title'], d['text'])
|
||||
for d in batch
|
||||
]
|
||||
batch_results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
results.extend(batch_results)
|
||||
|
||||
self.logger.info(f"引用抽取进度: {min(i+batch_size, len(documents))}/{len(documents)}")
|
||||
|
||||
return results
|
||||
@@ -0,0 +1,166 @@
|
||||
"""
|
||||
城市规划法律法规知识图谱构建 - 主入口
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
# 添加项目路径
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
|
||||
def step1_read_documents():
|
||||
"""Step1: 读取Word文档"""
|
||||
from importlib import import_module
|
||||
# 动态导入scripts目录下的模块
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts'))
|
||||
docx_reader = import_module('docx_reader')
|
||||
|
||||
source_dir = r"E:\Project\SI\2026_KG_PlanningLaw\data\城市规划法律法规\城市更新法规数据库"
|
||||
output_path = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output\all_documents.json"
|
||||
|
||||
reader = docx_reader.DocxReader(source_dir)
|
||||
reader.read_all_documents()
|
||||
reader.save_to_json(output_path)
|
||||
reader.print_summary()
|
||||
|
||||
|
||||
def step2_extract_metadata():
|
||||
"""Step2: 解析法规元数据"""
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts'))
|
||||
metadata_extractor = import_module('legal_metadata_extractor')
|
||||
|
||||
input_path = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output\all_documents.json"
|
||||
output_path = r"E:\Project\SI\2026_KG_PlanningLaw\dofile\kg_project\output\structured_documents.json"
|
||||
|
||||
extractor = metadata_extractor.LegalMetadataExtractor()
|
||||
extractor.process_all(input_path, output_path)
|
||||
|
||||
|
||||
def step3_generate_csv():
|
||||
"""Step3: 生成结构化CSV"""
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts'))
|
||||
extract_csv = import_module('extract_legal_csv')
|
||||
extract_csv.main()
|
||||
|
||||
|
||||
def step4_deep_extraction():
|
||||
"""Step4: LLM深度抽取"""
|
||||
from deep_extraction_pipeline import DeepExtractionPipeline
|
||||
|
||||
config_file = str(Path(__file__).parent.parent / 'config' / 'deep_extraction_config.yaml')
|
||||
structured_input = str(Path(__file__).parent.parent / 'output' / 'structured_documents.json')
|
||||
raw_input = str(Path(__file__).parent.parent / 'output' / 'all_documents.json')
|
||||
output_dir = str(Path(__file__).parent.parent / 'output')
|
||||
|
||||
pipeline = DeepExtractionPipeline(config_file)
|
||||
|
||||
results = asyncio.run(pipeline.run(
|
||||
structured_input, raw_input, output_dir,
|
||||
max_sections=None # 完整抽取
|
||||
))
|
||||
|
||||
print("\n结果:")
|
||||
print(json.dumps(results, indent=2, ensure_ascii=False))
|
||||
|
||||
|
||||
def step5_visualize():
|
||||
"""Step5: 可视化"""
|
||||
from importlib import import_module
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts'))
|
||||
visualize = import_module('visualize_legal_kg')
|
||||
visualize.main()
|
||||
|
||||
|
||||
def show_status():
|
||||
"""显示项目状态"""
|
||||
output_dir = Path(__file__).parent.parent / 'output'
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("城市规划法律法规知识图谱 - 项目状态")
|
||||
print("=" * 60)
|
||||
|
||||
files = {
|
||||
'all_documents.json': 'Step1: Word文档读取',
|
||||
'structured_documents.json': 'Step2: 法规元数据解析',
|
||||
'nodes_structured.csv': 'Step3: 结构化节点',
|
||||
'rels_structured.csv': 'Step3: 结构化关系',
|
||||
'nodes_llm.csv': 'Step4: LLM抽取节点',
|
||||
'rels_llm.csv': 'Step4: LLM抽取关系',
|
||||
'nodes_merged.csv': 'Step5: 合并节点',
|
||||
'rels_merged.csv': 'Step5: 合并关系',
|
||||
}
|
||||
|
||||
for filename, description in files.items():
|
||||
filepath = output_dir / filename
|
||||
if filepath.exists():
|
||||
import os
|
||||
size = os.path.getsize(filepath)
|
||||
print(f" [OK] {description} ({filename}, {size:,} bytes)")
|
||||
else:
|
||||
print(f" [--] {description} ({filename}, 未生成)")
|
||||
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
def importlib_safe_import(module_name):
|
||||
from importlib import import_module
|
||||
return import_module(module_name)
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description='城市规划法律法规知识图谱构建')
|
||||
parser.add_argument('--step', type=int, choices=[1, 2, 3, 4, 5], help='执行指定步骤')
|
||||
parser.add_argument('--full', action='store_true', help='运行完整流程')
|
||||
parser.add_argument('--status', action='store_true', help='查看项目状态')
|
||||
parser.add_argument('--test', action='store_true', help='测试模式(少量数据)')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.status:
|
||||
show_status()
|
||||
return
|
||||
|
||||
steps = {
|
||||
1: ("读取Word文档", step1_read_documents),
|
||||
2: ("解析法规元数据", step2_extract_metadata),
|
||||
3: ("生成结构化CSV", step3_generate_csv),
|
||||
4: ("LLM深度抽取", step4_deep_extraction),
|
||||
5: ("可视化", step5_visualize),
|
||||
}
|
||||
|
||||
if args.step:
|
||||
name, func = steps[args.step]
|
||||
print(f"\n{'='*60}")
|
||||
print(f"执行 Step {args.step}: {name}")
|
||||
print(f"{'='*60}")
|
||||
func()
|
||||
elif args.full:
|
||||
print("\n运行完整流程...")
|
||||
for step_num, (name, func) in steps.items():
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Step {step_num}: {name}")
|
||||
print(f"{'='*60}")
|
||||
try:
|
||||
func()
|
||||
except Exception as e:
|
||||
print(f"Step {step_num} 失败: {e}")
|
||||
break
|
||||
else:
|
||||
print("请指定操作:")
|
||||
print(" python src/main.py --step 1 # 读取Word文档")
|
||||
print(" python src/main.py --step 2 # 解析法规元数据")
|
||||
print(" python src/main.py --step 3 # 生成结构化CSV")
|
||||
print(" python src/main.py --step 4 # LLM深度抽取")
|
||||
print(" python src/main.py --step 5 # 可视化")
|
||||
print(" python src/main.py --full # 运行完整流程")
|
||||
print(" python src/main.py --status # 查看项目状态")
|
||||
print(" python src/main.py --test # 测试模式")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,29 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
echo ========================================
|
||||
echo 城市规划法律法规知识图谱构建工具
|
||||
echo ========================================
|
||||
echo.
|
||||
echo 请选择操作:
|
||||
echo 1. 读取Word文档 (Step1)
|
||||
echo 2. 解析法规元数据 (Step2)
|
||||
echo 3. 生成结构化CSV (Step3)
|
||||
echo 4. LLM深度抽取 (Step4)
|
||||
echo 5. 实体规范化与合并 (Step5)
|
||||
echo 6. 可视化 (Step6)
|
||||
echo 7. 运行完整流程
|
||||
echo 8. 查看进度
|
||||
echo.
|
||||
|
||||
set /p choice=请输入选项:
|
||||
|
||||
if "%choice%"=="1" python scripts/docx_reader.py
|
||||
if "%choice%"=="2" python scripts/legal_metadata_extractor.py
|
||||
if "%choice%"=="3" python scripts/extract_legal_csv.py
|
||||
if "%choice%"=="4" python src/deep_extraction_pipeline.py
|
||||
if "%choice%"=="5" python src/data_processing/merge_data.py
|
||||
if "%choice%"=="6" python scripts/visualize_legal_kg.py
|
||||
if "%choice%"=="7" python src/main.py --full
|
||||
if "%choice%"=="8" python src/main.py --status
|
||||
|
||||
pause
|
||||
Reference in New Issue
Block a user