fix: preserve data granularity and recover tasks safely
This commit is contained in:
@@ -0,0 +1,76 @@
|
|||||||
|
# 研途 · 考研数据工作台
|
||||||
|
|
||||||
|
将 YAM 的择校工作区和历史考情能力,与 yzw 的全国目录采集、代理池和断点功能整合为独立的本地网页应用。
|
||||||
|
|
||||||
|
当前安装位置:`F:\@Other\kaoyan-studio`。原 YAM、yzw 和 proxy_pool 源码不作修改。
|
||||||
|
|
||||||
|
## 启动
|
||||||
|
|
||||||
|
双击 **启动工作台.cmd**,浏览器打开 **http://127.0.0.1:8765**。窗口关闭后后台服务继续运行;双击 **停止工作台.cmd** 结束本项目服务。
|
||||||
|
|
||||||
|
本机已安装依赖并完成网页构建,可以直接启动。源码包包含已构建网页,复制到另一台 Windows 电脑后需要 Python 3.12+,首次运行会安装锁定的 Python 依赖。重新构建或修改网页需要 Node.js 22+;缺少网页构建时,脚本会安装前端依赖并构建。浏览器登录使用 Microsoft Edge。
|
||||||
|
|
||||||
|
在 PowerShell 中也可以运行:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
& 'F:\@Other\kaoyan-studio\start.ps1'
|
||||||
|
# 仅启动后台:start.ps1 -NoBrowser
|
||||||
|
# 改端口:start.ps1 -Port 8768
|
||||||
|
```
|
||||||
|
|
||||||
|
## 第一次采集
|
||||||
|
|
||||||
|
1. 在「代理 IP 池」刷新,查看池内 HTTPS 候选和当前路由。默认连接 `http://127.0.0.1:5010`;服务目录为 `C:\Users\Jayson\Desktop\proxy_pool`。
|
||||||
|
2. 在「设置与登录」选择网络策略。优先代理在池为空时允许直连;严格代理没有可用 IP 时等待处理。备用代理每行一个 `IP:端口`。代理池存在候选并不保证该 IP 对所有目标网站可用,连接失败会切换并隔离。
|
||||||
|
3. 点击「打开登录窗口」,在独立 Edge 窗口完成学信网登录和验证码。工作台会检测登录并保存 Cookie。也可粘贴已登录研招网的 Cookie 请求头,或使用账号密码登录。点击「验证目录访问」确认详情接口权限。
|
||||||
|
4. 在「采集任务」选择内容、目录年份、地区、专业和院校关键词。建议先按一个地区、一个专业或一所院校验证,再创建全国任务。专业代码可以从内置目录选择,也可直接输入 6 位代码。
|
||||||
|
5. 查看任务日志。暂停可继续;需要登录、代理或限流解除时,处理后点击恢复;取消保留已有数据。关闭或重启服务会保留可恢复断点。
|
||||||
|
|
||||||
|
研招网学校、专业索引有时可以匿名读取,但方向与科目详情需要登录。程序验证实际业务返回状态,不将 HTTP 200 当作登录成功。目录采集先核对研招页面配置中的年份,避免将当前目录标为历史年份。
|
||||||
|
|
||||||
|
## 数据查询
|
||||||
|
|
||||||
|
「院校工作区」支持院校/专业/院系关键词、年份、省份、专业代码、学习方式、学位类型、院校层次、科目、院系、方向、人数及参考线筛选和排序。可收藏候选、勾选 2–6 条记录对比,点击院校名查看来源、分数线和计划历史。
|
||||||
|
|
||||||
|
每条记录按 **年份 + 院校 + 院系 + 专业 + 方向 + 考试方式 + 科目组合 + 来源** 保存。不同年份、方向及可选科目组合独立保留,相同记录重新采集时更新。人数保留原文:明确「专业」人数可用于筛选;学院、方向、范围或待定口径不会被转换成专业总人数。不同方向和科目组合可能重复展示同一专业人数,不能累加。
|
||||||
|
|
||||||
|
历史分数线区分专业线、学科参考线和门类参考线,匹配院系后才展示;专项计划线独立保留在详情,不作为普通计划的参考线。表格显示不晚于目录年的最近匹配线,空字段显示「未提供」。历年图表按口径、院系、备注和来源分开,不将门类线标为专业线。
|
||||||
|
|
||||||
|
985/211 是历史工程参考名单。双一流优先采用数据源标识,同时补充已知的 2022 年名单;名单来源见 `data/school_levels.json`。学校更名、新名单或未识别名称可能需要更新参考元数据。
|
||||||
|
|
||||||
|
内置 2,255 条专业目录条目和 460 所院校公开索引,来自 YAM 的公共数据文件。**索引并不代表已经采集招生详情**;首次安装的目录和分数线数据库为空。全国采集不限定于这些索引,通过研招网查询全部 31 个省级地区的招生单位。
|
||||||
|
|
||||||
|
## 原项目数据互通
|
||||||
|
|
||||||
|
在工作区点击「导入数据」:
|
||||||
|
|
||||||
|
| 格式 | 用途 | 年份规则 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| yzw CSV | 原 `export_major_csv.py` 输出的全部字段 | 原文件无年,必须填写实际招生年份 |
|
||||||
|
| YAM SQLite | Python 数据库中的 schools/departments、score_lines、admission_plans | 无年份目录须填写;历史分数/计划保留文件年份 |
|
||||||
|
| YAM JSON 种子 | `data/seeds/yan_zhao_*_all_regions.json` | 只有学校列表时仅导入索引;有方向详情时须提供年份 |
|
||||||
|
| 工作台 JSON | 完整目录和分数线快照 | 保留各条记录年份 |
|
||||||
|
| CSV / XLSX | 工作台导出,或包含相应机器字段/中文表头的数据 | 优先文件年份,否则须明确填写 |
|
||||||
|
|
||||||
|
支持 UTF-8 / UTF-8 BOM / GB18030 CSV。一次最多 100 MB、30 万条。导入只读取所选文件,不打开原软件环境或修改源文件;SQLite 使用只读模式。其他版本 YAM 表结构不同的数据库,请从原软件导出兼容文件再导入。
|
||||||
|
|
||||||
|
可将当前筛选结果导出 CSV、Excel 或完整 JSON。表格导出保留院系/方向代码和字段口径,回导不会凭空增加同一记录。JSON 同时保留相关学校专业的历史分数线;凭据不进入导出。CSV/Excel 对可能被表格软件当作公式的文本加保护前缀。
|
||||||
|
|
||||||
|
## 本地数据与开发
|
||||||
|
|
||||||
|
所有运行数据位于项目 `.runtime`:`studio.db` 保存目录、分数、收藏、任务和断点;`settings.json` 保存非机密配置;`chsi-cookie.enc` 由当前 Windows 用户的 DPAPI 加密。密码不持久化,接口不返回 Cookie。备份数据库时先停止工作台,再复制 `.runtime`;恢复加密 Cookie 需要原 Windows 用户环境,否则重新登录即可。
|
||||||
|
|
||||||
|
服务默认仅监听 `127.0.0.1`,拒绝非本机 Host 和跨来源写入。代理池设置页使用现有 `manage.ps1` 控制本机服务;工作台停止脚本只结束自己管理且进程身份匹配的服务,不停止代理池。
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# 后端行为测试
|
||||||
|
.\.venv\Scripts\python.exe -m pytest -q
|
||||||
|
# 前端逻辑测试及生产构建
|
||||||
|
cd frontend
|
||||||
|
npm.cmd test
|
||||||
|
npm.cmd run build
|
||||||
|
```
|
||||||
|
|
||||||
|
浏览器验收使用独立的测试数据库,运行方式及结果见 `docs/verification.md`。没有真人登录的自动验收不能替代账号登录后的全国采集验证。
|
||||||
|
|
||||||
|
项目结构:`studio/` 为网络、数据源、存储和 API;`frontend/src/` 为 React 网页;`data/` 为公开参考元数据;`tests/` 为后端行为测试。来源与许可证见 `THIRD_PARTY_NOTICES.md`。
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
# 来源与第三方声明
|
||||||
|
|
||||||
|
本项目独立实现采集器、存储、任务和网页服务,参考以下项目的公开接口及字段设计,不直接运行原仓库模块。
|
||||||
|
|
||||||
|
- [YAM / 研喵](https://github.com/C-Y-T-258/yam):参考全国目录、掌上考研历史分数/计划接口及择校工作区设计。`data/majors.json` 与 `data/schools.json` 整理自其公共专业目录和院校种子,仅保留公开参考字段,未携带签名、Cookie 或账号。原 `pyproject.toml` 声明 MIT,作者 YAM Contributors;本地原仓库没有单独的 LICENSE 文件。
|
||||||
|
- [yzw](https://github.com/freecho/yzw):参考学校/专业/方向 API、CSV 字段和代理策略。学校层次参考数据以原文件为基础,补充学信网发布的教育部名单、名称别名,并修正中国中医科学院的 211 误标。原项目 MIT,Copyright (c) 2025 freecho,完整许可副本在 `licenses/yzw-MIT.txt`。
|
||||||
|
- [proxy_pool](https://github.com/jhao104/proxy_pool):使用用户已有的 Flask API 和本机 `manage.ps1`,不内嵌或修改代理池源码。
|
||||||
|
|
||||||
|
参考名单来源:[985 名单](https://www.moe.gov.cn/srcsite/A22/s7065/200612/t20061206_128833.html)、[学信网发布的教育部 211 名单](https://gaokao.chsi.com.cn/gkxx/gxmd/211gc.shtml)、[第二轮双一流通知](https://www.moe.gov.cn/srcsite/A22/s7065/202202/t20220211_598710.html)。985/211 用于历史工程参考;双一流标识优先采用采集来源,静态名单为已知参考子集。
|
||||||
|
|
||||||
|
Python / JavaScript 依赖保留其自身许可证。运行环境包括 FastAPI、Starlette、Uvicorn、requests、Beautiful Soup、openpyxl、Playwright、React、Vite、TypeScript、Lucide;版本记录在 `requirements.lock` 和 `frontend/package-lock.json`,许可证原文随安装包提供。
|
||||||
|
|
||||||
|
采集数据的准确性、完整性和开放年份取决于原网站,应用保留来源及口径,并明确标记未知字段。内置索引不等同于已经采集的招生数据。
|
||||||
@@ -0,0 +1,294 @@
|
|||||||
|
{
|
||||||
|
"985": [
|
||||||
|
"上海交通大学",
|
||||||
|
"东北大学",
|
||||||
|
"东南大学",
|
||||||
|
"中南大学",
|
||||||
|
"中国人民大学",
|
||||||
|
"中国农业大学",
|
||||||
|
"中国海洋大学",
|
||||||
|
"中国科学技术大学",
|
||||||
|
"中央民族大学",
|
||||||
|
"中山大学",
|
||||||
|
"兰州大学",
|
||||||
|
"北京大学",
|
||||||
|
"北京师范大学",
|
||||||
|
"北京理工大学",
|
||||||
|
"北京航空航天大学",
|
||||||
|
"华东师范大学",
|
||||||
|
"华中科技大学",
|
||||||
|
"华南理工大学",
|
||||||
|
"南京大学",
|
||||||
|
"南开大学",
|
||||||
|
"厦门大学",
|
||||||
|
"吉林大学",
|
||||||
|
"同济大学",
|
||||||
|
"哈尔滨工业大学",
|
||||||
|
"四川大学",
|
||||||
|
"国防科学技术大学",
|
||||||
|
"国防科技大学",
|
||||||
|
"复旦大学",
|
||||||
|
"大连理工大学",
|
||||||
|
"天津大学",
|
||||||
|
"山东大学",
|
||||||
|
"武汉大学",
|
||||||
|
"浙江大学",
|
||||||
|
"清华大学",
|
||||||
|
"湖南大学",
|
||||||
|
"电子科技大学",
|
||||||
|
"西北农林科技大学",
|
||||||
|
"西北工业大学",
|
||||||
|
"西安交通大学",
|
||||||
|
"重庆大学"
|
||||||
|
],
|
||||||
|
"211": [
|
||||||
|
"上海交通大学",
|
||||||
|
"上海外国语大学",
|
||||||
|
"上海大学",
|
||||||
|
"上海财经大学",
|
||||||
|
"东北农业大学",
|
||||||
|
"东北大学",
|
||||||
|
"东北师范大学",
|
||||||
|
"东北林业大学",
|
||||||
|
"东华大学",
|
||||||
|
"东南大学",
|
||||||
|
"中南大学",
|
||||||
|
"中南财经政法大学",
|
||||||
|
"中国人民大学",
|
||||||
|
"中国传媒大学",
|
||||||
|
"中国农业大学",
|
||||||
|
"中国地质大学(北京)",
|
||||||
|
"中国地质大学(武汉)",
|
||||||
|
"中国政法大学",
|
||||||
|
"中国海洋大学",
|
||||||
|
"中国石油大学(北京)",
|
||||||
|
"中国石油大学(华东)",
|
||||||
|
"中国矿业大学",
|
||||||
|
"中国矿业大学(北京)",
|
||||||
|
"中国矿业大学(徐州)",
|
||||||
|
"中国科学技术大学",
|
||||||
|
"中国药科大学",
|
||||||
|
"中央民族大学",
|
||||||
|
"中央财经大学",
|
||||||
|
"中央音乐学院",
|
||||||
|
"中山大学",
|
||||||
|
"云南大学",
|
||||||
|
"兰州大学",
|
||||||
|
"内蒙古大学",
|
||||||
|
"北京中医药大学",
|
||||||
|
"北京交通大学",
|
||||||
|
"北京体育大学",
|
||||||
|
"北京化工大学",
|
||||||
|
"北京外国语大学",
|
||||||
|
"北京大学",
|
||||||
|
"北京工业大学",
|
||||||
|
"北京师范大学",
|
||||||
|
"北京林业大学",
|
||||||
|
"北京理工大学",
|
||||||
|
"北京科技大学",
|
||||||
|
"北京航空航天大学",
|
||||||
|
"北京邮电大学",
|
||||||
|
"华东师范大学",
|
||||||
|
"华东理工大学",
|
||||||
|
"华中农业大学",
|
||||||
|
"华中师范大学",
|
||||||
|
"华中科技大学",
|
||||||
|
"华北电力大学",
|
||||||
|
"华北电力大学(保定)",
|
||||||
|
"华北电力大学(北京)",
|
||||||
|
"华南师范大学",
|
||||||
|
"华南理工大学",
|
||||||
|
"南京农业大学",
|
||||||
|
"南京大学",
|
||||||
|
"南京师范大学",
|
||||||
|
"南京理工大学",
|
||||||
|
"南京航空航天大学",
|
||||||
|
"南开大学",
|
||||||
|
"南昌大学",
|
||||||
|
"厦门大学",
|
||||||
|
"合肥工业大学",
|
||||||
|
"吉林大学",
|
||||||
|
"同济大学",
|
||||||
|
"哈尔滨工业大学",
|
||||||
|
"哈尔滨工程大学",
|
||||||
|
"四川农业大学",
|
||||||
|
"四川大学",
|
||||||
|
"国防科学技术大学",
|
||||||
|
"国防科技大学",
|
||||||
|
"复旦大学",
|
||||||
|
"大连海事大学",
|
||||||
|
"大连理工大学",
|
||||||
|
"天津医科大学",
|
||||||
|
"天津大学",
|
||||||
|
"太原理工大学",
|
||||||
|
"宁夏大学",
|
||||||
|
"安徽大学",
|
||||||
|
"对外经济贸易大学",
|
||||||
|
"山东大学",
|
||||||
|
"广西大学",
|
||||||
|
"延边大学",
|
||||||
|
"新疆大学",
|
||||||
|
"暨南大学",
|
||||||
|
"武汉大学",
|
||||||
|
"武汉理工大学",
|
||||||
|
"江南大学",
|
||||||
|
"河北工业大学",
|
||||||
|
"河海大学",
|
||||||
|
"浙江大学",
|
||||||
|
"海军军医大学",
|
||||||
|
"海南大学",
|
||||||
|
"清华大学",
|
||||||
|
"湖南大学",
|
||||||
|
"湖南师范大学",
|
||||||
|
"电子科技大学",
|
||||||
|
"石河子大学",
|
||||||
|
"福州大学",
|
||||||
|
"空军军医大学",
|
||||||
|
"第二军医大学",
|
||||||
|
"第四军医大学",
|
||||||
|
"苏州大学",
|
||||||
|
"西北农林科技大学",
|
||||||
|
"西北大学",
|
||||||
|
"西北工业大学",
|
||||||
|
"西南交通大学",
|
||||||
|
"西南大学",
|
||||||
|
"西南财经大学",
|
||||||
|
"西安交通大学",
|
||||||
|
"西安电子科技大学",
|
||||||
|
"西藏大学",
|
||||||
|
"贵州大学",
|
||||||
|
"辽宁大学",
|
||||||
|
"郑州大学",
|
||||||
|
"重庆大学",
|
||||||
|
"长安大学",
|
||||||
|
"陕西师范大学",
|
||||||
|
"青海大学"
|
||||||
|
],
|
||||||
|
"double_first_class_2022_known": [
|
||||||
|
"上海中医药大学",
|
||||||
|
"上海交通大学",
|
||||||
|
"上海外国语大学",
|
||||||
|
"上海大学",
|
||||||
|
"上海财经大学",
|
||||||
|
"东北农业大学",
|
||||||
|
"东北大学",
|
||||||
|
"东北师范大学",
|
||||||
|
"东北林业大学",
|
||||||
|
"东华大学",
|
||||||
|
"东南大学",
|
||||||
|
"中南大学",
|
||||||
|
"中南财经政法大学",
|
||||||
|
"中国人民大学",
|
||||||
|
"中国传媒大学",
|
||||||
|
"中国农业大学",
|
||||||
|
"中国地质大学(北京)",
|
||||||
|
"中国地质大学(武汉)",
|
||||||
|
"中国政法大学",
|
||||||
|
"中国海洋大学",
|
||||||
|
"中国石油大学(北京)",
|
||||||
|
"中国石油大学(华东)",
|
||||||
|
"中国矿业大学",
|
||||||
|
"中国矿业大学(北京)",
|
||||||
|
"中国科学技术大学",
|
||||||
|
"中国药科大学",
|
||||||
|
"中央民族大学",
|
||||||
|
"中央财经大学",
|
||||||
|
"中央音乐学院",
|
||||||
|
"中山大学",
|
||||||
|
"云南大学",
|
||||||
|
"兰州大学",
|
||||||
|
"内蒙古大学",
|
||||||
|
"北京中医药大学",
|
||||||
|
"北京交通大学",
|
||||||
|
"北京体育大学",
|
||||||
|
"北京化工大学",
|
||||||
|
"北京协和医学院",
|
||||||
|
"北京外国语大学",
|
||||||
|
"北京大学",
|
||||||
|
"北京工业大学",
|
||||||
|
"北京师范大学",
|
||||||
|
"北京林业大学",
|
||||||
|
"北京理工大学",
|
||||||
|
"北京科技大学",
|
||||||
|
"北京航空航天大学",
|
||||||
|
"北京邮电大学",
|
||||||
|
"华东师范大学",
|
||||||
|
"华东理工大学",
|
||||||
|
"华中农业大学",
|
||||||
|
"华中师范大学",
|
||||||
|
"华中科技大学",
|
||||||
|
"华北电力大学",
|
||||||
|
"华南师范大学",
|
||||||
|
"华南理工大学",
|
||||||
|
"南京农业大学",
|
||||||
|
"南京医科大学",
|
||||||
|
"南京大学",
|
||||||
|
"南京师范大学",
|
||||||
|
"南京理工大学",
|
||||||
|
"南京航空航天大学",
|
||||||
|
"南开大学",
|
||||||
|
"南方科技大学",
|
||||||
|
"南昌大学",
|
||||||
|
"厦门大学",
|
||||||
|
"合肥工业大学",
|
||||||
|
"吉林大学",
|
||||||
|
"同济大学",
|
||||||
|
"哈尔滨工业大学",
|
||||||
|
"哈尔滨工程大学",
|
||||||
|
"四川大学",
|
||||||
|
"国防科技大学",
|
||||||
|
"复旦大学",
|
||||||
|
"大连海事大学",
|
||||||
|
"大连理工大学",
|
||||||
|
"天津中医药大学",
|
||||||
|
"天津医科大学",
|
||||||
|
"天津大学",
|
||||||
|
"天津工业大学",
|
||||||
|
"太原理工大学",
|
||||||
|
"宁夏大学",
|
||||||
|
"安徽大学",
|
||||||
|
"对外经济贸易大学",
|
||||||
|
"山东大学",
|
||||||
|
"广州中医药大学",
|
||||||
|
"广西大学",
|
||||||
|
"延边大学",
|
||||||
|
"成都理工大学",
|
||||||
|
"新疆大学",
|
||||||
|
"暨南大学",
|
||||||
|
"武汉大学",
|
||||||
|
"武汉理工大学",
|
||||||
|
"江南大学",
|
||||||
|
"河北工业大学",
|
||||||
|
"河海大学",
|
||||||
|
"浙江大学",
|
||||||
|
"海南大学",
|
||||||
|
"清华大学",
|
||||||
|
"湖南大学",
|
||||||
|
"湖南师范大学",
|
||||||
|
"电子科技大学",
|
||||||
|
"石河子大学",
|
||||||
|
"福州大学",
|
||||||
|
"苏州大学",
|
||||||
|
"西北农林科技大学",
|
||||||
|
"西北大学",
|
||||||
|
"西北工业大学",
|
||||||
|
"西南交通大学",
|
||||||
|
"西南大学",
|
||||||
|
"西南财经大学",
|
||||||
|
"西安交通大学",
|
||||||
|
"西安电子科技大学",
|
||||||
|
"西藏大学",
|
||||||
|
"贵州大学",
|
||||||
|
"辽宁大学",
|
||||||
|
"郑州大学",
|
||||||
|
"重庆大学",
|
||||||
|
"长安大学",
|
||||||
|
"青海大学"
|
||||||
|
],
|
||||||
|
"sources": [
|
||||||
|
"https://www.moe.gov.cn/srcsite/A22/s7065/200612/t20061206_128833.html",
|
||||||
|
"https://gaokao.chsi.com.cn/gkxx/gxmd/211gc.shtml",
|
||||||
|
"https://www.moe.gov.cn/srcsite/A22/s7065/202202/t20220211_598710.html"
|
||||||
|
],
|
||||||
|
"note": "历史工程名单与已知名称别名,用于参考筛选。211以原yzw名单为基础并与学信网教育部名单核对补充、排除中国中医科学院误标。双一流优先实时研招网syl或导入字段;静态2022名单只是已知子集。"
|
||||||
|
}
|
||||||
+4
-2
@@ -7,8 +7,10 @@ Task 1: complete — 5 tests green; cross-year records, alternatives, conservati
|
|||||||
Task 2: complete — 5 tests green; shared proxy, strict route, fallback, rotation, DPAPI vault.
|
Task 2: complete — 5 tests green; shared proxy, strict route, fallback, rotation, DPAPI vault.
|
||||||
Task 3: complete — 5 tests green; pagination, business login status, year mismatch, exact campus, pause/resume.
|
Task 3: complete — 5 tests green; pagination, business login status, year mismatch, exact campus, pause/resume.
|
||||||
Task 4: complete — 5 tests green; file import/export, YAM SQLite, favorites, origin guard, formula-safe exports.
|
Task 4: complete — 5 tests green; file import/export, YAM SQLite, favorites, origin guard, formula-safe exports.
|
||||||
Task 5: pending
|
Task 5: complete — 3 frontend logic tests, production build, full Edge acceptance including mobile passed.
|
||||||
Task 6: pending
|
Task 6: final packaging — launch/reuse/stop/restart verified; README/notices ready; independent review findings fixed with regression tests.
|
||||||
|
|
||||||
Public index: 2,255 major entries and 460 unique schools extracted from YAM public data files. Seed index contains no admissions snapshots, cookies or signed request tokens.
|
Public index: 2,255 major entries and 460 unique schools extracted from YAM public data files. Seed index contains no admissions snapshots, cookies or signed request tokens.
|
||||||
|
|
||||||
|
Final verification: 35 backend tests passed; 3 frontend tests passed; build exit 0; browser QA PASS. Details: docs/verification.md.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# 验证记录
|
||||||
|
|
||||||
|
2026-10-11,Windows、本地独立虚拟环境,Python 3.12.13 / Node 24.16。
|
||||||
|
|
||||||
|
- 后端:`python -m pytest -q`,35 passed。Starlette TestClient 提示一条 httpx 迁移 warning,测试退出 0,生产服务不使用 TestClient。
|
||||||
|
- 前端:`npm test`,3 passed;`npm run build`,TypeScript 检查与 Vite 生产构建退出 0。
|
||||||
|
- Edge 实际浏览器:独立端口 8766、独立 `.runtime/qa/db`,验证导入→科目筛选→收藏→对比→历史趋势→Excel 下载;任务/代理/设置页面;设置保存;390px 手机宽度无页面溢出;无 pageerror。测试数据均标明「自动验收样例」,没有写入正式数据库。
|
||||||
|
- 启动脚本:Windows PowerShell 首启、重复启动、停止、重新启动均退出 0;正式服务在 127.0.0.1:8765,健康检查应用身份和项目路径匹配。
|
||||||
|
- 真实数据源:proxy_pool `/count/` 正常响应;研招页面 `curYear=2027` 可核实;匿名北京大学学校及 081200 专业索引可返回,方向接口要求登录,按 LoginRequired 处理;掌上考研精确匹配北京大学,2025 年 081200 查询返回 1 条门类参考线和 2 条历史计划。真实探测只读取少量公开数据,不写入正式目录。
|
||||||
|
- 独立只读审查:复现并修复专项线覆盖、同名跨域 Cookie、任务失败恢复竞态、服务退出被误取消、文件往返丢失身份字段、学院人数误升格、YAM 种子误识别、院校关键词范围、参考书 JSON 字符串、业务限流等问题,补充相应回归。
|
||||||
|
|
||||||
|
自动验收未使用真人学信网账号,未对全部院校进行实网抓取。登录后的全国采集能力取决于账号状态、代理可用性、开放目录年份和数据源限流。
|
||||||
|
|
||||||
|
复跑浏览器验收(只使用测试库):
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
.\.venv\Scripts\python.exe -m studio --port 8766 --data-dir .runtime\qa\db
|
||||||
|
# 在另一个终端运行
|
||||||
|
.\.venv\Scripts\python.exe tools\browser_qa.py
|
||||||
|
```
|
||||||
|
|
||||||
|
正式服务不会自动开始采集,需要用户在网页创建任务。
|
||||||
+16
-1
@@ -1,2 +1,17 @@
|
|||||||
<!doctype html>
|
<!doctype html>
|
||||||
<html lang="zh-CN"><head><meta charset="UTF-8" /><meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="theme-color" content="#132b3a" /><meta name="description" content="本地考研数据工作台:全国目录、历年考情、代理池与择校对比。" /><title>研途 · 考研数据工作台</title></head><body><div id="root"></div><script type="module" src="/src/main.tsx"></script></body></html>
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
|
<meta name="theme-color" content="#132b3a" />
|
||||||
|
<meta
|
||||||
|
name="description"
|
||||||
|
content="本地考研数据工作台:全国目录、历年考情、代理池与择校对比。"
|
||||||
|
/>
|
||||||
|
<title>研途 · 考研数据工作台</title>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div id="root"></div>
|
||||||
|
<script type="module" src="/src/main.tsx"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
|||||||
+1997
-74
File diff suppressed because one or more lines are too long
+21
-5
@@ -1,9 +1,25 @@
|
|||||||
export async function api<T>(path: string, options: RequestInit = {}): Promise<T> {
|
export async function api<T>(
|
||||||
const response = await fetch('/api' + path, { ...options, headers: options.body instanceof FormData ? options.headers : { 'Content-Type': 'application/json', ...options.headers } });
|
path: string,
|
||||||
|
options: RequestInit = {},
|
||||||
|
): Promise<T> {
|
||||||
|
const response = await fetch("/api" + path, {
|
||||||
|
...options,
|
||||||
|
headers:
|
||||||
|
options.body instanceof FormData
|
||||||
|
? options.headers
|
||||||
|
: { "Content-Type": "application/json", ...options.headers },
|
||||||
|
});
|
||||||
if (!response.ok) {
|
if (!response.ok) {
|
||||||
const body = await response.json().catch(() => ({ detail: `请求失败 (${response.status})` }));
|
const body = await response
|
||||||
throw new Error(typeof body.detail === 'string' ? body.detail : `参数无效 (${response.status})`);
|
.json()
|
||||||
|
.catch(() => ({ detail: `请求失败 (${response.status})` }));
|
||||||
|
throw new Error(
|
||||||
|
typeof body.detail === "string"
|
||||||
|
? body.detail
|
||||||
|
: `参数无效 (${response.status})`,
|
||||||
|
);
|
||||||
}
|
}
|
||||||
return response.json();
|
return response.json();
|
||||||
}
|
}
|
||||||
export const post = <T,>(path: string, body: unknown = {}) => api<T>(path, { method: 'POST', body: JSON.stringify(body) });
|
export const post = <T>(path: string, body: unknown = {}) =>
|
||||||
|
api<T>(path, { method: "POST", body: JSON.stringify(body) });
|
||||||
|
|||||||
+18
-12
@@ -1,17 +1,23 @@
|
|||||||
import { describe, expect, it } from 'vitest';
|
import { describe, expect, it } from "vitest";
|
||||||
import { display, scopeName, scoreSeries, queryString } from './format';
|
import { display, scopeName, scoreSeries, queryString } from "./format";
|
||||||
describe('unknown values and comparable history', () => {
|
describe("unknown values and comparable history", () => {
|
||||||
it('preserves zero and marks unknown', () => {
|
it("preserves zero and marks unknown", () => {
|
||||||
expect(display(0)).toBe('0');
|
expect(display(0)).toBe("0");
|
||||||
expect(display(null)).toBe('未提供');
|
expect(display(null)).toBe("未提供");
|
||||||
expect(scopeName('category')).toBe('门类参考线');
|
expect(scopeName("category")).toBe("门类参考线");
|
||||||
});
|
});
|
||||||
it('keeps different score scopes separate and omits unknown totals', () => {
|
it("keeps different score scopes separate and omits unknown totals", () => {
|
||||||
const result = scoreSeries([{ year: 2024, total: 310, scope: 'major' }, { year: 2025, total: 300, scope: 'category' }, { year: 2026, total: null, scope: 'major' }]);
|
const result = scoreSeries([
|
||||||
|
{ year: 2024, total: 310, scope: "major" },
|
||||||
|
{ year: 2025, total: 300, scope: "category" },
|
||||||
|
{ year: 2026, total: null, scope: "major" },
|
||||||
|
]);
|
||||||
expect(result).toHaveLength(2);
|
expect(result).toHaveLength(2);
|
||||||
expect(result.flatMap(r => r.points)).toHaveLength(2);
|
expect(result.flatMap((r) => r.points)).toHaveLength(2);
|
||||||
});
|
});
|
||||||
it('encodes filters and omits empty values', () => {
|
it("encodes filters and omits empty values", () => {
|
||||||
expect(queryString({ keyword: '北京 大学', year: '', favorites: false })).toBe('keyword=%E5%8C%97%E4%BA%AC+%E5%A4%A7%E5%AD%A6');
|
expect(
|
||||||
|
queryString({ keyword: "北京 大学", year: "", favorites: false }),
|
||||||
|
).toBe("keyword=%E5%8C%97%E4%BA%AC+%E5%A4%A7%E5%AD%A6");
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
+65
-14
@@ -1,16 +1,67 @@
|
|||||||
import type { Score, Filters } from './types';
|
import type { Score, Filters } from "./types";
|
||||||
export const display = (value: unknown): string => value === null || value === undefined || value === '' ? '未提供' : String(value);
|
export const display = (value: unknown): string =>
|
||||||
export const scopeName = (scope?: string | null) => ({ major: '专业线', discipline: '学科参考线', category: '门类参考线', unknown: '口径待核实' }[scope || ''] || '未提供');
|
value === null || value === undefined || value === ""
|
||||||
export const modeName = (mode: string) => ({ direct: '直接连接', prefer_proxy: '优先代理', proxy_only: '严格代理', catalog: '全国招生目录', scores: '历年考情', combined: '目录 + 历年考情' }[mode] || mode);
|
? "未提供"
|
||||||
export const statusName = (status: string) => ({ queued: '等待开始', running: '采集中', paused: '已暂停', waiting_login: '等待登录', waiting_proxy: '等待代理', completed: '已完成', partial: '部分失败', failed: '采集失败', cancelled: '已取消' }[status] || status);
|
: String(value);
|
||||||
export const queryString = (filters: Filters) => new URLSearchParams(Object.entries(filters).filter(([, v]) => v !== '' && v !== false).map(([k, v]) => [k, String(v)])).toString();
|
export const scopeName = (scope?: string | null) =>
|
||||||
|
({
|
||||||
|
major: "专业线",
|
||||||
|
discipline: "学科参考线",
|
||||||
|
category: "门类参考线",
|
||||||
|
unknown: "口径待核实",
|
||||||
|
})[scope || ""] || "未提供";
|
||||||
|
export const modeName = (mode: string) =>
|
||||||
|
({
|
||||||
|
direct: "直接连接",
|
||||||
|
prefer_proxy: "优先代理",
|
||||||
|
proxy_only: "严格代理",
|
||||||
|
catalog: "全国招生目录",
|
||||||
|
scores: "历年考情",
|
||||||
|
combined: "目录 + 历年考情",
|
||||||
|
})[mode] || mode;
|
||||||
|
export const statusName = (status: string) =>
|
||||||
|
({
|
||||||
|
queued: "等待开始",
|
||||||
|
running: "采集中",
|
||||||
|
paused: "已暂停",
|
||||||
|
waiting_login: "等待登录",
|
||||||
|
waiting_proxy: "等待代理",
|
||||||
|
waiting_source: "等待限流解除",
|
||||||
|
completed: "已完成",
|
||||||
|
partial: "部分失败",
|
||||||
|
failed: "采集失败",
|
||||||
|
cancelled: "已取消",
|
||||||
|
})[status] || status;
|
||||||
|
export const queryString = (filters: Filters) =>
|
||||||
|
new URLSearchParams(
|
||||||
|
Object.entries(filters)
|
||||||
|
.filter(([, v]) => v !== "" && v !== false)
|
||||||
|
.map(([k, v]) => [k, String(v)]),
|
||||||
|
).toString();
|
||||||
export function scoreSeries(scores: Score[]) {
|
export function scoreSeries(scores: Score[]) {
|
||||||
const grouped = new Map<string, { label: string; points: { year: number; total: number }[] }>();
|
const grouped = new Map<
|
||||||
scores.filter(s => s.total !== null).forEach(s => {
|
string,
|
||||||
const key = [s.scope, s.raw_code || '', s.department_name || '', s.source || '', s.note || ''].join('|');
|
{ label: string; points: { year: number; total: number }[] }
|
||||||
const group = grouped.get(key) || { label: scopeName(s.scope) + (s.note ? ` · ${s.note}` : ''), points: [] };
|
>();
|
||||||
group.points.push({ year: s.year, total: s.total! });
|
scores
|
||||||
grouped.set(key, group);
|
.filter((s) => s.total !== null)
|
||||||
});
|
.forEach((s) => {
|
||||||
return [...grouped.values()].map(g => ({ ...g, points: g.points.sort((a, b) => a.year - b.year) }));
|
const key = [
|
||||||
|
s.scope,
|
||||||
|
s.raw_code || "",
|
||||||
|
s.department_name || "",
|
||||||
|
s.source || "",
|
||||||
|
s.note || "",
|
||||||
|
].join("|");
|
||||||
|
const group = grouped.get(key) || {
|
||||||
|
label: scopeName(s.scope) + (s.note ? ` · ${s.note}` : ""),
|
||||||
|
points: [],
|
||||||
|
};
|
||||||
|
group.points.push({ year: s.year, total: s.total! });
|
||||||
|
grouped.set(key, group);
|
||||||
|
});
|
||||||
|
return [...grouped.values()].map((g) => ({
|
||||||
|
...g,
|
||||||
|
points: g.points.sort((a, b) => a.year - b.year),
|
||||||
|
}));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,9 @@
|
|||||||
import { StrictMode } from 'react';
|
import { StrictMode } from "react";
|
||||||
import { createRoot } from 'react-dom/client';
|
import { createRoot } from "react-dom/client";
|
||||||
import App from './App';
|
import App from "./App";
|
||||||
import './style.css';
|
import "./style.css";
|
||||||
createRoot(document.getElementById('root')!).render(<StrictMode><App /></StrictMode>);
|
createRoot(document.getElementById("root")!).render(
|
||||||
|
<StrictMode>
|
||||||
|
<App />
|
||||||
|
</StrictMode>,
|
||||||
|
);
|
||||||
|
|||||||
+1529
-243
File diff suppressed because it is too large
Load Diff
+113
-9
@@ -1,12 +1,116 @@
|
|||||||
export type Row = { id: number; year: number; school_name: string; school_code: string; province: string; level: string; major_code: string; major_name: string; department_name: string; department_code: string; direction: string; subjects: string[]; quota: number | null; quota_text: string; quota_scope: string; degree_type: string; study_mode: string; exam_type: string; advisor: string; special_plans: string[]; remarks: string; source: string; source_url: string; fetched_at: string; year_verified: boolean; favorite: boolean; score_total: number | null; score_year: number | null; score_scope: string | null; reference_books?: string[] };
|
export type Row = {
|
||||||
export type Score = { year: number; total: number | null; scope: string; raw_code?: string; department_name?: string; source?: string; note?: string; politics?: number | null; english?: number | null; special_one?: number | null; special_two?: number | null };
|
id: number;
|
||||||
|
year: number;
|
||||||
|
school_name: string;
|
||||||
|
school_code: string;
|
||||||
|
province: string;
|
||||||
|
level: string;
|
||||||
|
major_code: string;
|
||||||
|
major_name: string;
|
||||||
|
department_name: string;
|
||||||
|
department_code: string;
|
||||||
|
direction: string;
|
||||||
|
subjects: string[];
|
||||||
|
quota: number | null;
|
||||||
|
quota_text: string;
|
||||||
|
quota_scope: string;
|
||||||
|
degree_type: string;
|
||||||
|
study_mode: string;
|
||||||
|
exam_type: string;
|
||||||
|
advisor: string;
|
||||||
|
special_plans: string[];
|
||||||
|
remarks: string;
|
||||||
|
source: string;
|
||||||
|
source_url: string;
|
||||||
|
fetched_at: string;
|
||||||
|
year_verified: boolean;
|
||||||
|
favorite: boolean;
|
||||||
|
score_total: number | null;
|
||||||
|
score_year: number | null;
|
||||||
|
score_scope: string | null;
|
||||||
|
reference_books?: string[];
|
||||||
|
};
|
||||||
|
export type Score = {
|
||||||
|
year: number;
|
||||||
|
total: number | null;
|
||||||
|
scope: string;
|
||||||
|
raw_code?: string;
|
||||||
|
department_name?: string;
|
||||||
|
source?: string;
|
||||||
|
note?: string;
|
||||||
|
politics?: number | null;
|
||||||
|
english?: number | null;
|
||||||
|
special_one?: number | null;
|
||||||
|
special_two?: number | null;
|
||||||
|
};
|
||||||
export type History = { record: Row; scores: Score[]; plans: Row[] };
|
export type History = { record: Row; scores: Score[]; plans: Row[] };
|
||||||
export type Job = { id: string; params: { mode: string; year: number; major_code?: string; provinces: string[] }; status: string; message: string; created_at: string; updated_at: string; records: number; errors: number; completed_units: number };
|
export type Job = {
|
||||||
export type Log = { id: number; created_at: string; level: string; message: string };
|
id: string;
|
||||||
|
params: {
|
||||||
|
mode: string;
|
||||||
|
year: number;
|
||||||
|
major_code?: string;
|
||||||
|
provinces: string[];
|
||||||
|
};
|
||||||
|
status: string;
|
||||||
|
message: string;
|
||||||
|
created_at: string;
|
||||||
|
updated_at: string;
|
||||||
|
records: number;
|
||||||
|
errors: number;
|
||||||
|
completed_units: number;
|
||||||
|
};
|
||||||
|
export type Log = {
|
||||||
|
id: number;
|
||||||
|
created_at: string;
|
||||||
|
level: string;
|
||||||
|
message: string;
|
||||||
|
};
|
||||||
export type Province = { code: string; name: string };
|
export type Province = { code: string; name: string };
|
||||||
export type Major = { code: string; name: string; degree_type: string; category: string; discipline: string };
|
export type Major = {
|
||||||
export type Overview = { records: number; schools: number; majors: number; favorites: number; indexed_schools: number; scores: number; years: number[]; provinces: { province: string; count: number }[]; provinces_list: Province[]; sources: { name: string; role: string; url: string }[] };
|
code: string;
|
||||||
export type Config = { network_mode: string; pool_url: string; pool_project: string; backup_proxies: string[]; request_interval: number; timeout: number; retries: number };
|
name: string;
|
||||||
export type Proxy = { mode: string; route: string; active_proxy: string | null; fallback: boolean; counts: { count?: number; http_type?: { https: number; http: number } }; available: number; quarantined: number; error: string; proxies: { url: string; failed: boolean; active: boolean }[] };
|
degree_type: string;
|
||||||
export type Auth = { saved: boolean; valid: boolean; message: string; browser_state: string; browser_message: string };
|
category: string;
|
||||||
|
discipline: string;
|
||||||
|
};
|
||||||
|
export type Overview = {
|
||||||
|
records: number;
|
||||||
|
schools: number;
|
||||||
|
majors: number;
|
||||||
|
favorites: number;
|
||||||
|
indexed_schools: number;
|
||||||
|
scores: number;
|
||||||
|
years: number[];
|
||||||
|
provinces: { province: string; count: number }[];
|
||||||
|
provinces_list: Province[];
|
||||||
|
sources: { name: string; role: string; url: string }[];
|
||||||
|
};
|
||||||
|
export type Config = {
|
||||||
|
network_mode: string;
|
||||||
|
pool_url: string;
|
||||||
|
pool_project: string;
|
||||||
|
backup_proxies: string[];
|
||||||
|
request_interval: number;
|
||||||
|
timeout: number;
|
||||||
|
retries: number;
|
||||||
|
};
|
||||||
|
export type Proxy = {
|
||||||
|
mode: string;
|
||||||
|
route: string;
|
||||||
|
active_proxy: string | null;
|
||||||
|
fallback: boolean;
|
||||||
|
counts: { count?: number; http_type?: { https: number; http: number } };
|
||||||
|
available: number;
|
||||||
|
quarantined: number;
|
||||||
|
error: string;
|
||||||
|
proxies: { url: string; failed: boolean; active: boolean }[];
|
||||||
|
};
|
||||||
|
export type Auth = {
|
||||||
|
saved: boolean;
|
||||||
|
valid: boolean;
|
||||||
|
message: string;
|
||||||
|
browser_state: string;
|
||||||
|
browser_message: string;
|
||||||
|
};
|
||||||
export type Filters = Record<string, string | number | boolean>;
|
export type Filters = Record<string, string | number | boolean>;
|
||||||
|
|||||||
+14
-2
@@ -1,2 +1,14 @@
|
|||||||
{"compilerOptions":{"target":"ES2022","lib":["ES2022","DOM","DOM.Iterable"],"module":"ESNext","moduleResolution":"bundler","jsx":"react-jsx","strict":true,"skipLibCheck":true,"noEmit":true,"allowImportingTsExtensions":true},"include":["src","vite.config.ts"]}
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022",
|
||||||
|
"lib": ["ES2022", "DOM", "DOM.Iterable"],
|
||||||
|
"module": "ESNext",
|
||||||
|
"moduleResolution": "bundler",
|
||||||
|
"jsx": "react-jsx",
|
||||||
|
"strict": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"noEmit": true,
|
||||||
|
"allowImportingTsExtensions": true
|
||||||
|
},
|
||||||
|
"include": ["src", "vite.config.ts"]
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,4 +1,7 @@
|
|||||||
import { defineConfig } from 'vite';
|
import { defineConfig } from "vite";
|
||||||
import react from '@vitejs/plugin-react';
|
import react from "@vitejs/plugin-react";
|
||||||
export default defineConfig({plugins:[react()],server:{proxy:{'/api':'http://127.0.0.1:8765'}},build:{sourcemap:false}});
|
export default defineConfig({
|
||||||
|
plugins: [react()],
|
||||||
|
server: { proxy: { "/api": "http://127.0.0.1:8765" } },
|
||||||
|
build: { sourcemap: false },
|
||||||
|
});
|
||||||
|
|||||||
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2025 freecho
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
+6
-3
@@ -13,7 +13,7 @@ from studio.database import Database
|
|||||||
from studio.domain import PROVINCES
|
from studio.domain import PROVINCES
|
||||||
from studio.imports import import_file, export_sheet
|
from studio.imports import import_file, export_sheet
|
||||||
from studio.jobs import JobManager
|
from studio.jobs import JobManager
|
||||||
from studio.network import ProxyRouter, Transport, ProxyUnavailable, RemoteUnavailable
|
from studio.network import ProxyRouter, Transport, ProxyUnavailable, RemoteUnavailable, RateLimited
|
||||||
from studio.settings import SettingsStore
|
from studio.settings import SettingsStore
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parent.parent
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
@@ -60,6 +60,7 @@ def create_app(data_dir=None):
|
|||||||
|
|
||||||
@app.exception_handler(ProxyUnavailable)
|
@app.exception_handler(ProxyUnavailable)
|
||||||
@app.exception_handler(RemoteUnavailable)
|
@app.exception_handler(RemoteUnavailable)
|
||||||
|
@app.exception_handler(RateLimited)
|
||||||
async def remote_error(request, exc):
|
async def remote_error(request, exc):
|
||||||
return JSONResponse({"detail": str(exc)}, status_code=503)
|
return JSONResponse({"detail": str(exc)}, status_code=503)
|
||||||
|
|
||||||
@@ -188,10 +189,12 @@ def create_app(data_dir=None):
|
|||||||
if total > 100 * 1024 * 1024:
|
if total > 100 * 1024 * 1024:
|
||||||
raise ValueError("一次最多上传 100 MB,请拆分文件")
|
raise ValueError("一次最多上传 100 MB,请拆分文件")
|
||||||
stream.write(chunk)
|
stream.write(chunk)
|
||||||
records, scores = import_file(temporary, year)
|
result = import_file(temporary, year)
|
||||||
|
records, scores = result
|
||||||
db.upsert_catalog(records)
|
db.upsert_catalog(records)
|
||||||
db.upsert_scores(scores)
|
db.upsert_scores(scores)
|
||||||
return {"records": len(records), "scores": len(scores), "message": "导入完成,相同记录按年份更新"}
|
db.index_schools(result.schools)
|
||||||
|
return {"records": len(records), "scores": len(scores), "indexed_schools": len(result.schools), "message": "导入完成,相同记录按年份更新;无详情的 YAM 种子只作为院校索引"}
|
||||||
except (UnicodeError, json.JSONDecodeError, KeyError) as exc:
|
except (UnicodeError, json.JSONDecodeError, KeyError) as exc:
|
||||||
raise ValueError(f"文件编码或结构无法识别({type(exc).__name__})") from None
|
raise ValueError(f"文件编码或结构无法识别({type(exc).__name__})") from None
|
||||||
finally:
|
finally:
|
||||||
|
|||||||
+17
-6
@@ -60,14 +60,24 @@ class AuthService:
|
|||||||
self.browser_thread = None
|
self.browser_thread = None
|
||||||
try:
|
try:
|
||||||
self._load(self.vault.load())
|
self._load(self.vault.load())
|
||||||
except RuntimeError:
|
except (RuntimeError, OSError, ValueError):
|
||||||
self.message = "保存的 Cookie 无法解密,请重新登录"
|
self.message = "保存的 Cookie 无法解密,请重新登录"
|
||||||
|
|
||||||
def _load(self, cookies):
|
def _load(self, cookies):
|
||||||
|
if isinstance(cookies, dict):
|
||||||
|
cookies = [{"name": name, "value": value, "domain": ".chsi.com.cn", "path": "/"} for name, value in cookies.items()]
|
||||||
with self.transport.lock:
|
with self.transport.lock:
|
||||||
self.transport.session.cookies.clear()
|
self.transport.session.cookies.clear()
|
||||||
for name, value in cookies.items():
|
for cookie in cookies:
|
||||||
self.transport.session.cookies.set(name, value, domain=".chsi.com.cn", path="/")
|
domain = str(cookie.get("domain", ".chsi.com.cn"))
|
||||||
|
if domain.lstrip(".") != "chsi.com.cn" and not domain.endswith(".chsi.com.cn"):
|
||||||
|
continue
|
||||||
|
expires = cookie.get("expires")
|
||||||
|
self.transport.session.cookies.set(cookie["name"], cookie["value"], domain=domain, path=cookie.get("path", "/"), secure=bool(cookie.get("secure")), expires=int(expires) if expires and expires > 0 else None)
|
||||||
|
|
||||||
|
def _snapshot(self):
|
||||||
|
with self.transport.lock:
|
||||||
|
return [{"name": c.name, "value": c.value, "domain": c.domain, "path": c.path, "secure": c.secure, "expires": c.expires} for c in self.transport.session.cookies]
|
||||||
|
|
||||||
def status(self):
|
def status(self):
|
||||||
return {"saved": self.vault.path.exists(), "valid": self.valid, "message": self.message, "browser_state": self.browser_state, "browser_message": self.browser_message}
|
return {"saved": self.vault.path.exists(), "valid": self.valid, "message": self.message, "browser_state": self.browser_state, "browser_message": self.browser_message}
|
||||||
@@ -91,7 +101,8 @@ class AuthService:
|
|||||||
return self.status()
|
return self.status()
|
||||||
|
|
||||||
def check(self):
|
def check(self):
|
||||||
response = self.transport.request("POST", "https://yz.chsi.com.cn/zsml/rs/dws.do", data={"ssdm": "11", "curPage": 1, "start": 0, "pageSize": 1, "dwlxs[0]": "all"})
|
# The school/major indexes can be public while direction details require login.
|
||||||
|
response = self.transport.request("POST", "https://yz.chsi.com.cn/zsml/rs/yjfxs.do", data={"dwdm": "10001", "zydm": "081200", "zymc": "计算机科学与技术", "curPage": 1, "start": 0, "pageSize": 1, "dwlxs[0]": "all"})
|
||||||
try:
|
try:
|
||||||
body = response.json()
|
body = response.json()
|
||||||
except ValueError:
|
except ValueError:
|
||||||
@@ -101,7 +112,7 @@ class AuthService:
|
|||||||
self.valid = body.get("flag") is True and isinstance(body.get("msg"), dict)
|
self.valid = body.get("flag") is True and isinstance(body.get("msg"), dict)
|
||||||
self.message = "研招目录接口验证成功" if self.valid else "登录未生效或数据源限流,请在浏览器完成登录后重试"
|
self.message = "研招目录接口验证成功" if self.valid else "登录未生效或数据源限流,请在浏览器完成登录后重试"
|
||||||
if self.valid:
|
if self.valid:
|
||||||
self.vault.save(self.transport.session.cookies.get_dict())
|
self.vault.save(self._snapshot())
|
||||||
return self.status()
|
return self.status()
|
||||||
|
|
||||||
def login(self, username, password):
|
def login(self, username, password):
|
||||||
@@ -144,7 +155,7 @@ class AuthService:
|
|||||||
while time.monotonic() < deadline:
|
while time.monotonic() < deadline:
|
||||||
page.wait_for_timeout(2000)
|
page.wait_for_timeout(2000)
|
||||||
if "yz.chsi.com.cn" in page.url:
|
if "yz.chsi.com.cn" in page.url:
|
||||||
cookies = {c["name"]: c["value"] for c in context.cookies() if "chsi.com.cn" in c["domain"]}
|
cookies = [c for c in context.cookies() if c["domain"].lstrip(".") == "chsi.com.cn" or c["domain"].endswith(".chsi.com.cn")]
|
||||||
if cookies:
|
if cookies:
|
||||||
self._load(cookies)
|
self._load(cookies)
|
||||||
if self.check()["valid"]:
|
if self.check()["valid"]:
|
||||||
|
|||||||
+31
-16
@@ -2,7 +2,7 @@
|
|||||||
import json
|
import json
|
||||||
import re
|
import re
|
||||||
from studio.domain import finish_record, integer, normalized_name, select_score_lines
|
from studio.domain import finish_record, integer, normalized_name, select_score_lines
|
||||||
from studio.network import RemoteUnavailable
|
from studio.network import RemoteUnavailable, RateLimited
|
||||||
|
|
||||||
|
|
||||||
class LoginRequired(RuntimeError):
|
class LoginRequired(RuntimeError):
|
||||||
@@ -35,17 +35,25 @@ class CatalogClient:
|
|||||||
def pages(self, endpoint, payload, page_size=50):
|
def pages(self, endpoint, payload, page_size=50):
|
||||||
seen = set()
|
seen = set()
|
||||||
for page in range(1, 10001):
|
for page in range(1, 10001):
|
||||||
response = self.transport.request("POST", self.BASE + "rs/" + endpoint + ".do", data={**payload, "curPage": page, "start": (page - 1) * page_size, "pageSize": page_size})
|
for attempt in range(3):
|
||||||
try:
|
response = self.transport.request("POST", self.BASE + "rs/" + endpoint + ".do", data={**payload, "curPage": page, "start": (page - 1) * page_size, "pageSize": page_size})
|
||||||
body = response.json()
|
try:
|
||||||
except ValueError:
|
body = response.json()
|
||||||
raise LoginRequired("研招网返回登录/验证页面,请先完成登录") from None
|
except ValueError:
|
||||||
|
raise LoginRequired("研招网返回登录/验证页面,请先完成登录") from None
|
||||||
|
message = str(body.get('msg', ''))
|
||||||
|
if body.get('flag') is not True and '验证码' in message:
|
||||||
|
raise LoginRequired('研招网需要人工验证码,请在浏览器完成验证后恢复')
|
||||||
|
if body.get('flag') is not True and any(token in message for token in ('频繁', '稍后')):
|
||||||
|
if attempt == 2:
|
||||||
|
raise RateLimited('研招网业务限流持续出现,已暂停;请增加请求间隔后恢复')
|
||||||
|
self.transport.sleep(5 * 2 ** attempt)
|
||||||
|
continue
|
||||||
|
break
|
||||||
if body.get("flag") is not True:
|
if body.get("flag") is not True:
|
||||||
message = str(body.get("msg", ""))
|
message = str(body.get("msg", ""))
|
||||||
if any(token in message for token in ("登录", "登陆", "认证")):
|
if any(token in message for token in ("登录", "登陆", "认证")):
|
||||||
raise LoginRequired("研招网要求登录,请更新 Cookie 后恢复任务")
|
raise LoginRequired("研招网要求登录,请更新 Cookie 后恢复任务")
|
||||||
if any(token in message for token in ("频繁", "稍后", "验证码")):
|
|
||||||
raise RemoteUnavailable("研招网访问受限,请增加请求间隔并完成人工验证后恢复")
|
|
||||||
raise RemoteUnavailable("研招网目录接口未成功返回数据")
|
raise RemoteUnavailable("研招网目录接口未成功返回数据")
|
||||||
data = body.get("msg") or {}
|
data = body.get("msg") or {}
|
||||||
if not isinstance(data, dict) or not isinstance(data.get("list"), list):
|
if not isinstance(data, dict) or not isinstance(data.get("list"), list):
|
||||||
@@ -83,14 +91,21 @@ class ScoreClient:
|
|||||||
self.cache = {}
|
self.cache = {}
|
||||||
|
|
||||||
def post(self, endpoint, payload):
|
def post(self, endpoint, payload):
|
||||||
response = self.transport.request("POST", self.BASE + endpoint, data=payload, headers={"Referer": "https://www.kaoyan.cn/"})
|
for attempt in range(3):
|
||||||
try:
|
response = self.transport.request("POST", self.BASE + endpoint, data=payload, headers={"Referer": "https://www.kaoyan.cn/"})
|
||||||
body = response.json()
|
try:
|
||||||
except ValueError:
|
body = response.json()
|
||||||
raise RemoteUnavailable("掌上考研未返回有效 JSON") from None
|
except ValueError:
|
||||||
if str(body.get("code")) != "0000":
|
raise RemoteUnavailable("掌上考研未返回有效 JSON") from None
|
||||||
raise RemoteUnavailable("掌上考研接口未成功返回数据,请稍后重试")
|
if str(body.get("code")) == "0000":
|
||||||
return body.get("data") or {}
|
return body.get("data") or {}
|
||||||
|
message = str(body.get('message', body.get('msg', '')))
|
||||||
|
if any(token in message for token in ('频繁', '稍后', '限流', '繁忙')):
|
||||||
|
if attempt == 2:
|
||||||
|
raise RateLimited('掌上考研业务限流持续出现,请增加请求间隔后恢复')
|
||||||
|
self.transport.sleep(5 * 2 ** attempt)
|
||||||
|
else:
|
||||||
|
raise RemoteUnavailable("掌上考研接口未成功返回数据,请稍后重试")
|
||||||
|
|
||||||
def pages(self, endpoint, payload, limit=100):
|
def pages(self, endpoint, payload, limit=100):
|
||||||
seen = set()
|
seen = set()
|
||||||
|
|||||||
+23
-7
@@ -3,8 +3,15 @@ import json
|
|||||||
import sqlite3
|
import sqlite3
|
||||||
from contextlib import contextmanager
|
from contextlib import contextmanager
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from studio.domain import now, normalized_name
|
from studio.domain import now, normalized_name, is_special_score, school_level
|
||||||
|
|
||||||
|
SCORE_SCHEMA = """
|
||||||
|
CREATE TABLE IF NOT EXISTS scores(
|
||||||
|
id INTEGER PRIMARY KEY, school_code TEXT, major_code TEXT, year INTEGER,
|
||||||
|
department_id TEXT, department_name TEXT, scope TEXT, total INTEGER, source TEXT, data TEXT NOT NULL,
|
||||||
|
note TEXT NOT NULL DEFAULT '',raw_code TEXT NOT NULL DEFAULT '',
|
||||||
|
UNIQUE(school_code,major_code,year,department_id,department_name,scope,source,note,raw_code));
|
||||||
|
"""
|
||||||
SCHEMA = """
|
SCHEMA = """
|
||||||
CREATE TABLE IF NOT EXISTS catalog(
|
CREATE TABLE IF NOT EXISTS catalog(
|
||||||
id INTEGER PRIMARY KEY, record_key TEXT NOT NULL UNIQUE, year INTEGER NOT NULL,
|
id INTEGER PRIMARY KEY, record_key TEXT NOT NULL UNIQUE, year INTEGER NOT NULL,
|
||||||
@@ -12,10 +19,7 @@ CREATE TABLE IF NOT EXISTS catalog(
|
|||||||
department_code TEXT, department_name TEXT, study_mode TEXT, degree_type TEXT, level TEXT,
|
department_code TEXT, department_name TEXT, study_mode TEXT, degree_type TEXT, level TEXT,
|
||||||
subjects TEXT, quota INTEGER, source TEXT, favorite INTEGER NOT NULL DEFAULT 0, data TEXT NOT NULL);
|
subjects TEXT, quota INTEGER, source TEXT, favorite INTEGER NOT NULL DEFAULT 0, data TEXT NOT NULL);
|
||||||
CREATE INDEX IF NOT EXISTS catalog_filter ON catalog(year,province,major_code,school_code);
|
CREATE INDEX IF NOT EXISTS catalog_filter ON catalog(year,province,major_code,school_code);
|
||||||
CREATE TABLE IF NOT EXISTS scores(
|
""" + SCORE_SCHEMA + """
|
||||||
id INTEGER PRIMARY KEY, school_code TEXT, major_code TEXT, year INTEGER,
|
|
||||||
department_id TEXT, department_name TEXT, scope TEXT, total INTEGER, source TEXT, data TEXT NOT NULL,
|
|
||||||
UNIQUE(school_code,major_code,year,department_id,department_name,scope,source));
|
|
||||||
CREATE INDEX IF NOT EXISTS scores_lookup ON scores(school_code,major_code,year DESC);
|
CREATE INDEX IF NOT EXISTS scores_lookup ON scores(school_code,major_code,year DESC);
|
||||||
CREATE TABLE IF NOT EXISTS schools(school_code TEXT PRIMARY KEY, school_name TEXT, province TEXT, level TEXT, data TEXT);
|
CREATE TABLE IF NOT EXISTS schools(school_code TEXT PRIMARY KEY, school_name TEXT, province TEXT, level TEXT, data TEXT);
|
||||||
CREATE TABLE IF NOT EXISTS jobs(id TEXT PRIMARY KEY, params TEXT, status TEXT, created_at TEXT, updated_at TEXT, message TEXT, errors INTEGER DEFAULT 0, records INTEGER DEFAULT 0);
|
CREATE TABLE IF NOT EXISTS jobs(id TEXT PRIMARY KEY, params TEXT, status TEXT, created_at TEXT, updated_at TEXT, message TEXT, errors INTEGER DEFAULT 0, records INTEGER DEFAULT 0);
|
||||||
@@ -34,6 +38,13 @@ class Database:
|
|||||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
with self.connection() as conn:
|
with self.connection() as conn:
|
||||||
conn.executescript(SCHEMA)
|
conn.executescript(SCHEMA)
|
||||||
|
if 'note' not in {row['name'] for row in conn.execute('PRAGMA table_info(scores)')}:
|
||||||
|
# Rebuild atomically to replace the early schema's narrower unique key.
|
||||||
|
conn.execute(SCORE_SCHEMA.replace('IF NOT EXISTS scores', 'scores_new'))
|
||||||
|
conn.execute("""INSERT INTO scores_new(id,school_code,major_code,year,department_id,department_name,scope,total,source,data,note,raw_code) SELECT id,school_code,major_code,year,department_id,department_name,scope,total,source,data,coalesce(json_extract(data,'$.note'),''),coalesce(json_extract(data,'$.raw_code'),'') FROM scores""")
|
||||||
|
conn.execute('DROP TABLE scores')
|
||||||
|
conn.execute('ALTER TABLE scores_new RENAME TO scores')
|
||||||
|
conn.execute('CREATE INDEX scores_lookup ON scores(school_code,major_code,year DESC)')
|
||||||
conn.execute("UPDATE jobs SET status='paused',message='服务重启,任务可从断点恢复' WHERE status IN ('running','queued')")
|
conn.execute("UPDATE jobs SET status='paused',message='服务重启,任务可从断点恢复' WHERE status IN ('running','queued')")
|
||||||
|
|
||||||
@contextmanager
|
@contextmanager
|
||||||
@@ -41,6 +52,7 @@ class Database:
|
|||||||
conn = sqlite3.connect(self.path, timeout=30)
|
conn = sqlite3.connect(self.path, timeout=30)
|
||||||
conn.row_factory = sqlite3.Row
|
conn.row_factory = sqlite3.Row
|
||||||
conn.create_function("norm_name", 1, normalized_name, deterministic=True)
|
conn.create_function("norm_name", 1, normalized_name, deterministic=True)
|
||||||
|
conn.create_function("is_special_score", 1, is_special_score, deterministic=True)
|
||||||
conn.execute("PRAGMA journal_mode=WAL")
|
conn.execute("PRAGMA journal_mode=WAL")
|
||||||
try:
|
try:
|
||||||
yield conn
|
yield conn
|
||||||
@@ -66,6 +78,7 @@ class Database:
|
|||||||
self._school(conn, record)
|
self._school(conn, record)
|
||||||
|
|
||||||
def _school(self, conn, record):
|
def _school(self, conn, record):
|
||||||
|
record = {**record, 'level': school_level(record['school_name'], record.get('level', ''))}
|
||||||
conn.execute("INSERT INTO schools VALUES(?,?,?,?,?) ON CONFLICT(school_code) DO UPDATE SET school_name=excluded.school_name,province=CASE WHEN excluded.province<>'' THEN excluded.province ELSE schools.province END,level=CASE WHEN excluded.level<>'' THEN excluded.level ELSE schools.level END", (record["school_code"],record["school_name"],record.get("province", ""),record.get("level", ""),encode(record)))
|
conn.execute("INSERT INTO schools VALUES(?,?,?,?,?) ON CONFLICT(school_code) DO UPDATE SET school_name=excluded.school_name,province=CASE WHEN excluded.province<>'' THEN excluded.province ELSE schools.province END,level=CASE WHEN excluded.level<>'' THEN excluded.level ELSE schools.level END", (record["school_code"],record["school_name"],record.get("province", ""),record.get("level", ""),encode(record)))
|
||||||
|
|
||||||
def index_schools(self, rows):
|
def index_schools(self, rows):
|
||||||
@@ -76,7 +89,7 @@ class Database:
|
|||||||
def upsert_scores(self, records):
|
def upsert_scores(self, records):
|
||||||
with self.connection() as conn:
|
with self.connection() as conn:
|
||||||
for row in records:
|
for row in records:
|
||||||
conn.execute("""INSERT INTO scores(school_code,major_code,year,department_id,department_name,scope,total,source,data) VALUES(?,?,?,?,?,?,?,?,?) ON CONFLICT(school_code,major_code,year,department_id,department_name,scope,source) DO UPDATE SET total=excluded.total,data=excluded.data""",(row["school_code"],row["major_code"],row["year"],row.get("department_id", ""),row.get("department_name", ""),row["scope"],row.get("total"),row.get("source", "掌上考研"),encode(row)))
|
conn.execute("""INSERT INTO scores(school_code,major_code,year,department_id,department_name,scope,total,source,data,note,raw_code) VALUES(?,?,?,?,?,?,?,?,?,?,?) ON CONFLICT(school_code,major_code,year,department_id,department_name,scope,source,note,raw_code) DO UPDATE SET total=excluded.total,data=excluded.data""",(row["school_code"],row["major_code"],row["year"],row.get("department_id", "") or "",row.get("department_name", "") or "",row["scope"],row.get("total"),row.get("source", "掌上考研"),encode(row),row.get('note') or '',row.get('raw_code') or ''))
|
||||||
|
|
||||||
def _query(self, filters: dict):
|
def _query(self, filters: dict):
|
||||||
conditions, params = ["1=1"], []
|
conditions, params = ["1=1"], []
|
||||||
@@ -95,6 +108,9 @@ class Database:
|
|||||||
if filters.get("keyword"):
|
if filters.get("keyword"):
|
||||||
conditions.append("(c.school_name LIKE ? OR c.major_name LIKE ? OR c.department_name LIKE ? OR c.major_code LIKE ?)")
|
conditions.append("(c.school_name LIKE ? OR c.major_name LIKE ? OR c.department_name LIKE ? OR c.major_code LIKE ?)")
|
||||||
params.extend([f"%{filters['keyword']}%"] * 4)
|
params.extend([f"%{filters['keyword']}%"] * 4)
|
||||||
|
if filters.get('school_keyword'):
|
||||||
|
conditions.append('c.school_name LIKE ?')
|
||||||
|
params.append(f"%{filters['school_keyword']}%")
|
||||||
if filters.get("favorites"):
|
if filters.get("favorites"):
|
||||||
conditions.append("c.favorite=1")
|
conditions.append("c.favorite=1")
|
||||||
if filters.get("min_quota") is not None:
|
if filters.get("min_quota") is not None:
|
||||||
@@ -104,7 +120,7 @@ class Database:
|
|||||||
if filters.get(field):
|
if filters.get(field):
|
||||||
conditions.append(f"json_extract(c.data,'$.subjects[{index}]') LIKE ?")
|
conditions.append(f"json_extract(c.data,'$.subjects[{index}]') LIKE ?")
|
||||||
params.append(str(filters[field])+"%")
|
params.append(str(filters[field])+"%")
|
||||||
select = """SELECT c.*,s.data AS score_data,s.total AS score_total,s.year AS score_year,s.scope AS score_scope FROM catalog c LEFT JOIN scores s ON s.id=(SELECT x.id FROM scores x WHERE x.school_code=c.school_code AND x.major_code=c.major_code AND x.year<=c.year AND (x.department_name='' AND x.department_id='' OR x.department_id<>'' AND x.department_id=c.department_code OR x.department_name<>'' AND norm_name(x.department_name)=norm_name(c.department_name)) AND coalesce(json_extract(x.data,'$.note'),'') NOT LIKE '%专项%' AND coalesce(json_extract(x.data,'$.note'),'') NOT LIKE '%少干%' ORDER BY x.year DESC,CASE x.scope WHEN 'major' THEN 0 WHEN 'discipline' THEN 1 ELSE 2 END,x.id DESC LIMIT 1) WHERE """ + " AND ".join(conditions)
|
select = """SELECT c.*,s.data AS score_data,s.total AS score_total,s.year AS score_year,s.scope AS score_scope FROM catalog c LEFT JOIN scores s ON s.id=(SELECT x.id FROM scores x WHERE x.school_code=c.school_code AND x.major_code=c.major_code AND x.year<=c.year AND (x.department_name='' AND x.department_id='' OR x.department_id<>'' AND x.department_id=c.department_code OR x.department_name<>'' AND norm_name(x.department_name)=norm_name(c.department_name)) AND is_special_score(json_extract(x.data,'$.note'))=0 ORDER BY x.year DESC,CASE x.scope WHEN 'major' THEN 0 WHEN 'discipline' THEN 1 ELSE 2 END,x.id DESC LIMIT 1) WHERE """ + " AND ".join(conditions)
|
||||||
if filters.get("max_score") is not None:
|
if filters.get("max_score") is not None:
|
||||||
select = "SELECT * FROM ("+select+") q WHERE q.score_total<=?"
|
select = "SELECT * FROM ("+select+") q WHERE q.score_total<=?"
|
||||||
params.append(filters["max_score"])
|
params.append(filters["max_score"])
|
||||||
|
|||||||
@@ -4,11 +4,27 @@ import json
|
|||||||
import re
|
import re
|
||||||
from datetime import datetime, timezone, timedelta
|
from datetime import datetime, timezone, timedelta
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
from pathlib import Path
|
||||||
|
from functools import lru_cache
|
||||||
from urllib.parse import urlencode
|
from urllib.parse import urlencode
|
||||||
|
|
||||||
PROVINCES = {"11":"北京","12":"天津","13":"河北","14":"山西","15":"内蒙古","21":"辽宁","22":"吉林","23":"黑龙江","31":"上海","32":"江苏","33":"浙江","34":"安徽","35":"福建","36":"江西","37":"山东","41":"河南","42":"湖北","43":"湖南","44":"广东","45":"广西","46":"海南","50":"重庆","51":"四川","52":"贵州","53":"云南","54":"西藏","61":"陕西","62":"甘肃","63":"青海","64":"宁夏","65":"新疆"}
|
PROVINCES = {"11":"北京","12":"天津","13":"河北","14":"山西","15":"内蒙古","21":"辽宁","22":"吉林","23":"黑龙江","31":"上海","32":"江苏","33":"浙江","34":"安徽","35":"福建","36":"江西","37":"山东","41":"河南","42":"湖北","43":"湖南","44":"广东","45":"广西","46":"海南","50":"重庆","51":"四川","52":"贵州","53":"云南","54":"西藏","61":"陕西","62":"甘肃","63":"青海","64":"宁夏","65":"新疆"}
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def school_metadata():
|
||||||
|
path = Path(__file__).resolve().parent.parent / 'data/school_levels.json'
|
||||||
|
data = json.loads(path.read_text(encoding='utf-8')) if path.exists() else {}
|
||||||
|
return {k: {normalized_name(n) for n in data.get(k, [])} for k in ('985','211','double_first_class_2022_known')}
|
||||||
|
|
||||||
|
|
||||||
|
def school_level(name, existing=''):
|
||||||
|
metadata = school_metadata()
|
||||||
|
name = normalized_name(name)
|
||||||
|
levels = [label for key, label in [('985','985'),('211','211'),('double_first_class_2022_known','双一流')] if name in metadata[key] or label in str(existing).split('/')]
|
||||||
|
return '/'.join(levels) or existing
|
||||||
|
|
||||||
|
|
||||||
def now() -> str:
|
def now() -> str:
|
||||||
return datetime.now(timezone(timedelta(hours=8))).isoformat(timespec="seconds")
|
return datetime.now(timezone(timedelta(hours=8))).isoformat(timespec="seconds")
|
||||||
|
|
||||||
@@ -17,6 +33,16 @@ def normalized_name(value: str) -> str:
|
|||||||
return str(value or "").replace("(", "(").replace(")", ")").replace(" ", "").strip()
|
return str(value or "").replace("(", "(").replace(")", ")").replace(" ", "").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def is_special_score(note):
|
||||||
|
text = str(note or '')
|
||||||
|
tokens = ('专项', '少干', '少骨', '少数民族骨干', '退役大学生士兵', '退役士兵', '强军', '援藏', '单独考试')
|
||||||
|
for token in tokens:
|
||||||
|
cleaned = re.sub(r'(?:不含|不包括|除外)\s*' + re.escape(token), '', text)
|
||||||
|
if token in cleaned:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
def integer(value: Any) -> int | None:
|
def integer(value: Any) -> int | None:
|
||||||
if value is None or isinstance(value, bool):
|
if value is None or isinstance(value, bool):
|
||||||
return None
|
return None
|
||||||
@@ -47,6 +73,7 @@ def record_key(record: dict) -> str:
|
|||||||
def finish_record(record: dict) -> dict:
|
def finish_record(record: dict) -> dict:
|
||||||
defaults = {"school_code":"", "school_name":"", "province":"", "major_code":"", "major_name":"", "department_code":"", "department_name":"", "direction_code":"", "direction":"", "study_mode":"", "degree_type":"", "exam_type":"", "advisor":"", "quota_text":"", "quota":None, "quota_scope":"unknown", "subjects":[], "special_plans":[], "level":"", "remarks":"", "source":"", "source_url":"", "fetched_at":now(), "year_verified":False}
|
defaults = {"school_code":"", "school_name":"", "province":"", "major_code":"", "major_name":"", "department_code":"", "department_name":"", "direction_code":"", "direction":"", "study_mode":"", "degree_type":"", "exam_type":"", "advisor":"", "quota_text":"", "quota":None, "quota_scope":"unknown", "subjects":[], "special_plans":[], "level":"", "remarks":"", "source":"", "source_url":"", "fetched_at":now(), "year_verified":False}
|
||||||
row = {**defaults, **record}
|
row = {**defaults, **record}
|
||||||
|
row['level'] = school_level(row['school_name'], row['level'])
|
||||||
if not row["school_code"]:
|
if not row["school_code"]:
|
||||||
row["school_code"] = "name:" + normalized_name(row["school_name"])
|
row["school_code"] = "name:" + normalized_name(row["school_name"])
|
||||||
row["record_key"] = record_key(row)
|
row["record_key"] = record_key(row)
|
||||||
|
|||||||
+34
-6
@@ -3,15 +3,28 @@ import csv
|
|||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
import sqlite3
|
import sqlite3
|
||||||
|
from dataclasses import dataclass, field
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from openpyxl import load_workbook, Workbook
|
from openpyxl import load_workbook, Workbook
|
||||||
from studio.domain import finish_record, integer, parse_quota
|
from studio.domain import finish_record, integer, parse_quota, normalize_catalog, PROVINCES
|
||||||
|
|
||||||
LABELS = {"year": "年份", "school_name": "院校", "school_code": "院校代码", "province": "省份", "level": "院校层次", "major_code": "专业代码", "major_name": "专业", "degree_type": "学位类型", "department_name": "院系", "direction": "研究方向", "study_mode": "学习方式", "exam_type": "考试方式", "advisor": "导师", "quota_text": "招生人数原文", "quota": "明确人数", "quota_scope": "人数口径", "subjects": "考试科目", "special_plans": "专项计划", "score_year": "参考线年份", "score_total": "参考线总分", "score_scope": "分数线口径", "remarks": "备注", "source": "来源", "source_url": "来源链接", "year_verified": "来源年份已核实", "fetched_at": "采集时间"}
|
LABELS = {"year": "年份", "school_name": "院校", "school_code": "院校代码", "province": "省份", "level": "院校层次", "major_code": "专业代码", "major_name": "专业", "degree_type": "学位类型", "department_name": "院系", "direction": "研究方向", "study_mode": "学习方式", "exam_type": "考试方式", "advisor": "导师", "quota_text": "招生人数原文", "quota": "明确人数", "quota_scope": "人数口径", "subjects": "考试科目", "special_plans": "专项计划", "score_year": "参考线年份", "score_total": "参考线总分", "score_scope": "分数线口径", "remarks": "备注", "source": "来源", "source_url": "来源链接", "year_verified": "来源年份已核实", "fetched_at": "采集时间"}
|
||||||
|
LABELS.update({"department_code": "院系代码", "direction_code": "方向代码"})
|
||||||
ALIASES = {"院校名称": "school_name", "学校名称": "school_name", "专业名称": "major_name", "院系所": "department_name", "拟招生人数": "quota_text", "planned_enrollment": "quota_text", "nzsrsstr": "quota_text", "school_level": "level", "department": "department_name", "research_direction": "direction", "enrollment_text": "quota_text", "enrollment_count": "quota", "exam_subjects": "subjects"}
|
ALIASES = {"院校名称": "school_name", "学校名称": "school_name", "专业名称": "major_name", "院系所": "department_name", "拟招生人数": "quota_text", "planned_enrollment": "quota_text", "nzsrsstr": "quota_text", "school_level": "level", "department": "department_name", "research_direction": "direction", "enrollment_text": "quota_text", "enrollment_count": "quota", "exam_subjects": "subjects"}
|
||||||
ALIASES.update({v: k for k, v in LABELS.items()})
|
ALIASES.update({v: k for k, v in LABELS.items()})
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ImportResult:
|
||||||
|
records: list
|
||||||
|
scores: list
|
||||||
|
schools: list = field(default_factory=list)
|
||||||
|
|
||||||
|
def __iter__(self):
|
||||||
|
yield self.records
|
||||||
|
yield self.scores
|
||||||
|
|
||||||
|
|
||||||
def list_field(value):
|
def list_field(value):
|
||||||
if isinstance(value, list):
|
if isinstance(value, list):
|
||||||
return value
|
return value
|
||||||
@@ -44,7 +57,7 @@ def row_record(raw, year=None, source="文件导入"):
|
|||||||
if not subjects:
|
if not subjects:
|
||||||
subjects = [str(row.get(f"exam_subject{i}") or row.get(f"subject{i}") or row.get(f"考试科目{i}") or "") for i in range(1, 5)]
|
subjects = [str(row.get(f"exam_subject{i}") or row.get(f"subject{i}") or row.get(f"考试科目{i}") or "") for i in range(1, 5)]
|
||||||
quota, scope = parse_quota(row.get("quota_text", ""))
|
quota, scope = parse_quota(row.get("quota_text", ""))
|
||||||
if quota is None and row.get("quota") != "":
|
if quota is None and not str(row.get('quota_text') or '').strip():
|
||||||
quota = integer(row.get("quota"))
|
quota = integer(row.get("quota"))
|
||||||
scope = row.get("quota_scope") or "unspecified"
|
scope = row.get("quota_scope") or "unspecified"
|
||||||
plans = list_field(row.get("special_plans"))
|
plans = list_field(row.get("special_plans"))
|
||||||
@@ -52,7 +65,7 @@ def row_record(raw, year=None, source="文件导入"):
|
|||||||
plans.append("退役大学生士兵")
|
plans.append("退役大学生士兵")
|
||||||
if row.get("shaogu_program") in ("是", "1", 1):
|
if row.get("shaogu_program") in ("是", "1", 1):
|
||||||
plans.append("少数民族骨干")
|
plans.append("少数民族骨干")
|
||||||
return finish_record({**row, "year": explicit_year, "source": row.get("source") or source, "subjects": subjects, "special_plans": plans, "quota": quota, "quota_scope": scope, "year_verified": row.get("year_verified") is True or row.get("year_verified") == "True"})
|
return finish_record({**row, "year": explicit_year, "source": row.get("source") or source, "subjects": subjects, "reference_books": list_field(row.get('reference_books')), "special_plans": plans, "quota": quota, "quota_scope": scope, "year_verified": row.get("year_verified") is True or row.get("year_verified") == "True"})
|
||||||
|
|
||||||
|
|
||||||
def import_file(path, year=None):
|
def import_file(path, year=None):
|
||||||
@@ -92,14 +105,29 @@ def import_file(path, year=None):
|
|||||||
raise ValueError("支持 CSV、XLSX、JSON 和 YAM SQLite 文件")
|
raise ValueError("支持 CSV、XLSX、JSON 和 YAM SQLite 文件")
|
||||||
if len(rows) > 300000:
|
if len(rows) > 300000:
|
||||||
raise ValueError("一次最多导入 30 万条,请拆分文件")
|
raise ValueError("一次最多导入 30 万条,请拆分文件")
|
||||||
records = [row_record(row, year, "yzw 导入" if suffix == ".csv" else "文件导入") for row in rows]
|
records, schools = [], []
|
||||||
|
for row in rows:
|
||||||
|
if isinstance(row, dict) and row.get('dwdm') and row.get('dwmc'):
|
||||||
|
schools.append({'school_code': str(row['dwdm']), 'school_name': row['dwmc'], 'province': row.get('szss') or PROVINCES.get(str(row.get('szssm', '')), ''), 'level': '双一流' if str(row.get('syl')) == '1' else ''})
|
||||||
|
directions = row.get('data') or ([row] if row.get('kskmz') else [])
|
||||||
|
if not isinstance(directions, list):
|
||||||
|
raise ValueError('YAM 种子 data 应为方向数组')
|
||||||
|
for direction in directions:
|
||||||
|
if not isinstance(direction, dict) or not any(k in direction for k in ('yjfxmc', 'kskmz', 'yxsmc')):
|
||||||
|
raise ValueError('YAM 方向数据结构无法识别')
|
||||||
|
explicit = integer(row.get('year')) or integer(year)
|
||||||
|
if not explicit:
|
||||||
|
raise ValueError('种子含有目录详情,请指定其招生年份')
|
||||||
|
records.extend(normalize_catalog(direction, explicit, row))
|
||||||
|
else:
|
||||||
|
records.append(row_record(row, year, "yzw 导入" if suffix == ".csv" else "文件导入"))
|
||||||
for score in scores:
|
for score in scores:
|
||||||
if not all(k in score for k in ("school_code", "major_code", "year", "scope")):
|
if not all(k in score for k in ("school_code", "major_code", "year", "scope")):
|
||||||
raise ValueError("快照分数线缺少院校/专业/年份/口径")
|
raise ValueError("快照分数线缺少院校/专业/年份/口径")
|
||||||
score["year"] = int(score["year"])
|
score["year"] = int(score["year"])
|
||||||
score["total"] = integer(score.get("total"))
|
score["total"] = integer(score.get("total"))
|
||||||
score.setdefault("note", "")
|
score.setdefault("note", "")
|
||||||
return records, scores
|
return ImportResult(records, scores, schools)
|
||||||
|
|
||||||
|
|
||||||
def import_yam(path, year):
|
def import_yam(path, year):
|
||||||
@@ -133,7 +161,7 @@ def import_yam(path, year):
|
|||||||
row = dict(entry)
|
row = dict(entry)
|
||||||
school = lookup.get((str(row["school_id"]), str(row.get("major_code", ""))), {})
|
school = lookup.get((str(row["school_id"]), str(row.get("major_code", ""))), {})
|
||||||
records.append(row_record({**row, "school_code": str(row["school_id"]), "school_name": school.get("name", ""), "province": school.get("province", ""), "direction": row.get("research_direction", ""), "direction_code": str(row.get("spe_id") or row.get("plan_id") or ""), "department_code": str(row.get("department_id", "") or ""), "source": row.get("source") or "YAM 导入", "remarks": row.get("note", "")}, year))
|
records.append(row_record({**row, "school_code": str(row["school_id"]), "school_name": school.get("name", ""), "province": school.get("province", ""), "direction": row.get("research_direction", ""), "direction_code": str(row.get("spe_id") or row.get("plan_id") or ""), "department_code": str(row.get("department_id", "") or ""), "source": row.get("source") or "YAM 导入", "remarks": row.get("note", "")}, year))
|
||||||
return records, scores
|
return ImportResult(records, scores)
|
||||||
except sqlite3.DatabaseError:
|
except sqlite3.DatabaseError:
|
||||||
raise ValueError("SQLite 文件无法读取,请确认是完整导出的数据库") from None
|
raise ValueError("SQLite 文件无法读取,请确认是完整导出的数据库") from None
|
||||||
finally:
|
finally:
|
||||||
|
|||||||
+27
-7
@@ -3,13 +3,17 @@ import threading
|
|||||||
import uuid
|
import uuid
|
||||||
from studio.connectors import CatalogClient, ScoreClient, LoginRequired, SourceYearMismatch
|
from studio.connectors import CatalogClient, ScoreClient, LoginRequired, SourceYearMismatch
|
||||||
from studio.domain import PROVINCES, normalize_catalog
|
from studio.domain import PROVINCES, normalize_catalog
|
||||||
from studio.network import ProxyUnavailable, RemoteUnavailable
|
from studio.network import ProxyUnavailable, RemoteUnavailable, RateLimited
|
||||||
|
|
||||||
|
|
||||||
class Cancelled(Exception):
|
class Cancelled(Exception):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class Shutdown(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
class JobManager:
|
class JobManager:
|
||||||
def __init__(self, database, settings, transport=None, runner=None):
|
def __init__(self, database, settings, transport=None, runner=None):
|
||||||
self.db, self.settings, self.transport = database, settings, transport
|
self.db, self.settings, self.transport = database, settings, transport
|
||||||
@@ -54,7 +58,7 @@ class JobManager:
|
|||||||
|
|
||||||
def _launch(self, jid):
|
def _launch(self, jid):
|
||||||
condition = threading.Condition()
|
condition = threading.Condition()
|
||||||
control = {"condition": condition, "paused": False, "cancelled": False}
|
control = {"condition": condition, "paused": False, "cancelled": False, "shutdown": False}
|
||||||
thread = threading.Thread(target=self._run, args=(jid,), daemon=True, name="collector-" + jid[:8])
|
thread = threading.Thread(target=self._run, args=(jid,), daemon=True, name="collector-" + jid[:8])
|
||||||
control["thread"] = thread
|
control["thread"] = thread
|
||||||
self.controls[jid] = control
|
self.controls[jid] = control
|
||||||
@@ -71,6 +75,8 @@ class JobManager:
|
|||||||
self.checkpoint(jid)
|
self.checkpoint(jid)
|
||||||
errors = self.get(jid)["errors"]
|
errors = self.get(jid)["errors"]
|
||||||
self.db.job_update(jid, status="partial" if errors else "completed", message="部分单元失败,可恢复重试" if errors else "采集完成")
|
self.db.job_update(jid, status="partial" if errors else "completed", message="部分单元失败,可恢复重试" if errors else "采集完成")
|
||||||
|
except Shutdown:
|
||||||
|
self.db.job_update(jid, status="paused", message="服务停止,任务可从断点恢复")
|
||||||
except Cancelled:
|
except Cancelled:
|
||||||
self.db.job_update(jid, status="cancelled", message="已取消,已保存数据保留")
|
self.db.job_update(jid, status="cancelled", message="已取消,已保存数据保留")
|
||||||
except LoginRequired as exc:
|
except LoginRequired as exc:
|
||||||
@@ -79,6 +85,9 @@ class JobManager:
|
|||||||
except ProxyUnavailable as exc:
|
except ProxyUnavailable as exc:
|
||||||
self.db.job_update(jid, status="waiting_proxy", message=str(exc))
|
self.db.job_update(jid, status="waiting_proxy", message=str(exc))
|
||||||
self.db.log(jid, "warning", str(exc))
|
self.db.log(jid, "warning", str(exc))
|
||||||
|
except RateLimited as exc:
|
||||||
|
self.db.job_update(jid, status="waiting_source", message=str(exc))
|
||||||
|
self.db.log(jid, "warning", str(exc))
|
||||||
except (RemoteUnavailable, SourceYearMismatch, ValueError) as exc:
|
except (RemoteUnavailable, SourceYearMismatch, ValueError) as exc:
|
||||||
self.db.job_update(jid, status="failed", message=str(exc))
|
self.db.job_update(jid, status="failed", message=str(exc))
|
||||||
self.db.log(jid, "error", str(exc))
|
self.db.log(jid, "error", str(exc))
|
||||||
@@ -88,16 +97,20 @@ class JobManager:
|
|||||||
finally:
|
finally:
|
||||||
if self.transport:
|
if self.transport:
|
||||||
self.transport.local.guard = None
|
self.transport.local.guard = None
|
||||||
|
if self.controls[jid]["shutdown"] and self.get(jid)["status"] != 'cancelled':
|
||||||
|
self.db.job_update(jid, status="paused", message="服务停止,任务可从断点恢复")
|
||||||
|
|
||||||
def checkpoint(self, jid):
|
def checkpoint(self, jid):
|
||||||
control = self.controls.get(jid)
|
control = self.controls.get(jid)
|
||||||
if not control:
|
if not control:
|
||||||
return
|
return
|
||||||
with control["condition"]:
|
with control["condition"]:
|
||||||
while control["paused"] and not control["cancelled"]:
|
while control["paused"] and not control["cancelled"] and not control["shutdown"]:
|
||||||
control["condition"].wait(.25)
|
control["condition"].wait(.25)
|
||||||
if control["cancelled"]:
|
if control["cancelled"]:
|
||||||
raise Cancelled()
|
raise Cancelled()
|
||||||
|
if control["shutdown"]:
|
||||||
|
raise Shutdown()
|
||||||
|
|
||||||
def unit_done(self, jid, unit):
|
def unit_done(self, jid, unit):
|
||||||
return self.db.unit_done(jid, unit)
|
return self.db.unit_done(jid, unit)
|
||||||
@@ -117,16 +130,20 @@ class JobManager:
|
|||||||
def resume(self, jid):
|
def resume(self, jid):
|
||||||
with self.lock:
|
with self.lock:
|
||||||
job = self.get(jid)
|
job = self.get(jid)
|
||||||
if not job or job["status"] not in ("paused", "partial", "failed", "waiting_login", "waiting_proxy"):
|
if not job or job["status"] not in ("paused", "partial", "failed", "waiting_login", "waiting_proxy", "waiting_source"):
|
||||||
raise ValueError("此任务当前不能恢复")
|
raise ValueError("此任务当前不能恢复")
|
||||||
self._ensure_available(jid)
|
self._ensure_available(jid)
|
||||||
control = self.controls.get(jid)
|
control = self.controls.get(jid)
|
||||||
if control and control["thread"].is_alive():
|
if control and control["thread"].is_alive() and control["paused"] and job['status'] == 'paused':
|
||||||
with control["condition"]:
|
with control["condition"]:
|
||||||
self.db.job_update(jid, status="running", message="正在继续采集")
|
self.db.job_update(jid, status="running", message="正在继续采集")
|
||||||
control["paused"] = False
|
control["paused"] = False
|
||||||
control["condition"].notify_all()
|
control["condition"].notify_all()
|
||||||
else:
|
else:
|
||||||
|
if control and control['thread'].is_alive():
|
||||||
|
control['thread'].join(timeout=1)
|
||||||
|
if control['thread'].is_alive():
|
||||||
|
raise ValueError("旧任务正在结束,请稍后点击恢复")
|
||||||
self._launch(jid)
|
self._launch(jid)
|
||||||
return self.get(jid)
|
return self.get(jid)
|
||||||
|
|
||||||
@@ -147,7 +164,10 @@ class JobManager:
|
|||||||
self.closed = True
|
self.closed = True
|
||||||
for jid, control in list(self.controls.items()):
|
for jid, control in list(self.controls.items()):
|
||||||
if control["thread"].is_alive():
|
if control["thread"].is_alive():
|
||||||
self.cancel(jid)
|
with control['condition']:
|
||||||
|
control['shutdown'] = True
|
||||||
|
self.db.job_update(jid, status='paused', message='服务停止,任务可从断点恢复')
|
||||||
|
control['condition'].notify_all()
|
||||||
for control in list(self.controls.values()):
|
for control in list(self.controls.values()):
|
||||||
control["thread"].join(timeout=1)
|
control["thread"].join(timeout=1)
|
||||||
|
|
||||||
@@ -194,7 +214,7 @@ class JobManager:
|
|||||||
self._error(jid, f"{label}:{exc}")
|
self._error(jid, f"{label}:{exc}")
|
||||||
|
|
||||||
def _scores(self, jid, params):
|
def _scores(self, jid, params):
|
||||||
filters = {"year": params["year"], "major_code": params.get("major_code", ""), "keyword": params.get("school_keyword", "")}
|
filters = {"year": params["year"], "major_code": params.get("major_code", ""), "school_keyword": params.get("school_keyword", "")}
|
||||||
records = self.db.all_catalog(filters)
|
records = self.db.all_catalog(filters)
|
||||||
targets = {}
|
targets = {}
|
||||||
for row in records:
|
for row in records:
|
||||||
|
|||||||
+5
-1
@@ -13,6 +13,10 @@ class RemoteUnavailable(RuntimeError):
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class RateLimited(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
class ProxyRouter:
|
class ProxyRouter:
|
||||||
def __init__(self, settings, session=None):
|
def __init__(self, settings, session=None):
|
||||||
self.settings = settings
|
self.settings = settings
|
||||||
@@ -146,4 +150,4 @@ class Transport:
|
|||||||
raise RemoteUnavailable("网络连接失败,已尝试切换代理;请检查网络或恢复任务") from None
|
raise RemoteUnavailable("网络连接失败,已尝试切换代理;请检查网络或恢复任务") from None
|
||||||
except requests.HTTPError as exc:
|
except requests.HTTPError as exc:
|
||||||
raise RemoteUnavailable(f"数据源返回 HTTP {exc.response.status_code}") from None
|
raise RemoteUnavailable(f"数据源返回 HTTP {exc.response.status_code}") from None
|
||||||
raise RemoteUnavailable("数据源限流,请降低请求频率后重试")
|
raise RateLimited("数据源持续限流,任务已等待;请降低请求频率后恢复")
|
||||||
|
|||||||
@@ -66,3 +66,13 @@ def test_import_yam_sqlite_keeps_year_and_score_scope(client, tmp_path):
|
|||||||
assert rows[0]["year"] == 2026
|
assert rows[0]["year"] == 2026
|
||||||
assert rows[0]["score_year"] == 2025
|
assert rows[0]["score_year"] == 2025
|
||||||
assert rows[0]["score_scope"] == "discipline"
|
assert rows[0]["score_scope"] == "discipline"
|
||||||
|
|
||||||
|
|
||||||
|
def test_yam_seed_json_imports_school_index_without_inventing_admissions(client):
|
||||||
|
payload = [{'dwdm': '10001', 'dwmc': '北京大学', 'zydm': '081200', 'zymc': '计算机科学与技术', 'szss': '北京', 'data': [], 'sign': 'token-must-not-copy'}]
|
||||||
|
response = client.post('/api/import', files={'file': ('yam-seed.json', json.dumps(payload).encode())})
|
||||||
|
assert response.status_code == 200, response.text
|
||||||
|
assert response.json()['records'] == 0
|
||||||
|
assert response.json()['indexed_schools'] == 1
|
||||||
|
assert client.get('/api/catalog').json()['total'] == 0
|
||||||
|
assert 'token-must-not-copy' not in client.get('/api/schools').text
|
||||||
|
|||||||
@@ -79,3 +79,64 @@ def test_task_pause_resume_reuses_persisted_work_without_duplicate_entries(tmp_p
|
|||||||
assert jobs.get(job["id"])["completed_units"] == 2
|
assert jobs.get(job["id"])["completed_units"] == 2
|
||||||
jobs.close()
|
jobs.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_service_shutdown_pauses_instead_of_cancelling_persisted_job(tmp_path):
|
||||||
|
db, settings = Database(tmp_path / 'studio.db'), SettingsStore(tmp_path)
|
||||||
|
entered = threading.Event()
|
||||||
|
def runner(jid, params, manager):
|
||||||
|
entered.set()
|
||||||
|
while True:
|
||||||
|
manager.checkpoint(jid)
|
||||||
|
time.sleep(.01)
|
||||||
|
manager = JobManager(db, settings, runner=runner)
|
||||||
|
job = manager.start({'mode': 'catalog', 'year': 2027})
|
||||||
|
assert entered.wait(2)
|
||||||
|
manager.close()
|
||||||
|
assert Database(tmp_path / 'studio.db').job_get(job['id'])['status'] == 'paused'
|
||||||
|
|
||||||
|
|
||||||
|
def test_business_rate_limit_uses_bounded_backoff_without_skipping_a_page():
|
||||||
|
from studio.network import RateLimited
|
||||||
|
class Limited(CatalogTransport):
|
||||||
|
def __init__(self):
|
||||||
|
self.calls, self.waits = 0, []
|
||||||
|
def request(self, *args, **kwargs):
|
||||||
|
self.calls += 1
|
||||||
|
return Response({'flag': False, 'msg': '访问太频繁,请稍后重试'})
|
||||||
|
def sleep(self, seconds):
|
||||||
|
self.waits.append(seconds)
|
||||||
|
transport = Limited()
|
||||||
|
with pytest.raises(RateLimited):
|
||||||
|
list(CatalogClient(transport).schools('11'))
|
||||||
|
assert transport.calls == 3
|
||||||
|
assert transport.waits == [5, 10]
|
||||||
|
|
||||||
|
|
||||||
|
def test_recovery_waits_for_failed_worker_to_exit_before_launching_another(tmp_path):
|
||||||
|
db, settings = Database(tmp_path / 'studio.db'), SettingsStore(tmp_path)
|
||||||
|
entered, release = threading.Event(), threading.Event()
|
||||||
|
original_log = db.log
|
||||||
|
def guarded_log(jid, level, message):
|
||||||
|
if level == 'error':
|
||||||
|
entered.set()
|
||||||
|
release.wait(2)
|
||||||
|
original_log(jid, level, message)
|
||||||
|
db.log = guarded_log
|
||||||
|
attempts = []
|
||||||
|
def runner(jid, params, manager):
|
||||||
|
attempts.append(jid)
|
||||||
|
if len(attempts) == 1:
|
||||||
|
raise ValueError('failure fixture')
|
||||||
|
manager.complete_unit(jid, 'recovered')
|
||||||
|
manager = JobManager(db, settings, runner=runner)
|
||||||
|
job = manager.start({'mode': 'catalog', 'year': 2027})
|
||||||
|
assert entered.wait(2)
|
||||||
|
threading.Timer(.05, release.set).start()
|
||||||
|
manager.resume(job['id'])
|
||||||
|
deadline = time.monotonic() + 2
|
||||||
|
while manager.get(job['id'])['status'] != 'completed' and time.monotonic() < deadline:
|
||||||
|
time.sleep(.02)
|
||||||
|
assert manager.get(job['id'])['status'] == 'completed'
|
||||||
|
assert len(attempts) == 2
|
||||||
|
manager.close()
|
||||||
|
|
||||||
|
|||||||
@@ -64,3 +64,61 @@ def test_same_year_different_college_lines_are_not_attached_arbitrarily(tmp_path
|
|||||||
db.upsert_scores(select_score_lines(rows, "081200", "10001", "北京大学", 2026))
|
db.upsert_scores(select_score_lines(rows, "081200", "10001", "北京大学", 2026))
|
||||||
assert db.query_catalog({"year": 2026})["items"][0]["score_total"] is None
|
assert db.query_catalog({"year": 2026})["items"][0]["score_total"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_special_plan_score_does_not_overwrite_regular_admission_line(tmp_path):
|
||||||
|
db = Database(tmp_path / "studio.db")
|
||||||
|
db.upsert_catalog(normalize_catalog(raw_catalog(), 2026))
|
||||||
|
rows = [{"code": "081200", "total": "320", "depart_id": "001", "note": ""}, {"code": "081200", "total": "260", "depart_id": "001", "note": "少干专项计划"}]
|
||||||
|
db.upsert_scores(select_score_lines(rows, "081200", "10001", "北京大学", 2026))
|
||||||
|
assert db.overview()["scores"] == 2
|
||||||
|
assert db.query_catalog({})["items"][0]["score_total"] == 320
|
||||||
|
|
||||||
|
|
||||||
|
def test_special_plan_full_names_are_not_used_as_regular_reference_line(tmp_path):
|
||||||
|
db = Database(tmp_path / 'studio.db')
|
||||||
|
db.upsert_catalog(normalize_catalog(raw_catalog(), 2026))
|
||||||
|
rows = [{'code': '081200', 'total': 320, 'depart_id': '001', 'note': ''}, {'code': '081200', 'total': 260, 'depart_id': '001', 'note': '少数民族骨干计划'}, {'code': '081200', 'total': 250, 'depart_id': '001', 'note': '退役大学生士兵计划'}]
|
||||||
|
db.upsert_scores(select_score_lines(rows, '081200', '10001', '北京大学', 2026))
|
||||||
|
assert db.query_catalog({})['items'][0]['score_total'] == 320
|
||||||
|
|
||||||
|
|
||||||
|
def test_spreadsheet_roundtrip_preserves_record_identity(tmp_path):
|
||||||
|
from studio.imports import export_sheet, import_file
|
||||||
|
db = Database(tmp_path / 'studio.db')
|
||||||
|
records = normalize_catalog(raw_catalog(), 2027)
|
||||||
|
db.upsert_catalog(records)
|
||||||
|
for kind in ('csv', 'xlsx'):
|
||||||
|
path = tmp_path / ('data.' + kind)
|
||||||
|
path.write_bytes(export_sheet(records, kind))
|
||||||
|
imported, _ = import_file(path)
|
||||||
|
db.upsert_catalog(imported)
|
||||||
|
assert db.query_catalog({})['total'] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_known_historical_school_levels_are_available_in_filters(tmp_path):
|
||||||
|
db = Database(tmp_path / 'studio.db')
|
||||||
|
db.upsert_catalog(normalize_catalog(raw_catalog(), 2027))
|
||||||
|
assert db.query_catalog({'level': '985'})['total'] == 2
|
||||||
|
assert db.query_catalog({'level': '211'})['total'] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_imported_college_quota_cannot_be_promoted_by_numeric_column():
|
||||||
|
from studio.imports import row_record
|
||||||
|
row = row_record({'school_name':'Test','major_code':'081200','enrollment_text':'学院:100','enrollment_count':100},2026)
|
||||||
|
assert row['quota'] is None
|
||||||
|
assert row['quota_scope'] == 'college'
|
||||||
|
|
||||||
|
|
||||||
|
def test_job_school_keyword_does_not_match_department_names(tmp_path):
|
||||||
|
db = Database(tmp_path / 'studio.db')
|
||||||
|
item = raw_catalog()
|
||||||
|
item['yxsmc'] = '信息学院'
|
||||||
|
db.upsert_catalog(normalize_catalog(item, 2027))
|
||||||
|
assert db.query_catalog({'school_keyword': '信息'})['total'] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_yam_reference_books_are_arrays_for_the_detail_view():
|
||||||
|
from studio.imports import row_record
|
||||||
|
row = row_record({'school_name':'Test','major_code':'081200','reference_books':'[]'},2026)
|
||||||
|
assert row['reference_books'] == []
|
||||||
|
|
||||||
|
|||||||
+38
-1
@@ -3,7 +3,7 @@ import pytest
|
|||||||
|
|
||||||
from studio.network import ProxyRouter, Transport, ProxyUnavailable
|
from studio.network import ProxyRouter, Transport, ProxyUnavailable
|
||||||
from studio.settings import SettingsStore
|
from studio.settings import SettingsStore
|
||||||
from studio.auth import CookieVault
|
from studio.auth import CookieVault, AuthService
|
||||||
|
|
||||||
|
|
||||||
class Response:
|
class Response:
|
||||||
@@ -91,3 +91,40 @@ def test_cookie_is_encrypted_and_not_returned_in_settings(tmp_path):
|
|||||||
assert b"secret-test-cookie" not in vault.path.read_bytes()
|
assert b"secret-test-cookie" not in vault.path.read_bytes()
|
||||||
assert "cookie" not in SettingsStore(tmp_path).read()
|
assert "cookie" not in SettingsStore(tmp_path).read()
|
||||||
|
|
||||||
|
|
||||||
|
def test_rate_limiting_does_not_quarantine_a_healthy_proxy(tmp_path):
|
||||||
|
class Limited(Remote):
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__()
|
||||||
|
self.calls = 0
|
||||||
|
def request(self, *args, **kwargs):
|
||||||
|
self.calls += 1
|
||||||
|
if self.calls == 1:
|
||||||
|
return Response({}, status=429)
|
||||||
|
return super().request(*args, **kwargs)
|
||||||
|
_, router, transport = setup_network(tmp_path, remote=Limited())
|
||||||
|
original = router.resolve()
|
||||||
|
assert transport.request("GET", "https://yz.chsi.com.cn/zsml/").json()["routed_through"] == original
|
||||||
|
assert router.failed == set()
|
||||||
|
|
||||||
|
|
||||||
|
def test_browser_cookie_domains_preserve_same_name_sessions(tmp_path):
|
||||||
|
_, _, transport = setup_network(tmp_path)
|
||||||
|
vault = CookieVault(tmp_path)
|
||||||
|
vault.save([{"name": "JSESSIONID", "value": "account-cookie", "domain": "account.chsi.com.cn", "path": "/"}, {"name": "JSESSIONID", "value": "catalog-cookie", "domain": "yz.chsi.com.cn", "path": "/"}])
|
||||||
|
AuthService(transport, vault)
|
||||||
|
cookies = transport.session.cookies
|
||||||
|
assert cookies.get("JSESSIONID", domain="account.chsi.com.cn") == "account-cookie"
|
||||||
|
assert cookies.get("JSESSIONID", domain="yz.chsi.com.cn") == "catalog-cookie"
|
||||||
|
|
||||||
|
|
||||||
|
def test_auth_checks_protected_direction_api_rather_than_public_school_index(tmp_path):
|
||||||
|
class PublicIndex(Remote):
|
||||||
|
def request(self, method, url, **kwargs):
|
||||||
|
if url.endswith('/dws.do'):
|
||||||
|
return Response({'flag': True, 'msg': {'list': []}})
|
||||||
|
return Response({'flag': False, 'msg': '请登录'})
|
||||||
|
_, _, transport = setup_network(tmp_path, remote=PublicIndex())
|
||||||
|
auth = AuthService(transport, CookieVault(tmp_path))
|
||||||
|
assert auth.check()['valid'] is False
|
||||||
|
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ def main():
|
|||||||
expect(page.get_by_role("heading", name="从数据出发,找到适合的院校")).to_be_visible()
|
expect(page.get_by_role("heading", name="从数据出发,找到适合的院校")).to_be_visible()
|
||||||
page.get_by_role("navigation").get_by_role("button", name="院校工作区").click()
|
page.get_by_role("navigation").get_by_role("button", name="院校工作区").click()
|
||||||
page.get_by_role("button", name="导入数据", exact=True).click()
|
page.get_by_role("button", name="导入数据", exact=True).click()
|
||||||
|
expect(page.get_by_role('spinbutton', name='这份目录的招生年份')).to_have_value('')
|
||||||
rows = [{"year": 2026, "school_name": "验收样例甲大学", "school_code": "QA001", "province": "北京", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "计算机学院", "subjects": ["101思想政治理论", "201英语一", "301数学一", "408计算机学科专业基础"], "quota_text": "专业:6(不含推免)", "source": "自动验收样例"}, {"year": 2026, "school_name": "验收样例乙大学", "school_code": "QA002", "province": "上海", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "信息学院", "subjects": ["101思想政治理论", "204英语二", "302数学二", "自命题专业课"], "quota_text": "学院:100", "source": "自动验收样例"}]
|
rows = [{"year": 2026, "school_name": "验收样例甲大学", "school_code": "QA001", "province": "北京", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "计算机学院", "subjects": ["101思想政治理论", "201英语一", "301数学一", "408计算机学科专业基础"], "quota_text": "专业:6(不含推免)", "source": "自动验收样例"}, {"year": 2026, "school_name": "验收样例乙大学", "school_code": "QA002", "province": "上海", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "信息学院", "subjects": ["101思想政治理论", "204英语二", "302数学二", "自命题专业课"], "quota_text": "学院:100", "source": "自动验收样例"}]
|
||||||
scores = [{"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2024, "total": 305, "scope": "major", "note": ""}, {"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2025, "total": 315, "scope": "major", "note": ""}]
|
scores = [{"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2024, "total": 305, "scope": "major", "note": ""}, {"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2025, "total": 315, "scope": "major", "note": ""}]
|
||||||
page.locator('input[type=file]').set_input_files({"name": "acceptance.json", "mimeType": "application/json", "buffer": json.dumps({"catalog": rows, "scores": scores}, ensure_ascii=False).encode()})
|
page.locator('input[type=file]').set_input_files({"name": "acceptance.json", "mimeType": "application/json", "buffer": json.dumps({"catalog": rows, "scores": scores}, ensure_ascii=False).encode()})
|
||||||
|
|||||||
Reference in New Issue
Block a user