fix: preserve data granularity and recover tasks safely
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# 研途 · 考研数据工作台
|
||||
|
||||
将 YAM 的择校工作区和历史考情能力,与 yzw 的全国目录采集、代理池和断点功能整合为独立的本地网页应用。
|
||||
|
||||
当前安装位置:`F:\@Other\kaoyan-studio`。原 YAM、yzw 和 proxy_pool 源码不作修改。
|
||||
|
||||
## 启动
|
||||
|
||||
双击 **启动工作台.cmd**,浏览器打开 **http://127.0.0.1:8765**。窗口关闭后后台服务继续运行;双击 **停止工作台.cmd** 结束本项目服务。
|
||||
|
||||
本机已安装依赖并完成网页构建,可以直接启动。源码包包含已构建网页,复制到另一台 Windows 电脑后需要 Python 3.12+,首次运行会安装锁定的 Python 依赖。重新构建或修改网页需要 Node.js 22+;缺少网页构建时,脚本会安装前端依赖并构建。浏览器登录使用 Microsoft Edge。
|
||||
|
||||
在 PowerShell 中也可以运行:
|
||||
|
||||
```powershell
|
||||
& 'F:\@Other\kaoyan-studio\start.ps1'
|
||||
# 仅启动后台:start.ps1 -NoBrowser
|
||||
# 改端口:start.ps1 -Port 8768
|
||||
```
|
||||
|
||||
## 第一次采集
|
||||
|
||||
1. 在「代理 IP 池」刷新,查看池内 HTTPS 候选和当前路由。默认连接 `http://127.0.0.1:5010`;服务目录为 `C:\Users\Jayson\Desktop\proxy_pool`。
|
||||
2. 在「设置与登录」选择网络策略。优先代理在池为空时允许直连;严格代理没有可用 IP 时等待处理。备用代理每行一个 `IP:端口`。代理池存在候选并不保证该 IP 对所有目标网站可用,连接失败会切换并隔离。
|
||||
3. 点击「打开登录窗口」,在独立 Edge 窗口完成学信网登录和验证码。工作台会检测登录并保存 Cookie。也可粘贴已登录研招网的 Cookie 请求头,或使用账号密码登录。点击「验证目录访问」确认详情接口权限。
|
||||
4. 在「采集任务」选择内容、目录年份、地区、专业和院校关键词。建议先按一个地区、一个专业或一所院校验证,再创建全国任务。专业代码可以从内置目录选择,也可直接输入 6 位代码。
|
||||
5. 查看任务日志。暂停可继续;需要登录、代理或限流解除时,处理后点击恢复;取消保留已有数据。关闭或重启服务会保留可恢复断点。
|
||||
|
||||
研招网学校、专业索引有时可以匿名读取,但方向与科目详情需要登录。程序验证实际业务返回状态,不将 HTTP 200 当作登录成功。目录采集先核对研招页面配置中的年份,避免将当前目录标为历史年份。
|
||||
|
||||
## 数据查询
|
||||
|
||||
「院校工作区」支持院校/专业/院系关键词、年份、省份、专业代码、学习方式、学位类型、院校层次、科目、院系、方向、人数及参考线筛选和排序。可收藏候选、勾选 2–6 条记录对比,点击院校名查看来源、分数线和计划历史。
|
||||
|
||||
每条记录按 **年份 + 院校 + 院系 + 专业 + 方向 + 考试方式 + 科目组合 + 来源** 保存。不同年份、方向及可选科目组合独立保留,相同记录重新采集时更新。人数保留原文:明确「专业」人数可用于筛选;学院、方向、范围或待定口径不会被转换成专业总人数。不同方向和科目组合可能重复展示同一专业人数,不能累加。
|
||||
|
||||
历史分数线区分专业线、学科参考线和门类参考线,匹配院系后才展示;专项计划线独立保留在详情,不作为普通计划的参考线。表格显示不晚于目录年的最近匹配线,空字段显示「未提供」。历年图表按口径、院系、备注和来源分开,不将门类线标为专业线。
|
||||
|
||||
985/211 是历史工程参考名单。双一流优先采用数据源标识,同时补充已知的 2022 年名单;名单来源见 `data/school_levels.json`。学校更名、新名单或未识别名称可能需要更新参考元数据。
|
||||
|
||||
内置 2,255 条专业目录条目和 460 所院校公开索引,来自 YAM 的公共数据文件。**索引并不代表已经采集招生详情**;首次安装的目录和分数线数据库为空。全国采集不限定于这些索引,通过研招网查询全部 31 个省级地区的招生单位。
|
||||
|
||||
## 原项目数据互通
|
||||
|
||||
在工作区点击「导入数据」:
|
||||
|
||||
| 格式 | 用途 | 年份规则 |
|
||||
| --- | --- | --- |
|
||||
| yzw CSV | 原 `export_major_csv.py` 输出的全部字段 | 原文件无年,必须填写实际招生年份 |
|
||||
| YAM SQLite | Python 数据库中的 schools/departments、score_lines、admission_plans | 无年份目录须填写;历史分数/计划保留文件年份 |
|
||||
| YAM JSON 种子 | `data/seeds/yan_zhao_*_all_regions.json` | 只有学校列表时仅导入索引;有方向详情时须提供年份 |
|
||||
| 工作台 JSON | 完整目录和分数线快照 | 保留各条记录年份 |
|
||||
| CSV / XLSX | 工作台导出,或包含相应机器字段/中文表头的数据 | 优先文件年份,否则须明确填写 |
|
||||
|
||||
支持 UTF-8 / UTF-8 BOM / GB18030 CSV。一次最多 100 MB、30 万条。导入只读取所选文件,不打开原软件环境或修改源文件;SQLite 使用只读模式。其他版本 YAM 表结构不同的数据库,请从原软件导出兼容文件再导入。
|
||||
|
||||
可将当前筛选结果导出 CSV、Excel 或完整 JSON。表格导出保留院系/方向代码和字段口径,回导不会凭空增加同一记录。JSON 同时保留相关学校专业的历史分数线;凭据不进入导出。CSV/Excel 对可能被表格软件当作公式的文本加保护前缀。
|
||||
|
||||
## 本地数据与开发
|
||||
|
||||
所有运行数据位于项目 `.runtime`:`studio.db` 保存目录、分数、收藏、任务和断点;`settings.json` 保存非机密配置;`chsi-cookie.enc` 由当前 Windows 用户的 DPAPI 加密。密码不持久化,接口不返回 Cookie。备份数据库时先停止工作台,再复制 `.runtime`;恢复加密 Cookie 需要原 Windows 用户环境,否则重新登录即可。
|
||||
|
||||
服务默认仅监听 `127.0.0.1`,拒绝非本机 Host 和跨来源写入。代理池设置页使用现有 `manage.ps1` 控制本机服务;工作台停止脚本只结束自己管理且进程身份匹配的服务,不停止代理池。
|
||||
|
||||
```powershell
|
||||
# 后端行为测试
|
||||
.\.venv\Scripts\python.exe -m pytest -q
|
||||
# 前端逻辑测试及生产构建
|
||||
cd frontend
|
||||
npm.cmd test
|
||||
npm.cmd run build
|
||||
```
|
||||
|
||||
浏览器验收使用独立的测试数据库,运行方式及结果见 `docs/verification.md`。没有真人登录的自动验收不能替代账号登录后的全国采集验证。
|
||||
|
||||
项目结构:`studio/` 为网络、数据源、存储和 API;`frontend/src/` 为 React 网页;`data/` 为公开参考元数据;`tests/` 为后端行为测试。来源与许可证见 `THIRD_PARTY_NOTICES.md`。
|
||||
@@ -0,0 +1,13 @@
|
||||
# 来源与第三方声明
|
||||
|
||||
本项目独立实现采集器、存储、任务和网页服务,参考以下项目的公开接口及字段设计,不直接运行原仓库模块。
|
||||
|
||||
- [YAM / 研喵](https://github.com/C-Y-T-258/yam):参考全国目录、掌上考研历史分数/计划接口及择校工作区设计。`data/majors.json` 与 `data/schools.json` 整理自其公共专业目录和院校种子,仅保留公开参考字段,未携带签名、Cookie 或账号。原 `pyproject.toml` 声明 MIT,作者 YAM Contributors;本地原仓库没有单独的 LICENSE 文件。
|
||||
- [yzw](https://github.com/freecho/yzw):参考学校/专业/方向 API、CSV 字段和代理策略。学校层次参考数据以原文件为基础,补充学信网发布的教育部名单、名称别名,并修正中国中医科学院的 211 误标。原项目 MIT,Copyright (c) 2025 freecho,完整许可副本在 `licenses/yzw-MIT.txt`。
|
||||
- [proxy_pool](https://github.com/jhao104/proxy_pool):使用用户已有的 Flask API 和本机 `manage.ps1`,不内嵌或修改代理池源码。
|
||||
|
||||
参考名单来源:[985 名单](https://www.moe.gov.cn/srcsite/A22/s7065/200612/t20061206_128833.html)、[学信网发布的教育部 211 名单](https://gaokao.chsi.com.cn/gkxx/gxmd/211gc.shtml)、[第二轮双一流通知](https://www.moe.gov.cn/srcsite/A22/s7065/202202/t20220211_598710.html)。985/211 用于历史工程参考;双一流标识优先采用采集来源,静态名单为已知参考子集。
|
||||
|
||||
Python / JavaScript 依赖保留其自身许可证。运行环境包括 FastAPI、Starlette、Uvicorn、requests、Beautiful Soup、openpyxl、Playwright、React、Vite、TypeScript、Lucide;版本记录在 `requirements.lock` 和 `frontend/package-lock.json`,许可证原文随安装包提供。
|
||||
|
||||
采集数据的准确性、完整性和开放年份取决于原网站,应用保留来源及口径,并明确标记未知字段。内置索引不等同于已经采集的招生数据。
|
||||
@@ -0,0 +1,294 @@
|
||||
{
|
||||
"985": [
|
||||
"上海交通大学",
|
||||
"东北大学",
|
||||
"东南大学",
|
||||
"中南大学",
|
||||
"中国人民大学",
|
||||
"中国农业大学",
|
||||
"中国海洋大学",
|
||||
"中国科学技术大学",
|
||||
"中央民族大学",
|
||||
"中山大学",
|
||||
"兰州大学",
|
||||
"北京大学",
|
||||
"北京师范大学",
|
||||
"北京理工大学",
|
||||
"北京航空航天大学",
|
||||
"华东师范大学",
|
||||
"华中科技大学",
|
||||
"华南理工大学",
|
||||
"南京大学",
|
||||
"南开大学",
|
||||
"厦门大学",
|
||||
"吉林大学",
|
||||
"同济大学",
|
||||
"哈尔滨工业大学",
|
||||
"四川大学",
|
||||
"国防科学技术大学",
|
||||
"国防科技大学",
|
||||
"复旦大学",
|
||||
"大连理工大学",
|
||||
"天津大学",
|
||||
"山东大学",
|
||||
"武汉大学",
|
||||
"浙江大学",
|
||||
"清华大学",
|
||||
"湖南大学",
|
||||
"电子科技大学",
|
||||
"西北农林科技大学",
|
||||
"西北工业大学",
|
||||
"西安交通大学",
|
||||
"重庆大学"
|
||||
],
|
||||
"211": [
|
||||
"上海交通大学",
|
||||
"上海外国语大学",
|
||||
"上海大学",
|
||||
"上海财经大学",
|
||||
"东北农业大学",
|
||||
"东北大学",
|
||||
"东北师范大学",
|
||||
"东北林业大学",
|
||||
"东华大学",
|
||||
"东南大学",
|
||||
"中南大学",
|
||||
"中南财经政法大学",
|
||||
"中国人民大学",
|
||||
"中国传媒大学",
|
||||
"中国农业大学",
|
||||
"中国地质大学(北京)",
|
||||
"中国地质大学(武汉)",
|
||||
"中国政法大学",
|
||||
"中国海洋大学",
|
||||
"中国石油大学(北京)",
|
||||
"中国石油大学(华东)",
|
||||
"中国矿业大学",
|
||||
"中国矿业大学(北京)",
|
||||
"中国矿业大学(徐州)",
|
||||
"中国科学技术大学",
|
||||
"中国药科大学",
|
||||
"中央民族大学",
|
||||
"中央财经大学",
|
||||
"中央音乐学院",
|
||||
"中山大学",
|
||||
"云南大学",
|
||||
"兰州大学",
|
||||
"内蒙古大学",
|
||||
"北京中医药大学",
|
||||
"北京交通大学",
|
||||
"北京体育大学",
|
||||
"北京化工大学",
|
||||
"北京外国语大学",
|
||||
"北京大学",
|
||||
"北京工业大学",
|
||||
"北京师范大学",
|
||||
"北京林业大学",
|
||||
"北京理工大学",
|
||||
"北京科技大学",
|
||||
"北京航空航天大学",
|
||||
"北京邮电大学",
|
||||
"华东师范大学",
|
||||
"华东理工大学",
|
||||
"华中农业大学",
|
||||
"华中师范大学",
|
||||
"华中科技大学",
|
||||
"华北电力大学",
|
||||
"华北电力大学(保定)",
|
||||
"华北电力大学(北京)",
|
||||
"华南师范大学",
|
||||
"华南理工大学",
|
||||
"南京农业大学",
|
||||
"南京大学",
|
||||
"南京师范大学",
|
||||
"南京理工大学",
|
||||
"南京航空航天大学",
|
||||
"南开大学",
|
||||
"南昌大学",
|
||||
"厦门大学",
|
||||
"合肥工业大学",
|
||||
"吉林大学",
|
||||
"同济大学",
|
||||
"哈尔滨工业大学",
|
||||
"哈尔滨工程大学",
|
||||
"四川农业大学",
|
||||
"四川大学",
|
||||
"国防科学技术大学",
|
||||
"国防科技大学",
|
||||
"复旦大学",
|
||||
"大连海事大学",
|
||||
"大连理工大学",
|
||||
"天津医科大学",
|
||||
"天津大学",
|
||||
"太原理工大学",
|
||||
"宁夏大学",
|
||||
"安徽大学",
|
||||
"对外经济贸易大学",
|
||||
"山东大学",
|
||||
"广西大学",
|
||||
"延边大学",
|
||||
"新疆大学",
|
||||
"暨南大学",
|
||||
"武汉大学",
|
||||
"武汉理工大学",
|
||||
"江南大学",
|
||||
"河北工业大学",
|
||||
"河海大学",
|
||||
"浙江大学",
|
||||
"海军军医大学",
|
||||
"海南大学",
|
||||
"清华大学",
|
||||
"湖南大学",
|
||||
"湖南师范大学",
|
||||
"电子科技大学",
|
||||
"石河子大学",
|
||||
"福州大学",
|
||||
"空军军医大学",
|
||||
"第二军医大学",
|
||||
"第四军医大学",
|
||||
"苏州大学",
|
||||
"西北农林科技大学",
|
||||
"西北大学",
|
||||
"西北工业大学",
|
||||
"西南交通大学",
|
||||
"西南大学",
|
||||
"西南财经大学",
|
||||
"西安交通大学",
|
||||
"西安电子科技大学",
|
||||
"西藏大学",
|
||||
"贵州大学",
|
||||
"辽宁大学",
|
||||
"郑州大学",
|
||||
"重庆大学",
|
||||
"长安大学",
|
||||
"陕西师范大学",
|
||||
"青海大学"
|
||||
],
|
||||
"double_first_class_2022_known": [
|
||||
"上海中医药大学",
|
||||
"上海交通大学",
|
||||
"上海外国语大学",
|
||||
"上海大学",
|
||||
"上海财经大学",
|
||||
"东北农业大学",
|
||||
"东北大学",
|
||||
"东北师范大学",
|
||||
"东北林业大学",
|
||||
"东华大学",
|
||||
"东南大学",
|
||||
"中南大学",
|
||||
"中南财经政法大学",
|
||||
"中国人民大学",
|
||||
"中国传媒大学",
|
||||
"中国农业大学",
|
||||
"中国地质大学(北京)",
|
||||
"中国地质大学(武汉)",
|
||||
"中国政法大学",
|
||||
"中国海洋大学",
|
||||
"中国石油大学(北京)",
|
||||
"中国石油大学(华东)",
|
||||
"中国矿业大学",
|
||||
"中国矿业大学(北京)",
|
||||
"中国科学技术大学",
|
||||
"中国药科大学",
|
||||
"中央民族大学",
|
||||
"中央财经大学",
|
||||
"中央音乐学院",
|
||||
"中山大学",
|
||||
"云南大学",
|
||||
"兰州大学",
|
||||
"内蒙古大学",
|
||||
"北京中医药大学",
|
||||
"北京交通大学",
|
||||
"北京体育大学",
|
||||
"北京化工大学",
|
||||
"北京协和医学院",
|
||||
"北京外国语大学",
|
||||
"北京大学",
|
||||
"北京工业大学",
|
||||
"北京师范大学",
|
||||
"北京林业大学",
|
||||
"北京理工大学",
|
||||
"北京科技大学",
|
||||
"北京航空航天大学",
|
||||
"北京邮电大学",
|
||||
"华东师范大学",
|
||||
"华东理工大学",
|
||||
"华中农业大学",
|
||||
"华中师范大学",
|
||||
"华中科技大学",
|
||||
"华北电力大学",
|
||||
"华南师范大学",
|
||||
"华南理工大学",
|
||||
"南京农业大学",
|
||||
"南京医科大学",
|
||||
"南京大学",
|
||||
"南京师范大学",
|
||||
"南京理工大学",
|
||||
"南京航空航天大学",
|
||||
"南开大学",
|
||||
"南方科技大学",
|
||||
"南昌大学",
|
||||
"厦门大学",
|
||||
"合肥工业大学",
|
||||
"吉林大学",
|
||||
"同济大学",
|
||||
"哈尔滨工业大学",
|
||||
"哈尔滨工程大学",
|
||||
"四川大学",
|
||||
"国防科技大学",
|
||||
"复旦大学",
|
||||
"大连海事大学",
|
||||
"大连理工大学",
|
||||
"天津中医药大学",
|
||||
"天津医科大学",
|
||||
"天津大学",
|
||||
"天津工业大学",
|
||||
"太原理工大学",
|
||||
"宁夏大学",
|
||||
"安徽大学",
|
||||
"对外经济贸易大学",
|
||||
"山东大学",
|
||||
"广州中医药大学",
|
||||
"广西大学",
|
||||
"延边大学",
|
||||
"成都理工大学",
|
||||
"新疆大学",
|
||||
"暨南大学",
|
||||
"武汉大学",
|
||||
"武汉理工大学",
|
||||
"江南大学",
|
||||
"河北工业大学",
|
||||
"河海大学",
|
||||
"浙江大学",
|
||||
"海南大学",
|
||||
"清华大学",
|
||||
"湖南大学",
|
||||
"湖南师范大学",
|
||||
"电子科技大学",
|
||||
"石河子大学",
|
||||
"福州大学",
|
||||
"苏州大学",
|
||||
"西北农林科技大学",
|
||||
"西北大学",
|
||||
"西北工业大学",
|
||||
"西南交通大学",
|
||||
"西南大学",
|
||||
"西南财经大学",
|
||||
"西安交通大学",
|
||||
"西安电子科技大学",
|
||||
"西藏大学",
|
||||
"贵州大学",
|
||||
"辽宁大学",
|
||||
"郑州大学",
|
||||
"重庆大学",
|
||||
"长安大学",
|
||||
"青海大学"
|
||||
],
|
||||
"sources": [
|
||||
"https://www.moe.gov.cn/srcsite/A22/s7065/200612/t20061206_128833.html",
|
||||
"https://gaokao.chsi.com.cn/gkxx/gxmd/211gc.shtml",
|
||||
"https://www.moe.gov.cn/srcsite/A22/s7065/202202/t20220211_598710.html"
|
||||
],
|
||||
"note": "历史工程名单与已知名称别名,用于参考筛选。211以原yzw名单为基础并与学信网教育部名单核对补充、排除中国中医科学院误标。双一流优先实时研招网syl或导入字段;静态2022名单只是已知子集。"
|
||||
}
|
||||
+4
-2
@@ -7,8 +7,10 @@ Task 1: complete — 5 tests green; cross-year records, alternatives, conservati
|
||||
Task 2: complete — 5 tests green; shared proxy, strict route, fallback, rotation, DPAPI vault.
|
||||
Task 3: complete — 5 tests green; pagination, business login status, year mismatch, exact campus, pause/resume.
|
||||
Task 4: complete — 5 tests green; file import/export, YAM SQLite, favorites, origin guard, formula-safe exports.
|
||||
Task 5: pending
|
||||
Task 6: pending
|
||||
Task 5: complete — 3 frontend logic tests, production build, full Edge acceptance including mobile passed.
|
||||
Task 6: final packaging — launch/reuse/stop/restart verified; README/notices ready; independent review findings fixed with regression tests.
|
||||
|
||||
Public index: 2,255 major entries and 460 unique schools extracted from YAM public data files. Seed index contains no admissions snapshots, cookies or signed request tokens.
|
||||
|
||||
Final verification: 35 backend tests passed; 3 frontend tests passed; build exit 0; browser QA PASS. Details: docs/verification.md.
|
||||
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# 验证记录
|
||||
|
||||
2026-10-11,Windows、本地独立虚拟环境,Python 3.12.13 / Node 24.16。
|
||||
|
||||
- 后端:`python -m pytest -q`,35 passed。Starlette TestClient 提示一条 httpx 迁移 warning,测试退出 0,生产服务不使用 TestClient。
|
||||
- 前端:`npm test`,3 passed;`npm run build`,TypeScript 检查与 Vite 生产构建退出 0。
|
||||
- Edge 实际浏览器:独立端口 8766、独立 `.runtime/qa/db`,验证导入→科目筛选→收藏→对比→历史趋势→Excel 下载;任务/代理/设置页面;设置保存;390px 手机宽度无页面溢出;无 pageerror。测试数据均标明「自动验收样例」,没有写入正式数据库。
|
||||
- 启动脚本:Windows PowerShell 首启、重复启动、停止、重新启动均退出 0;正式服务在 127.0.0.1:8765,健康检查应用身份和项目路径匹配。
|
||||
- 真实数据源:proxy_pool `/count/` 正常响应;研招页面 `curYear=2027` 可核实;匿名北京大学学校及 081200 专业索引可返回,方向接口要求登录,按 LoginRequired 处理;掌上考研精确匹配北京大学,2025 年 081200 查询返回 1 条门类参考线和 2 条历史计划。真实探测只读取少量公开数据,不写入正式目录。
|
||||
- 独立只读审查:复现并修复专项线覆盖、同名跨域 Cookie、任务失败恢复竞态、服务退出被误取消、文件往返丢失身份字段、学院人数误升格、YAM 种子误识别、院校关键词范围、参考书 JSON 字符串、业务限流等问题,补充相应回归。
|
||||
|
||||
自动验收未使用真人学信网账号,未对全部院校进行实网抓取。登录后的全国采集能力取决于账号状态、代理可用性、开放目录年份和数据源限流。
|
||||
|
||||
复跑浏览器验收(只使用测试库):
|
||||
|
||||
```powershell
|
||||
.\.venv\Scripts\python.exe -m studio --port 8766 --data-dir .runtime\qa\db
|
||||
# 在另一个终端运行
|
||||
.\.venv\Scripts\python.exe tools\browser_qa.py
|
||||
```
|
||||
|
||||
正式服务不会自动开始采集,需要用户在网页创建任务。
|
||||
+16
-1
@@ -1,2 +1,17 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN"><head><meta charset="UTF-8" /><meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="theme-color" content="#132b3a" /><meta name="description" content="本地考研数据工作台:全国目录、历年考情、代理池与择校对比。" /><title>研途 · 考研数据工作台</title></head><body><div id="root"></div><script type="module" src="/src/main.tsx"></script></body></html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="theme-color" content="#132b3a" />
|
||||
<meta
|
||||
name="description"
|
||||
content="本地考研数据工作台:全国目录、历年考情、代理池与择校对比。"
|
||||
/>
|
||||
<title>研途 · 考研数据工作台</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/src/main.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
+1997
-74
File diff suppressed because one or more lines are too long
+21
-5
@@ -1,9 +1,25 @@
|
||||
export async function api<T>(path: string, options: RequestInit = {}): Promise<T> {
|
||||
const response = await fetch('/api' + path, { ...options, headers: options.body instanceof FormData ? options.headers : { 'Content-Type': 'application/json', ...options.headers } });
|
||||
export async function api<T>(
|
||||
path: string,
|
||||
options: RequestInit = {},
|
||||
): Promise<T> {
|
||||
const response = await fetch("/api" + path, {
|
||||
...options,
|
||||
headers:
|
||||
options.body instanceof FormData
|
||||
? options.headers
|
||||
: { "Content-Type": "application/json", ...options.headers },
|
||||
});
|
||||
if (!response.ok) {
|
||||
const body = await response.json().catch(() => ({ detail: `请求失败 (${response.status})` }));
|
||||
throw new Error(typeof body.detail === 'string' ? body.detail : `参数无效 (${response.status})`);
|
||||
const body = await response
|
||||
.json()
|
||||
.catch(() => ({ detail: `请求失败 (${response.status})` }));
|
||||
throw new Error(
|
||||
typeof body.detail === "string"
|
||||
? body.detail
|
||||
: `参数无效 (${response.status})`,
|
||||
);
|
||||
}
|
||||
return response.json();
|
||||
}
|
||||
export const post = <T,>(path: string, body: unknown = {}) => api<T>(path, { method: 'POST', body: JSON.stringify(body) });
|
||||
export const post = <T>(path: string, body: unknown = {}) =>
|
||||
api<T>(path, { method: "POST", body: JSON.stringify(body) });
|
||||
|
||||
+18
-12
@@ -1,17 +1,23 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { display, scopeName, scoreSeries, queryString } from './format';
|
||||
describe('unknown values and comparable history', () => {
|
||||
it('preserves zero and marks unknown', () => {
|
||||
expect(display(0)).toBe('0');
|
||||
expect(display(null)).toBe('未提供');
|
||||
expect(scopeName('category')).toBe('门类参考线');
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { display, scopeName, scoreSeries, queryString } from "./format";
|
||||
describe("unknown values and comparable history", () => {
|
||||
it("preserves zero and marks unknown", () => {
|
||||
expect(display(0)).toBe("0");
|
||||
expect(display(null)).toBe("未提供");
|
||||
expect(scopeName("category")).toBe("门类参考线");
|
||||
});
|
||||
it('keeps different score scopes separate and omits unknown totals', () => {
|
||||
const result = scoreSeries([{ year: 2024, total: 310, scope: 'major' }, { year: 2025, total: 300, scope: 'category' }, { year: 2026, total: null, scope: 'major' }]);
|
||||
it("keeps different score scopes separate and omits unknown totals", () => {
|
||||
const result = scoreSeries([
|
||||
{ year: 2024, total: 310, scope: "major" },
|
||||
{ year: 2025, total: 300, scope: "category" },
|
||||
{ year: 2026, total: null, scope: "major" },
|
||||
]);
|
||||
expect(result).toHaveLength(2);
|
||||
expect(result.flatMap(r => r.points)).toHaveLength(2);
|
||||
expect(result.flatMap((r) => r.points)).toHaveLength(2);
|
||||
});
|
||||
it('encodes filters and omits empty values', () => {
|
||||
expect(queryString({ keyword: '北京 大学', year: '', favorites: false })).toBe('keyword=%E5%8C%97%E4%BA%AC+%E5%A4%A7%E5%AD%A6');
|
||||
it("encodes filters and omits empty values", () => {
|
||||
expect(
|
||||
queryString({ keyword: "北京 大学", year: "", favorites: false }),
|
||||
).toBe("keyword=%E5%8C%97%E4%BA%AC+%E5%A4%A7%E5%AD%A6");
|
||||
});
|
||||
});
|
||||
|
||||
+62
-11
@@ -1,16 +1,67 @@
|
||||
import type { Score, Filters } from './types';
|
||||
export const display = (value: unknown): string => value === null || value === undefined || value === '' ? '未提供' : String(value);
|
||||
export const scopeName = (scope?: string | null) => ({ major: '专业线', discipline: '学科参考线', category: '门类参考线', unknown: '口径待核实' }[scope || ''] || '未提供');
|
||||
export const modeName = (mode: string) => ({ direct: '直接连接', prefer_proxy: '优先代理', proxy_only: '严格代理', catalog: '全国招生目录', scores: '历年考情', combined: '目录 + 历年考情' }[mode] || mode);
|
||||
export const statusName = (status: string) => ({ queued: '等待开始', running: '采集中', paused: '已暂停', waiting_login: '等待登录', waiting_proxy: '等待代理', completed: '已完成', partial: '部分失败', failed: '采集失败', cancelled: '已取消' }[status] || status);
|
||||
export const queryString = (filters: Filters) => new URLSearchParams(Object.entries(filters).filter(([, v]) => v !== '' && v !== false).map(([k, v]) => [k, String(v)])).toString();
|
||||
import type { Score, Filters } from "./types";
|
||||
export const display = (value: unknown): string =>
|
||||
value === null || value === undefined || value === ""
|
||||
? "未提供"
|
||||
: String(value);
|
||||
export const scopeName = (scope?: string | null) =>
|
||||
({
|
||||
major: "专业线",
|
||||
discipline: "学科参考线",
|
||||
category: "门类参考线",
|
||||
unknown: "口径待核实",
|
||||
})[scope || ""] || "未提供";
|
||||
export const modeName = (mode: string) =>
|
||||
({
|
||||
direct: "直接连接",
|
||||
prefer_proxy: "优先代理",
|
||||
proxy_only: "严格代理",
|
||||
catalog: "全国招生目录",
|
||||
scores: "历年考情",
|
||||
combined: "目录 + 历年考情",
|
||||
})[mode] || mode;
|
||||
export const statusName = (status: string) =>
|
||||
({
|
||||
queued: "等待开始",
|
||||
running: "采集中",
|
||||
paused: "已暂停",
|
||||
waiting_login: "等待登录",
|
||||
waiting_proxy: "等待代理",
|
||||
waiting_source: "等待限流解除",
|
||||
completed: "已完成",
|
||||
partial: "部分失败",
|
||||
failed: "采集失败",
|
||||
cancelled: "已取消",
|
||||
})[status] || status;
|
||||
export const queryString = (filters: Filters) =>
|
||||
new URLSearchParams(
|
||||
Object.entries(filters)
|
||||
.filter(([, v]) => v !== "" && v !== false)
|
||||
.map(([k, v]) => [k, String(v)]),
|
||||
).toString();
|
||||
export function scoreSeries(scores: Score[]) {
|
||||
const grouped = new Map<string, { label: string; points: { year: number; total: number }[] }>();
|
||||
scores.filter(s => s.total !== null).forEach(s => {
|
||||
const key = [s.scope, s.raw_code || '', s.department_name || '', s.source || '', s.note || ''].join('|');
|
||||
const group = grouped.get(key) || { label: scopeName(s.scope) + (s.note ? ` · ${s.note}` : ''), points: [] };
|
||||
const grouped = new Map<
|
||||
string,
|
||||
{ label: string; points: { year: number; total: number }[] }
|
||||
>();
|
||||
scores
|
||||
.filter((s) => s.total !== null)
|
||||
.forEach((s) => {
|
||||
const key = [
|
||||
s.scope,
|
||||
s.raw_code || "",
|
||||
s.department_name || "",
|
||||
s.source || "",
|
||||
s.note || "",
|
||||
].join("|");
|
||||
const group = grouped.get(key) || {
|
||||
label: scopeName(s.scope) + (s.note ? ` · ${s.note}` : ""),
|
||||
points: [],
|
||||
};
|
||||
group.points.push({ year: s.year, total: s.total! });
|
||||
grouped.set(key, group);
|
||||
});
|
||||
return [...grouped.values()].map(g => ({ ...g, points: g.points.sort((a, b) => a.year - b.year) }));
|
||||
return [...grouped.values()].map((g) => ({
|
||||
...g,
|
||||
points: g.points.sort((a, b) => a.year - b.year),
|
||||
}));
|
||||
}
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import { StrictMode } from 'react';
|
||||
import { createRoot } from 'react-dom/client';
|
||||
import App from './App';
|
||||
import './style.css';
|
||||
createRoot(document.getElementById('root')!).render(<StrictMode><App /></StrictMode>);
|
||||
import { StrictMode } from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
import App from "./App";
|
||||
import "./style.css";
|
||||
createRoot(document.getElementById("root")!).render(
|
||||
<StrictMode>
|
||||
<App />
|
||||
</StrictMode>,
|
||||
);
|
||||
|
||||
+1529
-243
File diff suppressed because it is too large
Load Diff
+113
-9
@@ -1,12 +1,116 @@
|
||||
export type Row = { id: number; year: number; school_name: string; school_code: string; province: string; level: string; major_code: string; major_name: string; department_name: string; department_code: string; direction: string; subjects: string[]; quota: number | null; quota_text: string; quota_scope: string; degree_type: string; study_mode: string; exam_type: string; advisor: string; special_plans: string[]; remarks: string; source: string; source_url: string; fetched_at: string; year_verified: boolean; favorite: boolean; score_total: number | null; score_year: number | null; score_scope: string | null; reference_books?: string[] };
|
||||
export type Score = { year: number; total: number | null; scope: string; raw_code?: string; department_name?: string; source?: string; note?: string; politics?: number | null; english?: number | null; special_one?: number | null; special_two?: number | null };
|
||||
export type Row = {
|
||||
id: number;
|
||||
year: number;
|
||||
school_name: string;
|
||||
school_code: string;
|
||||
province: string;
|
||||
level: string;
|
||||
major_code: string;
|
||||
major_name: string;
|
||||
department_name: string;
|
||||
department_code: string;
|
||||
direction: string;
|
||||
subjects: string[];
|
||||
quota: number | null;
|
||||
quota_text: string;
|
||||
quota_scope: string;
|
||||
degree_type: string;
|
||||
study_mode: string;
|
||||
exam_type: string;
|
||||
advisor: string;
|
||||
special_plans: string[];
|
||||
remarks: string;
|
||||
source: string;
|
||||
source_url: string;
|
||||
fetched_at: string;
|
||||
year_verified: boolean;
|
||||
favorite: boolean;
|
||||
score_total: number | null;
|
||||
score_year: number | null;
|
||||
score_scope: string | null;
|
||||
reference_books?: string[];
|
||||
};
|
||||
export type Score = {
|
||||
year: number;
|
||||
total: number | null;
|
||||
scope: string;
|
||||
raw_code?: string;
|
||||
department_name?: string;
|
||||
source?: string;
|
||||
note?: string;
|
||||
politics?: number | null;
|
||||
english?: number | null;
|
||||
special_one?: number | null;
|
||||
special_two?: number | null;
|
||||
};
|
||||
export type History = { record: Row; scores: Score[]; plans: Row[] };
|
||||
export type Job = { id: string; params: { mode: string; year: number; major_code?: string; provinces: string[] }; status: string; message: string; created_at: string; updated_at: string; records: number; errors: number; completed_units: number };
|
||||
export type Log = { id: number; created_at: string; level: string; message: string };
|
||||
export type Job = {
|
||||
id: string;
|
||||
params: {
|
||||
mode: string;
|
||||
year: number;
|
||||
major_code?: string;
|
||||
provinces: string[];
|
||||
};
|
||||
status: string;
|
||||
message: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
records: number;
|
||||
errors: number;
|
||||
completed_units: number;
|
||||
};
|
||||
export type Log = {
|
||||
id: number;
|
||||
created_at: string;
|
||||
level: string;
|
||||
message: string;
|
||||
};
|
||||
export type Province = { code: string; name: string };
|
||||
export type Major = { code: string; name: string; degree_type: string; category: string; discipline: string };
|
||||
export type Overview = { records: number; schools: number; majors: number; favorites: number; indexed_schools: number; scores: number; years: number[]; provinces: { province: string; count: number }[]; provinces_list: Province[]; sources: { name: string; role: string; url: string }[] };
|
||||
export type Config = { network_mode: string; pool_url: string; pool_project: string; backup_proxies: string[]; request_interval: number; timeout: number; retries: number };
|
||||
export type Proxy = { mode: string; route: string; active_proxy: string | null; fallback: boolean; counts: { count?: number; http_type?: { https: number; http: number } }; available: number; quarantined: number; error: string; proxies: { url: string; failed: boolean; active: boolean }[] };
|
||||
export type Auth = { saved: boolean; valid: boolean; message: string; browser_state: string; browser_message: string };
|
||||
export type Major = {
|
||||
code: string;
|
||||
name: string;
|
||||
degree_type: string;
|
||||
category: string;
|
||||
discipline: string;
|
||||
};
|
||||
export type Overview = {
|
||||
records: number;
|
||||
schools: number;
|
||||
majors: number;
|
||||
favorites: number;
|
||||
indexed_schools: number;
|
||||
scores: number;
|
||||
years: number[];
|
||||
provinces: { province: string; count: number }[];
|
||||
provinces_list: Province[];
|
||||
sources: { name: string; role: string; url: string }[];
|
||||
};
|
||||
export type Config = {
|
||||
network_mode: string;
|
||||
pool_url: string;
|
||||
pool_project: string;
|
||||
backup_proxies: string[];
|
||||
request_interval: number;
|
||||
timeout: number;
|
||||
retries: number;
|
||||
};
|
||||
export type Proxy = {
|
||||
mode: string;
|
||||
route: string;
|
||||
active_proxy: string | null;
|
||||
fallback: boolean;
|
||||
counts: { count?: number; http_type?: { https: number; http: number } };
|
||||
available: number;
|
||||
quarantined: number;
|
||||
error: string;
|
||||
proxies: { url: string; failed: boolean; active: boolean }[];
|
||||
};
|
||||
export type Auth = {
|
||||
saved: boolean;
|
||||
valid: boolean;
|
||||
message: string;
|
||||
browser_state: string;
|
||||
browser_message: string;
|
||||
};
|
||||
export type Filters = Record<string, string | number | boolean>;
|
||||
|
||||
+14
-2
@@ -1,2 +1,14 @@
|
||||
{"compilerOptions":{"target":"ES2022","lib":["ES2022","DOM","DOM.Iterable"],"module":"ESNext","moduleResolution":"bundler","jsx":"react-jsx","strict":true,"skipLibCheck":true,"noEmit":true,"allowImportingTsExtensions":true},"include":["src","vite.config.ts"]}
|
||||
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"lib": ["ES2022", "DOM", "DOM.Iterable"],
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"jsx": "react-jsx",
|
||||
"strict": true,
|
||||
"skipLibCheck": true,
|
||||
"noEmit": true,
|
||||
"allowImportingTsExtensions": true
|
||||
},
|
||||
"include": ["src", "vite.config.ts"]
|
||||
}
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
import { defineConfig } from 'vite';
|
||||
import react from '@vitejs/plugin-react';
|
||||
export default defineConfig({plugins:[react()],server:{proxy:{'/api':'http://127.0.0.1:8765'}},build:{sourcemap:false}});
|
||||
|
||||
import { defineConfig } from "vite";
|
||||
import react from "@vitejs/plugin-react";
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
server: { proxy: { "/api": "http://127.0.0.1:8765" } },
|
||||
build: { sourcemap: false },
|
||||
});
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 freecho
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
+6
-3
@@ -13,7 +13,7 @@ from studio.database import Database
|
||||
from studio.domain import PROVINCES
|
||||
from studio.imports import import_file, export_sheet
|
||||
from studio.jobs import JobManager
|
||||
from studio.network import ProxyRouter, Transport, ProxyUnavailable, RemoteUnavailable
|
||||
from studio.network import ProxyRouter, Transport, ProxyUnavailable, RemoteUnavailable, RateLimited
|
||||
from studio.settings import SettingsStore
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
@@ -60,6 +60,7 @@ def create_app(data_dir=None):
|
||||
|
||||
@app.exception_handler(ProxyUnavailable)
|
||||
@app.exception_handler(RemoteUnavailable)
|
||||
@app.exception_handler(RateLimited)
|
||||
async def remote_error(request, exc):
|
||||
return JSONResponse({"detail": str(exc)}, status_code=503)
|
||||
|
||||
@@ -188,10 +189,12 @@ def create_app(data_dir=None):
|
||||
if total > 100 * 1024 * 1024:
|
||||
raise ValueError("一次最多上传 100 MB,请拆分文件")
|
||||
stream.write(chunk)
|
||||
records, scores = import_file(temporary, year)
|
||||
result = import_file(temporary, year)
|
||||
records, scores = result
|
||||
db.upsert_catalog(records)
|
||||
db.upsert_scores(scores)
|
||||
return {"records": len(records), "scores": len(scores), "message": "导入完成,相同记录按年份更新"}
|
||||
db.index_schools(result.schools)
|
||||
return {"records": len(records), "scores": len(scores), "indexed_schools": len(result.schools), "message": "导入完成,相同记录按年份更新;无详情的 YAM 种子只作为院校索引"}
|
||||
except (UnicodeError, json.JSONDecodeError, KeyError) as exc:
|
||||
raise ValueError(f"文件编码或结构无法识别({type(exc).__name__})") from None
|
||||
finally:
|
||||
|
||||
+17
-6
@@ -60,14 +60,24 @@ class AuthService:
|
||||
self.browser_thread = None
|
||||
try:
|
||||
self._load(self.vault.load())
|
||||
except RuntimeError:
|
||||
except (RuntimeError, OSError, ValueError):
|
||||
self.message = "保存的 Cookie 无法解密,请重新登录"
|
||||
|
||||
def _load(self, cookies):
|
||||
if isinstance(cookies, dict):
|
||||
cookies = [{"name": name, "value": value, "domain": ".chsi.com.cn", "path": "/"} for name, value in cookies.items()]
|
||||
with self.transport.lock:
|
||||
self.transport.session.cookies.clear()
|
||||
for name, value in cookies.items():
|
||||
self.transport.session.cookies.set(name, value, domain=".chsi.com.cn", path="/")
|
||||
for cookie in cookies:
|
||||
domain = str(cookie.get("domain", ".chsi.com.cn"))
|
||||
if domain.lstrip(".") != "chsi.com.cn" and not domain.endswith(".chsi.com.cn"):
|
||||
continue
|
||||
expires = cookie.get("expires")
|
||||
self.transport.session.cookies.set(cookie["name"], cookie["value"], domain=domain, path=cookie.get("path", "/"), secure=bool(cookie.get("secure")), expires=int(expires) if expires and expires > 0 else None)
|
||||
|
||||
def _snapshot(self):
|
||||
with self.transport.lock:
|
||||
return [{"name": c.name, "value": c.value, "domain": c.domain, "path": c.path, "secure": c.secure, "expires": c.expires} for c in self.transport.session.cookies]
|
||||
|
||||
def status(self):
|
||||
return {"saved": self.vault.path.exists(), "valid": self.valid, "message": self.message, "browser_state": self.browser_state, "browser_message": self.browser_message}
|
||||
@@ -91,7 +101,8 @@ class AuthService:
|
||||
return self.status()
|
||||
|
||||
def check(self):
|
||||
response = self.transport.request("POST", "https://yz.chsi.com.cn/zsml/rs/dws.do", data={"ssdm": "11", "curPage": 1, "start": 0, "pageSize": 1, "dwlxs[0]": "all"})
|
||||
# The school/major indexes can be public while direction details require login.
|
||||
response = self.transport.request("POST", "https://yz.chsi.com.cn/zsml/rs/yjfxs.do", data={"dwdm": "10001", "zydm": "081200", "zymc": "计算机科学与技术", "curPage": 1, "start": 0, "pageSize": 1, "dwlxs[0]": "all"})
|
||||
try:
|
||||
body = response.json()
|
||||
except ValueError:
|
||||
@@ -101,7 +112,7 @@ class AuthService:
|
||||
self.valid = body.get("flag") is True and isinstance(body.get("msg"), dict)
|
||||
self.message = "研招目录接口验证成功" if self.valid else "登录未生效或数据源限流,请在浏览器完成登录后重试"
|
||||
if self.valid:
|
||||
self.vault.save(self.transport.session.cookies.get_dict())
|
||||
self.vault.save(self._snapshot())
|
||||
return self.status()
|
||||
|
||||
def login(self, username, password):
|
||||
@@ -144,7 +155,7 @@ class AuthService:
|
||||
while time.monotonic() < deadline:
|
||||
page.wait_for_timeout(2000)
|
||||
if "yz.chsi.com.cn" in page.url:
|
||||
cookies = {c["name"]: c["value"] for c in context.cookies() if "chsi.com.cn" in c["domain"]}
|
||||
cookies = [c for c in context.cookies() if c["domain"].lstrip(".") == "chsi.com.cn" or c["domain"].endswith(".chsi.com.cn")]
|
||||
if cookies:
|
||||
self._load(cookies)
|
||||
if self.check()["valid"]:
|
||||
|
||||
+20
-5
@@ -2,7 +2,7 @@
|
||||
import json
|
||||
import re
|
||||
from studio.domain import finish_record, integer, normalized_name, select_score_lines
|
||||
from studio.network import RemoteUnavailable
|
||||
from studio.network import RemoteUnavailable, RateLimited
|
||||
|
||||
|
||||
class LoginRequired(RuntimeError):
|
||||
@@ -35,17 +35,25 @@ class CatalogClient:
|
||||
def pages(self, endpoint, payload, page_size=50):
|
||||
seen = set()
|
||||
for page in range(1, 10001):
|
||||
for attempt in range(3):
|
||||
response = self.transport.request("POST", self.BASE + "rs/" + endpoint + ".do", data={**payload, "curPage": page, "start": (page - 1) * page_size, "pageSize": page_size})
|
||||
try:
|
||||
body = response.json()
|
||||
except ValueError:
|
||||
raise LoginRequired("研招网返回登录/验证页面,请先完成登录") from None
|
||||
message = str(body.get('msg', ''))
|
||||
if body.get('flag') is not True and '验证码' in message:
|
||||
raise LoginRequired('研招网需要人工验证码,请在浏览器完成验证后恢复')
|
||||
if body.get('flag') is not True and any(token in message for token in ('频繁', '稍后')):
|
||||
if attempt == 2:
|
||||
raise RateLimited('研招网业务限流持续出现,已暂停;请增加请求间隔后恢复')
|
||||
self.transport.sleep(5 * 2 ** attempt)
|
||||
continue
|
||||
break
|
||||
if body.get("flag") is not True:
|
||||
message = str(body.get("msg", ""))
|
||||
if any(token in message for token in ("登录", "登陆", "认证")):
|
||||
raise LoginRequired("研招网要求登录,请更新 Cookie 后恢复任务")
|
||||
if any(token in message for token in ("频繁", "稍后", "验证码")):
|
||||
raise RemoteUnavailable("研招网访问受限,请增加请求间隔并完成人工验证后恢复")
|
||||
raise RemoteUnavailable("研招网目录接口未成功返回数据")
|
||||
data = body.get("msg") or {}
|
||||
if not isinstance(data, dict) or not isinstance(data.get("list"), list):
|
||||
@@ -83,14 +91,21 @@ class ScoreClient:
|
||||
self.cache = {}
|
||||
|
||||
def post(self, endpoint, payload):
|
||||
for attempt in range(3):
|
||||
response = self.transport.request("POST", self.BASE + endpoint, data=payload, headers={"Referer": "https://www.kaoyan.cn/"})
|
||||
try:
|
||||
body = response.json()
|
||||
except ValueError:
|
||||
raise RemoteUnavailable("掌上考研未返回有效 JSON") from None
|
||||
if str(body.get("code")) != "0000":
|
||||
raise RemoteUnavailable("掌上考研接口未成功返回数据,请稍后重试")
|
||||
if str(body.get("code")) == "0000":
|
||||
return body.get("data") or {}
|
||||
message = str(body.get('message', body.get('msg', '')))
|
||||
if any(token in message for token in ('频繁', '稍后', '限流', '繁忙')):
|
||||
if attempt == 2:
|
||||
raise RateLimited('掌上考研业务限流持续出现,请增加请求间隔后恢复')
|
||||
self.transport.sleep(5 * 2 ** attempt)
|
||||
else:
|
||||
raise RemoteUnavailable("掌上考研接口未成功返回数据,请稍后重试")
|
||||
|
||||
def pages(self, endpoint, payload, limit=100):
|
||||
seen = set()
|
||||
|
||||
+23
-7
@@ -3,8 +3,15 @@ import json
|
||||
import sqlite3
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from studio.domain import now, normalized_name
|
||||
from studio.domain import now, normalized_name, is_special_score, school_level
|
||||
|
||||
SCORE_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS scores(
|
||||
id INTEGER PRIMARY KEY, school_code TEXT, major_code TEXT, year INTEGER,
|
||||
department_id TEXT, department_name TEXT, scope TEXT, total INTEGER, source TEXT, data TEXT NOT NULL,
|
||||
note TEXT NOT NULL DEFAULT '',raw_code TEXT NOT NULL DEFAULT '',
|
||||
UNIQUE(school_code,major_code,year,department_id,department_name,scope,source,note,raw_code));
|
||||
"""
|
||||
SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS catalog(
|
||||
id INTEGER PRIMARY KEY, record_key TEXT NOT NULL UNIQUE, year INTEGER NOT NULL,
|
||||
@@ -12,10 +19,7 @@ CREATE TABLE IF NOT EXISTS catalog(
|
||||
department_code TEXT, department_name TEXT, study_mode TEXT, degree_type TEXT, level TEXT,
|
||||
subjects TEXT, quota INTEGER, source TEXT, favorite INTEGER NOT NULL DEFAULT 0, data TEXT NOT NULL);
|
||||
CREATE INDEX IF NOT EXISTS catalog_filter ON catalog(year,province,major_code,school_code);
|
||||
CREATE TABLE IF NOT EXISTS scores(
|
||||
id INTEGER PRIMARY KEY, school_code TEXT, major_code TEXT, year INTEGER,
|
||||
department_id TEXT, department_name TEXT, scope TEXT, total INTEGER, source TEXT, data TEXT NOT NULL,
|
||||
UNIQUE(school_code,major_code,year,department_id,department_name,scope,source));
|
||||
""" + SCORE_SCHEMA + """
|
||||
CREATE INDEX IF NOT EXISTS scores_lookup ON scores(school_code,major_code,year DESC);
|
||||
CREATE TABLE IF NOT EXISTS schools(school_code TEXT PRIMARY KEY, school_name TEXT, province TEXT, level TEXT, data TEXT);
|
||||
CREATE TABLE IF NOT EXISTS jobs(id TEXT PRIMARY KEY, params TEXT, status TEXT, created_at TEXT, updated_at TEXT, message TEXT, errors INTEGER DEFAULT 0, records INTEGER DEFAULT 0);
|
||||
@@ -34,6 +38,13 @@ class Database:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with self.connection() as conn:
|
||||
conn.executescript(SCHEMA)
|
||||
if 'note' not in {row['name'] for row in conn.execute('PRAGMA table_info(scores)')}:
|
||||
# Rebuild atomically to replace the early schema's narrower unique key.
|
||||
conn.execute(SCORE_SCHEMA.replace('IF NOT EXISTS scores', 'scores_new'))
|
||||
conn.execute("""INSERT INTO scores_new(id,school_code,major_code,year,department_id,department_name,scope,total,source,data,note,raw_code) SELECT id,school_code,major_code,year,department_id,department_name,scope,total,source,data,coalesce(json_extract(data,'$.note'),''),coalesce(json_extract(data,'$.raw_code'),'') FROM scores""")
|
||||
conn.execute('DROP TABLE scores')
|
||||
conn.execute('ALTER TABLE scores_new RENAME TO scores')
|
||||
conn.execute('CREATE INDEX scores_lookup ON scores(school_code,major_code,year DESC)')
|
||||
conn.execute("UPDATE jobs SET status='paused',message='服务重启,任务可从断点恢复' WHERE status IN ('running','queued')")
|
||||
|
||||
@contextmanager
|
||||
@@ -41,6 +52,7 @@ class Database:
|
||||
conn = sqlite3.connect(self.path, timeout=30)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.create_function("norm_name", 1, normalized_name, deterministic=True)
|
||||
conn.create_function("is_special_score", 1, is_special_score, deterministic=True)
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
try:
|
||||
yield conn
|
||||
@@ -66,6 +78,7 @@ class Database:
|
||||
self._school(conn, record)
|
||||
|
||||
def _school(self, conn, record):
|
||||
record = {**record, 'level': school_level(record['school_name'], record.get('level', ''))}
|
||||
conn.execute("INSERT INTO schools VALUES(?,?,?,?,?) ON CONFLICT(school_code) DO UPDATE SET school_name=excluded.school_name,province=CASE WHEN excluded.province<>'' THEN excluded.province ELSE schools.province END,level=CASE WHEN excluded.level<>'' THEN excluded.level ELSE schools.level END", (record["school_code"],record["school_name"],record.get("province", ""),record.get("level", ""),encode(record)))
|
||||
|
||||
def index_schools(self, rows):
|
||||
@@ -76,7 +89,7 @@ class Database:
|
||||
def upsert_scores(self, records):
|
||||
with self.connection() as conn:
|
||||
for row in records:
|
||||
conn.execute("""INSERT INTO scores(school_code,major_code,year,department_id,department_name,scope,total,source,data) VALUES(?,?,?,?,?,?,?,?,?) ON CONFLICT(school_code,major_code,year,department_id,department_name,scope,source) DO UPDATE SET total=excluded.total,data=excluded.data""",(row["school_code"],row["major_code"],row["year"],row.get("department_id", ""),row.get("department_name", ""),row["scope"],row.get("total"),row.get("source", "掌上考研"),encode(row)))
|
||||
conn.execute("""INSERT INTO scores(school_code,major_code,year,department_id,department_name,scope,total,source,data,note,raw_code) VALUES(?,?,?,?,?,?,?,?,?,?,?) ON CONFLICT(school_code,major_code,year,department_id,department_name,scope,source,note,raw_code) DO UPDATE SET total=excluded.total,data=excluded.data""",(row["school_code"],row["major_code"],row["year"],row.get("department_id", "") or "",row.get("department_name", "") or "",row["scope"],row.get("total"),row.get("source", "掌上考研"),encode(row),row.get('note') or '',row.get('raw_code') or ''))
|
||||
|
||||
def _query(self, filters: dict):
|
||||
conditions, params = ["1=1"], []
|
||||
@@ -95,6 +108,9 @@ class Database:
|
||||
if filters.get("keyword"):
|
||||
conditions.append("(c.school_name LIKE ? OR c.major_name LIKE ? OR c.department_name LIKE ? OR c.major_code LIKE ?)")
|
||||
params.extend([f"%{filters['keyword']}%"] * 4)
|
||||
if filters.get('school_keyword'):
|
||||
conditions.append('c.school_name LIKE ?')
|
||||
params.append(f"%{filters['school_keyword']}%")
|
||||
if filters.get("favorites"):
|
||||
conditions.append("c.favorite=1")
|
||||
if filters.get("min_quota") is not None:
|
||||
@@ -104,7 +120,7 @@ class Database:
|
||||
if filters.get(field):
|
||||
conditions.append(f"json_extract(c.data,'$.subjects[{index}]') LIKE ?")
|
||||
params.append(str(filters[field])+"%")
|
||||
select = """SELECT c.*,s.data AS score_data,s.total AS score_total,s.year AS score_year,s.scope AS score_scope FROM catalog c LEFT JOIN scores s ON s.id=(SELECT x.id FROM scores x WHERE x.school_code=c.school_code AND x.major_code=c.major_code AND x.year<=c.year AND (x.department_name='' AND x.department_id='' OR x.department_id<>'' AND x.department_id=c.department_code OR x.department_name<>'' AND norm_name(x.department_name)=norm_name(c.department_name)) AND coalesce(json_extract(x.data,'$.note'),'') NOT LIKE '%专项%' AND coalesce(json_extract(x.data,'$.note'),'') NOT LIKE '%少干%' ORDER BY x.year DESC,CASE x.scope WHEN 'major' THEN 0 WHEN 'discipline' THEN 1 ELSE 2 END,x.id DESC LIMIT 1) WHERE """ + " AND ".join(conditions)
|
||||
select = """SELECT c.*,s.data AS score_data,s.total AS score_total,s.year AS score_year,s.scope AS score_scope FROM catalog c LEFT JOIN scores s ON s.id=(SELECT x.id FROM scores x WHERE x.school_code=c.school_code AND x.major_code=c.major_code AND x.year<=c.year AND (x.department_name='' AND x.department_id='' OR x.department_id<>'' AND x.department_id=c.department_code OR x.department_name<>'' AND norm_name(x.department_name)=norm_name(c.department_name)) AND is_special_score(json_extract(x.data,'$.note'))=0 ORDER BY x.year DESC,CASE x.scope WHEN 'major' THEN 0 WHEN 'discipline' THEN 1 ELSE 2 END,x.id DESC LIMIT 1) WHERE """ + " AND ".join(conditions)
|
||||
if filters.get("max_score") is not None:
|
||||
select = "SELECT * FROM ("+select+") q WHERE q.score_total<=?"
|
||||
params.append(filters["max_score"])
|
||||
|
||||
@@ -4,11 +4,27 @@ import json
|
||||
import re
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Any
|
||||
from pathlib import Path
|
||||
from functools import lru_cache
|
||||
from urllib.parse import urlencode
|
||||
|
||||
PROVINCES = {"11":"北京","12":"天津","13":"河北","14":"山西","15":"内蒙古","21":"辽宁","22":"吉林","23":"黑龙江","31":"上海","32":"江苏","33":"浙江","34":"安徽","35":"福建","36":"江西","37":"山东","41":"河南","42":"湖北","43":"湖南","44":"广东","45":"广西","46":"海南","50":"重庆","51":"四川","52":"贵州","53":"云南","54":"西藏","61":"陕西","62":"甘肃","63":"青海","64":"宁夏","65":"新疆"}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def school_metadata():
|
||||
path = Path(__file__).resolve().parent.parent / 'data/school_levels.json'
|
||||
data = json.loads(path.read_text(encoding='utf-8')) if path.exists() else {}
|
||||
return {k: {normalized_name(n) for n in data.get(k, [])} for k in ('985','211','double_first_class_2022_known')}
|
||||
|
||||
|
||||
def school_level(name, existing=''):
|
||||
metadata = school_metadata()
|
||||
name = normalized_name(name)
|
||||
levels = [label for key, label in [('985','985'),('211','211'),('double_first_class_2022_known','双一流')] if name in metadata[key] or label in str(existing).split('/')]
|
||||
return '/'.join(levels) or existing
|
||||
|
||||
|
||||
def now() -> str:
|
||||
return datetime.now(timezone(timedelta(hours=8))).isoformat(timespec="seconds")
|
||||
|
||||
@@ -17,6 +33,16 @@ def normalized_name(value: str) -> str:
|
||||
return str(value or "").replace("(", "(").replace(")", ")").replace(" ", "").strip()
|
||||
|
||||
|
||||
def is_special_score(note):
|
||||
text = str(note or '')
|
||||
tokens = ('专项', '少干', '少骨', '少数民族骨干', '退役大学生士兵', '退役士兵', '强军', '援藏', '单独考试')
|
||||
for token in tokens:
|
||||
cleaned = re.sub(r'(?:不含|不包括|除外)\s*' + re.escape(token), '', text)
|
||||
if token in cleaned:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def integer(value: Any) -> int | None:
|
||||
if value is None or isinstance(value, bool):
|
||||
return None
|
||||
@@ -47,6 +73,7 @@ def record_key(record: dict) -> str:
|
||||
def finish_record(record: dict) -> dict:
|
||||
defaults = {"school_code":"", "school_name":"", "province":"", "major_code":"", "major_name":"", "department_code":"", "department_name":"", "direction_code":"", "direction":"", "study_mode":"", "degree_type":"", "exam_type":"", "advisor":"", "quota_text":"", "quota":None, "quota_scope":"unknown", "subjects":[], "special_plans":[], "level":"", "remarks":"", "source":"", "source_url":"", "fetched_at":now(), "year_verified":False}
|
||||
row = {**defaults, **record}
|
||||
row['level'] = school_level(row['school_name'], row['level'])
|
||||
if not row["school_code"]:
|
||||
row["school_code"] = "name:" + normalized_name(row["school_name"])
|
||||
row["record_key"] = record_key(row)
|
||||
|
||||
+34
-6
@@ -3,15 +3,28 @@ import csv
|
||||
import io
|
||||
import json
|
||||
import sqlite3
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from openpyxl import load_workbook, Workbook
|
||||
from studio.domain import finish_record, integer, parse_quota
|
||||
from studio.domain import finish_record, integer, parse_quota, normalize_catalog, PROVINCES
|
||||
|
||||
LABELS = {"year": "年份", "school_name": "院校", "school_code": "院校代码", "province": "省份", "level": "院校层次", "major_code": "专业代码", "major_name": "专业", "degree_type": "学位类型", "department_name": "院系", "direction": "研究方向", "study_mode": "学习方式", "exam_type": "考试方式", "advisor": "导师", "quota_text": "招生人数原文", "quota": "明确人数", "quota_scope": "人数口径", "subjects": "考试科目", "special_plans": "专项计划", "score_year": "参考线年份", "score_total": "参考线总分", "score_scope": "分数线口径", "remarks": "备注", "source": "来源", "source_url": "来源链接", "year_verified": "来源年份已核实", "fetched_at": "采集时间"}
|
||||
LABELS.update({"department_code": "院系代码", "direction_code": "方向代码"})
|
||||
ALIASES = {"院校名称": "school_name", "学校名称": "school_name", "专业名称": "major_name", "院系所": "department_name", "拟招生人数": "quota_text", "planned_enrollment": "quota_text", "nzsrsstr": "quota_text", "school_level": "level", "department": "department_name", "research_direction": "direction", "enrollment_text": "quota_text", "enrollment_count": "quota", "exam_subjects": "subjects"}
|
||||
ALIASES.update({v: k for k, v in LABELS.items()})
|
||||
|
||||
|
||||
@dataclass
|
||||
class ImportResult:
|
||||
records: list
|
||||
scores: list
|
||||
schools: list = field(default_factory=list)
|
||||
|
||||
def __iter__(self):
|
||||
yield self.records
|
||||
yield self.scores
|
||||
|
||||
|
||||
def list_field(value):
|
||||
if isinstance(value, list):
|
||||
return value
|
||||
@@ -44,7 +57,7 @@ def row_record(raw, year=None, source="文件导入"):
|
||||
if not subjects:
|
||||
subjects = [str(row.get(f"exam_subject{i}") or row.get(f"subject{i}") or row.get(f"考试科目{i}") or "") for i in range(1, 5)]
|
||||
quota, scope = parse_quota(row.get("quota_text", ""))
|
||||
if quota is None and row.get("quota") != "":
|
||||
if quota is None and not str(row.get('quota_text') or '').strip():
|
||||
quota = integer(row.get("quota"))
|
||||
scope = row.get("quota_scope") or "unspecified"
|
||||
plans = list_field(row.get("special_plans"))
|
||||
@@ -52,7 +65,7 @@ def row_record(raw, year=None, source="文件导入"):
|
||||
plans.append("退役大学生士兵")
|
||||
if row.get("shaogu_program") in ("是", "1", 1):
|
||||
plans.append("少数民族骨干")
|
||||
return finish_record({**row, "year": explicit_year, "source": row.get("source") or source, "subjects": subjects, "special_plans": plans, "quota": quota, "quota_scope": scope, "year_verified": row.get("year_verified") is True or row.get("year_verified") == "True"})
|
||||
return finish_record({**row, "year": explicit_year, "source": row.get("source") or source, "subjects": subjects, "reference_books": list_field(row.get('reference_books')), "special_plans": plans, "quota": quota, "quota_scope": scope, "year_verified": row.get("year_verified") is True or row.get("year_verified") == "True"})
|
||||
|
||||
|
||||
def import_file(path, year=None):
|
||||
@@ -92,14 +105,29 @@ def import_file(path, year=None):
|
||||
raise ValueError("支持 CSV、XLSX、JSON 和 YAM SQLite 文件")
|
||||
if len(rows) > 300000:
|
||||
raise ValueError("一次最多导入 30 万条,请拆分文件")
|
||||
records = [row_record(row, year, "yzw 导入" if suffix == ".csv" else "文件导入") for row in rows]
|
||||
records, schools = [], []
|
||||
for row in rows:
|
||||
if isinstance(row, dict) and row.get('dwdm') and row.get('dwmc'):
|
||||
schools.append({'school_code': str(row['dwdm']), 'school_name': row['dwmc'], 'province': row.get('szss') or PROVINCES.get(str(row.get('szssm', '')), ''), 'level': '双一流' if str(row.get('syl')) == '1' else ''})
|
||||
directions = row.get('data') or ([row] if row.get('kskmz') else [])
|
||||
if not isinstance(directions, list):
|
||||
raise ValueError('YAM 种子 data 应为方向数组')
|
||||
for direction in directions:
|
||||
if not isinstance(direction, dict) or not any(k in direction for k in ('yjfxmc', 'kskmz', 'yxsmc')):
|
||||
raise ValueError('YAM 方向数据结构无法识别')
|
||||
explicit = integer(row.get('year')) or integer(year)
|
||||
if not explicit:
|
||||
raise ValueError('种子含有目录详情,请指定其招生年份')
|
||||
records.extend(normalize_catalog(direction, explicit, row))
|
||||
else:
|
||||
records.append(row_record(row, year, "yzw 导入" if suffix == ".csv" else "文件导入"))
|
||||
for score in scores:
|
||||
if not all(k in score for k in ("school_code", "major_code", "year", "scope")):
|
||||
raise ValueError("快照分数线缺少院校/专业/年份/口径")
|
||||
score["year"] = int(score["year"])
|
||||
score["total"] = integer(score.get("total"))
|
||||
score.setdefault("note", "")
|
||||
return records, scores
|
||||
return ImportResult(records, scores, schools)
|
||||
|
||||
|
||||
def import_yam(path, year):
|
||||
@@ -133,7 +161,7 @@ def import_yam(path, year):
|
||||
row = dict(entry)
|
||||
school = lookup.get((str(row["school_id"]), str(row.get("major_code", ""))), {})
|
||||
records.append(row_record({**row, "school_code": str(row["school_id"]), "school_name": school.get("name", ""), "province": school.get("province", ""), "direction": row.get("research_direction", ""), "direction_code": str(row.get("spe_id") or row.get("plan_id") or ""), "department_code": str(row.get("department_id", "") or ""), "source": row.get("source") or "YAM 导入", "remarks": row.get("note", "")}, year))
|
||||
return records, scores
|
||||
return ImportResult(records, scores)
|
||||
except sqlite3.DatabaseError:
|
||||
raise ValueError("SQLite 文件无法读取,请确认是完整导出的数据库") from None
|
||||
finally:
|
||||
|
||||
+27
-7
@@ -3,13 +3,17 @@ import threading
|
||||
import uuid
|
||||
from studio.connectors import CatalogClient, ScoreClient, LoginRequired, SourceYearMismatch
|
||||
from studio.domain import PROVINCES, normalize_catalog
|
||||
from studio.network import ProxyUnavailable, RemoteUnavailable
|
||||
from studio.network import ProxyUnavailable, RemoteUnavailable, RateLimited
|
||||
|
||||
|
||||
class Cancelled(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Shutdown(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class JobManager:
|
||||
def __init__(self, database, settings, transport=None, runner=None):
|
||||
self.db, self.settings, self.transport = database, settings, transport
|
||||
@@ -54,7 +58,7 @@ class JobManager:
|
||||
|
||||
def _launch(self, jid):
|
||||
condition = threading.Condition()
|
||||
control = {"condition": condition, "paused": False, "cancelled": False}
|
||||
control = {"condition": condition, "paused": False, "cancelled": False, "shutdown": False}
|
||||
thread = threading.Thread(target=self._run, args=(jid,), daemon=True, name="collector-" + jid[:8])
|
||||
control["thread"] = thread
|
||||
self.controls[jid] = control
|
||||
@@ -71,6 +75,8 @@ class JobManager:
|
||||
self.checkpoint(jid)
|
||||
errors = self.get(jid)["errors"]
|
||||
self.db.job_update(jid, status="partial" if errors else "completed", message="部分单元失败,可恢复重试" if errors else "采集完成")
|
||||
except Shutdown:
|
||||
self.db.job_update(jid, status="paused", message="服务停止,任务可从断点恢复")
|
||||
except Cancelled:
|
||||
self.db.job_update(jid, status="cancelled", message="已取消,已保存数据保留")
|
||||
except LoginRequired as exc:
|
||||
@@ -79,6 +85,9 @@ class JobManager:
|
||||
except ProxyUnavailable as exc:
|
||||
self.db.job_update(jid, status="waiting_proxy", message=str(exc))
|
||||
self.db.log(jid, "warning", str(exc))
|
||||
except RateLimited as exc:
|
||||
self.db.job_update(jid, status="waiting_source", message=str(exc))
|
||||
self.db.log(jid, "warning", str(exc))
|
||||
except (RemoteUnavailable, SourceYearMismatch, ValueError) as exc:
|
||||
self.db.job_update(jid, status="failed", message=str(exc))
|
||||
self.db.log(jid, "error", str(exc))
|
||||
@@ -88,16 +97,20 @@ class JobManager:
|
||||
finally:
|
||||
if self.transport:
|
||||
self.transport.local.guard = None
|
||||
if self.controls[jid]["shutdown"] and self.get(jid)["status"] != 'cancelled':
|
||||
self.db.job_update(jid, status="paused", message="服务停止,任务可从断点恢复")
|
||||
|
||||
def checkpoint(self, jid):
|
||||
control = self.controls.get(jid)
|
||||
if not control:
|
||||
return
|
||||
with control["condition"]:
|
||||
while control["paused"] and not control["cancelled"]:
|
||||
while control["paused"] and not control["cancelled"] and not control["shutdown"]:
|
||||
control["condition"].wait(.25)
|
||||
if control["cancelled"]:
|
||||
raise Cancelled()
|
||||
if control["shutdown"]:
|
||||
raise Shutdown()
|
||||
|
||||
def unit_done(self, jid, unit):
|
||||
return self.db.unit_done(jid, unit)
|
||||
@@ -117,16 +130,20 @@ class JobManager:
|
||||
def resume(self, jid):
|
||||
with self.lock:
|
||||
job = self.get(jid)
|
||||
if not job or job["status"] not in ("paused", "partial", "failed", "waiting_login", "waiting_proxy"):
|
||||
if not job or job["status"] not in ("paused", "partial", "failed", "waiting_login", "waiting_proxy", "waiting_source"):
|
||||
raise ValueError("此任务当前不能恢复")
|
||||
self._ensure_available(jid)
|
||||
control = self.controls.get(jid)
|
||||
if control and control["thread"].is_alive():
|
||||
if control and control["thread"].is_alive() and control["paused"] and job['status'] == 'paused':
|
||||
with control["condition"]:
|
||||
self.db.job_update(jid, status="running", message="正在继续采集")
|
||||
control["paused"] = False
|
||||
control["condition"].notify_all()
|
||||
else:
|
||||
if control and control['thread'].is_alive():
|
||||
control['thread'].join(timeout=1)
|
||||
if control['thread'].is_alive():
|
||||
raise ValueError("旧任务正在结束,请稍后点击恢复")
|
||||
self._launch(jid)
|
||||
return self.get(jid)
|
||||
|
||||
@@ -147,7 +164,10 @@ class JobManager:
|
||||
self.closed = True
|
||||
for jid, control in list(self.controls.items()):
|
||||
if control["thread"].is_alive():
|
||||
self.cancel(jid)
|
||||
with control['condition']:
|
||||
control['shutdown'] = True
|
||||
self.db.job_update(jid, status='paused', message='服务停止,任务可从断点恢复')
|
||||
control['condition'].notify_all()
|
||||
for control in list(self.controls.values()):
|
||||
control["thread"].join(timeout=1)
|
||||
|
||||
@@ -194,7 +214,7 @@ class JobManager:
|
||||
self._error(jid, f"{label}:{exc}")
|
||||
|
||||
def _scores(self, jid, params):
|
||||
filters = {"year": params["year"], "major_code": params.get("major_code", ""), "keyword": params.get("school_keyword", "")}
|
||||
filters = {"year": params["year"], "major_code": params.get("major_code", ""), "school_keyword": params.get("school_keyword", "")}
|
||||
records = self.db.all_catalog(filters)
|
||||
targets = {}
|
||||
for row in records:
|
||||
|
||||
+5
-1
@@ -13,6 +13,10 @@ class RemoteUnavailable(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class RateLimited(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class ProxyRouter:
|
||||
def __init__(self, settings, session=None):
|
||||
self.settings = settings
|
||||
@@ -146,4 +150,4 @@ class Transport:
|
||||
raise RemoteUnavailable("网络连接失败,已尝试切换代理;请检查网络或恢复任务") from None
|
||||
except requests.HTTPError as exc:
|
||||
raise RemoteUnavailable(f"数据源返回 HTTP {exc.response.status_code}") from None
|
||||
raise RemoteUnavailable("数据源限流,请降低请求频率后重试")
|
||||
raise RateLimited("数据源持续限流,任务已等待;请降低请求频率后恢复")
|
||||
|
||||
@@ -66,3 +66,13 @@ def test_import_yam_sqlite_keeps_year_and_score_scope(client, tmp_path):
|
||||
assert rows[0]["year"] == 2026
|
||||
assert rows[0]["score_year"] == 2025
|
||||
assert rows[0]["score_scope"] == "discipline"
|
||||
|
||||
|
||||
def test_yam_seed_json_imports_school_index_without_inventing_admissions(client):
|
||||
payload = [{'dwdm': '10001', 'dwmc': '北京大学', 'zydm': '081200', 'zymc': '计算机科学与技术', 'szss': '北京', 'data': [], 'sign': 'token-must-not-copy'}]
|
||||
response = client.post('/api/import', files={'file': ('yam-seed.json', json.dumps(payload).encode())})
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()['records'] == 0
|
||||
assert response.json()['indexed_schools'] == 1
|
||||
assert client.get('/api/catalog').json()['total'] == 0
|
||||
assert 'token-must-not-copy' not in client.get('/api/schools').text
|
||||
|
||||
@@ -79,3 +79,64 @@ def test_task_pause_resume_reuses_persisted_work_without_duplicate_entries(tmp_p
|
||||
assert jobs.get(job["id"])["completed_units"] == 2
|
||||
jobs.close()
|
||||
|
||||
|
||||
def test_service_shutdown_pauses_instead_of_cancelling_persisted_job(tmp_path):
|
||||
db, settings = Database(tmp_path / 'studio.db'), SettingsStore(tmp_path)
|
||||
entered = threading.Event()
|
||||
def runner(jid, params, manager):
|
||||
entered.set()
|
||||
while True:
|
||||
manager.checkpoint(jid)
|
||||
time.sleep(.01)
|
||||
manager = JobManager(db, settings, runner=runner)
|
||||
job = manager.start({'mode': 'catalog', 'year': 2027})
|
||||
assert entered.wait(2)
|
||||
manager.close()
|
||||
assert Database(tmp_path / 'studio.db').job_get(job['id'])['status'] == 'paused'
|
||||
|
||||
|
||||
def test_business_rate_limit_uses_bounded_backoff_without_skipping_a_page():
|
||||
from studio.network import RateLimited
|
||||
class Limited(CatalogTransport):
|
||||
def __init__(self):
|
||||
self.calls, self.waits = 0, []
|
||||
def request(self, *args, **kwargs):
|
||||
self.calls += 1
|
||||
return Response({'flag': False, 'msg': '访问太频繁,请稍后重试'})
|
||||
def sleep(self, seconds):
|
||||
self.waits.append(seconds)
|
||||
transport = Limited()
|
||||
with pytest.raises(RateLimited):
|
||||
list(CatalogClient(transport).schools('11'))
|
||||
assert transport.calls == 3
|
||||
assert transport.waits == [5, 10]
|
||||
|
||||
|
||||
def test_recovery_waits_for_failed_worker_to_exit_before_launching_another(tmp_path):
|
||||
db, settings = Database(tmp_path / 'studio.db'), SettingsStore(tmp_path)
|
||||
entered, release = threading.Event(), threading.Event()
|
||||
original_log = db.log
|
||||
def guarded_log(jid, level, message):
|
||||
if level == 'error':
|
||||
entered.set()
|
||||
release.wait(2)
|
||||
original_log(jid, level, message)
|
||||
db.log = guarded_log
|
||||
attempts = []
|
||||
def runner(jid, params, manager):
|
||||
attempts.append(jid)
|
||||
if len(attempts) == 1:
|
||||
raise ValueError('failure fixture')
|
||||
manager.complete_unit(jid, 'recovered')
|
||||
manager = JobManager(db, settings, runner=runner)
|
||||
job = manager.start({'mode': 'catalog', 'year': 2027})
|
||||
assert entered.wait(2)
|
||||
threading.Timer(.05, release.set).start()
|
||||
manager.resume(job['id'])
|
||||
deadline = time.monotonic() + 2
|
||||
while manager.get(job['id'])['status'] != 'completed' and time.monotonic() < deadline:
|
||||
time.sleep(.02)
|
||||
assert manager.get(job['id'])['status'] == 'completed'
|
||||
assert len(attempts) == 2
|
||||
manager.close()
|
||||
|
||||
|
||||
@@ -64,3 +64,61 @@ def test_same_year_different_college_lines_are_not_attached_arbitrarily(tmp_path
|
||||
db.upsert_scores(select_score_lines(rows, "081200", "10001", "北京大学", 2026))
|
||||
assert db.query_catalog({"year": 2026})["items"][0]["score_total"] is None
|
||||
|
||||
|
||||
def test_special_plan_score_does_not_overwrite_regular_admission_line(tmp_path):
|
||||
db = Database(tmp_path / "studio.db")
|
||||
db.upsert_catalog(normalize_catalog(raw_catalog(), 2026))
|
||||
rows = [{"code": "081200", "total": "320", "depart_id": "001", "note": ""}, {"code": "081200", "total": "260", "depart_id": "001", "note": "少干专项计划"}]
|
||||
db.upsert_scores(select_score_lines(rows, "081200", "10001", "北京大学", 2026))
|
||||
assert db.overview()["scores"] == 2
|
||||
assert db.query_catalog({})["items"][0]["score_total"] == 320
|
||||
|
||||
|
||||
def test_special_plan_full_names_are_not_used_as_regular_reference_line(tmp_path):
|
||||
db = Database(tmp_path / 'studio.db')
|
||||
db.upsert_catalog(normalize_catalog(raw_catalog(), 2026))
|
||||
rows = [{'code': '081200', 'total': 320, 'depart_id': '001', 'note': ''}, {'code': '081200', 'total': 260, 'depart_id': '001', 'note': '少数民族骨干计划'}, {'code': '081200', 'total': 250, 'depart_id': '001', 'note': '退役大学生士兵计划'}]
|
||||
db.upsert_scores(select_score_lines(rows, '081200', '10001', '北京大学', 2026))
|
||||
assert db.query_catalog({})['items'][0]['score_total'] == 320
|
||||
|
||||
|
||||
def test_spreadsheet_roundtrip_preserves_record_identity(tmp_path):
|
||||
from studio.imports import export_sheet, import_file
|
||||
db = Database(tmp_path / 'studio.db')
|
||||
records = normalize_catalog(raw_catalog(), 2027)
|
||||
db.upsert_catalog(records)
|
||||
for kind in ('csv', 'xlsx'):
|
||||
path = tmp_path / ('data.' + kind)
|
||||
path.write_bytes(export_sheet(records, kind))
|
||||
imported, _ = import_file(path)
|
||||
db.upsert_catalog(imported)
|
||||
assert db.query_catalog({})['total'] == 2
|
||||
|
||||
|
||||
def test_known_historical_school_levels_are_available_in_filters(tmp_path):
|
||||
db = Database(tmp_path / 'studio.db')
|
||||
db.upsert_catalog(normalize_catalog(raw_catalog(), 2027))
|
||||
assert db.query_catalog({'level': '985'})['total'] == 2
|
||||
assert db.query_catalog({'level': '211'})['total'] == 2
|
||||
|
||||
|
||||
def test_imported_college_quota_cannot_be_promoted_by_numeric_column():
|
||||
from studio.imports import row_record
|
||||
row = row_record({'school_name':'Test','major_code':'081200','enrollment_text':'学院:100','enrollment_count':100},2026)
|
||||
assert row['quota'] is None
|
||||
assert row['quota_scope'] == 'college'
|
||||
|
||||
|
||||
def test_job_school_keyword_does_not_match_department_names(tmp_path):
|
||||
db = Database(tmp_path / 'studio.db')
|
||||
item = raw_catalog()
|
||||
item['yxsmc'] = '信息学院'
|
||||
db.upsert_catalog(normalize_catalog(item, 2027))
|
||||
assert db.query_catalog({'school_keyword': '信息'})['total'] == 0
|
||||
|
||||
|
||||
def test_yam_reference_books_are_arrays_for_the_detail_view():
|
||||
from studio.imports import row_record
|
||||
row = row_record({'school_name':'Test','major_code':'081200','reference_books':'[]'},2026)
|
||||
assert row['reference_books'] == []
|
||||
|
||||
|
||||
+38
-1
@@ -3,7 +3,7 @@ import pytest
|
||||
|
||||
from studio.network import ProxyRouter, Transport, ProxyUnavailable
|
||||
from studio.settings import SettingsStore
|
||||
from studio.auth import CookieVault
|
||||
from studio.auth import CookieVault, AuthService
|
||||
|
||||
|
||||
class Response:
|
||||
@@ -91,3 +91,40 @@ def test_cookie_is_encrypted_and_not_returned_in_settings(tmp_path):
|
||||
assert b"secret-test-cookie" not in vault.path.read_bytes()
|
||||
assert "cookie" not in SettingsStore(tmp_path).read()
|
||||
|
||||
|
||||
def test_rate_limiting_does_not_quarantine_a_healthy_proxy(tmp_path):
|
||||
class Limited(Remote):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.calls = 0
|
||||
def request(self, *args, **kwargs):
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
return Response({}, status=429)
|
||||
return super().request(*args, **kwargs)
|
||||
_, router, transport = setup_network(tmp_path, remote=Limited())
|
||||
original = router.resolve()
|
||||
assert transport.request("GET", "https://yz.chsi.com.cn/zsml/").json()["routed_through"] == original
|
||||
assert router.failed == set()
|
||||
|
||||
|
||||
def test_browser_cookie_domains_preserve_same_name_sessions(tmp_path):
|
||||
_, _, transport = setup_network(tmp_path)
|
||||
vault = CookieVault(tmp_path)
|
||||
vault.save([{"name": "JSESSIONID", "value": "account-cookie", "domain": "account.chsi.com.cn", "path": "/"}, {"name": "JSESSIONID", "value": "catalog-cookie", "domain": "yz.chsi.com.cn", "path": "/"}])
|
||||
AuthService(transport, vault)
|
||||
cookies = transport.session.cookies
|
||||
assert cookies.get("JSESSIONID", domain="account.chsi.com.cn") == "account-cookie"
|
||||
assert cookies.get("JSESSIONID", domain="yz.chsi.com.cn") == "catalog-cookie"
|
||||
|
||||
|
||||
def test_auth_checks_protected_direction_api_rather_than_public_school_index(tmp_path):
|
||||
class PublicIndex(Remote):
|
||||
def request(self, method, url, **kwargs):
|
||||
if url.endswith('/dws.do'):
|
||||
return Response({'flag': True, 'msg': {'list': []}})
|
||||
return Response({'flag': False, 'msg': '请登录'})
|
||||
_, _, transport = setup_network(tmp_path, remote=PublicIndex())
|
||||
auth = AuthService(transport, CookieVault(tmp_path))
|
||||
assert auth.check()['valid'] is False
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ def main():
|
||||
expect(page.get_by_role("heading", name="从数据出发,找到适合的院校")).to_be_visible()
|
||||
page.get_by_role("navigation").get_by_role("button", name="院校工作区").click()
|
||||
page.get_by_role("button", name="导入数据", exact=True).click()
|
||||
expect(page.get_by_role('spinbutton', name='这份目录的招生年份')).to_have_value('')
|
||||
rows = [{"year": 2026, "school_name": "验收样例甲大学", "school_code": "QA001", "province": "北京", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "计算机学院", "subjects": ["101思想政治理论", "201英语一", "301数学一", "408计算机学科专业基础"], "quota_text": "专业:6(不含推免)", "source": "自动验收样例"}, {"year": 2026, "school_name": "验收样例乙大学", "school_code": "QA002", "province": "上海", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "信息学院", "subjects": ["101思想政治理论", "204英语二", "302数学二", "自命题专业课"], "quota_text": "学院:100", "source": "自动验收样例"}]
|
||||
scores = [{"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2024, "total": 305, "scope": "major", "note": ""}, {"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2025, "total": 315, "scope": "major", "note": ""}]
|
||||
page.locator('input[type=file]').set_input_files({"name": "acceptance.json", "mimeType": "application/json", "buffer": json.dumps({"catalog": rows, "scores": scores}, ensure_ascii=False).encode()})
|
||||
|
||||
Reference in New Issue
Block a user