fix: preserve data granularity and recover tasks safely

This commit is contained in:
Codex
2026-10-11 14:18:56 +08:00
parent 23f3a15a15
commit 603184549b
29 changed files with 4557 additions and 418 deletions
+76
View File
@@ -0,0 +1,76 @@
# 研途 · 考研数据工作台
将 YAM 的择校工作区和历史考情能力,与 yzw 的全国目录采集、代理池和断点功能整合为独立的本地网页应用。
当前安装位置:`F:\@Other\kaoyan-studio`。原 YAM、yzw 和 proxy_pool 源码不作修改。
## 启动
双击 **启动工作台.cmd**,浏览器打开 **http://127.0.0.1:8765**。窗口关闭后后台服务继续运行;双击 **停止工作台.cmd** 结束本项目服务。
本机已安装依赖并完成网页构建,可以直接启动。源码包包含已构建网页,复制到另一台 Windows 电脑后需要 Python 3.12+,首次运行会安装锁定的 Python 依赖。重新构建或修改网页需要 Node.js 22+;缺少网页构建时,脚本会安装前端依赖并构建。浏览器登录使用 Microsoft Edge。
在 PowerShell 中也可以运行:
```powershell
& 'F:\@Other\kaoyan-studio\start.ps1'
# 仅启动后台:start.ps1 -NoBrowser
# 改端口:start.ps1 -Port 8768
```
## 第一次采集
1. 在「代理 IP 池」刷新,查看池内 HTTPS 候选和当前路由。默认连接 `http://127.0.0.1:5010`;服务目录为 `C:\Users\Jayson\Desktop\proxy_pool`。
2. 在「设置与登录」选择网络策略。优先代理在池为空时允许直连;严格代理没有可用 IP 时等待处理。备用代理每行一个 `IP:端口`。代理池存在候选并不保证该 IP 对所有目标网站可用,连接失败会切换并隔离。
3. 点击「打开登录窗口」,在独立 Edge 窗口完成学信网登录和验证码。工作台会检测登录并保存 Cookie。也可粘贴已登录研招网的 Cookie 请求头,或使用账号密码登录。点击「验证目录访问」确认详情接口权限。
4. 在「采集任务」选择内容、目录年份、地区、专业和院校关键词。建议先按一个地区、一个专业或一所院校验证,再创建全国任务。专业代码可以从内置目录选择,也可直接输入 6 位代码。
5. 查看任务日志。暂停可继续;需要登录、代理或限流解除时,处理后点击恢复;取消保留已有数据。关闭或重启服务会保留可恢复断点。
研招网学校、专业索引有时可以匿名读取,但方向与科目详情需要登录。程序验证实际业务返回状态,不将 HTTP 200 当作登录成功。目录采集先核对研招页面配置中的年份,避免将当前目录标为历史年份。
## 数据查询
「院校工作区」支持院校/专业/院系关键词、年份、省份、专业代码、学习方式、学位类型、院校层次、科目、院系、方向、人数及参考线筛选和排序。可收藏候选、勾选 2–6 条记录对比,点击院校名查看来源、分数线和计划历史。
每条记录按 **年份 + 院校 + 院系 + 专业 + 方向 + 考试方式 + 科目组合 + 来源** 保存。不同年份、方向及可选科目组合独立保留,相同记录重新采集时更新。人数保留原文:明确「专业」人数可用于筛选;学院、方向、范围或待定口径不会被转换成专业总人数。不同方向和科目组合可能重复展示同一专业人数,不能累加。
历史分数线区分专业线、学科参考线和门类参考线,匹配院系后才展示;专项计划线独立保留在详情,不作为普通计划的参考线。表格显示不晚于目录年的最近匹配线,空字段显示「未提供」。历年图表按口径、院系、备注和来源分开,不将门类线标为专业线。
985/211 是历史工程参考名单。双一流优先采用数据源标识,同时补充已知的 2022 年名单;名单来源见 `data/school_levels.json`。学校更名、新名单或未识别名称可能需要更新参考元数据。
内置 2,255 条专业目录条目和 460 所院校公开索引,来自 YAM 的公共数据文件。**索引并不代表已经采集招生详情**;首次安装的目录和分数线数据库为空。全国采集不限定于这些索引,通过研招网查询全部 31 个省级地区的招生单位。
## 原项目数据互通
在工作区点击「导入数据」:
| 格式 | 用途 | 年份规则 |
| --- | --- | --- |
| yzw CSV | 原 `export_major_csv.py` 输出的全部字段 | 原文件无年,必须填写实际招生年份 |
| YAM SQLite | Python 数据库中的 schools/departments、score_lines、admission_plans | 无年份目录须填写;历史分数/计划保留文件年份 |
| YAM JSON 种子 | `data/seeds/yan_zhao_*_all_regions.json` | 只有学校列表时仅导入索引;有方向详情时须提供年份 |
| 工作台 JSON | 完整目录和分数线快照 | 保留各条记录年份 |
| CSV / XLSX | 工作台导出,或包含相应机器字段/中文表头的数据 | 优先文件年份,否则须明确填写 |
支持 UTF-8 / UTF-8 BOM / GB18030 CSV。一次最多 100 MB、30 万条。导入只读取所选文件,不打开原软件环境或修改源文件;SQLite 使用只读模式。其他版本 YAM 表结构不同的数据库,请从原软件导出兼容文件再导入。
可将当前筛选结果导出 CSV、Excel 或完整 JSON。表格导出保留院系/方向代码和字段口径,回导不会凭空增加同一记录。JSON 同时保留相关学校专业的历史分数线;凭据不进入导出。CSV/Excel 对可能被表格软件当作公式的文本加保护前缀。
## 本地数据与开发
所有运行数据位于项目 `.runtime`:`studio.db` 保存目录、分数、收藏、任务和断点;`settings.json` 保存非机密配置;`chsi-cookie.enc` 由当前 Windows 用户的 DPAPI 加密。密码不持久化,接口不返回 Cookie。备份数据库时先停止工作台,再复制 `.runtime`;恢复加密 Cookie 需要原 Windows 用户环境,否则重新登录即可。
服务默认仅监听 `127.0.0.1`,拒绝非本机 Host 和跨来源写入。代理池设置页使用现有 `manage.ps1` 控制本机服务;工作台停止脚本只结束自己管理且进程身份匹配的服务,不停止代理池。
```powershell
# 后端行为测试
.\.venv\Scripts\python.exe -m pytest -q
# 前端逻辑测试及生产构建
cd frontend
npm.cmd test
npm.cmd run build
```
浏览器验收使用独立的测试数据库,运行方式及结果见 `docs/verification.md`。没有真人登录的自动验收不能替代账号登录后的全国采集验证。
项目结构:`studio/` 为网络、数据源、存储和 API;`frontend/src/` 为 React 网页;`data/` 为公开参考元数据;`tests/` 为后端行为测试。来源与许可证见 `THIRD_PARTY_NOTICES.md`。
+13
View File
@@ -0,0 +1,13 @@
# 来源与第三方声明
本项目独立实现采集器、存储、任务和网页服务,参考以下项目的公开接口及字段设计,不直接运行原仓库模块。
- [YAM / 研喵](https://github.com/C-Y-T-258/yam):参考全国目录、掌上考研历史分数/计划接口及择校工作区设计。`data/majors.json` 与 `data/schools.json` 整理自其公共专业目录和院校种子,仅保留公开参考字段,未携带签名、Cookie 或账号。原 `pyproject.toml` 声明 MIT,作者 YAM Contributors;本地原仓库没有单独的 LICENSE 文件。
- [yzw](https://github.com/freecho/yzw):参考学校/专业/方向 API、CSV 字段和代理策略。学校层次参考数据以原文件为基础,补充学信网发布的教育部名单、名称别名,并修正中国中医科学院的 211 误标。原项目 MIT,Copyright (c) 2025 freecho,完整许可副本在 `licenses/yzw-MIT.txt`。
- [proxy_pool](https://github.com/jhao104/proxy_pool):使用用户已有的 Flask API 和本机 `manage.ps1`,不内嵌或修改代理池源码。
参考名单来源:[985 名单](https://www.moe.gov.cn/srcsite/A22/s7065/200612/t20061206_128833.html)、[学信网发布的教育部 211 名单](https://gaokao.chsi.com.cn/gkxx/gxmd/211gc.shtml)、[第二轮双一流通知](https://www.moe.gov.cn/srcsite/A22/s7065/202202/t20220211_598710.html)。985/211 用于历史工程参考;双一流标识优先采用采集来源,静态名单为已知参考子集。
Python / JavaScript 依赖保留其自身许可证。运行环境包括 FastAPI、Starlette、Uvicorn、requests、Beautiful Soup、openpyxl、Playwright、React、Vite、TypeScript、Lucide;版本记录在 `requirements.lock` 和 `frontend/package-lock.json`,许可证原文随安装包提供。
采集数据的准确性、完整性和开放年份取决于原网站,应用保留来源及口径,并明确标记未知字段。内置索引不等同于已经采集的招生数据。
+294
View File
@@ -0,0 +1,294 @@
{
"985": [
"上海交通大学",
"东北大学",
"东南大学",
"中南大学",
"中国人民大学",
"中国农业大学",
"中国海洋大学",
"中国科学技术大学",
"中央民族大学",
"中山大学",
"兰州大学",
"北京大学",
"北京师范大学",
"北京理工大学",
"北京航空航天大学",
"华东师范大学",
"华中科技大学",
"华南理工大学",
"南京大学",
"南开大学",
"厦门大学",
"吉林大学",
"同济大学",
"哈尔滨工业大学",
"四川大学",
"国防科学技术大学",
"国防科技大学",
"复旦大学",
"大连理工大学",
"天津大学",
"山东大学",
"武汉大学",
"浙江大学",
"清华大学",
"湖南大学",
"电子科技大学",
"西北农林科技大学",
"西北工业大学",
"西安交通大学",
"重庆大学"
],
"211": [
"上海交通大学",
"上海外国语大学",
"上海大学",
"上海财经大学",
"东北农业大学",
"东北大学",
"东北师范大学",
"东北林业大学",
"东华大学",
"东南大学",
"中南大学",
"中南财经政法大学",
"中国人民大学",
"中国传媒大学",
"中国农业大学",
"中国地质大学(北京)",
"中国地质大学(武汉)",
"中国政法大学",
"中国海洋大学",
"中国石油大学(北京)",
"中国石油大学(华东)",
"中国矿业大学",
"中国矿业大学(北京)",
"中国矿业大学(徐州)",
"中国科学技术大学",
"中国药科大学",
"中央民族大学",
"中央财经大学",
"中央音乐学院",
"中山大学",
"云南大学",
"兰州大学",
"内蒙古大学",
"北京中医药大学",
"北京交通大学",
"北京体育大学",
"北京化工大学",
"北京外国语大学",
"北京大学",
"北京工业大学",
"北京师范大学",
"北京林业大学",
"北京理工大学",
"北京科技大学",
"北京航空航天大学",
"北京邮电大学",
"华东师范大学",
"华东理工大学",
"华中农业大学",
"华中师范大学",
"华中科技大学",
"华北电力大学",
"华北电力大学(保定)",
"华北电力大学(北京)",
"华南师范大学",
"华南理工大学",
"南京农业大学",
"南京大学",
"南京师范大学",
"南京理工大学",
"南京航空航天大学",
"南开大学",
"南昌大学",
"厦门大学",
"合肥工业大学",
"吉林大学",
"同济大学",
"哈尔滨工业大学",
"哈尔滨工程大学",
"四川农业大学",
"四川大学",
"国防科学技术大学",
"国防科技大学",
"复旦大学",
"大连海事大学",
"大连理工大学",
"天津医科大学",
"天津大学",
"太原理工大学",
"宁夏大学",
"安徽大学",
"对外经济贸易大学",
"山东大学",
"广西大学",
"延边大学",
"新疆大学",
"暨南大学",
"武汉大学",
"武汉理工大学",
"江南大学",
"河北工业大学",
"河海大学",
"浙江大学",
"海军军医大学",
"海南大学",
"清华大学",
"湖南大学",
"湖南师范大学",
"电子科技大学",
"石河子大学",
"福州大学",
"空军军医大学",
"第二军医大学",
"第四军医大学",
"苏州大学",
"西北农林科技大学",
"西北大学",
"西北工业大学",
"西南交通大学",
"西南大学",
"西南财经大学",
"西安交通大学",
"西安电子科技大学",
"西藏大学",
"贵州大学",
"辽宁大学",
"郑州大学",
"重庆大学",
"长安大学",
"陕西师范大学",
"青海大学"
],
"double_first_class_2022_known": [
"上海中医药大学",
"上海交通大学",
"上海外国语大学",
"上海大学",
"上海财经大学",
"东北农业大学",
"东北大学",
"东北师范大学",
"东北林业大学",
"东华大学",
"东南大学",
"中南大学",
"中南财经政法大学",
"中国人民大学",
"中国传媒大学",
"中国农业大学",
"中国地质大学(北京)",
"中国地质大学(武汉)",
"中国政法大学",
"中国海洋大学",
"中国石油大学(北京)",
"中国石油大学(华东)",
"中国矿业大学",
"中国矿业大学(北京)",
"中国科学技术大学",
"中国药科大学",
"中央民族大学",
"中央财经大学",
"中央音乐学院",
"中山大学",
"云南大学",
"兰州大学",
"内蒙古大学",
"北京中医药大学",
"北京交通大学",
"北京体育大学",
"北京化工大学",
"北京协和医学院",
"北京外国语大学",
"北京大学",
"北京工业大学",
"北京师范大学",
"北京林业大学",
"北京理工大学",
"北京科技大学",
"北京航空航天大学",
"北京邮电大学",
"华东师范大学",
"华东理工大学",
"华中农业大学",
"华中师范大学",
"华中科技大学",
"华北电力大学",
"华南师范大学",
"华南理工大学",
"南京农业大学",
"南京医科大学",
"南京大学",
"南京师范大学",
"南京理工大学",
"南京航空航天大学",
"南开大学",
"南方科技大学",
"南昌大学",
"厦门大学",
"合肥工业大学",
"吉林大学",
"同济大学",
"哈尔滨工业大学",
"哈尔滨工程大学",
"四川大学",
"国防科技大学",
"复旦大学",
"大连海事大学",
"大连理工大学",
"天津中医药大学",
"天津医科大学",
"天津大学",
"天津工业大学",
"太原理工大学",
"宁夏大学",
"安徽大学",
"对外经济贸易大学",
"山东大学",
"广州中医药大学",
"广西大学",
"延边大学",
"成都理工大学",
"新疆大学",
"暨南大学",
"武汉大学",
"武汉理工大学",
"江南大学",
"河北工业大学",
"河海大学",
"浙江大学",
"海南大学",
"清华大学",
"湖南大学",
"湖南师范大学",
"电子科技大学",
"石河子大学",
"福州大学",
"苏州大学",
"西北农林科技大学",
"西北大学",
"西北工业大学",
"西南交通大学",
"西南大学",
"西南财经大学",
"西安交通大学",
"西安电子科技大学",
"西藏大学",
"贵州大学",
"辽宁大学",
"郑州大学",
"重庆大学",
"长安大学",
"青海大学"
],
"sources": [
"https://www.moe.gov.cn/srcsite/A22/s7065/200612/t20061206_128833.html",
"https://gaokao.chsi.com.cn/gkxx/gxmd/211gc.shtml",
"https://www.moe.gov.cn/srcsite/A22/s7065/202202/t20220211_598710.html"
],
"note": "历史工程名单与已知名称别名,用于参考筛选。211以原yzw名单为基础并与学信网教育部名单核对补充、排除中国中医科学院误标。双一流优先实时研招网syl或导入字段;静态2022名单只是已知子集。"
}
+4 -2
View File
@@ -7,8 +7,10 @@ Task 1: complete — 5 tests green; cross-year records, alternatives, conservati
Task 2: complete — 5 tests green; shared proxy, strict route, fallback, rotation, DPAPI vault. Task 2: complete — 5 tests green; shared proxy, strict route, fallback, rotation, DPAPI vault.
Task 3: complete — 5 tests green; pagination, business login status, year mismatch, exact campus, pause/resume. Task 3: complete — 5 tests green; pagination, business login status, year mismatch, exact campus, pause/resume.
Task 4: complete — 5 tests green; file import/export, YAM SQLite, favorites, origin guard, formula-safe exports. Task 4: complete — 5 tests green; file import/export, YAM SQLite, favorites, origin guard, formula-safe exports.
Task 5: pending Task 5: complete — 3 frontend logic tests, production build, full Edge acceptance including mobile passed.
Task 6: pending Task 6: final packaging — launch/reuse/stop/restart verified; README/notices ready; independent review findings fixed with regression tests.
Public index: 2,255 major entries and 460 unique schools extracted from YAM public data files. Seed index contains no admissions snapshots, cookies or signed request tokens. Public index: 2,255 major entries and 460 unique schools extracted from YAM public data files. Seed index contains no admissions snapshots, cookies or signed request tokens.
Final verification: 35 backend tests passed; 3 frontend tests passed; build exit 0; browser QA PASS. Details: docs/verification.md.
+22
View File
@@ -0,0 +1,22 @@
# 验证记录
2026-10-11,Windows、本地独立虚拟环境,Python 3.12.13 / Node 24.16。
- 后端:`python -m pytest -q`,35 passed。Starlette TestClient 提示一条 httpx 迁移 warning,测试退出 0,生产服务不使用 TestClient。
- 前端:`npm test`,3 passed;`npm run build`,TypeScript 检查与 Vite 生产构建退出 0。
- Edge 实际浏览器:独立端口 8766、独立 `.runtime/qa/db`,验证导入→科目筛选→收藏→对比→历史趋势→Excel 下载;任务/代理/设置页面;设置保存;390px 手机宽度无页面溢出;无 pageerror。测试数据均标明「自动验收样例」,没有写入正式数据库。
- 启动脚本:Windows PowerShell 首启、重复启动、停止、重新启动均退出 0;正式服务在 127.0.0.1:8765,健康检查应用身份和项目路径匹配。
- 真实数据源:proxy_pool `/count/` 正常响应;研招页面 `curYear=2027` 可核实;匿名北京大学学校及 081200 专业索引可返回,方向接口要求登录,按 LoginRequired 处理;掌上考研精确匹配北京大学,2025 年 081200 查询返回 1 条门类参考线和 2 条历史计划。真实探测只读取少量公开数据,不写入正式目录。
- 独立只读审查:复现并修复专项线覆盖、同名跨域 Cookie、任务失败恢复竞态、服务退出被误取消、文件往返丢失身份字段、学院人数误升格、YAM 种子误识别、院校关键词范围、参考书 JSON 字符串、业务限流等问题,补充相应回归。
自动验收未使用真人学信网账号,未对全部院校进行实网抓取。登录后的全国采集能力取决于账号状态、代理可用性、开放目录年份和数据源限流。
复跑浏览器验收(只使用测试库):
```powershell
.\.venv\Scripts\python.exe -m studio --port 8766 --data-dir .runtime\qa\db
# 在另一个终端运行
.\.venv\Scripts\python.exe tools\browser_qa.py
```
正式服务不会自动开始采集,需要用户在网页创建任务。
+16 -1
View File
@@ -1,2 +1,17 @@
<!doctype html> <!doctype html>
<html lang="zh-CN"><head><meta charset="UTF-8" /><meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="theme-color" content="#132b3a" /><meta name="description" content="本地考研数据工作台:全国目录、历年考情、代理池与择校对比。" /><title>研途 · 考研数据工作台</title></head><body><div id="root"></div><script type="module" src="/src/main.tsx"></script></body></html> <html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="theme-color" content="#132b3a" />
<meta
name="description"
content="本地考研数据工作台:全国目录、历年考情、代理池与择校对比。"
/>
<title>研途 · 考研数据工作台</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>
+1997 -74
View File
File diff suppressed because one or more lines are too long
+21 -5
View File
@@ -1,9 +1,25 @@
export async function api<T>(path: string, options: RequestInit = {}): Promise<T> { export async function api<T>(
const response = await fetch('/api' + path, { ...options, headers: options.body instanceof FormData ? options.headers : { 'Content-Type': 'application/json', ...options.headers } }); path: string,
options: RequestInit = {},
): Promise<T> {
const response = await fetch("/api" + path, {
...options,
headers:
options.body instanceof FormData
? options.headers
: { "Content-Type": "application/json", ...options.headers },
});
if (!response.ok) { if (!response.ok) {
const body = await response.json().catch(() => ({ detail: `请求失败 (${response.status})` })); const body = await response
throw new Error(typeof body.detail === 'string' ? body.detail : `参数无效 (${response.status})`); .json()
.catch(() => ({ detail: `请求失败 (${response.status})` }));
throw new Error(
typeof body.detail === "string"
? body.detail
: `参数无效 (${response.status})`,
);
} }
return response.json(); return response.json();
} }
export const post = <T,>(path: string, body: unknown = {}) => api<T>(path, { method: 'POST', body: JSON.stringify(body) }); export const post = <T>(path: string, body: unknown = {}) =>
api<T>(path, { method: "POST", body: JSON.stringify(body) });
+18 -12
View File
@@ -1,17 +1,23 @@
import { describe, expect, it } from 'vitest'; import { describe, expect, it } from "vitest";
import { display, scopeName, scoreSeries, queryString } from './format'; import { display, scopeName, scoreSeries, queryString } from "./format";
describe('unknown values and comparable history', () => { describe("unknown values and comparable history", () => {
it('preserves zero and marks unknown', () => { it("preserves zero and marks unknown", () => {
expect(display(0)).toBe('0'); expect(display(0)).toBe("0");
expect(display(null)).toBe('未提供'); expect(display(null)).toBe("未提供");
expect(scopeName('category')).toBe('门类参考线'); expect(scopeName("category")).toBe("门类参考线");
}); });
it('keeps different score scopes separate and omits unknown totals', () => { it("keeps different score scopes separate and omits unknown totals", () => {
const result = scoreSeries([{ year: 2024, total: 310, scope: 'major' }, { year: 2025, total: 300, scope: 'category' }, { year: 2026, total: null, scope: 'major' }]); const result = scoreSeries([
{ year: 2024, total: 310, scope: "major" },
{ year: 2025, total: 300, scope: "category" },
{ year: 2026, total: null, scope: "major" },
]);
expect(result).toHaveLength(2); expect(result).toHaveLength(2);
expect(result.flatMap(r => r.points)).toHaveLength(2); expect(result.flatMap((r) => r.points)).toHaveLength(2);
}); });
it('encodes filters and omits empty values', () => { it("encodes filters and omits empty values", () => {
expect(queryString({ keyword: '北京 大学', year: '', favorites: false })).toBe('keyword=%E5%8C%97%E4%BA%AC+%E5%A4%A7%E5%AD%A6'); expect(
queryString({ keyword: "北京 大学", year: "", favorites: false }),
).toBe("keyword=%E5%8C%97%E4%BA%AC+%E5%A4%A7%E5%AD%A6");
}); });
}); });
+62 -11
View File
@@ -1,16 +1,67 @@
import type { Score, Filters } from './types'; import type { Score, Filters } from "./types";
export const display = (value: unknown): string => value === null || value === undefined || value === '' ? '未提供' : String(value); export const display = (value: unknown): string =>
export const scopeName = (scope?: string | null) => ({ major: '专业线', discipline: '学科参考线', category: '门类参考线', unknown: '口径待核实' }[scope || ''] || '未提供'); value === null || value === undefined || value === ""
export const modeName = (mode: string) => ({ direct: '直接连接', prefer_proxy: '优先代理', proxy_only: '严格代理', catalog: '全国招生目录', scores: '历年考情', combined: '目录 + 历年考情' }[mode] || mode); ? "未提供"
export const statusName = (status: string) => ({ queued: '等待开始', running: '采集中', paused: '已暂停', waiting_login: '等待登录', waiting_proxy: '等待代理', completed: '已完成', partial: '部分失败', failed: '采集失败', cancelled: '已取消' }[status] || status); : String(value);
export const queryString = (filters: Filters) => new URLSearchParams(Object.entries(filters).filter(([, v]) => v !== '' && v !== false).map(([k, v]) => [k, String(v)])).toString(); export const scopeName = (scope?: string | null) =>
({
major: "专业线",
discipline: "学科参考线",
category: "门类参考线",
unknown: "口径待核实",
})[scope || ""] || "未提供";
export const modeName = (mode: string) =>
({
direct: "直接连接",
prefer_proxy: "优先代理",
proxy_only: "严格代理",
catalog: "全国招生目录",
scores: "历年考情",
combined: "目录 + 历年考情",
})[mode] || mode;
export const statusName = (status: string) =>
({
queued: "等待开始",
running: "采集中",
paused: "已暂停",
waiting_login: "等待登录",
waiting_proxy: "等待代理",
waiting_source: "等待限流解除",
completed: "已完成",
partial: "部分失败",
failed: "采集失败",
cancelled: "已取消",
})[status] || status;
export const queryString = (filters: Filters) =>
new URLSearchParams(
Object.entries(filters)
.filter(([, v]) => v !== "" && v !== false)
.map(([k, v]) => [k, String(v)]),
).toString();
export function scoreSeries(scores: Score[]) { export function scoreSeries(scores: Score[]) {
const grouped = new Map<string, { label: string; points: { year: number; total: number }[] }>(); const grouped = new Map<
scores.filter(s => s.total !== null).forEach(s => { string,
const key = [s.scope, s.raw_code || '', s.department_name || '', s.source || '', s.note || ''].join('|'); { label: string; points: { year: number; total: number }[] }
const group = grouped.get(key) || { label: scopeName(s.scope) + (s.note ? ` · ${s.note}` : ''), points: [] }; >();
scores
.filter((s) => s.total !== null)
.forEach((s) => {
const key = [
s.scope,
s.raw_code || "",
s.department_name || "",
s.source || "",
s.note || "",
].join("|");
const group = grouped.get(key) || {
label: scopeName(s.scope) + (s.note ? ` · ${s.note}` : ""),
points: [],
};
group.points.push({ year: s.year, total: s.total! }); group.points.push({ year: s.year, total: s.total! });
grouped.set(key, group); grouped.set(key, group);
}); });
return [...grouped.values()].map(g => ({ ...g, points: g.points.sort((a, b) => a.year - b.year) })); return [...grouped.values()].map((g) => ({
...g,
points: g.points.sort((a, b) => a.year - b.year),
}));
} }
+9 -5
View File
@@ -1,5 +1,9 @@
import { StrictMode } from 'react'; import { StrictMode } from "react";
import { createRoot } from 'react-dom/client'; import { createRoot } from "react-dom/client";
import App from './App'; import App from "./App";
import './style.css'; import "./style.css";
createRoot(document.getElementById('root')!).render(<StrictMode><App /></StrictMode>); createRoot(document.getElementById("root")!).render(
<StrictMode>
<App />
</StrictMode>,
);
+1529 -243
View File
File diff suppressed because it is too large Load Diff
+113 -9
View File
@@ -1,12 +1,116 @@
export type Row = { id: number; year: number; school_name: string; school_code: string; province: string; level: string; major_code: string; major_name: string; department_name: string; department_code: string; direction: string; subjects: string[]; quota: number | null; quota_text: string; quota_scope: string; degree_type: string; study_mode: string; exam_type: string; advisor: string; special_plans: string[]; remarks: string; source: string; source_url: string; fetched_at: string; year_verified: boolean; favorite: boolean; score_total: number | null; score_year: number | null; score_scope: string | null; reference_books?: string[] }; export type Row = {
export type Score = { year: number; total: number | null; scope: string; raw_code?: string; department_name?: string; source?: string; note?: string; politics?: number | null; english?: number | null; special_one?: number | null; special_two?: number | null }; id: number;
year: number;
school_name: string;
school_code: string;
province: string;
level: string;
major_code: string;
major_name: string;
department_name: string;
department_code: string;
direction: string;
subjects: string[];
quota: number | null;
quota_text: string;
quota_scope: string;
degree_type: string;
study_mode: string;
exam_type: string;
advisor: string;
special_plans: string[];
remarks: string;
source: string;
source_url: string;
fetched_at: string;
year_verified: boolean;
favorite: boolean;
score_total: number | null;
score_year: number | null;
score_scope: string | null;
reference_books?: string[];
};
export type Score = {
year: number;
total: number | null;
scope: string;
raw_code?: string;
department_name?: string;
source?: string;
note?: string;
politics?: number | null;
english?: number | null;
special_one?: number | null;
special_two?: number | null;
};
export type History = { record: Row; scores: Score[]; plans: Row[] }; export type History = { record: Row; scores: Score[]; plans: Row[] };
export type Job = { id: string; params: { mode: string; year: number; major_code?: string; provinces: string[] }; status: string; message: string; created_at: string; updated_at: string; records: number; errors: number; completed_units: number }; export type Job = {
export type Log = { id: number; created_at: string; level: string; message: string }; id: string;
params: {
mode: string;
year: number;
major_code?: string;
provinces: string[];
};
status: string;
message: string;
created_at: string;
updated_at: string;
records: number;
errors: number;
completed_units: number;
};
export type Log = {
id: number;
created_at: string;
level: string;
message: string;
};
export type Province = { code: string; name: string }; export type Province = { code: string; name: string };
export type Major = { code: string; name: string; degree_type: string; category: string; discipline: string }; export type Major = {
export type Overview = { records: number; schools: number; majors: number; favorites: number; indexed_schools: number; scores: number; years: number[]; provinces: { province: string; count: number }[]; provinces_list: Province[]; sources: { name: string; role: string; url: string }[] }; code: string;
export type Config = { network_mode: string; pool_url: string; pool_project: string; backup_proxies: string[]; request_interval: number; timeout: number; retries: number }; name: string;
export type Proxy = { mode: string; route: string; active_proxy: string | null; fallback: boolean; counts: { count?: number; http_type?: { https: number; http: number } }; available: number; quarantined: number; error: string; proxies: { url: string; failed: boolean; active: boolean }[] }; degree_type: string;
export type Auth = { saved: boolean; valid: boolean; message: string; browser_state: string; browser_message: string }; category: string;
discipline: string;
};
export type Overview = {
records: number;
schools: number;
majors: number;
favorites: number;
indexed_schools: number;
scores: number;
years: number[];
provinces: { province: string; count: number }[];
provinces_list: Province[];
sources: { name: string; role: string; url: string }[];
};
export type Config = {
network_mode: string;
pool_url: string;
pool_project: string;
backup_proxies: string[];
request_interval: number;
timeout: number;
retries: number;
};
export type Proxy = {
mode: string;
route: string;
active_proxy: string | null;
fallback: boolean;
counts: { count?: number; http_type?: { https: number; http: number } };
available: number;
quarantined: number;
error: string;
proxies: { url: string; failed: boolean; active: boolean }[];
};
export type Auth = {
saved: boolean;
valid: boolean;
message: string;
browser_state: string;
browser_message: string;
};
export type Filters = Record<string, string | number | boolean>; export type Filters = Record<string, string | number | boolean>;
+14 -2
View File
@@ -1,2 +1,14 @@
{"compilerOptions":{"target":"ES2022","lib":["ES2022","DOM","DOM.Iterable"],"module":"ESNext","moduleResolution":"bundler","jsx":"react-jsx","strict":true,"skipLibCheck":true,"noEmit":true,"allowImportingTsExtensions":true},"include":["src","vite.config.ts"]} {
"compilerOptions": {
"target": "ES2022",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"module": "ESNext",
"moduleResolution": "bundler",
"jsx": "react-jsx",
"strict": true,
"skipLibCheck": true,
"noEmit": true,
"allowImportingTsExtensions": true
},
"include": ["src", "vite.config.ts"]
}
+7 -4
View File
@@ -1,4 +1,7 @@
import { defineConfig } from 'vite'; import { defineConfig } from "vite";
import react from '@vitejs/plugin-react'; import react from "@vitejs/plugin-react";
export default defineConfig({plugins:[react()],server:{proxy:{'/api':'http://127.0.0.1:8765'}},build:{sourcemap:false}}); export default defineConfig({
plugins: [react()],
server: { proxy: { "/api": "http://127.0.0.1:8765" } },
build: { sourcemap: false },
});
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 freecho
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+6 -3
View File
@@ -13,7 +13,7 @@ from studio.database import Database
from studio.domain import PROVINCES from studio.domain import PROVINCES
from studio.imports import import_file, export_sheet from studio.imports import import_file, export_sheet
from studio.jobs import JobManager from studio.jobs import JobManager
from studio.network import ProxyRouter, Transport, ProxyUnavailable, RemoteUnavailable from studio.network import ProxyRouter, Transport, ProxyUnavailable, RemoteUnavailable, RateLimited
from studio.settings import SettingsStore from studio.settings import SettingsStore
ROOT = Path(__file__).resolve().parent.parent ROOT = Path(__file__).resolve().parent.parent
@@ -60,6 +60,7 @@ def create_app(data_dir=None):
@app.exception_handler(ProxyUnavailable) @app.exception_handler(ProxyUnavailable)
@app.exception_handler(RemoteUnavailable) @app.exception_handler(RemoteUnavailable)
@app.exception_handler(RateLimited)
async def remote_error(request, exc): async def remote_error(request, exc):
return JSONResponse({"detail": str(exc)}, status_code=503) return JSONResponse({"detail": str(exc)}, status_code=503)
@@ -188,10 +189,12 @@ def create_app(data_dir=None):
if total > 100 * 1024 * 1024: if total > 100 * 1024 * 1024:
raise ValueError("一次最多上传 100 MB,请拆分文件") raise ValueError("一次最多上传 100 MB,请拆分文件")
stream.write(chunk) stream.write(chunk)
records, scores = import_file(temporary, year) result = import_file(temporary, year)
records, scores = result
db.upsert_catalog(records) db.upsert_catalog(records)
db.upsert_scores(scores) db.upsert_scores(scores)
return {"records": len(records), "scores": len(scores), "message": "导入完成,相同记录按年份更新"} db.index_schools(result.schools)
return {"records": len(records), "scores": len(scores), "indexed_schools": len(result.schools), "message": "导入完成,相同记录按年份更新;无详情的 YAM 种子只作为院校索引"}
except (UnicodeError, json.JSONDecodeError, KeyError) as exc: except (UnicodeError, json.JSONDecodeError, KeyError) as exc:
raise ValueError(f"文件编码或结构无法识别({type(exc).__name__})") from None raise ValueError(f"文件编码或结构无法识别({type(exc).__name__})") from None
finally: finally:
+17 -6
View File
@@ -60,14 +60,24 @@ class AuthService:
self.browser_thread = None self.browser_thread = None
try: try:
self._load(self.vault.load()) self._load(self.vault.load())
except RuntimeError: except (RuntimeError, OSError, ValueError):
self.message = "保存的 Cookie 无法解密,请重新登录" self.message = "保存的 Cookie 无法解密,请重新登录"
def _load(self, cookies): def _load(self, cookies):
if isinstance(cookies, dict):
cookies = [{"name": name, "value": value, "domain": ".chsi.com.cn", "path": "/"} for name, value in cookies.items()]
with self.transport.lock: with self.transport.lock:
self.transport.session.cookies.clear() self.transport.session.cookies.clear()
for name, value in cookies.items(): for cookie in cookies:
self.transport.session.cookies.set(name, value, domain=".chsi.com.cn", path="/") domain = str(cookie.get("domain", ".chsi.com.cn"))
if domain.lstrip(".") != "chsi.com.cn" and not domain.endswith(".chsi.com.cn"):
continue
expires = cookie.get("expires")
self.transport.session.cookies.set(cookie["name"], cookie["value"], domain=domain, path=cookie.get("path", "/"), secure=bool(cookie.get("secure")), expires=int(expires) if expires and expires > 0 else None)
def _snapshot(self):
with self.transport.lock:
return [{"name": c.name, "value": c.value, "domain": c.domain, "path": c.path, "secure": c.secure, "expires": c.expires} for c in self.transport.session.cookies]
def status(self): def status(self):
return {"saved": self.vault.path.exists(), "valid": self.valid, "message": self.message, "browser_state": self.browser_state, "browser_message": self.browser_message} return {"saved": self.vault.path.exists(), "valid": self.valid, "message": self.message, "browser_state": self.browser_state, "browser_message": self.browser_message}
@@ -91,7 +101,8 @@ class AuthService:
return self.status() return self.status()
def check(self): def check(self):
response = self.transport.request("POST", "https://yz.chsi.com.cn/zsml/rs/dws.do", data={"ssdm": "11", "curPage": 1, "start": 0, "pageSize": 1, "dwlxs[0]": "all"}) # The school/major indexes can be public while direction details require login.
response = self.transport.request("POST", "https://yz.chsi.com.cn/zsml/rs/yjfxs.do", data={"dwdm": "10001", "zydm": "081200", "zymc": "计算机科学与技术", "curPage": 1, "start": 0, "pageSize": 1, "dwlxs[0]": "all"})
try: try:
body = response.json() body = response.json()
except ValueError: except ValueError:
@@ -101,7 +112,7 @@ class AuthService:
self.valid = body.get("flag") is True and isinstance(body.get("msg"), dict) self.valid = body.get("flag") is True and isinstance(body.get("msg"), dict)
self.message = "研招目录接口验证成功" if self.valid else "登录未生效或数据源限流,请在浏览器完成登录后重试" self.message = "研招目录接口验证成功" if self.valid else "登录未生效或数据源限流,请在浏览器完成登录后重试"
if self.valid: if self.valid:
self.vault.save(self.transport.session.cookies.get_dict()) self.vault.save(self._snapshot())
return self.status() return self.status()
def login(self, username, password): def login(self, username, password):
@@ -144,7 +155,7 @@ class AuthService:
while time.monotonic() < deadline: while time.monotonic() < deadline:
page.wait_for_timeout(2000) page.wait_for_timeout(2000)
if "yz.chsi.com.cn" in page.url: if "yz.chsi.com.cn" in page.url:
cookies = {c["name"]: c["value"] for c in context.cookies() if "chsi.com.cn" in c["domain"]} cookies = [c for c in context.cookies() if c["domain"].lstrip(".") == "chsi.com.cn" or c["domain"].endswith(".chsi.com.cn")]
if cookies: if cookies:
self._load(cookies) self._load(cookies)
if self.check()["valid"]: if self.check()["valid"]:
+20 -5
View File
@@ -2,7 +2,7 @@
import json import json
import re import re
from studio.domain import finish_record, integer, normalized_name, select_score_lines from studio.domain import finish_record, integer, normalized_name, select_score_lines
from studio.network import RemoteUnavailable from studio.network import RemoteUnavailable, RateLimited
class LoginRequired(RuntimeError): class LoginRequired(RuntimeError):
@@ -35,17 +35,25 @@ class CatalogClient:
def pages(self, endpoint, payload, page_size=50): def pages(self, endpoint, payload, page_size=50):
seen = set() seen = set()
for page in range(1, 10001): for page in range(1, 10001):
for attempt in range(3):
response = self.transport.request("POST", self.BASE + "rs/" + endpoint + ".do", data={**payload, "curPage": page, "start": (page - 1) * page_size, "pageSize": page_size}) response = self.transport.request("POST", self.BASE + "rs/" + endpoint + ".do", data={**payload, "curPage": page, "start": (page - 1) * page_size, "pageSize": page_size})
try: try:
body = response.json() body = response.json()
except ValueError: except ValueError:
raise LoginRequired("研招网返回登录/验证页面,请先完成登录") from None raise LoginRequired("研招网返回登录/验证页面,请先完成登录") from None
message = str(body.get('msg', ''))
if body.get('flag') is not True and '验证码' in message:
raise LoginRequired('研招网需要人工验证码,请在浏览器完成验证后恢复')
if body.get('flag') is not True and any(token in message for token in ('频繁', '稍后')):
if attempt == 2:
raise RateLimited('研招网业务限流持续出现,已暂停;请增加请求间隔后恢复')
self.transport.sleep(5 * 2 ** attempt)
continue
break
if body.get("flag") is not True: if body.get("flag") is not True:
message = str(body.get("msg", "")) message = str(body.get("msg", ""))
if any(token in message for token in ("登录", "登陆", "认证")): if any(token in message for token in ("登录", "登陆", "认证")):
raise LoginRequired("研招网要求登录,请更新 Cookie 后恢复任务") raise LoginRequired("研招网要求登录,请更新 Cookie 后恢复任务")
if any(token in message for token in ("频繁", "稍后", "验证码")):
raise RemoteUnavailable("研招网访问受限,请增加请求间隔并完成人工验证后恢复")
raise RemoteUnavailable("研招网目录接口未成功返回数据") raise RemoteUnavailable("研招网目录接口未成功返回数据")
data = body.get("msg") or {} data = body.get("msg") or {}
if not isinstance(data, dict) or not isinstance(data.get("list"), list): if not isinstance(data, dict) or not isinstance(data.get("list"), list):
@@ -83,14 +91,21 @@ class ScoreClient:
self.cache = {} self.cache = {}
def post(self, endpoint, payload): def post(self, endpoint, payload):
for attempt in range(3):
response = self.transport.request("POST", self.BASE + endpoint, data=payload, headers={"Referer": "https://www.kaoyan.cn/"}) response = self.transport.request("POST", self.BASE + endpoint, data=payload, headers={"Referer": "https://www.kaoyan.cn/"})
try: try:
body = response.json() body = response.json()
except ValueError: except ValueError:
raise RemoteUnavailable("掌上考研未返回有效 JSON") from None raise RemoteUnavailable("掌上考研未返回有效 JSON") from None
if str(body.get("code")) != "0000": if str(body.get("code")) == "0000":
raise RemoteUnavailable("掌上考研接口未成功返回数据,请稍后重试")
return body.get("data") or {} return body.get("data") or {}
message = str(body.get('message', body.get('msg', '')))
if any(token in message for token in ('频繁', '稍后', '限流', '繁忙')):
if attempt == 2:
raise RateLimited('掌上考研业务限流持续出现,请增加请求间隔后恢复')
self.transport.sleep(5 * 2 ** attempt)
else:
raise RemoteUnavailable("掌上考研接口未成功返回数据,请稍后重试")
def pages(self, endpoint, payload, limit=100): def pages(self, endpoint, payload, limit=100):
seen = set() seen = set()
+23 -7
View File
@@ -3,8 +3,15 @@ import json
import sqlite3 import sqlite3
from contextlib import contextmanager from contextlib import contextmanager
from pathlib import Path from pathlib import Path
from studio.domain import now, normalized_name from studio.domain import now, normalized_name, is_special_score, school_level
SCORE_SCHEMA = """
CREATE TABLE IF NOT EXISTS scores(
id INTEGER PRIMARY KEY, school_code TEXT, major_code TEXT, year INTEGER,
department_id TEXT, department_name TEXT, scope TEXT, total INTEGER, source TEXT, data TEXT NOT NULL,
note TEXT NOT NULL DEFAULT '',raw_code TEXT NOT NULL DEFAULT '',
UNIQUE(school_code,major_code,year,department_id,department_name,scope,source,note,raw_code));
"""
SCHEMA = """ SCHEMA = """
CREATE TABLE IF NOT EXISTS catalog( CREATE TABLE IF NOT EXISTS catalog(
id INTEGER PRIMARY KEY, record_key TEXT NOT NULL UNIQUE, year INTEGER NOT NULL, id INTEGER PRIMARY KEY, record_key TEXT NOT NULL UNIQUE, year INTEGER NOT NULL,
@@ -12,10 +19,7 @@ CREATE TABLE IF NOT EXISTS catalog(
department_code TEXT, department_name TEXT, study_mode TEXT, degree_type TEXT, level TEXT, department_code TEXT, department_name TEXT, study_mode TEXT, degree_type TEXT, level TEXT,
subjects TEXT, quota INTEGER, source TEXT, favorite INTEGER NOT NULL DEFAULT 0, data TEXT NOT NULL); subjects TEXT, quota INTEGER, source TEXT, favorite INTEGER NOT NULL DEFAULT 0, data TEXT NOT NULL);
CREATE INDEX IF NOT EXISTS catalog_filter ON catalog(year,province,major_code,school_code); CREATE INDEX IF NOT EXISTS catalog_filter ON catalog(year,province,major_code,school_code);
CREATE TABLE IF NOT EXISTS scores( """ + SCORE_SCHEMA + """
id INTEGER PRIMARY KEY, school_code TEXT, major_code TEXT, year INTEGER,
department_id TEXT, department_name TEXT, scope TEXT, total INTEGER, source TEXT, data TEXT NOT NULL,
UNIQUE(school_code,major_code,year,department_id,department_name,scope,source));
CREATE INDEX IF NOT EXISTS scores_lookup ON scores(school_code,major_code,year DESC); CREATE INDEX IF NOT EXISTS scores_lookup ON scores(school_code,major_code,year DESC);
CREATE TABLE IF NOT EXISTS schools(school_code TEXT PRIMARY KEY, school_name TEXT, province TEXT, level TEXT, data TEXT); CREATE TABLE IF NOT EXISTS schools(school_code TEXT PRIMARY KEY, school_name TEXT, province TEXT, level TEXT, data TEXT);
CREATE TABLE IF NOT EXISTS jobs(id TEXT PRIMARY KEY, params TEXT, status TEXT, created_at TEXT, updated_at TEXT, message TEXT, errors INTEGER DEFAULT 0, records INTEGER DEFAULT 0); CREATE TABLE IF NOT EXISTS jobs(id TEXT PRIMARY KEY, params TEXT, status TEXT, created_at TEXT, updated_at TEXT, message TEXT, errors INTEGER DEFAULT 0, records INTEGER DEFAULT 0);
@@ -34,6 +38,13 @@ class Database:
self.path.parent.mkdir(parents=True, exist_ok=True) self.path.parent.mkdir(parents=True, exist_ok=True)
with self.connection() as conn: with self.connection() as conn:
conn.executescript(SCHEMA) conn.executescript(SCHEMA)
if 'note' not in {row['name'] for row in conn.execute('PRAGMA table_info(scores)')}:
# Rebuild atomically to replace the early schema's narrower unique key.
conn.execute(SCORE_SCHEMA.replace('IF NOT EXISTS scores', 'scores_new'))
conn.execute("""INSERT INTO scores_new(id,school_code,major_code,year,department_id,department_name,scope,total,source,data,note,raw_code) SELECT id,school_code,major_code,year,department_id,department_name,scope,total,source,data,coalesce(json_extract(data,'$.note'),''),coalesce(json_extract(data,'$.raw_code'),'') FROM scores""")
conn.execute('DROP TABLE scores')
conn.execute('ALTER TABLE scores_new RENAME TO scores')
conn.execute('CREATE INDEX scores_lookup ON scores(school_code,major_code,year DESC)')
conn.execute("UPDATE jobs SET status='paused',message='服务重启,任务可从断点恢复' WHERE status IN ('running','queued')") conn.execute("UPDATE jobs SET status='paused',message='服务重启,任务可从断点恢复' WHERE status IN ('running','queued')")
@contextmanager @contextmanager
@@ -41,6 +52,7 @@ class Database:
conn = sqlite3.connect(self.path, timeout=30) conn = sqlite3.connect(self.path, timeout=30)
conn.row_factory = sqlite3.Row conn.row_factory = sqlite3.Row
conn.create_function("norm_name", 1, normalized_name, deterministic=True) conn.create_function("norm_name", 1, normalized_name, deterministic=True)
conn.create_function("is_special_score", 1, is_special_score, deterministic=True)
conn.execute("PRAGMA journal_mode=WAL") conn.execute("PRAGMA journal_mode=WAL")
try: try:
yield conn yield conn
@@ -66,6 +78,7 @@ class Database:
self._school(conn, record) self._school(conn, record)
def _school(self, conn, record): def _school(self, conn, record):
record = {**record, 'level': school_level(record['school_name'], record.get('level', ''))}
conn.execute("INSERT INTO schools VALUES(?,?,?,?,?) ON CONFLICT(school_code) DO UPDATE SET school_name=excluded.school_name,province=CASE WHEN excluded.province<>'' THEN excluded.province ELSE schools.province END,level=CASE WHEN excluded.level<>'' THEN excluded.level ELSE schools.level END", (record["school_code"],record["school_name"],record.get("province", ""),record.get("level", ""),encode(record))) conn.execute("INSERT INTO schools VALUES(?,?,?,?,?) ON CONFLICT(school_code) DO UPDATE SET school_name=excluded.school_name,province=CASE WHEN excluded.province<>'' THEN excluded.province ELSE schools.province END,level=CASE WHEN excluded.level<>'' THEN excluded.level ELSE schools.level END", (record["school_code"],record["school_name"],record.get("province", ""),record.get("level", ""),encode(record)))
def index_schools(self, rows): def index_schools(self, rows):
@@ -76,7 +89,7 @@ class Database:
def upsert_scores(self, records): def upsert_scores(self, records):
with self.connection() as conn: with self.connection() as conn:
for row in records: for row in records:
conn.execute("""INSERT INTO scores(school_code,major_code,year,department_id,department_name,scope,total,source,data) VALUES(?,?,?,?,?,?,?,?,?) ON CONFLICT(school_code,major_code,year,department_id,department_name,scope,source) DO UPDATE SET total=excluded.total,data=excluded.data""",(row["school_code"],row["major_code"],row["year"],row.get("department_id", ""),row.get("department_name", ""),row["scope"],row.get("total"),row.get("source", "掌上考研"),encode(row))) conn.execute("""INSERT INTO scores(school_code,major_code,year,department_id,department_name,scope,total,source,data,note,raw_code) VALUES(?,?,?,?,?,?,?,?,?,?,?) ON CONFLICT(school_code,major_code,year,department_id,department_name,scope,source,note,raw_code) DO UPDATE SET total=excluded.total,data=excluded.data""",(row["school_code"],row["major_code"],row["year"],row.get("department_id", "") or "",row.get("department_name", "") or "",row["scope"],row.get("total"),row.get("source", "掌上考研"),encode(row),row.get('note') or '',row.get('raw_code') or ''))
def _query(self, filters: dict): def _query(self, filters: dict):
conditions, params = ["1=1"], [] conditions, params = ["1=1"], []
@@ -95,6 +108,9 @@ class Database:
if filters.get("keyword"): if filters.get("keyword"):
conditions.append("(c.school_name LIKE ? OR c.major_name LIKE ? OR c.department_name LIKE ? OR c.major_code LIKE ?)") conditions.append("(c.school_name LIKE ? OR c.major_name LIKE ? OR c.department_name LIKE ? OR c.major_code LIKE ?)")
params.extend([f"%{filters['keyword']}%"] * 4) params.extend([f"%{filters['keyword']}%"] * 4)
if filters.get('school_keyword'):
conditions.append('c.school_name LIKE ?')
params.append(f"%{filters['school_keyword']}%")
if filters.get("favorites"): if filters.get("favorites"):
conditions.append("c.favorite=1") conditions.append("c.favorite=1")
if filters.get("min_quota") is not None: if filters.get("min_quota") is not None:
@@ -104,7 +120,7 @@ class Database:
if filters.get(field): if filters.get(field):
conditions.append(f"json_extract(c.data,'$.subjects[{index}]') LIKE ?") conditions.append(f"json_extract(c.data,'$.subjects[{index}]') LIKE ?")
params.append(str(filters[field])+"%") params.append(str(filters[field])+"%")
select = """SELECT c.*,s.data AS score_data,s.total AS score_total,s.year AS score_year,s.scope AS score_scope FROM catalog c LEFT JOIN scores s ON s.id=(SELECT x.id FROM scores x WHERE x.school_code=c.school_code AND x.major_code=c.major_code AND x.year<=c.year AND (x.department_name='' AND x.department_id='' OR x.department_id<>'' AND x.department_id=c.department_code OR x.department_name<>'' AND norm_name(x.department_name)=norm_name(c.department_name)) AND coalesce(json_extract(x.data,'$.note'),'') NOT LIKE '%专项%' AND coalesce(json_extract(x.data,'$.note'),'') NOT LIKE '%少干%' ORDER BY x.year DESC,CASE x.scope WHEN 'major' THEN 0 WHEN 'discipline' THEN 1 ELSE 2 END,x.id DESC LIMIT 1) WHERE """ + " AND ".join(conditions) select = """SELECT c.*,s.data AS score_data,s.total AS score_total,s.year AS score_year,s.scope AS score_scope FROM catalog c LEFT JOIN scores s ON s.id=(SELECT x.id FROM scores x WHERE x.school_code=c.school_code AND x.major_code=c.major_code AND x.year<=c.year AND (x.department_name='' AND x.department_id='' OR x.department_id<>'' AND x.department_id=c.department_code OR x.department_name<>'' AND norm_name(x.department_name)=norm_name(c.department_name)) AND is_special_score(json_extract(x.data,'$.note'))=0 ORDER BY x.year DESC,CASE x.scope WHEN 'major' THEN 0 WHEN 'discipline' THEN 1 ELSE 2 END,x.id DESC LIMIT 1) WHERE """ + " AND ".join(conditions)
if filters.get("max_score") is not None: if filters.get("max_score") is not None:
select = "SELECT * FROM ("+select+") q WHERE q.score_total<=?" select = "SELECT * FROM ("+select+") q WHERE q.score_total<=?"
params.append(filters["max_score"]) params.append(filters["max_score"])
+27
View File
@@ -4,11 +4,27 @@ import json
import re import re
from datetime import datetime, timezone, timedelta from datetime import datetime, timezone, timedelta
from typing import Any from typing import Any
from pathlib import Path
from functools import lru_cache
from urllib.parse import urlencode from urllib.parse import urlencode
PROVINCES = {"11":"北京","12":"天津","13":"河北","14":"山西","15":"内蒙古","21":"辽宁","22":"吉林","23":"黑龙江","31":"上海","32":"江苏","33":"浙江","34":"安徽","35":"福建","36":"江西","37":"山东","41":"河南","42":"湖北","43":"湖南","44":"广东","45":"广西","46":"海南","50":"重庆","51":"四川","52":"贵州","53":"云南","54":"西藏","61":"陕西","62":"甘肃","63":"青海","64":"宁夏","65":"新疆"} PROVINCES = {"11":"北京","12":"天津","13":"河北","14":"山西","15":"内蒙古","21":"辽宁","22":"吉林","23":"黑龙江","31":"上海","32":"江苏","33":"浙江","34":"安徽","35":"福建","36":"江西","37":"山东","41":"河南","42":"湖北","43":"湖南","44":"广东","45":"广西","46":"海南","50":"重庆","51":"四川","52":"贵州","53":"云南","54":"西藏","61":"陕西","62":"甘肃","63":"青海","64":"宁夏","65":"新疆"}
@lru_cache(maxsize=1)
def school_metadata():
path = Path(__file__).resolve().parent.parent / 'data/school_levels.json'
data = json.loads(path.read_text(encoding='utf-8')) if path.exists() else {}
return {k: {normalized_name(n) for n in data.get(k, [])} for k in ('985','211','double_first_class_2022_known')}
def school_level(name, existing=''):
metadata = school_metadata()
name = normalized_name(name)
levels = [label for key, label in [('985','985'),('211','211'),('double_first_class_2022_known','双一流')] if name in metadata[key] or label in str(existing).split('/')]
return '/'.join(levels) or existing
def now() -> str: def now() -> str:
return datetime.now(timezone(timedelta(hours=8))).isoformat(timespec="seconds") return datetime.now(timezone(timedelta(hours=8))).isoformat(timespec="seconds")
@@ -17,6 +33,16 @@ def normalized_name(value: str) -> str:
return str(value or "").replace("(", "(").replace(")", ")").replace(" ", "").strip() return str(value or "").replace("(", "(").replace(")", ")").replace(" ", "").strip()
def is_special_score(note):
text = str(note or '')
tokens = ('专项', '少干', '少骨', '少数民族骨干', '退役大学生士兵', '退役士兵', '强军', '援藏', '单独考试')
for token in tokens:
cleaned = re.sub(r'(?:不含|不包括|除外)\s*' + re.escape(token), '', text)
if token in cleaned:
return True
return False
def integer(value: Any) -> int | None: def integer(value: Any) -> int | None:
if value is None or isinstance(value, bool): if value is None or isinstance(value, bool):
return None return None
@@ -47,6 +73,7 @@ def record_key(record: dict) -> str:
def finish_record(record: dict) -> dict: def finish_record(record: dict) -> dict:
defaults = {"school_code":"", "school_name":"", "province":"", "major_code":"", "major_name":"", "department_code":"", "department_name":"", "direction_code":"", "direction":"", "study_mode":"", "degree_type":"", "exam_type":"", "advisor":"", "quota_text":"", "quota":None, "quota_scope":"unknown", "subjects":[], "special_plans":[], "level":"", "remarks":"", "source":"", "source_url":"", "fetched_at":now(), "year_verified":False} defaults = {"school_code":"", "school_name":"", "province":"", "major_code":"", "major_name":"", "department_code":"", "department_name":"", "direction_code":"", "direction":"", "study_mode":"", "degree_type":"", "exam_type":"", "advisor":"", "quota_text":"", "quota":None, "quota_scope":"unknown", "subjects":[], "special_plans":[], "level":"", "remarks":"", "source":"", "source_url":"", "fetched_at":now(), "year_verified":False}
row = {**defaults, **record} row = {**defaults, **record}
row['level'] = school_level(row['school_name'], row['level'])
if not row["school_code"]: if not row["school_code"]:
row["school_code"] = "name:" + normalized_name(row["school_name"]) row["school_code"] = "name:" + normalized_name(row["school_name"])
row["record_key"] = record_key(row) row["record_key"] = record_key(row)
+34 -6
View File
@@ -3,15 +3,28 @@ import csv
import io import io
import json import json
import sqlite3 import sqlite3
from dataclasses import dataclass, field
from pathlib import Path from pathlib import Path
from openpyxl import load_workbook, Workbook from openpyxl import load_workbook, Workbook
from studio.domain import finish_record, integer, parse_quota from studio.domain import finish_record, integer, parse_quota, normalize_catalog, PROVINCES
LABELS = {"year": "年份", "school_name": "院校", "school_code": "院校代码", "province": "省份", "level": "院校层次", "major_code": "专业代码", "major_name": "专业", "degree_type": "学位类型", "department_name": "院系", "direction": "研究方向", "study_mode": "学习方式", "exam_type": "考试方式", "advisor": "导师", "quota_text": "招生人数原文", "quota": "明确人数", "quota_scope": "人数口径", "subjects": "考试科目", "special_plans": "专项计划", "score_year": "参考线年份", "score_total": "参考线总分", "score_scope": "分数线口径", "remarks": "备注", "source": "来源", "source_url": "来源链接", "year_verified": "来源年份已核实", "fetched_at": "采集时间"} LABELS = {"year": "年份", "school_name": "院校", "school_code": "院校代码", "province": "省份", "level": "院校层次", "major_code": "专业代码", "major_name": "专业", "degree_type": "学位类型", "department_name": "院系", "direction": "研究方向", "study_mode": "学习方式", "exam_type": "考试方式", "advisor": "导师", "quota_text": "招生人数原文", "quota": "明确人数", "quota_scope": "人数口径", "subjects": "考试科目", "special_plans": "专项计划", "score_year": "参考线年份", "score_total": "参考线总分", "score_scope": "分数线口径", "remarks": "备注", "source": "来源", "source_url": "来源链接", "year_verified": "来源年份已核实", "fetched_at": "采集时间"}
LABELS.update({"department_code": "院系代码", "direction_code": "方向代码"})
ALIASES = {"院校名称": "school_name", "学校名称": "school_name", "专业名称": "major_name", "院系所": "department_name", "拟招生人数": "quota_text", "planned_enrollment": "quota_text", "nzsrsstr": "quota_text", "school_level": "level", "department": "department_name", "research_direction": "direction", "enrollment_text": "quota_text", "enrollment_count": "quota", "exam_subjects": "subjects"} ALIASES = {"院校名称": "school_name", "学校名称": "school_name", "专业名称": "major_name", "院系所": "department_name", "拟招生人数": "quota_text", "planned_enrollment": "quota_text", "nzsrsstr": "quota_text", "school_level": "level", "department": "department_name", "research_direction": "direction", "enrollment_text": "quota_text", "enrollment_count": "quota", "exam_subjects": "subjects"}
ALIASES.update({v: k for k, v in LABELS.items()}) ALIASES.update({v: k for k, v in LABELS.items()})
@dataclass
class ImportResult:
records: list
scores: list
schools: list = field(default_factory=list)
def __iter__(self):
yield self.records
yield self.scores
def list_field(value): def list_field(value):
if isinstance(value, list): if isinstance(value, list):
return value return value
@@ -44,7 +57,7 @@ def row_record(raw, year=None, source="文件导入"):
if not subjects: if not subjects:
subjects = [str(row.get(f"exam_subject{i}") or row.get(f"subject{i}") or row.get(f"考试科目{i}") or "") for i in range(1, 5)] subjects = [str(row.get(f"exam_subject{i}") or row.get(f"subject{i}") or row.get(f"考试科目{i}") or "") for i in range(1, 5)]
quota, scope = parse_quota(row.get("quota_text", "")) quota, scope = parse_quota(row.get("quota_text", ""))
if quota is None and row.get("quota") != "": if quota is None and not str(row.get('quota_text') or '').strip():
quota = integer(row.get("quota")) quota = integer(row.get("quota"))
scope = row.get("quota_scope") or "unspecified" scope = row.get("quota_scope") or "unspecified"
plans = list_field(row.get("special_plans")) plans = list_field(row.get("special_plans"))
@@ -52,7 +65,7 @@ def row_record(raw, year=None, source="文件导入"):
plans.append("退役大学生士兵") plans.append("退役大学生士兵")
if row.get("shaogu_program") in ("是", "1", 1): if row.get("shaogu_program") in ("是", "1", 1):
plans.append("少数民族骨干") plans.append("少数民族骨干")
return finish_record({**row, "year": explicit_year, "source": row.get("source") or source, "subjects": subjects, "special_plans": plans, "quota": quota, "quota_scope": scope, "year_verified": row.get("year_verified") is True or row.get("year_verified") == "True"}) return finish_record({**row, "year": explicit_year, "source": row.get("source") or source, "subjects": subjects, "reference_books": list_field(row.get('reference_books')), "special_plans": plans, "quota": quota, "quota_scope": scope, "year_verified": row.get("year_verified") is True or row.get("year_verified") == "True"})
def import_file(path, year=None): def import_file(path, year=None):
@@ -92,14 +105,29 @@ def import_file(path, year=None):
raise ValueError("支持 CSV、XLSX、JSON 和 YAM SQLite 文件") raise ValueError("支持 CSV、XLSX、JSON 和 YAM SQLite 文件")
if len(rows) > 300000: if len(rows) > 300000:
raise ValueError("一次最多导入 30 万条,请拆分文件") raise ValueError("一次最多导入 30 万条,请拆分文件")
records = [row_record(row, year, "yzw 导入" if suffix == ".csv" else "文件导入") for row in rows] records, schools = [], []
for row in rows:
if isinstance(row, dict) and row.get('dwdm') and row.get('dwmc'):
schools.append({'school_code': str(row['dwdm']), 'school_name': row['dwmc'], 'province': row.get('szss') or PROVINCES.get(str(row.get('szssm', '')), ''), 'level': '双一流' if str(row.get('syl')) == '1' else ''})
directions = row.get('data') or ([row] if row.get('kskmz') else [])
if not isinstance(directions, list):
raise ValueError('YAM 种子 data 应为方向数组')
for direction in directions:
if not isinstance(direction, dict) or not any(k in direction for k in ('yjfxmc', 'kskmz', 'yxsmc')):
raise ValueError('YAM 方向数据结构无法识别')
explicit = integer(row.get('year')) or integer(year)
if not explicit:
raise ValueError('种子含有目录详情,请指定其招生年份')
records.extend(normalize_catalog(direction, explicit, row))
else:
records.append(row_record(row, year, "yzw 导入" if suffix == ".csv" else "文件导入"))
for score in scores: for score in scores:
if not all(k in score for k in ("school_code", "major_code", "year", "scope")): if not all(k in score for k in ("school_code", "major_code", "year", "scope")):
raise ValueError("快照分数线缺少院校/专业/年份/口径") raise ValueError("快照分数线缺少院校/专业/年份/口径")
score["year"] = int(score["year"]) score["year"] = int(score["year"])
score["total"] = integer(score.get("total")) score["total"] = integer(score.get("total"))
score.setdefault("note", "") score.setdefault("note", "")
return records, scores return ImportResult(records, scores, schools)
def import_yam(path, year): def import_yam(path, year):
@@ -133,7 +161,7 @@ def import_yam(path, year):
row = dict(entry) row = dict(entry)
school = lookup.get((str(row["school_id"]), str(row.get("major_code", ""))), {}) school = lookup.get((str(row["school_id"]), str(row.get("major_code", ""))), {})
records.append(row_record({**row, "school_code": str(row["school_id"]), "school_name": school.get("name", ""), "province": school.get("province", ""), "direction": row.get("research_direction", ""), "direction_code": str(row.get("spe_id") or row.get("plan_id") or ""), "department_code": str(row.get("department_id", "") or ""), "source": row.get("source") or "YAM 导入", "remarks": row.get("note", "")}, year)) records.append(row_record({**row, "school_code": str(row["school_id"]), "school_name": school.get("name", ""), "province": school.get("province", ""), "direction": row.get("research_direction", ""), "direction_code": str(row.get("spe_id") or row.get("plan_id") or ""), "department_code": str(row.get("department_id", "") or ""), "source": row.get("source") or "YAM 导入", "remarks": row.get("note", "")}, year))
return records, scores return ImportResult(records, scores)
except sqlite3.DatabaseError: except sqlite3.DatabaseError:
raise ValueError("SQLite 文件无法读取,请确认是完整导出的数据库") from None raise ValueError("SQLite 文件无法读取,请确认是完整导出的数据库") from None
finally: finally:
+27 -7
View File
@@ -3,13 +3,17 @@ import threading
import uuid import uuid
from studio.connectors import CatalogClient, ScoreClient, LoginRequired, SourceYearMismatch from studio.connectors import CatalogClient, ScoreClient, LoginRequired, SourceYearMismatch
from studio.domain import PROVINCES, normalize_catalog from studio.domain import PROVINCES, normalize_catalog
from studio.network import ProxyUnavailable, RemoteUnavailable from studio.network import ProxyUnavailable, RemoteUnavailable, RateLimited
class Cancelled(Exception): class Cancelled(Exception):
pass pass
class Shutdown(Exception):
pass
class JobManager: class JobManager:
def __init__(self, database, settings, transport=None, runner=None): def __init__(self, database, settings, transport=None, runner=None):
self.db, self.settings, self.transport = database, settings, transport self.db, self.settings, self.transport = database, settings, transport
@@ -54,7 +58,7 @@ class JobManager:
def _launch(self, jid): def _launch(self, jid):
condition = threading.Condition() condition = threading.Condition()
control = {"condition": condition, "paused": False, "cancelled": False} control = {"condition": condition, "paused": False, "cancelled": False, "shutdown": False}
thread = threading.Thread(target=self._run, args=(jid,), daemon=True, name="collector-" + jid[:8]) thread = threading.Thread(target=self._run, args=(jid,), daemon=True, name="collector-" + jid[:8])
control["thread"] = thread control["thread"] = thread
self.controls[jid] = control self.controls[jid] = control
@@ -71,6 +75,8 @@ class JobManager:
self.checkpoint(jid) self.checkpoint(jid)
errors = self.get(jid)["errors"] errors = self.get(jid)["errors"]
self.db.job_update(jid, status="partial" if errors else "completed", message="部分单元失败,可恢复重试" if errors else "采集完成") self.db.job_update(jid, status="partial" if errors else "completed", message="部分单元失败,可恢复重试" if errors else "采集完成")
except Shutdown:
self.db.job_update(jid, status="paused", message="服务停止,任务可从断点恢复")
except Cancelled: except Cancelled:
self.db.job_update(jid, status="cancelled", message="已取消,已保存数据保留") self.db.job_update(jid, status="cancelled", message="已取消,已保存数据保留")
except LoginRequired as exc: except LoginRequired as exc:
@@ -79,6 +85,9 @@ class JobManager:
except ProxyUnavailable as exc: except ProxyUnavailable as exc:
self.db.job_update(jid, status="waiting_proxy", message=str(exc)) self.db.job_update(jid, status="waiting_proxy", message=str(exc))
self.db.log(jid, "warning", str(exc)) self.db.log(jid, "warning", str(exc))
except RateLimited as exc:
self.db.job_update(jid, status="waiting_source", message=str(exc))
self.db.log(jid, "warning", str(exc))
except (RemoteUnavailable, SourceYearMismatch, ValueError) as exc: except (RemoteUnavailable, SourceYearMismatch, ValueError) as exc:
self.db.job_update(jid, status="failed", message=str(exc)) self.db.job_update(jid, status="failed", message=str(exc))
self.db.log(jid, "error", str(exc)) self.db.log(jid, "error", str(exc))
@@ -88,16 +97,20 @@ class JobManager:
finally: finally:
if self.transport: if self.transport:
self.transport.local.guard = None self.transport.local.guard = None
if self.controls[jid]["shutdown"] and self.get(jid)["status"] != 'cancelled':
self.db.job_update(jid, status="paused", message="服务停止,任务可从断点恢复")
def checkpoint(self, jid): def checkpoint(self, jid):
control = self.controls.get(jid) control = self.controls.get(jid)
if not control: if not control:
return return
with control["condition"]: with control["condition"]:
while control["paused"] and not control["cancelled"]: while control["paused"] and not control["cancelled"] and not control["shutdown"]:
control["condition"].wait(.25) control["condition"].wait(.25)
if control["cancelled"]: if control["cancelled"]:
raise Cancelled() raise Cancelled()
if control["shutdown"]:
raise Shutdown()
def unit_done(self, jid, unit): def unit_done(self, jid, unit):
return self.db.unit_done(jid, unit) return self.db.unit_done(jid, unit)
@@ -117,16 +130,20 @@ class JobManager:
def resume(self, jid): def resume(self, jid):
with self.lock: with self.lock:
job = self.get(jid) job = self.get(jid)
if not job or job["status"] not in ("paused", "partial", "failed", "waiting_login", "waiting_proxy"): if not job or job["status"] not in ("paused", "partial", "failed", "waiting_login", "waiting_proxy", "waiting_source"):
raise ValueError("此任务当前不能恢复") raise ValueError("此任务当前不能恢复")
self._ensure_available(jid) self._ensure_available(jid)
control = self.controls.get(jid) control = self.controls.get(jid)
if control and control["thread"].is_alive(): if control and control["thread"].is_alive() and control["paused"] and job['status'] == 'paused':
with control["condition"]: with control["condition"]:
self.db.job_update(jid, status="running", message="正在继续采集") self.db.job_update(jid, status="running", message="正在继续采集")
control["paused"] = False control["paused"] = False
control["condition"].notify_all() control["condition"].notify_all()
else: else:
if control and control['thread'].is_alive():
control['thread'].join(timeout=1)
if control['thread'].is_alive():
raise ValueError("旧任务正在结束,请稍后点击恢复")
self._launch(jid) self._launch(jid)
return self.get(jid) return self.get(jid)
@@ -147,7 +164,10 @@ class JobManager:
self.closed = True self.closed = True
for jid, control in list(self.controls.items()): for jid, control in list(self.controls.items()):
if control["thread"].is_alive(): if control["thread"].is_alive():
self.cancel(jid) with control['condition']:
control['shutdown'] = True
self.db.job_update(jid, status='paused', message='服务停止,任务可从断点恢复')
control['condition'].notify_all()
for control in list(self.controls.values()): for control in list(self.controls.values()):
control["thread"].join(timeout=1) control["thread"].join(timeout=1)
@@ -194,7 +214,7 @@ class JobManager:
self._error(jid, f"{label}:{exc}") self._error(jid, f"{label}:{exc}")
def _scores(self, jid, params): def _scores(self, jid, params):
filters = {"year": params["year"], "major_code": params.get("major_code", ""), "keyword": params.get("school_keyword", "")} filters = {"year": params["year"], "major_code": params.get("major_code", ""), "school_keyword": params.get("school_keyword", "")}
records = self.db.all_catalog(filters) records = self.db.all_catalog(filters)
targets = {} targets = {}
for row in records: for row in records:
+5 -1
View File
@@ -13,6 +13,10 @@ class RemoteUnavailable(RuntimeError):
pass pass
class RateLimited(RuntimeError):
pass
class ProxyRouter: class ProxyRouter:
def __init__(self, settings, session=None): def __init__(self, settings, session=None):
self.settings = settings self.settings = settings
@@ -146,4 +150,4 @@ class Transport:
raise RemoteUnavailable("网络连接失败,已尝试切换代理;请检查网络或恢复任务") from None raise RemoteUnavailable("网络连接失败,已尝试切换代理;请检查网络或恢复任务") from None
except requests.HTTPError as exc: except requests.HTTPError as exc:
raise RemoteUnavailable(f"数据源返回 HTTP {exc.response.status_code}") from None raise RemoteUnavailable(f"数据源返回 HTTP {exc.response.status_code}") from None
raise RemoteUnavailable("数据源限流,请降低请求频率后重试") raise RateLimited("数据源持续限流,任务已等待;请降低请求频率后恢复")
+10
View File
@@ -66,3 +66,13 @@ def test_import_yam_sqlite_keeps_year_and_score_scope(client, tmp_path):
assert rows[0]["year"] == 2026 assert rows[0]["year"] == 2026
assert rows[0]["score_year"] == 2025 assert rows[0]["score_year"] == 2025
assert rows[0]["score_scope"] == "discipline" assert rows[0]["score_scope"] == "discipline"
def test_yam_seed_json_imports_school_index_without_inventing_admissions(client):
payload = [{'dwdm': '10001', 'dwmc': '北京大学', 'zydm': '081200', 'zymc': '计算机科学与技术', 'szss': '北京', 'data': [], 'sign': 'token-must-not-copy'}]
response = client.post('/api/import', files={'file': ('yam-seed.json', json.dumps(payload).encode())})
assert response.status_code == 200, response.text
assert response.json()['records'] == 0
assert response.json()['indexed_schools'] == 1
assert client.get('/api/catalog').json()['total'] == 0
assert 'token-must-not-copy' not in client.get('/api/schools').text
+61
View File
@@ -79,3 +79,64 @@ def test_task_pause_resume_reuses_persisted_work_without_duplicate_entries(tmp_p
assert jobs.get(job["id"])["completed_units"] == 2 assert jobs.get(job["id"])["completed_units"] == 2
jobs.close() jobs.close()
def test_service_shutdown_pauses_instead_of_cancelling_persisted_job(tmp_path):
db, settings = Database(tmp_path / 'studio.db'), SettingsStore(tmp_path)
entered = threading.Event()
def runner(jid, params, manager):
entered.set()
while True:
manager.checkpoint(jid)
time.sleep(.01)
manager = JobManager(db, settings, runner=runner)
job = manager.start({'mode': 'catalog', 'year': 2027})
assert entered.wait(2)
manager.close()
assert Database(tmp_path / 'studio.db').job_get(job['id'])['status'] == 'paused'
def test_business_rate_limit_uses_bounded_backoff_without_skipping_a_page():
from studio.network import RateLimited
class Limited(CatalogTransport):
def __init__(self):
self.calls, self.waits = 0, []
def request(self, *args, **kwargs):
self.calls += 1
return Response({'flag': False, 'msg': '访问太频繁,请稍后重试'})
def sleep(self, seconds):
self.waits.append(seconds)
transport = Limited()
with pytest.raises(RateLimited):
list(CatalogClient(transport).schools('11'))
assert transport.calls == 3
assert transport.waits == [5, 10]
def test_recovery_waits_for_failed_worker_to_exit_before_launching_another(tmp_path):
db, settings = Database(tmp_path / 'studio.db'), SettingsStore(tmp_path)
entered, release = threading.Event(), threading.Event()
original_log = db.log
def guarded_log(jid, level, message):
if level == 'error':
entered.set()
release.wait(2)
original_log(jid, level, message)
db.log = guarded_log
attempts = []
def runner(jid, params, manager):
attempts.append(jid)
if len(attempts) == 1:
raise ValueError('failure fixture')
manager.complete_unit(jid, 'recovered')
manager = JobManager(db, settings, runner=runner)
job = manager.start({'mode': 'catalog', 'year': 2027})
assert entered.wait(2)
threading.Timer(.05, release.set).start()
manager.resume(job['id'])
deadline = time.monotonic() + 2
while manager.get(job['id'])['status'] != 'completed' and time.monotonic() < deadline:
time.sleep(.02)
assert manager.get(job['id'])['status'] == 'completed'
assert len(attempts) == 2
manager.close()
+58
View File
@@ -64,3 +64,61 @@ def test_same_year_different_college_lines_are_not_attached_arbitrarily(tmp_path
db.upsert_scores(select_score_lines(rows, "081200", "10001", "北京大学", 2026)) db.upsert_scores(select_score_lines(rows, "081200", "10001", "北京大学", 2026))
assert db.query_catalog({"year": 2026})["items"][0]["score_total"] is None assert db.query_catalog({"year": 2026})["items"][0]["score_total"] is None
def test_special_plan_score_does_not_overwrite_regular_admission_line(tmp_path):
db = Database(tmp_path / "studio.db")
db.upsert_catalog(normalize_catalog(raw_catalog(), 2026))
rows = [{"code": "081200", "total": "320", "depart_id": "001", "note": ""}, {"code": "081200", "total": "260", "depart_id": "001", "note": "少干专项计划"}]
db.upsert_scores(select_score_lines(rows, "081200", "10001", "北京大学", 2026))
assert db.overview()["scores"] == 2
assert db.query_catalog({})["items"][0]["score_total"] == 320
def test_special_plan_full_names_are_not_used_as_regular_reference_line(tmp_path):
db = Database(tmp_path / 'studio.db')
db.upsert_catalog(normalize_catalog(raw_catalog(), 2026))
rows = [{'code': '081200', 'total': 320, 'depart_id': '001', 'note': ''}, {'code': '081200', 'total': 260, 'depart_id': '001', 'note': '少数民族骨干计划'}, {'code': '081200', 'total': 250, 'depart_id': '001', 'note': '退役大学生士兵计划'}]
db.upsert_scores(select_score_lines(rows, '081200', '10001', '北京大学', 2026))
assert db.query_catalog({})['items'][0]['score_total'] == 320
def test_spreadsheet_roundtrip_preserves_record_identity(tmp_path):
from studio.imports import export_sheet, import_file
db = Database(tmp_path / 'studio.db')
records = normalize_catalog(raw_catalog(), 2027)
db.upsert_catalog(records)
for kind in ('csv', 'xlsx'):
path = tmp_path / ('data.' + kind)
path.write_bytes(export_sheet(records, kind))
imported, _ = import_file(path)
db.upsert_catalog(imported)
assert db.query_catalog({})['total'] == 2
def test_known_historical_school_levels_are_available_in_filters(tmp_path):
db = Database(tmp_path / 'studio.db')
db.upsert_catalog(normalize_catalog(raw_catalog(), 2027))
assert db.query_catalog({'level': '985'})['total'] == 2
assert db.query_catalog({'level': '211'})['total'] == 2
def test_imported_college_quota_cannot_be_promoted_by_numeric_column():
from studio.imports import row_record
row = row_record({'school_name':'Test','major_code':'081200','enrollment_text':'学院:100','enrollment_count':100},2026)
assert row['quota'] is None
assert row['quota_scope'] == 'college'
def test_job_school_keyword_does_not_match_department_names(tmp_path):
db = Database(tmp_path / 'studio.db')
item = raw_catalog()
item['yxsmc'] = '信息学院'
db.upsert_catalog(normalize_catalog(item, 2027))
assert db.query_catalog({'school_keyword': '信息'})['total'] == 0
def test_yam_reference_books_are_arrays_for_the_detail_view():
from studio.imports import row_record
row = row_record({'school_name':'Test','major_code':'081200','reference_books':'[]'},2026)
assert row['reference_books'] == []
+38 -1
View File
@@ -3,7 +3,7 @@ import pytest
from studio.network import ProxyRouter, Transport, ProxyUnavailable from studio.network import ProxyRouter, Transport, ProxyUnavailable
from studio.settings import SettingsStore from studio.settings import SettingsStore
from studio.auth import CookieVault from studio.auth import CookieVault, AuthService
class Response: class Response:
@@ -91,3 +91,40 @@ def test_cookie_is_encrypted_and_not_returned_in_settings(tmp_path):
assert b"secret-test-cookie" not in vault.path.read_bytes() assert b"secret-test-cookie" not in vault.path.read_bytes()
assert "cookie" not in SettingsStore(tmp_path).read() assert "cookie" not in SettingsStore(tmp_path).read()
def test_rate_limiting_does_not_quarantine_a_healthy_proxy(tmp_path):
class Limited(Remote):
def __init__(self):
super().__init__()
self.calls = 0
def request(self, *args, **kwargs):
self.calls += 1
if self.calls == 1:
return Response({}, status=429)
return super().request(*args, **kwargs)
_, router, transport = setup_network(tmp_path, remote=Limited())
original = router.resolve()
assert transport.request("GET", "https://yz.chsi.com.cn/zsml/").json()["routed_through"] == original
assert router.failed == set()
def test_browser_cookie_domains_preserve_same_name_sessions(tmp_path):
_, _, transport = setup_network(tmp_path)
vault = CookieVault(tmp_path)
vault.save([{"name": "JSESSIONID", "value": "account-cookie", "domain": "account.chsi.com.cn", "path": "/"}, {"name": "JSESSIONID", "value": "catalog-cookie", "domain": "yz.chsi.com.cn", "path": "/"}])
AuthService(transport, vault)
cookies = transport.session.cookies
assert cookies.get("JSESSIONID", domain="account.chsi.com.cn") == "account-cookie"
assert cookies.get("JSESSIONID", domain="yz.chsi.com.cn") == "catalog-cookie"
def test_auth_checks_protected_direction_api_rather_than_public_school_index(tmp_path):
class PublicIndex(Remote):
def request(self, method, url, **kwargs):
if url.endswith('/dws.do'):
return Response({'flag': True, 'msg': {'list': []}})
return Response({'flag': False, 'msg': '请登录'})
_, _, transport = setup_network(tmp_path, remote=PublicIndex())
auth = AuthService(transport, CookieVault(tmp_path))
assert auth.check()['valid'] is False
+1
View File
@@ -17,6 +17,7 @@ def main():
expect(page.get_by_role("heading", name="从数据出发,找到适合的院校")).to_be_visible() expect(page.get_by_role("heading", name="从数据出发,找到适合的院校")).to_be_visible()
page.get_by_role("navigation").get_by_role("button", name="院校工作区").click() page.get_by_role("navigation").get_by_role("button", name="院校工作区").click()
page.get_by_role("button", name="导入数据", exact=True).click() page.get_by_role("button", name="导入数据", exact=True).click()
expect(page.get_by_role('spinbutton', name='这份目录的招生年份')).to_have_value('')
rows = [{"year": 2026, "school_name": "验收样例甲大学", "school_code": "QA001", "province": "北京", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "计算机学院", "subjects": ["101思想政治理论", "201英语一", "301数学一", "408计算机学科专业基础"], "quota_text": "专业:6(不含推免)", "source": "自动验收样例"}, {"year": 2026, "school_name": "验收样例乙大学", "school_code": "QA002", "province": "上海", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "信息学院", "subjects": ["101思想政治理论", "204英语二", "302数学二", "自命题专业课"], "quota_text": "学院:100", "source": "自动验收样例"}] rows = [{"year": 2026, "school_name": "验收样例甲大学", "school_code": "QA001", "province": "北京", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "计算机学院", "subjects": ["101思想政治理论", "201英语一", "301数学一", "408计算机学科专业基础"], "quota_text": "专业:6(不含推免)", "source": "自动验收样例"}, {"year": 2026, "school_name": "验收样例乙大学", "school_code": "QA002", "province": "上海", "major_code": "081200", "major_name": "计算机科学与技术", "department_name": "信息学院", "subjects": ["101思想政治理论", "204英语二", "302数学二", "自命题专业课"], "quota_text": "学院:100", "source": "自动验收样例"}]
scores = [{"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2024, "total": 305, "scope": "major", "note": ""}, {"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2025, "total": 315, "scope": "major", "note": ""}] scores = [{"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2024, "total": 305, "scope": "major", "note": ""}, {"school_code": "QA001", "major_code": "081200", "department_name": "计算机学院", "year": 2025, "total": 315, "scope": "major", "note": ""}]
page.locator('input[type=file]').set_input_files({"name": "acceptance.json", "mimeType": "application/json", "buffer": json.dumps({"catalog": rows, "scores": scores}, ensure_ascii=False).encode()}) page.locator('input[type=file]').set_input_files({"name": "acceptance.json", "mimeType": "application/json", "buffer": json.dumps({"catalog": rows, "scores": scores}, ensure_ascii=False).encode()})