Fix CHSI catalogue session verification and pagination
This commit is contained in:
@@ -22,11 +22,13 @@
|
|||||||
|
|
||||||
1. 在「代理 IP 池」刷新,查看池内 HTTPS 候选和当前路由。默认连接 `http://127.0.0.1:5010`;服务目录为 `C:\Users\Jayson\Desktop\proxy_pool`。
|
1. 在「代理 IP 池」刷新,查看池内 HTTPS 候选和当前路由。默认连接 `http://127.0.0.1:5010`;服务目录为 `C:\Users\Jayson\Desktop\proxy_pool`。
|
||||||
2. 在「设置与登录」选择网络策略。优先代理在池为空时允许直连;严格代理没有可用 IP 时等待处理。备用代理每行一个 `IP:端口`。代理池存在候选并不保证该 IP 对所有目标网站可用,连接失败会切换并隔离。
|
2. 在「设置与登录」选择网络策略。优先代理在池为空时允许直连;严格代理没有可用 IP 时等待处理。备用代理每行一个 `IP:端口`。代理池存在候选并不保证该 IP 对所有目标网站可用,连接失败会切换并隔离。
|
||||||
3. 点击「打开登录窗口」,在独立 Edge 窗口完成学信网登录和验证码。工作台会检测登录并保存 Cookie。也可粘贴已登录研招网的 Cookie 请求头,或使用账号密码登录。点击「验证目录访问」确认详情接口权限。
|
3. 点击「打开登录窗口」,在独立 Edge 窗口完成学信网登录和验证码。工作台会同步研招目录登录会话、验证第二页访问权限,再加密保存 Cookie。也可粘贴已登录研招网的 Cookie 请求头,或使用账号密码登录。点击「验证目录访问」会重新同步并检查权限。
|
||||||
4. 在「采集任务」选择内容、目录年份、地区、专业和院校关键词。建议先按一个地区、一个专业或一所院校验证,再创建全国任务。专业代码可以从内置目录选择,也可直接输入 6 位代码。
|
4. 在「采集任务」选择内容、目录年份、地区、专业和院校关键词。建议先按一个地区、一个专业或一所院校验证,再创建全国任务。专业代码可以从内置目录选择,也可直接输入 6 位代码。
|
||||||
5. 查看任务日志。暂停可继续;需要登录、代理或限流解除时,处理后点击恢复;取消保留已有数据。关闭或重启服务会保留可恢复断点。
|
5. 查看任务日志。暂停可继续;需要登录、代理或限流解除时,处理后点击恢复;取消保留已有数据。关闭或重启服务会保留可恢复断点。
|
||||||
|
|
||||||
研招网学校、专业索引有时可以匿名读取,但方向与科目详情需要登录。程序验证实际业务返回状态,不将 HTTP 200 当作登录成功。目录采集先核对研招页面配置中的年份,避免将当前目录标为历史年份。
|
研招网目录第一页可匿名返回,包括部分研究方向;不能用第一页判断登录成功。工作台访问研招目录的登录入口,将已有学信网 Cookie 同步到目录会话,再验证需要登录的第二页。采集开始/恢复时自动完成此检查;途中遇到登录提示,只同步一次并重试原页,失败后保留断点并提示重新登录。验证码仍需在浏览器人工完成,限流和网络失败会单独提示。登录卡片会持续更新状态;切换网络出口后需要重新验证,原 Cookie 保留。
|
||||||
|
|
||||||
|
分页使用接口返回的下一页位置和实际页大小。研招网可能将每页条数限制为 10,程序按实际位置继续,避免请求较大页时遗漏记录。目录采集先核对研招页面配置中的年份,避免将当前目录标为历史年份。
|
||||||
|
|
||||||
## 数据查询
|
## 数据查询
|
||||||
|
|
||||||
|
|||||||
+13
-3
@@ -2,14 +2,14 @@
|
|||||||
|
|
||||||
2026-10-11,Windows、本地独立虚拟环境,Python 3.12.13 / Node 24.16。
|
2026-10-11,Windows、本地独立虚拟环境,Python 3.12.13 / Node 24.16。
|
||||||
|
|
||||||
- 后端:`python -m pytest -q`,49 passed。Starlette TestClient 提示一条 httpx 迁移 warning,测试退出 0,生产服务不使用 TestClient。
|
- 后端:`python -m pytest -q`,62 passed。Starlette TestClient 提示一条 httpx 迁移 warning,测试退出 0,生产服务不使用 TestClient。
|
||||||
- 前端:`npm test`,3 passed;`npm run build`,TypeScript 检查与 Vite 生产构建退出 0。
|
- 前端:`npm test`,3 passed;`npm run build`,TypeScript 检查与 Vite 生产构建退出 0。
|
||||||
- Edge 实际浏览器:独立端口 8766、独立 `.runtime/qa/db`,验证导入→科目筛选→收藏→对比→历史趋势→Excel 下载;任务/代理/设置页面;设置保存;390px 手机宽度无页面溢出;无 pageerror。测试数据均标明「自动验收样例」,没有写入正式数据库。
|
- Edge 实际浏览器:独立端口 8766、独立 `.runtime/qa/db`,验证导入→科目筛选→收藏→对比→历史趋势→Excel 下载;任务/代理/设置页面;设置保存;390px 手机宽度无页面溢出;无 pageerror。测试数据均标明「自动验收样例」,没有写入正式数据库。
|
||||||
- 启动脚本:Windows PowerShell 首启、重复启动、停止、重新启动均退出 0;正式服务在 127.0.0.1:8765,健康检查应用身份和项目路径匹配。
|
- 启动脚本:Windows PowerShell 首启、重复启动、停止、重新启动均退出 0;正式服务在 127.0.0.1:8765,健康检查应用身份和项目路径匹配。
|
||||||
- 真实数据源:proxy_pool `/count/` 正常响应;研招页面 `curYear=2027` 可核实;匿名北京大学学校及 081200 专业索引可返回,方向接口要求登录,按 LoginRequired 处理;掌上考研精确匹配北京大学,2025 年 081200 查询返回 1 条门类参考线和 2 条历史计划。真实探测只读取少量公开数据,不写入正式目录。
|
- 真实数据源:proxy_pool `/count/` 正常响应;研招页面 `curYear=2027` 可核实;研招目录第一页(包括 081200 方向)可以匿名返回,第二页需要登录,详见下方登录会话修复;掌上考研精确匹配北京大学,2025 年 081200 查询返回 1 条门类参考线和 2 条历史计划。真实探测不写入正式目录。
|
||||||
- 独立只读审查:复现并修复专项线覆盖、同名跨域 Cookie、任务失败恢复竞态、服务退出被误取消、文件往返丢失身份字段、学院人数误升格、YAM 种子误识别、院校关键词范围、参考书 JSON 字符串、业务限流等问题,补充相应回归。
|
- 独立只读审查:复现并修复专项线覆盖、同名跨域 Cookie、任务失败恢复竞态、服务退出被误取消、文件往返丢失身份字段、学院人数误升格、YAM 种子误识别、院校关键词范围、参考书 JSON 字符串、业务限流等问题,补充相应回归。
|
||||||
|
|
||||||
自动验收未使用真人学信网账号,未对全部院校进行实网抓取。登录后的全国采集能力取决于账号状态、代理可用性、开放目录年份和数据源限流。
|
初次自动验收未使用真人学信网账号。后续登录修复使用用户已保存的 Cookie 进行了有限的只读分页验证,未对全部院校进行实网抓取。登录后的全国采集能力取决于账号状态、代理可用性、开放目录年份和数据源限流。
|
||||||
|
|
||||||
复跑浏览器验收(只使用测试库):
|
复跑浏览器验收(只使用测试库):
|
||||||
|
|
||||||
@@ -28,3 +28,13 @@
|
|||||||
- `tools/proxy_service_qa.py`:复制管理脚本到临时测试目录,启动无抓取行为的假 API / 调度器;实际验证普通权限 start → 重复 start → status → stop → API 关闭,PASS。仅只读检查现有 Redis,正式代理池未停止。
|
- `tools/proxy_service_qa.py`:复制管理脚本到临时测试目录,启动无抓取行为的假 API / 调度器;实际验证普通权限 start → 重复 start → status → stop → API 关闭,PASS。仅只读检查现有 Redis,正式代理池未停止。
|
||||||
- 独立代码审查确认真实脚本入口校验和一次操作使用同一配置快照;最终无 Critical / Important。
|
- 独立代码审查确认真实脚本入口校验和一次操作使用同一配置快照;最终无 Critical / Important。
|
||||||
- 正式界面 Edge 验收:显示 API 正常、两个进程的管理员权限、外部 Redis;点击检查状态成功,点击启动复用现有进程,无乱码或原错误堆栈,状态检查与复用保留当前路由,停止按钮给出权限限制说明。390px 代理页无溢出,无浏览器错误;截图保存为「代理池修复预览.png」。完整独立测试库浏览器流程亦再次通过。
|
- 正式界面 Edge 验收:显示 API 正常、两个进程的管理员权限、外部 Redis;点击检查状态成功,点击启动复用现有进程,无乱码或原错误堆栈,状态检查与复用保留当前路由,停止按钮给出权限限制说明。390px 代理页无溢出,无浏览器错误;截图保存为「代理池修复预览.png」。完整独立测试库浏览器流程亦再次通过。
|
||||||
|
|
||||||
|
登录会话与分页修复(2026-10-11):
|
||||||
|
|
||||||
|
- 复现:两次任务都在北京大学的前 10 个专业、107 条记录后收到「请登录」。匿名方向第一页也返回 flag=true;原来的第一页验证不能证明登录成功,且任务的登录失败没有更新登录卡片。
|
||||||
|
- 使用相同代理和已保存 Cookie,只读请求专业第二页仍提示登录;访问 `/zsml/a/dw.do` 同步 CAS 会话后,专业第二页和方向第二页均正常。没有要求用户重新提供凭据,没有输出 Cookie 或密码。
|
||||||
|
- 请求 pageSize=50 时,真实接口返回 size=10、startOfNextPage=10。修复按实际下一页位置继续,防止用 50 作为偏移跳过记录。
|
||||||
|
- 13 项新增行为回归覆盖匿名第一页误判、已有 SSO Cookie 初始化、服务端页大小限制、原页自动恢复、恢复次数限制、任务与登录状态同步及断点保留、限流分类、人工验证码、网络出口变化,以及验证中切换出口、清除/替换凭据、新验证码拦截覆盖旧证明的并发情形。
|
||||||
|
- 修正后实网只读验证:15 个专业跨两页、081200 的全部 15 条原始研究方向跨两页,当前代理保持不变,正式数据库写入 0 条。
|
||||||
|
- `tools/auth_browser_qa.py`:模拟登录已完成后收到失效状态,原版因不再轮询而失败;修正后登录卡片自动显示「需重新登录」并清除旧成功提示。独立端口 8766 和正式端口的 Edge 验收通过;390px 无溢出,无浏览器错误。原完整工作区浏览器验收再次通过。
|
||||||
|
- 正式服务重启后,使用已有加密 Cookie 再次验证第二页成功,登录卡片显示最新检查结果。原任务保留 107 条记录和 10 个已完成单元,仍可从断点恢复,未自动启动全国采集。截图为「登录修复预览.png」。最终 62 项后端测试、3 项前端测试和生产构建通过;独立只读审查未发现剩余 Critical / Important。
|
||||||
|
|||||||
+27
-9
@@ -1857,16 +1857,22 @@ function SettingsPanel({ notify }: { notify: Notify }) {
|
|||||||
const browserWaiting =
|
const browserWaiting =
|
||||||
auth && ["opening", "waiting"].includes(auth.browser_state);
|
auth && ["opening", "waiting"].includes(auth.browser_state);
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (!browserWaiting) return;
|
if (busy) return;
|
||||||
const timer = setInterval(
|
let active = true;
|
||||||
() =>
|
const refreshAuth = () =>
|
||||||
api<Auth>("/auth")
|
api<Auth>("/auth")
|
||||||
.then(setAuth)
|
.then((result) => {
|
||||||
.catch(() => {}),
|
if (active) setAuth(result);
|
||||||
2000,
|
})
|
||||||
);
|
.catch(() => {});
|
||||||
return () => clearInterval(timer);
|
const timer = setInterval(refreshAuth, 2000);
|
||||||
}, [browserWaiting]);
|
window.addEventListener("focus", refreshAuth);
|
||||||
|
return () => {
|
||||||
|
active = false;
|
||||||
|
clearInterval(timer);
|
||||||
|
window.removeEventListener("focus", refreshAuth);
|
||||||
|
};
|
||||||
|
}, [busy]);
|
||||||
async function save(e: React.FormEvent) {
|
async function save(e: React.FormEvent) {
|
||||||
e.preventDefault();
|
e.preventDefault();
|
||||||
setBusy("save");
|
setBusy("save");
|
||||||
@@ -2020,12 +2026,24 @@ function SettingsPanel({ notify }: { notify: Notify }) {
|
|||||||
<Badge tone={auth?.valid ? "green" : "amber"}>
|
<Badge tone={auth?.valid ? "green" : "amber"}>
|
||||||
{auth?.valid
|
{auth?.valid
|
||||||
? "已验证"
|
? "已验证"
|
||||||
|
: auth?.state === "captcha_required"
|
||||||
|
? "需人工验证"
|
||||||
|
: auth?.state === "login_required"
|
||||||
|
? "需重新登录"
|
||||||
: auth?.saved
|
: auth?.saved
|
||||||
? "已保存,待验证"
|
? "已保存,待验证"
|
||||||
: "尚未登录"}
|
: "尚未登录"}
|
||||||
</Badge>
|
</Badge>
|
||||||
</div>
|
</div>
|
||||||
<p>{auth?.message}</p>
|
<p>{auth?.message}</p>
|
||||||
|
{auth?.checked_at && (
|
||||||
|
<p className="form-hint">
|
||||||
|
最近检查:
|
||||||
|
{new Date(auth.checked_at).toLocaleString("zh-CN", {
|
||||||
|
hour12: false,
|
||||||
|
})}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
<div className="browser-login">
|
<div className="browser-login">
|
||||||
<div>
|
<div>
|
||||||
<Globe2 size={22} />
|
<Globe2 size={22} />
|
||||||
|
|||||||
@@ -138,6 +138,8 @@ export type PoolManageResult = {
|
|||||||
export type Auth = {
|
export type Auth = {
|
||||||
saved: boolean;
|
saved: boolean;
|
||||||
valid: boolean;
|
valid: boolean;
|
||||||
|
state: "unverified" | "verified" | "login_required" | "captcha_required";
|
||||||
|
checked_at: string | null;
|
||||||
message: string;
|
message: string;
|
||||||
browser_state: string;
|
browser_state: string;
|
||||||
browser_message: string;
|
browser_message: string;
|
||||||
|
|||||||
+2
-1
@@ -28,7 +28,7 @@ def create_app(data_dir=None):
|
|||||||
router = ProxyRouter(settings)
|
router = ProxyRouter(settings)
|
||||||
transport = Transport(settings, router)
|
transport = Transport(settings, router)
|
||||||
auth = AuthService(transport, CookieVault(directory))
|
auth = AuthService(transport, CookieVault(directory))
|
||||||
jobs = JobManager(db, settings, transport)
|
jobs = JobManager(db, settings, transport, auth=auth)
|
||||||
if (ROOT / "data/schools.json").exists():
|
if (ROOT / "data/schools.json").exists():
|
||||||
db.index_schools(json.loads((ROOT / "data/schools.json").read_text(encoding="utf-8")))
|
db.index_schools(json.loads((ROOT / "data/schools.json").read_text(encoding="utf-8")))
|
||||||
|
|
||||||
@@ -41,6 +41,7 @@ def create_app(data_dir=None):
|
|||||||
app.state.db, app.state.jobs = db, jobs
|
app.state.db, app.state.jobs = db, jobs
|
||||||
app.state.settings, app.state.router = settings, router
|
app.state.settings, app.state.router = settings, router
|
||||||
app.state.pool = pool
|
app.state.pool = pool
|
||||||
|
app.state.auth = auth
|
||||||
|
|
||||||
@app.middleware("http")
|
@app.middleware("http")
|
||||||
async def local_access(request: Request, call_next):
|
async def local_access(request: Request, call_next):
|
||||||
|
|||||||
+77
-18
@@ -6,8 +6,12 @@ import threading
|
|||||||
import time
|
import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from bs4 import BeautifulSoup
|
from bs4 import BeautifulSoup
|
||||||
|
from studio.connectors import CatalogClient, LoginRequired
|
||||||
|
from studio.domain import now
|
||||||
|
from studio.network import ProxyUnavailable, RemoteUnavailable, RateLimited
|
||||||
|
|
||||||
LOGIN_URL = "https://account.chsi.com.cn/passport/login?entrytype=yzgr&service=https%3A%2F%2Fyz.chsi.com.cn%2Fj_spring_cas_security_check"
|
LOGIN_URL = "https://account.chsi.com.cn/passport/login?entrytype=yzgr&service=https%3A%2F%2Fyz.chsi.com.cn%2Fj_spring_cas_security_check"
|
||||||
|
CATALOG_ENTRY = "https://yz.chsi.com.cn/zsml/a/dw.do"
|
||||||
|
|
||||||
|
|
||||||
class Blob(ctypes.Structure):
|
class Blob(ctypes.Structure):
|
||||||
@@ -55,9 +59,16 @@ class AuthService:
|
|||||||
self.transport, self.vault = transport, vault
|
self.transport, self.vault = transport, vault
|
||||||
self.message = "尚未验证登录状态"
|
self.message = "尚未验证登录状态"
|
||||||
self.valid = False
|
self.valid = False
|
||||||
|
self.state = "unverified"
|
||||||
|
self.checked_at = None
|
||||||
|
self.verified_route = None
|
||||||
|
self.revision = 0
|
||||||
|
self.state_lock = threading.RLock()
|
||||||
|
self.check_lock = threading.RLock()
|
||||||
self.browser_state = "idle"
|
self.browser_state = "idle"
|
||||||
self.browser_message = ""
|
self.browser_message = ""
|
||||||
self.browser_thread = None
|
self.browser_thread = None
|
||||||
|
self.browser_cancel = None
|
||||||
try:
|
try:
|
||||||
self._load(self.vault.load())
|
self._load(self.vault.load())
|
||||||
except (RuntimeError, OSError, ValueError):
|
except (RuntimeError, OSError, ValueError):
|
||||||
@@ -80,7 +91,19 @@ class AuthService:
|
|||||||
return [{"name": c.name, "value": c.value, "domain": c.domain, "path": c.path, "secure": c.secure, "expires": c.expires} for c in self.transport.session.cookies]
|
return [{"name": c.name, "value": c.value, "domain": c.domain, "path": c.path, "secure": c.secure, "expires": c.expires} for c in self.transport.session.cookies]
|
||||||
|
|
||||||
def status(self):
|
def status(self):
|
||||||
return {"saved": self.vault.path.exists(), "valid": self.valid, "message": self.message, "browser_state": self.browser_state, "browser_message": self.browser_message}
|
with self.state_lock:
|
||||||
|
if self.valid and self.verified_route != self.transport.router.status()["route"]:
|
||||||
|
self.invalidate("网络出口已变化,请验证目录访问后继续采集", state="unverified")
|
||||||
|
return {"saved": self.vault.path.exists(), "valid": self.valid, "state": self.state, "message": self.message, "checked_at": self.checked_at, "browser_state": self.browser_state, "browser_message": self.browser_message}
|
||||||
|
|
||||||
|
def invalidate(self, message, state="login_required"):
|
||||||
|
with self.state_lock:
|
||||||
|
self.revision += 1
|
||||||
|
self.valid = False
|
||||||
|
self.state, self.message = state, message
|
||||||
|
self.verified_route = None
|
||||||
|
if self.browser_state == "completed":
|
||||||
|
self.browser_state, self.browser_message = "idle", ""
|
||||||
|
|
||||||
def cookie(self, text):
|
def cookie(self, text):
|
||||||
if not text.strip() or "\n" in text or "\r" in text:
|
if not text.strip() or "\n" in text or "\r" in text:
|
||||||
@@ -94,30 +117,50 @@ class AuthService:
|
|||||||
cookies[name] = value
|
cookies[name] = value
|
||||||
if not cookies:
|
if not cookies:
|
||||||
raise ValueError("未识别到 Cookie 键值")
|
raise ValueError("未识别到 Cookie 键值")
|
||||||
|
if self.browser_cancel:
|
||||||
|
self.browser_cancel.set()
|
||||||
|
with self.check_lock:
|
||||||
self._load(cookies)
|
self._load(cookies)
|
||||||
self.vault.save(cookies)
|
self.vault.save(cookies)
|
||||||
self.valid = False
|
self.invalidate("Cookie 已加密保存,点击验证确认目录访问权限", state="unverified")
|
||||||
self.message = "Cookie 已加密保存,点击验证确认目录访问权限"
|
self.checked_at = None
|
||||||
|
self.browser_state, self.browser_message = "idle", ""
|
||||||
return self.status()
|
return self.status()
|
||||||
|
|
||||||
def check(self):
|
def check(self):
|
||||||
# The school/major indexes can be public while direction details require login.
|
with self.check_lock:
|
||||||
response = self.transport.request("POST", "https://yz.chsi.com.cn/zsml/rs/yjfxs.do", data={"dwdm": "10001", "zydm": "081200", "zymc": "计算机科学与技术", "curPage": 1, "start": 0, "pageSize": 1, "dwlxs[0]": "all"})
|
with self.state_lock:
|
||||||
|
self.invalidate("正在同步研招目录会话并验证访问权限", state="unverified")
|
||||||
|
revision = self.revision
|
||||||
try:
|
try:
|
||||||
body = response.json()
|
# The first ten rows are available anonymously, even for directions.
|
||||||
except ValueError:
|
# Enter the CAS-protected page to bind cookies, then prove page-two access.
|
||||||
self.valid = False
|
self.transport.request("GET", CATALOG_ENTRY)
|
||||||
self.message = "目录接口未返回有效数据,可能需要登录或验证码"
|
CatalogClient(self.transport).fetch_page("yjfxs", {"dwdm": "10001", "zydm": "081200", "zymc": "计算机科学与技术", "curPage": 2, "start": 10, "pageSize": 10, "dwlxs[0]": "all"})
|
||||||
return self.status()
|
proof_route = self.transport.local.response_route
|
||||||
self.valid = body.get("flag") is True and isinstance(body.get("msg"), dict)
|
|
||||||
self.message = "研招目录接口验证成功" if self.valid else "登录未生效或数据源限流,请在浏览器完成登录后重试"
|
|
||||||
if self.valid:
|
|
||||||
self.vault.save(self._snapshot())
|
self.vault.save(self._snapshot())
|
||||||
|
except LoginRequired as exc:
|
||||||
|
self.invalidate(str(exc), state="captcha_required" if exc.reason == "captcha" else "login_required")
|
||||||
|
except (ProxyUnavailable, RemoteUnavailable, RateLimited):
|
||||||
|
self.invalidate("暂时无法验证研招目录,请检查网络或稍后重试;已保存的 Cookie 保留", state="unverified")
|
||||||
|
raise
|
||||||
|
else:
|
||||||
|
with self.state_lock:
|
||||||
|
# A later crawler login/captcha failure supersedes this older proof.
|
||||||
|
if self.revision == revision:
|
||||||
|
self.valid, self.state = True, "verified"
|
||||||
|
self.message = "研招目录登录会话已同步,第二页访问验证成功"
|
||||||
|
self.verified_route = proof_route
|
||||||
|
finally:
|
||||||
|
self.checked_at = now()
|
||||||
return self.status()
|
return self.status()
|
||||||
|
|
||||||
def login(self, username, password):
|
def login(self, username, password):
|
||||||
if not username or not password:
|
if not username or not password:
|
||||||
raise ValueError("请输入学信网账号和密码")
|
raise ValueError("请输入学信网账号和密码")
|
||||||
|
if self.browser_cancel:
|
||||||
|
self.browser_cancel.set()
|
||||||
|
with self.check_lock:
|
||||||
response = self.transport.request("GET", LOGIN_URL)
|
response = self.transport.request("GET", LOGIN_URL)
|
||||||
soup = BeautifulSoup(response.text, "html.parser")
|
soup = BeautifulSoup(response.text, "html.parser")
|
||||||
form = {node.get("name"): node.get("value", "") for node in soup.select("input[type=hidden][name]")}
|
form = {node.get("name"): node.get("value", "") for node in soup.select("input[type=hidden][name]")}
|
||||||
@@ -128,16 +171,22 @@ class AuthService:
|
|||||||
return self.check()
|
return self.check()
|
||||||
|
|
||||||
def clear(self):
|
def clear(self):
|
||||||
|
if self.browser_cancel:
|
||||||
|
self.browser_cancel.set()
|
||||||
|
with self.check_lock:
|
||||||
self._load({})
|
self._load({})
|
||||||
self.vault.clear()
|
self.vault.clear()
|
||||||
self.valid = False
|
self.invalidate("已清除本项目保存的登录凭据", state="unverified")
|
||||||
self.message = "已清除本项目保存的登录凭据"
|
self.checked_at = None
|
||||||
|
self.browser_state, self.browser_message = "idle", ""
|
||||||
return self.status()
|
return self.status()
|
||||||
|
|
||||||
def browser(self):
|
def browser(self):
|
||||||
if self.browser_thread and self.browser_thread.is_alive():
|
if self.browser_thread and self.browser_thread.is_alive() and not self.browser_cancel.is_set():
|
||||||
return self.status()
|
return self.status()
|
||||||
proxy = self.transport.router.resolve()
|
proxy = self.transport.router.resolve()
|
||||||
|
cancelled = threading.Event()
|
||||||
|
self.browser_cancel = cancelled
|
||||||
self.browser_state = "opening"
|
self.browser_state = "opening"
|
||||||
self.browser_message = "正在打开独立 Edge 登录窗口,完成登录后可关闭窗口"
|
self.browser_message = "正在打开独立 Edge 登录窗口,完成登录后可关闭窗口"
|
||||||
|
|
||||||
@@ -149,24 +198,34 @@ class AuthService:
|
|||||||
context = browser.new_context()
|
context = browser.new_context()
|
||||||
page = context.new_page()
|
page = context.new_page()
|
||||||
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
|
page.goto(LOGIN_URL, wait_until="domcontentloaded", timeout=60000)
|
||||||
|
if cancelled.is_set():
|
||||||
|
browser.close()
|
||||||
|
return
|
||||||
self.browser_state = "waiting"
|
self.browser_state = "waiting"
|
||||||
self.browser_message = "请在 Edge 完成学信网登录和验证码;窗口保持打开,工作台自动检测"
|
self.browser_message = "请在 Edge 完成学信网登录和验证码;窗口保持打开,工作台自动检测"
|
||||||
deadline = time.monotonic() + 300
|
deadline = time.monotonic() + 300
|
||||||
while time.monotonic() < deadline:
|
while time.monotonic() < deadline and not cancelled.is_set():
|
||||||
page.wait_for_timeout(2000)
|
page.wait_for_timeout(2000)
|
||||||
if "yz.chsi.com.cn" in page.url:
|
if "yz.chsi.com.cn" in page.url:
|
||||||
cookies = [c for c in context.cookies() if c["domain"].lstrip(".") == "chsi.com.cn" or c["domain"].endswith(".chsi.com.cn")]
|
cookies = [c for c in context.cookies() if c["domain"].lstrip(".") == "chsi.com.cn" or c["domain"].endswith(".chsi.com.cn")]
|
||||||
if cookies:
|
if cookies:
|
||||||
|
with self.check_lock:
|
||||||
|
if cancelled.is_set():
|
||||||
|
break
|
||||||
self._load(cookies)
|
self._load(cookies)
|
||||||
if self.check()["valid"]:
|
if self.check()["valid"] and not cancelled.is_set():
|
||||||
self.browser_state = "completed"
|
self.browser_state = "completed"
|
||||||
self.browser_message = "已获取登录 Cookie,目录接口验证成功"
|
self.browser_message = "已获取登录 Cookie,目录接口验证成功"
|
||||||
browser.close()
|
browser.close()
|
||||||
return
|
return
|
||||||
browser.close()
|
browser.close()
|
||||||
|
if cancelled.is_set():
|
||||||
|
return
|
||||||
self.browser_state = "expired"
|
self.browser_state = "expired"
|
||||||
self.browser_message = "登录等待超时,可再次打开浏览器"
|
self.browser_message = "登录等待超时,可再次打开浏览器"
|
||||||
except Exception:
|
except Exception:
|
||||||
|
if cancelled.is_set():
|
||||||
|
return
|
||||||
self.browser_state = "failed"
|
self.browser_state = "failed"
|
||||||
self.browser_message = "登录窗口已关闭或连接失败,可重试或粘贴 Cookie"
|
self.browser_message = "登录窗口已关闭或连接失败,可重试或粘贴 Cookie"
|
||||||
self.browser_thread = threading.Thread(target=run, daemon=True, name="browser-login")
|
self.browser_thread = threading.Thread(target=run, daemon=True, name="browser-login")
|
||||||
|
|||||||
+43
-14
@@ -6,7 +6,9 @@ from studio.network import RemoteUnavailable, RateLimited
|
|||||||
|
|
||||||
|
|
||||||
class LoginRequired(RuntimeError):
|
class LoginRequired(RuntimeError):
|
||||||
pass
|
def __init__(self, message, reason="login"):
|
||||||
|
super().__init__(message)
|
||||||
|
self.reason = reason
|
||||||
|
|
||||||
|
|
||||||
class SourceYearMismatch(RuntimeError):
|
class SourceYearMismatch(RuntimeError):
|
||||||
@@ -16,8 +18,8 @@ class SourceYearMismatch(RuntimeError):
|
|||||||
class CatalogClient:
|
class CatalogClient:
|
||||||
BASE = "https://yz.chsi.com.cn/zsml/"
|
BASE = "https://yz.chsi.com.cn/zsml/"
|
||||||
|
|
||||||
def __init__(self, transport):
|
def __init__(self, transport, auth=None):
|
||||||
self.transport = transport
|
self.transport, self.auth = transport, auth
|
||||||
|
|
||||||
def verify_year(self, year):
|
def verify_year(self, year):
|
||||||
response = self.transport.request("GET", self.BASE)
|
response = self.transport.request("GET", self.BASE)
|
||||||
@@ -32,32 +34,53 @@ class CatalogClient:
|
|||||||
raise SourceYearMismatch(f"研招网当前公开目录为 {current} 年,任务选择了 {year} 年;历史目录请导入已有快照")
|
raise SourceYearMismatch(f"研招网当前公开目录为 {current} 年,任务选择了 {year} 年;历史目录请导入已有快照")
|
||||||
return current
|
return current
|
||||||
|
|
||||||
def pages(self, endpoint, payload, page_size=50):
|
def _fetch_page(self, endpoint, payload):
|
||||||
seen = set()
|
|
||||||
for page in range(1, 10001):
|
|
||||||
for attempt in range(3):
|
for attempt in range(3):
|
||||||
response = self.transport.request("POST", self.BASE + "rs/" + endpoint + ".do", data={**payload, "curPage": page, "start": (page - 1) * page_size, "pageSize": page_size})
|
response = self.transport.request("POST", self.BASE + "rs/" + endpoint + ".do", data=payload)
|
||||||
try:
|
try:
|
||||||
body = response.json()
|
body = response.json()
|
||||||
except ValueError:
|
except ValueError:
|
||||||
raise LoginRequired("研招网返回登录/验证页面,请先完成登录") from None
|
if '验证码' in response.text:
|
||||||
|
raise LoginRequired('研招网需要人工验证码,请在浏览器完成验证后恢复', reason="captcha") from None
|
||||||
|
if re.search(r"passport/login|j_spring_cas_security_check|name=[\"']password[\"']", response.text) or '/passport/login' in response.url:
|
||||||
|
raise LoginRequired("研招网返回登录页面,请在浏览器完成登录后恢复") from None
|
||||||
|
raise RemoteUnavailable("研招目录接口未返回有效 JSON,请稍后重试") from None
|
||||||
|
if not isinstance(body, dict):
|
||||||
|
raise RemoteUnavailable("研招网目录响应结构已变化,未写入不完整数据")
|
||||||
message = str(body.get('msg', ''))
|
message = str(body.get('msg', ''))
|
||||||
if body.get('flag') is not True and '验证码' in message:
|
if body.get('flag') is not True and '验证码' in message:
|
||||||
raise LoginRequired('研招网需要人工验证码,请在浏览器完成验证后恢复')
|
raise LoginRequired('研招网需要人工验证码,请在浏览器完成验证后恢复', reason="captcha")
|
||||||
if body.get('flag') is not True and any(token in message for token in ('频繁', '稍后')):
|
if body.get('flag') is not True and any(token in message for token in ('频繁', '稍后')):
|
||||||
if attempt == 2:
|
if attempt == 2:
|
||||||
raise RateLimited('研招网业务限流持续出现,已暂停;请增加请求间隔后恢复')
|
raise RateLimited('研招网业务限流持续出现,已暂停;请增加请求间隔后恢复')
|
||||||
self.transport.sleep(5 * 2 ** attempt)
|
self.transport.sleep(5 * 2 ** attempt)
|
||||||
continue
|
continue
|
||||||
break
|
|
||||||
if body.get("flag") is not True:
|
if body.get("flag") is not True:
|
||||||
message = str(body.get("msg", ""))
|
|
||||||
if any(token in message for token in ("登录", "登陆", "认证")):
|
if any(token in message for token in ("登录", "登陆", "认证")):
|
||||||
raise LoginRequired("研招网要求登录,请更新 Cookie 后恢复任务")
|
raise LoginRequired("研招网要求登录,请重新验证或在浏览器登录后恢复任务")
|
||||||
raise RemoteUnavailable("研招网目录接口未成功返回数据")
|
raise RemoteUnavailable("研招网目录接口未成功返回数据")
|
||||||
data = body.get("msg") or {}
|
data = body.get("msg")
|
||||||
if not isinstance(data, dict) or not isinstance(data.get("list"), list):
|
if not isinstance(data, dict) or not isinstance(data.get("list"), list):
|
||||||
raise RemoteUnavailable("研招网目录响应结构已变化,未写入不完整数据")
|
raise RemoteUnavailable("研招网目录响应结构已变化,未写入不完整数据")
|
||||||
|
return data
|
||||||
|
|
||||||
|
def fetch_page(self, endpoint, payload):
|
||||||
|
try:
|
||||||
|
return self._fetch_page(endpoint, payload)
|
||||||
|
except LoginRequired as exc:
|
||||||
|
if not self.auth or exc.reason == "captcha":
|
||||||
|
raise
|
||||||
|
# A CAS cookie may still be valid while the catalogue session has expired.
|
||||||
|
# Refresh once, then retry exactly the same page; never skip its records.
|
||||||
|
status = self.auth.check()
|
||||||
|
if not status["valid"]:
|
||||||
|
raise LoginRequired(status["message"], reason="captcha" if status["state"] == "captcha_required" else "login") from None
|
||||||
|
return self._fetch_page(endpoint, payload)
|
||||||
|
|
||||||
|
def pages(self, endpoint, payload, page_size=10):
|
||||||
|
seen, start = set(), 0
|
||||||
|
for page in range(1, 10001):
|
||||||
|
data = self.fetch_page(endpoint, {**payload, "curPage": page, "start": start, "pageSize": page_size})
|
||||||
rows = data["list"]
|
rows = data["list"]
|
||||||
signature = json.dumps(rows, ensure_ascii=False, sort_keys=True)
|
signature = json.dumps(rows, ensure_ascii=False, sort_keys=True)
|
||||||
if rows and signature in seen:
|
if rows and signature in seen:
|
||||||
@@ -66,8 +89,14 @@ class CatalogClient:
|
|||||||
yield from rows
|
yield from rows
|
||||||
more = data.get("nextPageAvailable")
|
more = data.get("nextPageAvailable")
|
||||||
total = integer(data.get("totalCount"))
|
total = integer(data.get("totalCount"))
|
||||||
if not rows or more is False or more in ("false", "0", 0) or more is None and ((total is not None and page * page_size >= total) or len(rows) < page_size):
|
size = integer(data.get("size")) or page_size
|
||||||
|
if not rows or more is False or more in ("false", "0", 0) or more is None and ((total is not None and start + len(rows) >= total) or len(rows) < size):
|
||||||
return
|
return
|
||||||
|
next_start = integer(data.get("startOfNextPage"))
|
||||||
|
next_start = start + len(rows) if next_start is None else next_start
|
||||||
|
if next_start <= start:
|
||||||
|
raise RemoteUnavailable("研招目录下一页位置无效,已停止以避免遗漏分页")
|
||||||
|
start, page_size = next_start, size
|
||||||
raise RemoteUnavailable("目录分页超过安全上限")
|
raise RemoteUnavailable("目录分页超过安全上限")
|
||||||
|
|
||||||
def schools(self, province, keyword=""):
|
def schools(self, province, keyword=""):
|
||||||
|
|||||||
+10
-2
@@ -15,8 +15,9 @@ class Shutdown(Exception):
|
|||||||
|
|
||||||
|
|
||||||
class JobManager:
|
class JobManager:
|
||||||
def __init__(self, database, settings, transport=None, runner=None):
|
def __init__(self, database, settings, transport=None, runner=None, auth=None):
|
||||||
self.db, self.settings, self.transport = database, settings, transport
|
self.db, self.settings, self.transport = database, settings, transport
|
||||||
|
self.auth = auth
|
||||||
self.runner = runner or self._collect
|
self.runner = runner or self._collect
|
||||||
self.controls = {}
|
self.controls = {}
|
||||||
self.lock = threading.RLock()
|
self.lock = threading.RLock()
|
||||||
@@ -80,6 +81,8 @@ class JobManager:
|
|||||||
except Cancelled:
|
except Cancelled:
|
||||||
self.db.job_update(jid, status="cancelled", message="已取消,已保存数据保留")
|
self.db.job_update(jid, status="cancelled", message="已取消,已保存数据保留")
|
||||||
except LoginRequired as exc:
|
except LoginRequired as exc:
|
||||||
|
if self.auth:
|
||||||
|
self.auth.invalidate(str(exc), state="captcha_required" if exc.reason == "captcha" else "login_required")
|
||||||
self.db.job_update(jid, status="waiting_login", message=str(exc))
|
self.db.job_update(jid, status="waiting_login", message=str(exc))
|
||||||
self.db.log(jid, "warning", str(exc))
|
self.db.log(jid, "warning", str(exc))
|
||||||
except ProxyUnavailable as exc:
|
except ProxyUnavailable as exc:
|
||||||
@@ -186,8 +189,13 @@ class JobManager:
|
|||||||
self._scores(jid, params)
|
self._scores(jid, params)
|
||||||
|
|
||||||
def _catalog(self, jid, params):
|
def _catalog(self, jid, params):
|
||||||
client = CatalogClient(self.transport)
|
client = CatalogClient(self.transport, auth=self.auth)
|
||||||
client.verify_year(params["year"])
|
client.verify_year(params["year"])
|
||||||
|
if self.auth:
|
||||||
|
status = self.auth.check()
|
||||||
|
if not status["valid"]:
|
||||||
|
raise LoginRequired(status["message"], reason="captcha" if status["state"] == "captcha_required" else "login")
|
||||||
|
self.db.log(jid, "info", "研招目录会话已同步,第二页访问验证成功")
|
||||||
for province in params["provinces"]:
|
for province in params["provinces"]:
|
||||||
self.checkpoint(jid)
|
self.checkpoint(jid)
|
||||||
self.db.log(jid, "info", "查询 " + PROVINCES[province] + " 院校")
|
self.db.log(jid, "info", "查询 " + PROVINCES[province] + " 院校")
|
||||||
|
|||||||
@@ -142,6 +142,7 @@ class Transport:
|
|||||||
self.sleep(min(30, 5 * (attempt + 1)))
|
self.sleep(min(30, 5 * (attempt + 1)))
|
||||||
continue
|
continue
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
|
self.local.response_route = proxy or "direct"
|
||||||
return response
|
return response
|
||||||
except (requests.exceptions.ProxyError, requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
except (requests.exceptions.ProxyError, requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||||
if proxy:
|
if proxy:
|
||||||
|
|||||||
@@ -0,0 +1,242 @@
|
|||||||
|
"""CHSI exposes page one publicly; page two proves the catalogue session."""
|
||||||
|
import threading
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from studio.auth import AuthService, CookieVault
|
||||||
|
from studio.connectors import CatalogClient, LoginRequired
|
||||||
|
from studio.database import Database
|
||||||
|
from studio.jobs import JobManager
|
||||||
|
from studio.network import RateLimited
|
||||||
|
from tests.test_network import Remote, Response, setup_network
|
||||||
|
|
||||||
|
|
||||||
|
class CatalogSession(Remote):
|
||||||
|
def __init__(self, accepts_sso=True):
|
||||||
|
super().__init__()
|
||||||
|
self.accepts_sso = accepts_sso
|
||||||
|
self.authorized = False
|
||||||
|
self.calls = []
|
||||||
|
self.blocked = None
|
||||||
|
|
||||||
|
def request(self, method, url, **kwargs):
|
||||||
|
data = kwargs.get('data', {})
|
||||||
|
self.calls.append((method, url, data.copy()))
|
||||||
|
if method == 'GET':
|
||||||
|
if url.endswith('/a/dw.do'):
|
||||||
|
self.authorized = self.accepts_sso
|
||||||
|
return Response({}, text='2027年硕士专业目录查询')
|
||||||
|
if self.blocked:
|
||||||
|
return Response({'flag': False, 'msg': self.blocked})
|
||||||
|
start = int(data.get('start', 0))
|
||||||
|
if start > 0 and not self.authorized:
|
||||||
|
return Response({'flag': False, 'msg': '请登录'})
|
||||||
|
# The server caps the requested 50 rows at ten and supplies its next offset.
|
||||||
|
rows = [{'dwdm': str(10000 + i)} for i in range(start, min(start + 10, 23))]
|
||||||
|
return Response({'flag': True, 'msg': {'list': rows, 'size': 10,
|
||||||
|
'startOfNextPage': start + 10, 'totalCount': 23,
|
||||||
|
'nextPageAvailable': start + 10 < 23}})
|
||||||
|
|
||||||
|
|
||||||
|
def service(tmp_path, remote):
|
||||||
|
settings, router, transport = setup_network(tmp_path, remote=remote)
|
||||||
|
auth = AuthService(transport, CookieVault(tmp_path))
|
||||||
|
auth.cookie('CASTGC=fixture-sso-cookie')
|
||||||
|
return auth, settings, router, transport
|
||||||
|
|
||||||
|
|
||||||
|
def test_public_first_page_is_not_valid_login_evidence(tmp_path):
|
||||||
|
remote = CatalogSession(accepts_sso=False)
|
||||||
|
auth, _, _, _ = service(tmp_path, remote)
|
||||||
|
result = auth.check()
|
||||||
|
assert result['valid'] is False
|
||||||
|
assert result['saved'] is True
|
||||||
|
assert any(int(data.get('start', 0)) > 0 for method, _, data in remote.calls if method == 'POST')
|
||||||
|
|
||||||
|
|
||||||
|
def test_existing_sso_cookie_is_bound_before_testing_page_two(tmp_path):
|
||||||
|
remote = CatalogSession()
|
||||||
|
auth, _, _, _ = service(tmp_path, remote)
|
||||||
|
assert auth.check()['valid'] is True
|
||||||
|
assert remote.calls[0][0] == 'GET'
|
||||||
|
assert remote.calls[0][1].endswith('/a/dw.do')
|
||||||
|
assert int(remote.calls[-1][2]['start']) == 10
|
||||||
|
|
||||||
|
|
||||||
|
def test_server_pagination_offsets_keep_every_row_when_page_size_is_capped(tmp_path):
|
||||||
|
remote = CatalogSession()
|
||||||
|
remote.authorized = True
|
||||||
|
_, _, transport = setup_network(tmp_path, remote=remote)
|
||||||
|
rows = list(CatalogClient(transport).pages('dws', {}, page_size=50))
|
||||||
|
assert [r['dwdm'] for r in rows] == [str(10000 + i) for i in range(23)]
|
||||||
|
assert [data['start'] for _, _, data in remote.calls] == [0, 10, 20]
|
||||||
|
|
||||||
|
|
||||||
|
def test_sso_recovery_retries_the_same_failed_page_without_skipping_rows(tmp_path):
|
||||||
|
remote = CatalogSession()
|
||||||
|
auth, _, _, transport = service(tmp_path, remote)
|
||||||
|
rows = list(CatalogClient(transport, auth=auth).pages('dws', {}))
|
||||||
|
assert [r['dwdm'] for r in rows] == [str(10000 + i) for i in range(23)]
|
||||||
|
assert auth.status()['valid'] is True
|
||||||
|
assert sum(url.endswith('/a/dw.do') for _, url, _ in remote.calls) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_unrecoverable_login_stops_after_one_sso_attempt(tmp_path):
|
||||||
|
remote = CatalogSession(accepts_sso=False)
|
||||||
|
auth, _, _, transport = service(tmp_path, remote)
|
||||||
|
with pytest.raises(LoginRequired):
|
||||||
|
list(CatalogClient(transport, auth=auth).pages('dws', {}))
|
||||||
|
assert sum(url.endswith('/a/dw.do') for _, url, _ in remote.calls) == 1
|
||||||
|
assert auth.status()['valid'] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_job_login_failure_invalidates_verified_badge_and_preserves_checkpoint(tmp_path):
|
||||||
|
auth, settings, _, transport = service(tmp_path, CatalogSession())
|
||||||
|
assert auth.check()['valid'] is True
|
||||||
|
auth.browser_state = 'completed'
|
||||||
|
auth.browser_message = '已获取登录 Cookie,目录接口验证成功'
|
||||||
|
db = Database(tmp_path / 'studio.db')
|
||||||
|
|
||||||
|
def runner(jid, params, manager):
|
||||||
|
manager.complete_unit(jid, 'already-saved')
|
||||||
|
db.job_update(jid, records=107)
|
||||||
|
raise LoginRequired('研招网要求登录,请重新验证后恢复任务')
|
||||||
|
|
||||||
|
jobs = JobManager(db, settings, transport, runner=runner, auth=auth)
|
||||||
|
try:
|
||||||
|
job = jobs.start({'mode': 'catalog', 'year': 2027})
|
||||||
|
jobs.controls[job['id']]['thread'].join(timeout=3)
|
||||||
|
result = jobs.get(job['id'])
|
||||||
|
assert result['status'] == 'waiting_login'
|
||||||
|
assert result['records'] == 107 and result['completed_units'] == 1
|
||||||
|
status = auth.status()
|
||||||
|
assert status['valid'] is False
|
||||||
|
assert status['state'] == 'login_required'
|
||||||
|
assert status['saved'] is True
|
||||||
|
assert status['browser_message'] == ''
|
||||||
|
finally:
|
||||||
|
jobs.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_rate_limiting_during_validation_is_not_reported_as_bad_cookie(tmp_path):
|
||||||
|
remote = CatalogSession()
|
||||||
|
auth, _, _, _ = service(tmp_path, remote)
|
||||||
|
assert auth.check()['valid'] is True
|
||||||
|
remote.blocked = '访问太频繁,请稍后重试'
|
||||||
|
with pytest.raises(RateLimited):
|
||||||
|
auth.check()
|
||||||
|
assert auth.status()['valid'] is False
|
||||||
|
assert auth.status()['state'] == 'unverified'
|
||||||
|
assert auth.status()['saved'] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_captcha_requires_human_verification_without_sso_retry(tmp_path):
|
||||||
|
remote = CatalogSession()
|
||||||
|
auth, _, _, transport = service(tmp_path, remote)
|
||||||
|
remote.blocked = '需要验证码'
|
||||||
|
with pytest.raises(LoginRequired):
|
||||||
|
list(CatalogClient(transport, auth=auth).pages('dws', {}))
|
||||||
|
assert not any(url.endswith('/a/dw.do') for _, url, _ in remote.calls)
|
||||||
|
|
||||||
|
|
||||||
|
def test_changing_route_invalidates_previous_verification_without_deleting_cookie(tmp_path):
|
||||||
|
auth, _, router, _ = service(tmp_path, CatalogSession())
|
||||||
|
assert auth.check()['valid'] is True
|
||||||
|
router.rotate()
|
||||||
|
status = auth.status()
|
||||||
|
assert status['valid'] is False
|
||||||
|
assert status['state'] == 'unverified'
|
||||||
|
assert status['saved'] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_route_changed_after_proof_is_not_marked_as_verified(tmp_path):
|
||||||
|
auth, _, router, _ = service(tmp_path, CatalogSession())
|
||||||
|
original_save = auth.vault.save
|
||||||
|
|
||||||
|
def switch_before_saving(cookies):
|
||||||
|
router.rotate()
|
||||||
|
original_save(cookies)
|
||||||
|
|
||||||
|
auth.vault.save = switch_before_saving
|
||||||
|
status = auth.check()
|
||||||
|
assert status['valid'] is False
|
||||||
|
assert status['state'] == 'unverified'
|
||||||
|
assert status['saved'] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_later_captcha_invalidation_is_not_overwritten_by_pending_verification(tmp_path):
|
||||||
|
auth, _, _, _ = service(tmp_path, CatalogSession())
|
||||||
|
before_save, finish_save = threading.Event(), threading.Event()
|
||||||
|
original_save = auth.vault.save
|
||||||
|
|
||||||
|
def delayed_save(cookies):
|
||||||
|
before_save.set()
|
||||||
|
assert finish_save.wait(3)
|
||||||
|
original_save(cookies)
|
||||||
|
|
||||||
|
auth.vault.save = delayed_save
|
||||||
|
result = []
|
||||||
|
checking = threading.Thread(target=lambda: result.append(auth.check()))
|
||||||
|
checking.start()
|
||||||
|
try:
|
||||||
|
assert before_save.wait(3)
|
||||||
|
auth.invalidate('采集中发现验证码,请在浏览器完成人工验证', state='captcha_required')
|
||||||
|
finally:
|
||||||
|
finish_save.set()
|
||||||
|
checking.join(timeout=3)
|
||||||
|
assert not checking.is_alive()
|
||||||
|
assert result[0]['valid'] is False
|
||||||
|
assert result[0]['state'] == 'captcha_required'
|
||||||
|
assert '验证码' in result[0]['message']
|
||||||
|
assert result[0]['saved'] is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('action', ['clear', 'replace'])
|
||||||
|
def test_pending_check_cannot_override_a_later_credential_change(tmp_path, action):
|
||||||
|
auth, _, _, _ = service(tmp_path, CatalogSession())
|
||||||
|
before_save, finish_save, change_started = threading.Event(), threading.Event(), threading.Event()
|
||||||
|
original_save = auth.vault.save
|
||||||
|
|
||||||
|
def delayed_save(cookies):
|
||||||
|
# Only delay the verifier, not the user's later replacement.
|
||||||
|
if threading.current_thread().name == 'verification':
|
||||||
|
before_save.set()
|
||||||
|
assert finish_save.wait(3)
|
||||||
|
original_save(cookies)
|
||||||
|
|
||||||
|
auth.vault.save = delayed_save
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
def check():
|
||||||
|
try:
|
||||||
|
auth.check()
|
||||||
|
except Exception as error:
|
||||||
|
errors.append(error)
|
||||||
|
|
||||||
|
def change():
|
||||||
|
change_started.set()
|
||||||
|
if action == 'clear':
|
||||||
|
auth.clear()
|
||||||
|
else:
|
||||||
|
auth.cookie('CASTGC=new-fixture-cookie')
|
||||||
|
|
||||||
|
checking = threading.Thread(target=check, name='verification')
|
||||||
|
changing = threading.Thread(target=change, name='credential-change')
|
||||||
|
checking.start()
|
||||||
|
try:
|
||||||
|
assert before_save.wait(3)
|
||||||
|
changing.start()
|
||||||
|
assert change_started.wait(3)
|
||||||
|
changing.join(timeout=.1)
|
||||||
|
finally:
|
||||||
|
finish_save.set()
|
||||||
|
checking.join(timeout=3)
|
||||||
|
if changing.ident:
|
||||||
|
changing.join(timeout=3)
|
||||||
|
assert not checking.is_alive() and not changing.is_alive()
|
||||||
|
assert not errors, errors
|
||||||
|
status = auth.status()
|
||||||
|
assert status['valid'] is False
|
||||||
|
assert status['state'] == 'unverified'
|
||||||
|
assert status['saved'] is (action == 'replace')
|
||||||
|
if action == 'replace':
|
||||||
|
assert auth.vault.load()['CASTGC'] == 'new-fixture-cookie'
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
"""Read-only UI regression: a completed login must keep tracking auth state."""
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from playwright.sync_api import sync_playwright, expect
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument('--url', default='http://127.0.0.1:8765')
|
||||||
|
args = parser.parse_args()
|
||||||
|
status = {'saved': True, 'valid': True, 'state': 'verified',
|
||||||
|
'message': '研招目录第二页接口验证成功', 'checked_at': '2026-10-11T15:00:00+08:00',
|
||||||
|
'browser_state': 'completed', 'browser_message': '已获取登录 Cookie,目录接口验证成功'}
|
||||||
|
with sync_playwright() as p:
|
||||||
|
browser = p.chromium.launch(channel='msedge', headless=True)
|
||||||
|
page = browser.new_page(viewport={'width': 1480, 'height': 1050})
|
||||||
|
errors = []
|
||||||
|
page.on('pageerror', lambda error: errors.append(str(error)))
|
||||||
|
page.route('**/api/auth', lambda route: route.fulfill(content_type='application/json', body=json.dumps(status)))
|
||||||
|
page.goto(args.url, wait_until='networkidle')
|
||||||
|
page.get_by_role('navigation').get_by_role('button', name='设置与登录').click()
|
||||||
|
card = page.locator('.auth-card')
|
||||||
|
expect(card).to_contain_text('已验证')
|
||||||
|
status.update(valid=False, state='login_required', message='研招网要求登录,请重新验证后恢复任务', browser_state='idle', browser_message='')
|
||||||
|
expect(card).to_contain_text('需重新登录', timeout=7000)
|
||||||
|
expect(card).not_to_contain_text('目录接口验证成功')
|
||||||
|
page.set_viewport_size({'width': 390, 'height': 844})
|
||||||
|
assert page.evaluate('document.documentElement.scrollWidth <= window.innerWidth'), 'mobile overflow'
|
||||||
|
assert not errors, errors
|
||||||
|
out = Path(__file__).resolve().parents[1] / '.runtime/qa'
|
||||||
|
out.mkdir(parents=True, exist_ok=True)
|
||||||
|
page.screenshot(path=str(out / 'auth-state.png'), full_page=True)
|
||||||
|
browser.close()
|
||||||
|
print('PASS: completed login polls invalidation, badge/message stay consistent, mobile layout, no browser errors')
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
Reference in New Issue
Block a user