Add Debian deployment, MariaDB storage and randomized proxy routing
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
# Debian server implementation plan
|
||||
|
||||
> **For agentic workers:** Use superpowers:executing-plans to implement this plan inline, with a fresh whole-branch reviewer.
|
||||
|
||||
**Goal:** Run the existing workbench continuously on the user's Debian 12 server with MariaDB, a managed proxy pool, randomized pacing, and preserved collected data.
|
||||
|
||||
**Architecture:** Keep SQLite as the local default. Add a MySQL database adapter behind the existing Database interface, Linux credential encryption and fixed systemd pool management. Serve the single-worker application on loopback behind authenticated HTTPS.
|
||||
|
||||
**Tech Stack:** Python 3.11+, FastAPI, PyMySQL, cryptography, React, MariaDB 10.11, systemd, Nginx, proxy_pool, dedicated Redis.
|
||||
|
||||
**Spec:** ../specs/2026-10-11-debian-server-design.md
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Preserve the currently running Windows collection.
|
||||
- Never print or commit cookies, encryption keys, database passwords, or access passwords.
|
||||
- Reuse SSH connections; do not alter existing databases or unrelated services.
|
||||
- Random request bounds: 0.3–60 seconds; server proxy rotation: 100 successful requests.
|
||||
- No automated captcha solving; no inherited login verification after migration.
|
||||
|
||||
## Review Focus
|
||||
|
||||
- Same-name cookies across CHSI domains must survive encrypted migration.
|
||||
- Normalized department names and special-plan score rules must match SQLite results.
|
||||
- Failed migrations must not leave a partly imported database; source remains untouched.
|
||||
- Automatic rotation must not split the SSO binding and access proof across proxies.
|
||||
- Manual pause and login-wait states must survive service restart without automatic resumption.
|
||||
|
||||
### Task 1: Random pacing and proxy rotation
|
||||
|
||||
Files: studio/settings.py, studio/network.py, studio/auth.py, tests/test_server_features.py.
|
||||
|
||||
- [ ] Add failing tests for varied bounded delays, invalid bounds, automatic route changes, and pinned verification.
|
||||
- [ ] Implement interval maximum and proxy rotation settings, actual request route/count reporting, and route pinning during auth verification.
|
||||
- [ ] Run network and auth tests, then commit.
|
||||
|
||||
### Task 2: MariaDB backend and migration
|
||||
|
||||
Files: studio/mysql_database.py, studio/migrate.py, studio/api.py, tests/test_mysql.py, pyproject.toml.
|
||||
|
||||
- [ ] Write tests against a real isolated MariaDB database for upserts, historical scopes, favorites, jobs, and atomic migration.
|
||||
- [ ] Implement MySQLDatabase.connection(), preserving the existing Database API; load private database configuration through STUDIO_DATABASE_CONFIG.
|
||||
- [ ] Implement migrate_sqlite(source, target) preserving IDs, logs and unit keys and rejecting nonempty targets.
|
||||
- [ ] Verify all existing tests and real MariaDB integration tests, then commit.
|
||||
|
||||
### Task 3: Linux operation and user interface
|
||||
|
||||
Files: studio/auth.py, studio/linux_pool.py, studio/pool_service.py, studio/__main__.py, studio/api.py, frontend/src/App.tsx, frontend/src/types.ts.
|
||||
|
||||
- [ ] Add failing tests for Linux vault roundtrip/permissions, server host filtering, and interrupted-task recovery selection.
|
||||
- [ ] Implement encrypted Linux storage, explicit manual-cookie login instructions, Linux systemd pool controls, configured hosts/data directory, and opt-in restart resumption.
|
||||
- [ ] Add interval range, rotation settings and counters to the UI; use capability metadata for platform instructions.
|
||||
- [ ] Run full backend tests and frontend tests/build, then commit.
|
||||
|
||||
### Task 4: Provision, migrate, and deploy
|
||||
|
||||
Files: deploy/*, docs/debian-deployment.md, README.md.
|
||||
|
||||
- [ ] Prepare reproducible systemd, Nginx and private configuration templates with rollback and backup instructions.
|
||||
- [ ] Provision independent MariaDB account/database, dedicated loopback Redis, proxy pool and workbench services.
|
||||
- [ ] Transfer online SQLite backup and credentials through SSH without exposing secrets; verify migrated data and server login.
|
||||
- [ ] Validate authenticated HTTPS, browser workflows, proxy API/scheduler and restart persistence; resume migrated task only after valid authentication.
|
||||
- [ ] Fresh reviewer checks the whole branch; fix material findings and verify again. Publish the deployment branch to the user's repository.
|
||||
@@ -0,0 +1,29 @@
|
||||
# Debian 12 持续采集部署
|
||||
|
||||
用户要求将工作台部署到 186.241.68.230,通过已有 SSH 密钥操作,避免频繁连接触发 fail2ban;使用服务器实际 MariaDB,部署 jhao104/proxy_pool,核实代理复用,并加入随机请求间隔。保留现有本机运行任务。
|
||||
|
||||
## 运行与访问
|
||||
|
||||
使用 Debian 的 Python 3.11、独立虚拟环境和 systemd 服务,普通专用用户运行工作台。现有 MariaDB 10.11 创建独立数据库与最小权限账号,不改其他项目数据库。Nginx 提供 HTTPS 和访问认证;优先指定域名,DNS 未就绪时使用已有有效 IP 证书及独立 9443 端口。后端、代理池与 Redis 仅监听回环地址。操作 SSH 尽量复用持续连接。
|
||||
|
||||
## 数据
|
||||
|
||||
增加 MariaDB/MySQL 存储实现,复用现有数据查询与任务接口;Windows 默认继续使用 SQLite。MariaDB 保存目录、分数线、院校索引、收藏、任务、日志和完成断点。使用 utf8mb4 与稳定记录键,保留原文、年份和特殊计划的匹配规则。
|
||||
|
||||
SQLite 在线 backup 生成一致性快照;显式迁移到空的新库,保留记录 ID 和全部断点。迁移完成核对每张表的行数及记录内容。旧电脑任务保持原状态,服务器副本初次迁移暂停,登录与网络验证成功后才恢复。重启时仅自动恢复原先运行的任务,用户暂停、取消、等待登录任务不自动启动。
|
||||
|
||||
## 凭据与无桌面登录
|
||||
|
||||
Windows 继续 DPAPI;Linux 使用本地权限 0600 的密钥和加密 Cookie。密钥与数据库配置存放运行目录,排除 Git 和公开下载。服务器无需桌面:界面说明在用户电脑完成学信网登录并粘贴 Cookie;账号登录仍可用,验证码保留人工流程。迁移原有 Cookie 通过 SSH 安全传输,并在服务器重新验证,不能继承原“已验证”标记。
|
||||
|
||||
## 请求与代理
|
||||
|
||||
每次请求从配置的最小与最大间隔随机采样,最大不得小于最小,两者均在 0.3–60 秒;相等时允许固定间隔。保留限流退避与人工验证码处理。
|
||||
|
||||
增加“每 N 次成功请求轮换代理”,0 表示保留原策略;服务器默认 100。显示当前代理的请求计数、累计切换次数及实际成功请求路由。会话同步和第二页验证期间延迟自动轮换,轮换后的目录会话重新验证。健康轮换保留旧 IP,不将限流误认成失效。
|
||||
|
||||
Linux 代理 API 与调度器使用专门 systemd 单元,独立 Redis 容器只绑定 127.0.0.1:6381。界面管理固定单元名称,只允许启动和停止本项目 API/调度器,不接受任意 shell 命令,不停止其他项目 Redis。
|
||||
|
||||
## 验证
|
||||
|
||||
现有测试继续通过。新增随机间隔边界、轮换与会话固定、Linux 凭据往返、主机与跨来源保护测试。在实际 MariaDB 临时测试库验证重复更新、收藏、历史线匹配、特殊线隔离、任务断点和 SQLite 迁移。构建网页,在服务器验证 HTTPS 认证、页面、数据查询、代理池、登录状态、systemd 重启恢复和现有站点未受影响。
|
||||
Reference in New Issue
Block a user