feat:上传到阿里云oss
feat:把以前的图片迁移到阿里云oss
This commit is contained in:
dzc committed
1 parent
2de8fa02b2
commit
f55561c98d
47 files changed
+7672
-96
No files matched your search
@@ -0,0 +1,4 @@
|
||||
node_modules/
|
||||
config.local.json
|
||||
*.local.json
|
||||
reports/
|
||||
@@ -0,0 +1,155 @@
|
||||
# 数据库图片迁移到阿里云 OSS
|
||||
|
||||
纯 JavaScript / Node.js 工具。默认自动读取项目当前环境的主数据库连接，从数据库识别旧数据库存储和默认阿里云 OSS，直接读取原图上传并校验，然后更新文件表和业务字段中的旧地址。
|
||||
|
||||
**通常不需要填写配置，不需要设置数据库密码环境变量，也不需要启动本地后端。** 需要 Node.js 20.19+、MySQL 5.7/8.x 和项目已设置好的默认阿里云 OSS。
|
||||
|
||||
## 直接使用
|
||||
|
||||
在本目录执行：
|
||||
|
||||
```sh
|
||||
npm ci --ignore-scripts
|
||||
node migrate.mjs init
|
||||
```
|
||||
|
||||
这会读取代码里的数据库连接，查询存储配置和已有迁移批次，自动生成 `config.local.json`。不上传图片、不更新数据库、不创建迁移表。数据库密码仍从项目 YAML 读取，OSS 密钥仍从存储配置表读取，不重复写入生成文件。
|
||||
|
||||
接着只读预检，或开始迁移：
|
||||
|
||||
```sh
|
||||
node migrate.mjs
|
||||
node migrate.mjs apply --limit 50
|
||||
node migrate.mjs apply
|
||||
```
|
||||
|
||||
第一条没有参数，默认只读预检；第二条实际迁移少量图片；第三条继续完成剩余迁移。可跳过 init，预检时也会自动生成配置。中断或失败后重跑相同 apply 命令即可。
|
||||
|
||||
预检只读文件元数据，不读取图片二进制内容或计算 SHA-256。图片分组、文件记录和路径拼写由三次批量查询完成，不再逐张反复访问远程数据库。输出包含当前阶段、扫描表名和行数；等待较长查询时每 5 秒显示耗时。`estimatedBytes` 是文件表记录的容量估算，`sizePending` 是缺少容量信息的文件数，实际长度和哈希在 apply 时校验。
|
||||
|
||||
上传时的源文件复核也分两步：先只查最新记录编号，再按主键读取该图片的长度和哈希，避免在没有路径索引的排序查询中读取图片大字段。每张图片完成后会显示源读取/复核、OSS 上传/校验、最终地址校验及总耗时，便于区分数据库等待与网络传输。
|
||||
|
||||
“更新已校验文件记录”是数据库事务阶段，不是再次上传。脚本每批最多处理 50 张：按主键范围加锁，批量复核源文件最新版本和哈希，再批量更新文件记录、写入逐条回滚日志及迁移状态，一起提交。保留原值比较和事务保护，不需要新增配置或给业务表加索引。该阶段重新计时，并显示每批源校验、记录及日志、提交的耗时。已知源变化或原值冲突会回滚整批，再逐项定位；连接或提交异常则停止，避免重复提交。
|
||||
|
||||
按 Ctrl+C 中断后，保留 `config.local.json` 和同一个 runId，重新执行 `node migrate.mjs apply` 即可续跑。已提交记录保留，未提交事务由数据库回滚。处于 `verified` 的已上传文件采用最多 4 张并发，重新读取 OSS 内容和最终访问地址核对哈希，不重新下载数据库图片，也不重新上传；切换时仍在事务里复核源内容。已切换文件也会批量核对文件记录，并并发检查访问地址。校验失败的图片不会用于更新业务引用。更新脚本后，需要停止旧进程再执行，新代码才会生效。
|
||||
|
||||
文件总大小不等于网络总传输量：脚本在运行电脑上从数据库读取、上传 OSS，并读取 OSS 及最终访问地址进行校验。macOS 系统代理开关也不代表 Node 默认会使用代理；需要结合进程代理变量、实际连接及系统路由判断。此脚本不修改系统代理设置。
|
||||
|
||||
如需回滚：
|
||||
|
||||
```sh
|
||||
node migrate.mjs rollback
|
||||
```
|
||||
|
||||
也可以使用 `npm run init`、`npm run plan`、`npm run apply`、`npm run rollback`。生成的配置和报告已忽略，不进入 Git。
|
||||
|
||||
## 自动识别的内容
|
||||
|
||||
- 环境：显式 `--profile` 优先，其次 `SPRING_PROFILES_ACTIVE`，否则按项目 `application.yaml`。当前源码默认是 dev；生成文件会固定已选环境，避免下一次切换环境后误续跑。
|
||||
- 数据库：合并基础 YAML 和对应环境 YAML，读取 dynamic.primary 指定的主数据源，包括地址、端口、数据库、用户名、密码。按需要解析环境变量占位符。
|
||||
- 源配置：从仍有未删除 BLOB 的数据库存储配置中识别，不依赖迁移过程中会变化的文件表 config_id。
|
||||
- 目标：唯一的默认存储配置，且必须是阿里云 OSS。不能确定时停止，不自动猜选。
|
||||
- 历史域名：从旧文件地址提取；OSS Bucket、访问域名和密钥直接使用已有配置。
|
||||
- 批次：已有批次优先续跑；首次生成稳定批次名称。保留服务器身份、存储编号和扫描范围，避免换库或换目标后误续跑。
|
||||
- 下载接口：根据项目端口和 API 前缀推导。当前源码端口是 48083；默认直接读数据库，不依赖这个接口。
|
||||
|
||||
项目 dev/local 的数据库可能是远程地址，不能把环境名 local 当成本机数据库。每次执行会显示实际连接地址和库名。工具读取仓库配置及支持的环境变量，无法知道部署服务器额外的 Docker/命令行覆盖。
|
||||
|
||||
多个未完成批次、默认 OSS 改变、数据库服务器改变等情况会明确提示。若固定批次建立后又出现新 BLOB，可显式新建批次：
|
||||
|
||||
```sh
|
||||
node migrate.mjs apply --run-id db-images-next
|
||||
```
|
||||
|
||||
新批次名称会保存到自动配置。完成回滚的批次不会自动重新迁移。
|
||||
|
||||
## 仅在需要时覆盖
|
||||
|
||||
正常不需要复制示例或逐项编辑。`config.example.json` 为 `{}`，表示全部自动识别；`config.manual.example.json` 保留完全手工配置方式，兼容旧用法。
|
||||
|
||||
例如切换项目环境：
|
||||
|
||||
```sh
|
||||
node migrate.mjs init --profile local
|
||||
```
|
||||
|
||||
同一生成配置绑定原数据库；如果切换后指向不同数据库，会停止，不能把原批次用于另一台服务器。
|
||||
|
||||
其他可选设置仍可写入配置文件：`referenceSchemas` 添加同一 MySQL 实例的 IM 库，`imagesOnly: false` 包含非图片，`sourceMode: "http"` 使用原下载接口。手动连接另一数据库时可用 `--config config.manual.local.json`，其中 mysql.passwordEnv 指定的环境变量需要自行提供。
|
||||
|
||||
迁移前备份数据库，保持原图片和目标配置稳定。回滚只恢复本批次写入且没有被后来修改的值，不删除原 BLOB 或 OSS 对象。最终清理仍是独立步骤。
|
||||
|
||||
## 实际处理流程
|
||||
|
||||
1. 按数据库自身的路径比较规则，选取每个 `config_id + path` 最大的未删除内容 ID，与 `DBFileClient.getContent` 一致。大整数 ID 全程使用字符串。
|
||||
2. 默认处理 MIME 为 `image/*` 或常见图片扩展名的内容；没有文件表记录但可以识别为图片的 BLOB 也会处理。其他文件计入跳过数量；如确需迁移全部数据库文件，设置 `imagesOnly: false`。无法识别的无扩展名内容需要单独核查。
|
||||
3. 建立固定迁移清单，保存原文件记录和旧地址。默认直接读取数据库原始字节；HTTP 模式通过旧下载接口读取。两种方式均与数据库计算的长度、SHA-256 比较。
|
||||
4. 使用独立目录 `objectPrefix/旧配置编号/路径哈希/内容哈希.扩展名` 上传；保留原始文件字节，不压缩图片。上传前后均检查对象，不覆盖不同内容。
|
||||
5. 从 OSS 重新读取校验，并匿名访问最终将写入数据库的 URL 再次校验。SDK 上传成功但浏览器地址不可访问时，不切换数据库记录。
|
||||
6. 本次选中文件全部通过上传校验后，每批最多 50 张用一个事务更新 `infra_file.config_id/path/url/type/size`，保留文件记录 ID 和自动更新时间。逐条回滚日志与文件记录、迁移状态同时提交。
|
||||
7. 扫描指定数据库的字符、文本、JSON 字段，按精确地址映射替换单 URL、逗号列表、JSON 值及可识别的富文本地址。每行更新都检查原值，避免覆盖用户刚修改的数据。原生 JSON 的日志保存数据库实际持久化结果，便于正确回滚。
|
||||
|
||||
上传图片是逐个进行的；正式迁移按主键分批计算内容哈希，不会一次加载整个 BLOB 表。文件元数据和 URL 映射保存在内存；引用读取每页至少 1000 行，更新仍逐行执行。MySQL 当前会话的单条 SELECT 默认最多执行 60 秒，沿用 `timeoutMs` 配置，不修改服务器全局设置。
|
||||
|
||||
## OSS 访问方式
|
||||
|
||||
- 默认 `targetUrlMode: "direct"`，业务字段保存 OSS/CDN 稳定地址。地址优先使用 `targetPublicDomain`，其次使用存储配置的 `domain`，最后根据 Bucket 和 OSS 节点生成公网域名。Bucket/CDN 应允许这些图片按该地址读取；脚本不会修改 Bucket 权限。
|
||||
- 私有 Bucket 可使用 `targetUrlMode: "backend"`，业务字段保存现有后端下载入口加目标配置编号的地址，后端通过自己的 OSS 凭据读取。`downloadBaseUrl` 此时必须是用户设备能访问的稳定地址，不能填仅运行脚本机器可访问的 localhost。现有接口是 `@PermitAll`，此模式沿用项目已有访问规则，不新增鉴权。
|
||||
- 默认 `sourceMode: "database"`，无需启动旧后端。也可设置 `sourceMode: "http"` 使用原下载接口。使用 backend 目标模式时，后端仍须在线。
|
||||
|
||||
## 旧地址与迁移范围
|
||||
|
||||
修改文件表不会自动改变旧接口的行为：它仍按 URL 中的配置编号直接读取存储。因此脚本始终保留 `infra_file_content` 和原数据库配置，让旧客户端缓存及未替换引用仍有机会继续读取。
|
||||
|
||||
过渡期通过文件管理删除 OSS 文件，不会自动删除旧数据库副本。历史副本的删除和最终清理需要关联处理；本脚本不承担清理工作。续跑会重新检查已迁移对象和文件记录，目标已被删除或改动时停止复用对应映射。
|
||||
|
||||
旧接口会解码路径一次，并将裸 `+` 当空格。脚本下载时逐段编码；旧 URL 只有在其实际解码结果对应当前文件时才加入映射。裸 `+`、已有 `%HH`、`?/#` 等存在歧义的旧地址不会被静默指向另一张图片，未解决的引用会进入复核统计。
|
||||
|
||||
未知查询参数、片段、相对地址和无法识别的文本组合不做猜测替换。JSON 属性名不改。历史域名别名通过 `oldDomains` 补充。默认扫描主数据库，额外数据库只扫描同一个 MySQL 实例且当前账号有权限的 `referenceSchemas`；不同实例的 IM 库需要单独处理。
|
||||
|
||||
IM 自己上传到 MinIO 的文件不属于本次 BLOB 搬迁，但 IM 中保存的主库旧图片 URL 可以随引用扫描替换。Redis、客户端本地缓存、其他数据库实例和外部系统不在扫描范围，旧 BLOB 不能在脚本结束后立即清理。
|
||||
|
||||
业务引用表需要单列主键；无主键、复合主键表会列入待复核。更新要求 InnoDB。视图、生成列、文件基础表和迁移日志表不参与业务引用替换。`excludeReferenceTables` 可以填写明确不需要扫描的表名或 `库名.表名`；这些主动排除项会在报告中列出。
|
||||
|
||||
接口访问日志、接口错误日志、定时任务日志、系统操作日志和登录日志也默认排除：它们不是页面图片引用，迁移不应改写历史记录。具体为 `infra_api_access_log`、`infra_api_error_log`、`infra_job_log`、`system_operate_log`、`system_login_log`，不会泛化排除其他名字包含 log 的业务表。
|
||||
|
||||
## 结果、恢复与权限
|
||||
|
||||
每次命令在配置文件旁的 `reports/` 写 JSON 报告：
|
||||
|
||||
| 状态 | 含义 |
|
||||
| --- | --- |
|
||||
| `planned` | 数据库预检通过，尚未校验 HTTP/OSS 连通性 |
|
||||
| `complete` | 本批文件已切换，已扫描范围未检测到剩余旧路由引用 |
|
||||
| `partial` | 限量试迁移、上传失败、文件记录冲突或业务更新冲突，需要续跑/处理 |
|
||||
| `needs-review` | 预检/迁移扫描仍发现旧地址或不支持的表；可能包含有意跳过的非图片、未知 URL 变体，不能据此清理源数据 |
|
||||
| `rolled_back` / `rollback_partial` | 回滚完成 / 仍有并发修改冲突 |
|
||||
|
||||
退出码：0 为本次命令成功；2 为部分完成或需要复核；1 为执行错误。
|
||||
失败时只输出简短错误，不输出配置密钥；详细原值、映射及逐项状态保存在主库的三个表：
|
||||
|
||||
```text
|
||||
infra_file_oss_migration_runs
|
||||
infra_file_oss_migration_items
|
||||
infra_file_oss_migration_changes
|
||||
```
|
||||
|
||||
`plan` 不创建这些表、不上传、不更新业务数据；`apply` 首次执行会创建日志表。更新与日志在同一事务中提交，回滚不依赖本地报告。保留这些表及数据库备份直到验收结束。
|
||||
|
||||
同一主库的 apply/rollback 使用 MySQL 命名锁避免重复运行。同一 runId 绑定原数据库服务器、存储目标和扫描范围，不能换库或换 Bucket 后继续执行。需要 SELECT/UPDATE、主库创建和写入日志表的权限，以及 OSS 对指定对象的读取/写入权限；不需要 OSS 删除权限。
|
||||
|
||||
脚本会开启当前数据库连接的严格模式，并在 SQL 出现截断等警告时回滚，防止较短的业务字段保存损坏 URL。apply 将当前连接的隔离级别设为 REPEATABLE READ，要求源内容表、文件表和日志表为 InnoDB；批次源校验的主键范围锁覆盖可能新增或恢复的较新版本，防止校验期间源内容变化。不修改数据库全局隔离级别。配置文件、依赖目录和报告已通过此目录的 `.gitignore` 排除；回滚日志包含原业务值，应按业务数据库的权限保护。
|
||||
|
||||
## 验证
|
||||
|
||||
```sh
|
||||
npm test
|
||||
```
|
||||
|
||||
默认运行本地 HTTP、模拟 OSS 和事务接口的测试。真实 MySQL 集成测试默认跳过；仅可指向专用临时 MySQL 容器的端口：
|
||||
|
||||
```sh
|
||||
MIGRATION_TEST_DB_PORT=临时容器端口 node --test test/integration.test.mjs
|
||||
```
|
||||
|
||||
集成测试会创建并删除固定的 `codex_oss_migration_test` 测试库，不可指向业务数据库服务器。其 OSS 是内存模拟，HTTP 下载接口由本地测试服务模拟；真实阿里云连接和实际业务页面仍需通过少量试迁移验收。
|
||||
@@ -0,0 +1 @@
|
||||
{}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"runId": "db-images-20261105",
|
||||
"mysql": {
|
||||
"host": "127.0.0.1",
|
||||
"port": 3306,
|
||||
"user": "root",
|
||||
"passwordEnv": "MIGRATION_DB_PASSWORD",
|
||||
"database": "YOUR_DATABASE"
|
||||
},
|
||||
"sourceConfigIds": [],
|
||||
"targetConfigId": null,
|
||||
"sourceMode": "http",
|
||||
"downloadBaseUrl": "http://127.0.0.1:48083/admin-api/infra/file",
|
||||
"oldDomains": [],
|
||||
"targetUrlMode": "direct",
|
||||
"targetPublicDomain": "",
|
||||
"objectPrefix": "migration/db-images",
|
||||
"referenceSchemas": [],
|
||||
"excludeReferenceTables": [],
|
||||
"imagesOnly": true,
|
||||
"batchSize": 100,
|
||||
"timeoutMs": 60000,
|
||||
"maxFileBytes": 67108864,
|
||||
"retries": 3
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
import { baseUrl, bit, id, normalizeConfig, ossSettings, qualified, sha256 } from './core.mjs';
|
||||
|
||||
const RUNS_TABLE = 'infra_file_oss_migration_runs';
|
||||
const RESTORED_FIELDS = [
|
||||
'sourceConfigIds', 'targetConfigId', 'sourceMode', 'downloadBaseUrl', 'oldDomains',
|
||||
'objectPrefix', 'imagesOnly', 'referenceSchemas', 'excludeReferenceTables', 'targetUrlMode'
|
||||
];
|
||||
const hasValue = (value, key) => Object.prototype.hasOwnProperty.call(value, key) && value[key] !== undefined;
|
||||
const sortedIds = values => [...new Set(values.map(id))].sort((left, right) => BigInt(left) < BigInt(right) ? -1 : BigInt(left) > BigInt(right) ? 1 : 0);
|
||||
|
||||
function assertIdentity(expected, current) {
|
||||
if (!expected || expected.server !== current.server || expected.schema !== current.schema) {
|
||||
throw new Error('数据库身份与原迁移配置不一致，请检查数据库连接和批次编号');
|
||||
}
|
||||
}
|
||||
|
||||
function parseRunSettings(value) {
|
||||
let settings;
|
||||
try {
|
||||
settings = typeof value === 'string' ? JSON.parse(value) : value;
|
||||
} catch {
|
||||
throw new Error('迁移批次的配置记录损坏，不能自动恢复');
|
||||
}
|
||||
if (!settings || typeof settings !== 'object' || !settings.binding || typeof settings.binding !== 'object') {
|
||||
throw new Error('迁移批次缺少原始配置绑定，不能自动恢复');
|
||||
}
|
||||
return settings;
|
||||
}
|
||||
|
||||
function selectRun(rows, requestedRunId) {
|
||||
if (requestedRunId !== undefined) return rows.find(row => row.run_id === requestedRunId);
|
||||
const unfinished = rows.filter(row => !['complete', 'rolled_back'].includes(row.status));
|
||||
if (unfinished.length > 1) throw new Error('存在多个未完成迁移批次，请使用 --run-id 指定要继续的批次');
|
||||
if (unfinished.length === 1) return unfinished[0];
|
||||
if (rows.length > 1) throw new Error('存在多个历史迁移批次，请使用 --run-id 指定批次，或指定新的批次编号');
|
||||
return rows[0];
|
||||
}
|
||||
|
||||
function contentId(value) {
|
||||
if ((typeof value === 'number' && !Number.isSafeInteger(value)) || !/^\d+$/.test(String(value))) {
|
||||
throw new Error('迁移批次的内容上界无效，不能安全判断增量数据');
|
||||
}
|
||||
return BigInt(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read-only discovery of storage configuration and a stable migration run.
|
||||
* This function never emits logs, persists configuration, or returns the OSS
|
||||
* credentials read from infra_file_config. Explicit input fields take priority
|
||||
* over a restored binding; loadSettings/runApply validate the final fingerprint.
|
||||
*/
|
||||
export async function discoverStorageConfig(db, input, { command = 'plan' } = {}) {
|
||||
if (!input?.mysql?.database) throw new Error('缺少本地数据库名称，无法自动识别文件存储');
|
||||
const schema = input.mysql.database;
|
||||
const [[serverRow]] = await db.execute('SELECT @@server_uuid AS server_uuid');
|
||||
if (!serverRow?.server_uuid) throw new Error('无法读取数据库服务器身份');
|
||||
const identity = { server: serverRow.server_uuid, schema };
|
||||
if (hasValue(input, '_databaseIdentity')) assertIdentity(input._databaseIdentity, identity);
|
||||
|
||||
const [runTable] = await db.execute(
|
||||
"SELECT 1 AS present FROM information_schema.TABLES WHERE TABLE_SCHEMA=? AND TABLE_NAME=? AND TABLE_TYPE='BASE TABLE' LIMIT 1",
|
||||
[schema, RUNS_TABLE]
|
||||
);
|
||||
let history = [];
|
||||
if (runTable.length) {
|
||||
[history] = await db.execute(`SELECT run_id,settings,status FROM ${qualified(schema, RUNS_TABLE)}`);
|
||||
}
|
||||
const selected = selectRun(history, input.runId);
|
||||
const settings = selected ? parseRunSettings(selected.settings) : undefined;
|
||||
const config = { ...input, mysql: { ...input.mysql } };
|
||||
if (selected) {
|
||||
assertIdentity(settings.binding, identity);
|
||||
for (const key of RESTORED_FIELDS) {
|
||||
if (!hasValue(input, key) && hasValue(settings.binding, key)) {
|
||||
config[key] = structuredClone(settings.binding[key]);
|
||||
}
|
||||
}
|
||||
if (!hasValue(input, 'targetPublicDomain') && hasValue(settings.binding, 'publicDomain')) {
|
||||
config.targetPublicDomain = settings.binding.publicDomain;
|
||||
}
|
||||
config.runId = selected.run_id;
|
||||
if (command === 'apply' && (selected.status.startsWith('rollback') || selected.status === 'rolled_back')) {
|
||||
throw new Error('该批次已开始或完成回滚，不能继续迁移；请继续 rollback，或使用 --run-id 指定新批次');
|
||||
}
|
||||
} else if (command === 'rollback') {
|
||||
throw new Error('未找到可回滚的历史迁移批次，请使用 --run-id 指定已有批次');
|
||||
}
|
||||
|
||||
config._databaseIdentity = identity;
|
||||
config._autoDiscovery = true;
|
||||
delete config._projectDownloadBaseUrl;
|
||||
|
||||
// Rollback only restores journalled database values. Current storage rows,
|
||||
// default targets and OSS credentials are deliberately irrelevant here.
|
||||
if (command === 'rollback') return normalizeConfig(config);
|
||||
|
||||
if (!hasValue(config, 'sourceMode')) config.sourceMode = 'database';
|
||||
if (!hasValue(config, 'downloadBaseUrl') && input._projectDownloadBaseUrl) {
|
||||
config.downloadBaseUrl = input._projectDownloadBaseUrl;
|
||||
}
|
||||
const [storageRows] = await db.execute(`SELECT id,storage,master,config FROM ${qualified(schema, 'infra_file_config')} WHERE deleted=0`);
|
||||
const byId = new Map(storageRows.map(row => [id(row.id), row]));
|
||||
const masters = storageRows.filter(row => bit(row.master));
|
||||
const requireDefault = !hasValue(config, 'targetConfigId') || input._autoDiscovery === true || Boolean(selected);
|
||||
if (requireDefault && masters.length !== 1) {
|
||||
throw new Error('默认文件存储配置必须唯一，请先在文件配置中设置唯一的阿里云 OSS 默认存储');
|
||||
}
|
||||
if (!hasValue(config, 'targetConfigId')) config.targetConfigId = id(masters[0].id);
|
||||
config.targetConfigId = id(config.targetConfigId);
|
||||
if (requireDefault && (id(masters[0].id) !== config.targetConfigId
|
||||
|| (selected && id(masters[0].id) !== id(settings.binding.targetConfigId)))) {
|
||||
throw new Error('当前默认 OSS 与原迁移目标不同，不能自动续跑；请恢复原默认配置或使用 --run-id 建立新批次');
|
||||
}
|
||||
const target = byId.get(config.targetConfigId);
|
||||
if (!target) throw new Error('目标文件存储配置不存在或已删除');
|
||||
try {
|
||||
ossSettings(target, config);
|
||||
} catch {
|
||||
// JSON/URL parsing errors must not include raw configuration or secrets.
|
||||
throw new Error('目标默认存储不是有效的阿里云 OSS 配置，请检查类型、endpoint、Bucket 和访问凭据');
|
||||
}
|
||||
|
||||
if (!hasValue(config, 'sourceConfigIds')) {
|
||||
const databaseIds = sortedIds(storageRows.filter(row => Number(row.storage) === 1).map(row => row.id));
|
||||
let contents = [];
|
||||
if (databaseIds.length) {
|
||||
[contents] = await db.execute(
|
||||
`SELECT DISTINCT config_id FROM ${qualified(schema, 'infra_file_content')} WHERE deleted=0 AND config_id IN (${databaseIds.map(() => '?').join(',')})`,
|
||||
databaseIds
|
||||
);
|
||||
}
|
||||
config.sourceConfigIds = sortedIds(contents.map(row => row.config_id));
|
||||
}
|
||||
if (!Array.isArray(config.sourceConfigIds) || !config.sourceConfigIds.length) {
|
||||
throw new Error('未发现包含未删除内容的数据库存储配置，无需迁移');
|
||||
}
|
||||
config.sourceConfigIds = [...new Set(config.sourceConfigIds.map(id))];
|
||||
for (const sourceId of config.sourceConfigIds) {
|
||||
const source = byId.get(sourceId);
|
||||
if (!source || Number(source.storage) !== 1) throw new Error('原数据库存储配置不存在、已删除或类型发生变化，请检查原迁移绑定');
|
||||
if (bit(source.master)) throw new Error('源配置仍是默认上传配置，请先切换默认配置到阿里云 OSS');
|
||||
}
|
||||
|
||||
if (selected && selected.status !== 'inventory' && command === 'apply') {
|
||||
const sourceIds = settings.binding.sourceConfigIds.map(id);
|
||||
if (!sourceIds.length) throw new Error('历史迁移批次的源配置列表为空，不能安全续跑');
|
||||
const [[maximum]] = await db.execute(
|
||||
`SELECT COALESCE(MAX(id),0) AS max_id FROM ${qualified(schema, 'infra_file_content')} WHERE deleted=0 AND config_id IN (${sourceIds.map(() => '?').join(',')})`,
|
||||
sourceIds
|
||||
);
|
||||
if (contentId(maximum.max_id) > contentId(settings.maxId)) {
|
||||
throw new Error('原批次建立后出现新的数据库文件，请使用 --run-id 指定新批次，避免遗漏新增内容');
|
||||
}
|
||||
}
|
||||
|
||||
if (!hasValue(config, 'oldDomains')) {
|
||||
const [domains] = await db.execute(
|
||||
`SELECT DISTINCT SUBSTRING_INDEX(url,'/admin-api/infra/file/',1) AS domain FROM ${qualified(schema, 'infra_file')} WHERE deleted=0 AND config_id IN (${config.sourceConfigIds.map(() => '?').join(',')}) AND url LIKE '%/admin-api/infra/file/%'`,
|
||||
config.sourceConfigIds
|
||||
);
|
||||
const validDomains = new Set();
|
||||
for (const row of domains) {
|
||||
try { validDomains.add(baseUrl(row.domain)); } catch { /* Ignore malformed and credential-bearing historical URLs. */ }
|
||||
}
|
||||
config.oldDomains = [...validDomains].sort();
|
||||
}
|
||||
|
||||
if (!hasValue(config, 'runId')) {
|
||||
const fingerprint = { ...identity, sourceConfigIds: sortedIds(config.sourceConfigIds), targetConfigId: config.targetConfigId };
|
||||
config.runId = `db-images-${sha256(JSON.stringify(fingerprint)).slice(0, 16)}`;
|
||||
}
|
||||
return normalizeConfig(config);
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
/**
|
||||
* Map in input order with bounded parallelism. After the first failure, stop
|
||||
* scheduling work and drain every in-flight worker before rejecting.
|
||||
*/
|
||||
export async function mapBounded(items, limit, worker) {
|
||||
if (!Number.isSafeInteger(limit) || limit < 1) throw new Error('并发数量必须为正整数');
|
||||
const source = Array.from(items);
|
||||
const results = new Array(source.length);
|
||||
let next = 0;
|
||||
let failed = false;
|
||||
let firstError;
|
||||
|
||||
async function run() {
|
||||
while (!failed && next < source.length) {
|
||||
const index = next++;
|
||||
try {
|
||||
results[index] = await worker(source[index], index);
|
||||
} catch (error) {
|
||||
if (!failed) {
|
||||
failed = true;
|
||||
firstError = error;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await Promise.all(Array.from({ length: Math.min(limit, source.length) }, () => run()));
|
||||
if (failed) throw firstError;
|
||||
return results;
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
|
||||
export const sha256 = value => createHash('sha256').update(value).digest('hex');
|
||||
export const quote = name => '`' + String(name).replaceAll('`', '``') + '`';
|
||||
export const qualified = (schema, table) => `${quote(schema)}.${quote(table)}`;
|
||||
export const bit = value => Buffer.isBuffer(value) ? value.some(byte => byte !== 0) : value === true || value === 1 || value === '1';
|
||||
export const errorText = error => String(error.code || error.message || error.name || '未知错误').replace(/https?:\/\/\S+/g, '[URL]').slice(0, 300);
|
||||
|
||||
export function id(value) {
|
||||
if (typeof value === 'number' && !Number.isSafeInteger(value)) throw new Error('配置编号超出安全整数范围，请使用字符串');
|
||||
if (!/^[1-9]\d*$/.test(String(value))) throw new Error('配置编号必须为正整数，不能使用存储类型代替配置编号');
|
||||
return String(value);
|
||||
}
|
||||
|
||||
export function baseUrl(value) {
|
||||
const parsed = new URL(value);
|
||||
if (!['http:', 'https:'].includes(parsed.protocol) || parsed.username || parsed.password || parsed.search || parsed.hash) {
|
||||
throw new Error('访问基础地址必须是无账号、查询参数和片段的 HTTP(S) URL');
|
||||
}
|
||||
return parsed.href.replace(/\/+$/, '');
|
||||
}
|
||||
|
||||
export function encodePath(path) {
|
||||
if (!path || path.startsWith('/') || path.includes('\\') || path.split('/').some(part => ['.', '..', ''].includes(part))) {
|
||||
throw new Error('文件路径为空或含不支持的路径段，需单独处理');
|
||||
}
|
||||
return path.split('/').map(encodeURIComponent).join('/');
|
||||
}
|
||||
|
||||
export const downloadUrl = (base, configId, path) => `${baseUrl(base)}/${id(configId)}/get/${encodePath(path)}`;
|
||||
|
||||
export function normalizeConfig(input) {
|
||||
const config = {
|
||||
sourceMode: 'http', targetUrlMode: 'direct', objectPrefix: 'migration/db-images',
|
||||
imagesOnly: true, batchSize: 100, timeoutMs: 60000, maxFileBytes: 64 * 1024 * 1024,
|
||||
retries: 3, oldDomains: [], excludeReferenceTables: [], ...input
|
||||
};
|
||||
if (!/^[a-zA-Z0-9_-]{1,64}$/.test(config.runId || '')) throw new Error('runId 必须是 1～64 位字母、数字、横线或下划线');
|
||||
if (!config.mysql?.database || !config.mysql?.user) throw new Error('请填写 mysql.database 和 mysql.user');
|
||||
if (!Array.isArray(config.sourceConfigIds) || !config.sourceConfigIds.length) throw new Error('请填写 sourceConfigIds（原数据库存储配置编号）');
|
||||
config.sourceConfigIds = [...new Set(config.sourceConfigIds.map(id))];
|
||||
config.targetConfigId = id(config.targetConfigId);
|
||||
if (config.sourceConfigIds.includes(config.targetConfigId)) throw new Error('源配置和 OSS 目标配置不能相同');
|
||||
if (!['http', 'database'].includes(config.sourceMode)) throw new Error('sourceMode 只能为 http 或 database');
|
||||
if (!['direct', 'backend'].includes(config.targetUrlMode)) throw new Error('targetUrlMode 只能为 direct 或 backend');
|
||||
if (config.sourceMode === 'http' || config.targetUrlMode === 'backend') config.downloadBaseUrl = baseUrl(config.downloadBaseUrl);
|
||||
if (!/^[a-zA-Z0-9][a-zA-Z0-9/_-]{0,119}$/.test(config.objectPrefix) || config.objectPrefix.endsWith('/')) throw new Error('objectPrefix 只能包含字母、数字、横线、下划线和目录分隔符');
|
||||
for (const key of ['batchSize', 'timeoutMs', 'maxFileBytes', 'retries']) {
|
||||
if (!Number.isSafeInteger(config[key]) || config[key] < 1) throw new Error(`${key} 必须为正整数`);
|
||||
}
|
||||
if (config.batchSize > 1000 || config.retries > 10) throw new Error('batchSize 上限为 1000，retries 上限为 10');
|
||||
config.oldDomains = config.oldDomains.map(baseUrl);
|
||||
config.referenceSchemas = [...new Set([config.mysql.database, ...(config.referenceSchemas || [])])];
|
||||
return config;
|
||||
}
|
||||
|
||||
export function parseStorage(row) {
|
||||
return typeof row.config === 'string' ? JSON.parse(row.config) : row.config;
|
||||
}
|
||||
|
||||
export function ossSettings(row, config) {
|
||||
if (Number(row.storage) !== 20) throw new Error('目标配置不是 S3/OSS 存储（storage=20）');
|
||||
const value = parseStorage(row);
|
||||
const endpoint = new URL(/^https?:\/\//.test(value.endpoint) ? value.endpoint : `https://${value.endpoint}`);
|
||||
if (!/^oss-[a-z0-9-]+\.aliyuncs\.com$/.test(endpoint.hostname) || endpoint.pathname !== '/' || endpoint.search || endpoint.hash || endpoint.username || endpoint.password) {
|
||||
throw new Error('目标 endpoint 必须为阿里云 OSS 区域/加速节点，不能是其他 S3 服务或 Bucket 域名');
|
||||
}
|
||||
if (!value.bucket || !value.accessKey || !value.accessSecret) throw new Error('OSS 配置缺少 bucket/accessKey/accessSecret');
|
||||
const publicDomain = baseUrl(config.targetPublicDomain || value.domain || `https://${value.bucket}.${endpoint.hostname.replace('-internal.', '.')}`);
|
||||
return { endpoint: endpoint.origin, bucket: value.bucket, accessKeyId: value.accessKey, accessKeySecret: value.accessSecret, publicDomain };
|
||||
}
|
||||
|
||||
export function objectKey(config, item) {
|
||||
const extension = /\.([a-zA-Z0-9]{1,10})$/.exec(item.path)?.[1]?.toLowerCase();
|
||||
return `${config.objectPrefix}/${item.configId}/${sha256(item.path)}/${item.sha256}${extension ? '.' + extension : ''}`;
|
||||
}
|
||||
|
||||
export function aliases(item, rows, sourceDomain, config) {
|
||||
const result = new Set(rows.map(row => row.url).filter(Boolean));
|
||||
const domains = [...new Set([sourceDomain, ...config.oldDomains].filter(Boolean).map(baseUrl))];
|
||||
const paths = [...new Set([item.path, ...rows.map(row => row.path)])];
|
||||
const sourcePaths = new Set(paths);
|
||||
const addPathAliases = (prefix, path) => {
|
||||
const encoded = encodePath(path);
|
||||
result.add(prefix + path);
|
||||
result.add(prefix + encoded);
|
||||
// FileController uses form-style decoding: a literal '+' means a space.
|
||||
result.add(prefix + encoded.replaceAll('%20', '+'));
|
||||
};
|
||||
for (const domain of domains) for (const path of paths) {
|
||||
const prefix = `${domain}/admin-api/infra/file/${item.configId}/get/`;
|
||||
addPathAliases(prefix, path);
|
||||
}
|
||||
if (config.downloadBaseUrl) {
|
||||
const prefix = baseUrl(config.downloadBaseUrl) + '/' + id(item.configId) + '/get/';
|
||||
for (const path of paths) addPathAliases(prefix, path);
|
||||
}
|
||||
return [...result].filter(value => {
|
||||
try {
|
||||
// Parse the actual request path: raw '?' / '#' end the path, and the
|
||||
// browser encodes spaces/Unicode before FileController sees requestURI.
|
||||
const parsed = new URL(value, 'https://migration-alias.invalid');
|
||||
const route = parsed.pathname.match(/\/infra\/file\/([^/]+)\/get(?:\/|$)/);
|
||||
if (!route) return true; // Keep pre-existing URLs from other file routes.
|
||||
if (BigInt(decodeURIComponent(route[1])) !== BigInt(item.configId)) return false;
|
||||
const marker = parsed.pathname.indexOf('/get/');
|
||||
if (marker < 0) return false;
|
||||
const path = decodeURIComponent(parsed.pathname.slice(marker + 5).replaceAll('+', ' '));
|
||||
return sourcePaths.has(path);
|
||||
} catch {
|
||||
// Unverifiable aliases remain untouched and are reported by the caller.
|
||||
return false;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
export function mimeType(path, records) {
|
||||
const stored = records.map(row => row.type).find(type => /^image\//i.test(type || ''));
|
||||
if (stored) return stored;
|
||||
const ext = path.split('.').pop().toLowerCase();
|
||||
const types = { png: 'image/png', jpg: 'image/jpeg', jpeg: 'image/jpeg', gif: 'image/gif', webp: 'image/webp', svg: 'image/svg+xml', bmp: 'image/bmp', ico: 'image/x-icon', tif: 'image/tiff', tiff: 'image/tiff', avif: 'image/avif', heic: 'image/heic', heif: 'image/heif' };
|
||||
return types[ext] || records.find(row => row.type)?.type || 'application/octet-stream';
|
||||
}
|
||||
|
||||
export async function readHttp(url, { timeoutMs, maxFileBytes }, expectedSize) {
|
||||
const response = await fetch(url, { signal: AbortSignal.timeout(timeoutMs), redirect: 'error' });
|
||||
if (!response.ok) { await response.body?.cancel(); throw new Error(`文件访问返回 HTTP ${response.status}`); }
|
||||
const declared = Number(response.headers.get('content-length'));
|
||||
if (declared > maxFileBytes || (expectedSize !== undefined && declared > expectedSize)) {
|
||||
await response.body?.cancel(); throw new Error('HTTP 文件长度超过限制');
|
||||
}
|
||||
let size = 0;
|
||||
const chunks = [];
|
||||
for await (const chunk of response.body) {
|
||||
size += chunk.length;
|
||||
if (size > maxFileBytes || (expectedSize !== undefined && size > expectedSize)) throw new Error('HTTP 文件长度超过限制');
|
||||
chunks.push(chunk);
|
||||
}
|
||||
return Buffer.concat(chunks);
|
||||
}
|
||||
|
||||
export function verifyContent(bytes, expected) {
|
||||
if (bytes.length !== Number(expected.size) || sha256(bytes) !== expected.sha256) throw new Error('文件长度或 SHA-256 不一致');
|
||||
}
|
||||
|
||||
export async function ensureObject(oss, key, bytes, item) {
|
||||
let exists = true;
|
||||
try {
|
||||
const result = await oss.head(key);
|
||||
if (Number(result.res.headers['content-length']) !== Number(item.size)) throw new Error('OSS 同名对象长度不同，禁止覆盖');
|
||||
} catch (error) {
|
||||
if (error.code === 'NoSuchKey' || error.status === 404) exists = false;
|
||||
else throw error;
|
||||
}
|
||||
if (!exists) {
|
||||
try {
|
||||
await oss.put(key, bytes, { headers: { 'Content-Type': item.type, 'x-oss-forbid-overwrite': 'true' } });
|
||||
} catch (error) {
|
||||
if (error.code !== 'FileAlreadyExists' && error.status !== 409) throw error;
|
||||
}
|
||||
}
|
||||
const result = await oss.head(key);
|
||||
if (Number(result.res.headers['content-length']) !== Number(item.size)) throw new Error('OSS 文件长度不一致');
|
||||
verifyContent((await oss.get(key)).content, item);
|
||||
}
|
||||
|
||||
export async function retry(action, attempts, wait = ms => new Promise(resolve => setTimeout(resolve, ms))) {
|
||||
for (let i = 0; ; i++) {
|
||||
try { return await action(); } catch (error) {
|
||||
if (i + 1 >= attempts) throw error;
|
||||
await wait(Math.min(1000 * 2 ** i, 5000));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,318 @@
|
||||
import { qualified, quote } from './core.mjs';
|
||||
|
||||
const RUNS = 'infra_file_oss_migration_runs';
|
||||
const ITEMS = 'infra_file_oss_migration_items';
|
||||
const CHANGES = 'infra_file_oss_migration_changes';
|
||||
const metadataCache = new WeakMap();
|
||||
const own = (object, key) => Object.prototype.hasOwnProperty.call(object, key);
|
||||
|
||||
// Snapshots must preserve BIT/Buffer and 64-bit keys without Number conversion.
|
||||
function pack(value) {
|
||||
if (Buffer.isBuffer(value)) return { type: 'buffer', value: value.toString('base64') };
|
||||
if (typeof value === 'bigint') return { type: 'bigint', value: value.toString() };
|
||||
if (value instanceof Date) return { type: 'date', value: value.toISOString() };
|
||||
if (value === undefined) throw new Error('变更快照不能包含 undefined');
|
||||
return { type: 'json', value };
|
||||
}
|
||||
|
||||
function unpack(value) {
|
||||
if (value.type === 'buffer') return Buffer.from(value.value, 'base64');
|
||||
if (value.type === 'bigint') return BigInt(value.value);
|
||||
if (value.type === 'date') return new Date(value.value);
|
||||
if (value.type === 'json') return value.value;
|
||||
throw new Error('变更日志值类型不受支持');
|
||||
}
|
||||
|
||||
function snapshot(record) {
|
||||
return JSON.stringify(Object.fromEntries(Object.entries(record).map(([key, value]) => [key, pack(value)])));
|
||||
}
|
||||
|
||||
function restoreSnapshot(text) {
|
||||
return Object.fromEntries(Object.entries(JSON.parse(text)).map(([key, value]) => [key, unpack(value)]));
|
||||
}
|
||||
|
||||
function parameter(value) {
|
||||
if (typeof value === 'bigint') return value.toString();
|
||||
if (value !== null && typeof value === 'object' && !Buffer.isBuffer(value) && !(value instanceof Date)) return JSON.stringify(value);
|
||||
return value;
|
||||
}
|
||||
|
||||
const equal = (left, right) => JSON.stringify(pack(left)) === JSON.stringify(pack(right));
|
||||
|
||||
function condition(values) {
|
||||
const entries = Object.entries(values);
|
||||
return {
|
||||
sql: entries.map(([key]) => `CAST(${quote(key)} AS BINARY) <=> CAST(? AS BINARY)`).join(' AND '),
|
||||
params: entries.map(([, value]) => parameter(value))
|
||||
};
|
||||
}
|
||||
|
||||
async function tableMetadata(connection, schema, table) {
|
||||
let cache = metadataCache.get(connection);
|
||||
if (!cache) { cache = new Map(); metadataCache.set(connection, cache); }
|
||||
const key = JSON.stringify([schema, table]);
|
||||
if (!cache.has(key)) {
|
||||
const [rows] = await connection.execute(
|
||||
`SELECT c.COLUMN_NAME AS name, c.EXTRA AS extra, t.ENGINE AS engine
|
||||
FROM information_schema.COLUMNS c JOIN information_schema.TABLES t
|
||||
ON t.TABLE_SCHEMA = c.TABLE_SCHEMA AND t.TABLE_NAME = c.TABLE_NAME
|
||||
WHERE c.TABLE_SCHEMA = ? AND c.TABLE_NAME = ?`, [schema, table]);
|
||||
if (!rows.length || rows.some(row => String(row.engine).toLowerCase() !== 'innodb')) {
|
||||
throw new Error(`表 ${schema}.${table} 必须为 InnoDB，才能保证更新和日志原子提交`);
|
||||
}
|
||||
cache.set(key, {
|
||||
names: new Set(rows.map(row => row.name)),
|
||||
automatic: rows.filter(row => /on update/i.test(row.extra || '')).map(row => row.name)
|
||||
});
|
||||
}
|
||||
return cache.get(key);
|
||||
}
|
||||
|
||||
function assertFields(metadata, primaryKey, before, after) {
|
||||
if (own(after, primaryKey)) throw new Error('迁移不能修改主键');
|
||||
for (const name of [primaryKey, ...Object.keys(before), ...Object.keys(after)]) {
|
||||
if (!metadata.names.has(name)) throw new Error(`变更列 ${name} 不存在`);
|
||||
}
|
||||
for (const name of Object.keys(after)) {
|
||||
if (!own(before, name)) throw new Error(`变更列 ${name} 缺少原值快照`);
|
||||
}
|
||||
}
|
||||
|
||||
function assignments(values, metadata) {
|
||||
const entries = Object.entries(values);
|
||||
return {
|
||||
sql: [
|
||||
...entries.map(([name]) => `${quote(name)} = ?`),
|
||||
...metadata.automatic.filter(name => !own(values, name)).map(name => `${quote(name)} = ${quote(name)}`)
|
||||
].join(', '),
|
||||
params: entries.map(([, value]) => parameter(value))
|
||||
};
|
||||
}
|
||||
|
||||
export async function ensureJournal(connection, schema) {
|
||||
await connection.execute(`CREATE TABLE IF NOT EXISTS ${qualified(schema, RUNS)} (
|
||||
run_id VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL PRIMARY KEY,
|
||||
settings LONGTEXT NOT NULL,
|
||||
status VARCHAR(32) NOT NULL DEFAULT 'active',
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin`);
|
||||
await connection.execute(`CREATE TABLE IF NOT EXISTS ${qualified(schema, ITEMS)} (
|
||||
run_id VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
|
||||
item_key CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
|
||||
data LONGTEXT NOT NULL,
|
||||
status VARCHAR(32) NOT NULL,
|
||||
error TEXT NULL,
|
||||
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (run_id, item_key)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin`);
|
||||
await connection.execute(`CREATE TABLE IF NOT EXISTS ${qualified(schema, CHANGES)} (
|
||||
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
|
||||
run_id VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
|
||||
schema_name VARCHAR(64) NOT NULL,
|
||||
table_name VARCHAR(64) NOT NULL,
|
||||
pk_name VARCHAR(64) NOT NULL,
|
||||
pk_value TEXT NOT NULL,
|
||||
before_json LONGTEXT NOT NULL,
|
||||
after_json LONGTEXT NOT NULL,
|
||||
rolled_back TINYINT NOT NULL DEFAULT 0,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
KEY ix_run_id_id (run_id, id)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin`);
|
||||
for (const table of [RUNS, ITEMS, CHANGES]) await tableMetadata(connection, schema, table);
|
||||
}
|
||||
|
||||
/** The caller owns BEGIN/COMMIT/ROLLBACK, including the change-journal INSERT. */
|
||||
export async function applyChange(connection, journalSchema, runId, change) {
|
||||
const { schema, table, primaryKey, pkValue, before, after } = change;
|
||||
const metadata = await tableMetadata(connection, schema, table);
|
||||
assertFields(metadata, primaryKey, before, after);
|
||||
const changed = Object.fromEntries(Object.entries(after).filter(([name, value]) => !equal(before[name], value)));
|
||||
if (!Object.keys(changed).length) return { changed: false };
|
||||
const update = assignments(changed, metadata);
|
||||
const guard = condition(before);
|
||||
const [result] = await connection.execute(
|
||||
`UPDATE ${qualified(schema, table)} SET ${update.sql}
|
||||
WHERE ${quote(primaryKey)} = ?${guard.sql ? ` AND ${guard.sql}` : ''}`,
|
||||
[...update.params, parameter(pkValue), ...guard.params]);
|
||||
if (result.affectedRows !== 1) {
|
||||
const error = new Error(`数据已变化或不存在：${schema}.${table}，本次更新已停止`);
|
||||
error.code = 'MIGRATION_CAS_CONFLICT';
|
||||
throw error;
|
||||
}
|
||||
if (result.warningStatus > 0) {
|
||||
throw new Error('数据库更新产生警告，可能发生字段截断或类型转换；事务必须回滚');
|
||||
}
|
||||
// Native JSON columns normalize whitespace/key order, and MySQL may convert
|
||||
// numeric request values. Rollback must compare against the bytes/type that
|
||||
// the database actually persisted, not the JavaScript UPDATE parameters.
|
||||
const [storedRows] = await connection.execute(
|
||||
`SELECT ${Object.keys(changed).map(quote).join(', ')} FROM ${qualified(schema, table)}
|
||||
WHERE ${quote(primaryKey)} = ? FOR UPDATE`, [parameter(pkValue)]);
|
||||
if (storedRows.length !== 1 || Object.keys(changed).some(name => !own(storedRows[0], name))) {
|
||||
throw new Error('无法读取更新后的记录，事务必须回滚');
|
||||
}
|
||||
const persisted = Object.fromEntries(Object.keys(changed).map(name => [name, storedRows[0][name]]));
|
||||
const [journal] = await connection.execute(
|
||||
`INSERT INTO ${qualified(journalSchema, CHANGES)}
|
||||
(run_id, schema_name, table_name, pk_name, pk_value, before_json, after_json)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
[runId, schema, table, primaryKey, JSON.stringify(pack(pkValue)), snapshot(before), snapshot(persisted)]);
|
||||
return { changed: true, journalId: String(journal.insertId) };
|
||||
}
|
||||
|
||||
function primaryKeyIdentity(value) {
|
||||
if (Buffer.isBuffer(value)) return `buffer:${value.toString('base64')}`;
|
||||
if (!['string', 'number', 'bigint'].includes(typeof value)
|
||||
|| (typeof value === 'number' && !Number.isSafeInteger(value))) {
|
||||
throw new Error('批量变更主键必须是字符串、安全整数、BigInt 或 Buffer');
|
||||
}
|
||||
// mysql2 may return integer primary keys as either strings or numbers.
|
||||
return `scalar:${String(value)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply one homogeneous batch with one CASE UPDATE, one persisted-value read,
|
||||
* and one journal INSERT. The caller must roll back the entire transaction on
|
||||
* any error: the UPDATE can have matched only part of a concurrently edited
|
||||
* batch before affectedRows reports the conflict.
|
||||
*/
|
||||
export async function applyChanges(connection, journalSchema, runId, changes) {
|
||||
if (!Array.isArray(changes)) throw new Error('批量变更必须为数组');
|
||||
if (!changes.length) return { changed: 0, skipped: 0 };
|
||||
const { schema, table, primaryKey } = changes[0];
|
||||
const afterFields = Object.keys(changes[0].after).sort();
|
||||
const identities = new Set();
|
||||
for (const change of changes) {
|
||||
if (change.schema !== schema || change.table !== table || change.primaryKey !== primaryKey) {
|
||||
throw new Error('批量变更必须使用相同的数据库、表和主键列');
|
||||
}
|
||||
if (JSON.stringify(Object.keys(change.after).sort()) !== JSON.stringify(afterFields)) {
|
||||
throw new Error('批量变更的目标字段集合必须一致');
|
||||
}
|
||||
const identity = primaryKeyIdentity(change.pkValue);
|
||||
if (identities.has(identity)) throw new Error('批量变更包含重复主键');
|
||||
identities.add(identity);
|
||||
}
|
||||
const metadata = await tableMetadata(connection, schema, table);
|
||||
const active = [];
|
||||
for (const change of changes) {
|
||||
assertFields(metadata, primaryKey, change.before, change.after);
|
||||
const changed = Object.fromEntries(Object.entries(change.after).filter(([name, value]) => !equal(change.before[name], value)));
|
||||
if (Object.keys(changed).length) active.push({ ...change, changed });
|
||||
}
|
||||
const skipped = changes.length - active.length;
|
||||
if (!active.length) return { changed: 0, skipped };
|
||||
|
||||
const fields = afterFields.filter(name => active.some(change => own(change.changed, name)));
|
||||
const updateParams = [];
|
||||
const set = fields.map(name => {
|
||||
const branches = active.filter(change => own(change.changed, name)).map(change => {
|
||||
updateParams.push(parameter(change.pkValue), parameter(change.changed[name]));
|
||||
return 'WHEN ? THEN ?';
|
||||
});
|
||||
return `${quote(name)} = CASE ${quote(primaryKey)} ${branches.join(' ')} ELSE ${quote(name)} END`;
|
||||
});
|
||||
set.push(...metadata.automatic.filter(name => !fields.includes(name)).map(name => `${quote(name)} = ${quote(name)}`));
|
||||
const where = active.map(change => {
|
||||
const guard = condition(change.before);
|
||||
updateParams.push(parameter(change.pkValue), ...guard.params);
|
||||
return `(${quote(primaryKey)} = ?${guard.sql ? ` AND ${guard.sql}` : ''})`;
|
||||
});
|
||||
const [updated] = await connection.execute(
|
||||
`UPDATE ${qualified(schema, table)} SET ${set.join(', ')} WHERE ${where.join(' OR ')}`, updateParams);
|
||||
if (updated.affectedRows !== active.length) {
|
||||
const error = new Error(`批量更新发现数据已变化或不存在：${schema}.${table}，必须回滚整批事务`);
|
||||
error.code = 'MIGRATION_CAS_CONFLICT';
|
||||
throw error;
|
||||
}
|
||||
if (updated.warningStatus > 0) throw new Error('数据库批量更新产生警告，可能发生字段截断或类型转换；事务必须回滚');
|
||||
|
||||
const [storedRows] = await connection.execute(
|
||||
`SELECT ${[primaryKey, ...fields].map(quote).join(', ')} FROM ${qualified(schema, table)}
|
||||
WHERE ${quote(primaryKey)} IN (${active.map(() => '?').join(', ')}) FOR UPDATE`,
|
||||
active.map(change => parameter(change.pkValue)));
|
||||
if (storedRows.length !== active.length) throw new Error('无法读取批量更新后的全部记录，事务必须回滚');
|
||||
const storedByKey = new Map();
|
||||
for (const row of storedRows) {
|
||||
if (![primaryKey, ...fields].every(name => own(row, name))) throw new Error('批量更新读回记录缺少字段，事务必须回滚');
|
||||
const identity = primaryKeyIdentity(row[primaryKey]);
|
||||
if (storedByKey.has(identity)) throw new Error('批量更新读回记录包含重复主键，事务必须回滚');
|
||||
storedByKey.set(identity, row);
|
||||
}
|
||||
const journalParams = [];
|
||||
for (const change of active) {
|
||||
const row = storedByKey.get(primaryKeyIdentity(change.pkValue));
|
||||
if (!row) throw new Error('批量更新读回主键与请求不一致，事务必须回滚');
|
||||
const persisted = Object.fromEntries(Object.keys(change.changed).map(name => [name, row[name]]));
|
||||
journalParams.push(runId, schema, table, primaryKey, JSON.stringify(pack(change.pkValue)), snapshot(change.before), snapshot(persisted));
|
||||
}
|
||||
const [journal] = await connection.execute(
|
||||
`INSERT INTO ${qualified(journalSchema, CHANGES)}
|
||||
(run_id, schema_name, table_name, pk_name, pk_value, before_json, after_json)
|
||||
VALUES ${active.map(() => '(?, ?, ?, ?, ?, ?, ?)').join(', ')}`, journalParams);
|
||||
if (journal.affectedRows !== active.length || journal.warningStatus > 0) {
|
||||
throw new Error('批量回滚日志未完整保存，事务必须回滚');
|
||||
}
|
||||
return { changed: active.length, skipped };
|
||||
}
|
||||
|
||||
export async function rollbackChanges(connection, journalSchema, runId) {
|
||||
const result = { restored: 0, alreadyRestored: 0, conflicts: [] };
|
||||
let cursor = null;
|
||||
const journal = qualified(journalSchema, CHANGES);
|
||||
while (true) {
|
||||
const [batch] = await connection.execute(
|
||||
`SELECT CAST(id AS CHAR) AS id FROM ${journal}
|
||||
WHERE run_id = ? AND rolled_back = 0${cursor === null ? '' : ' AND id < ?'} ORDER BY id DESC LIMIT 100`,
|
||||
cursor === null ? [runId] : [runId, cursor]);
|
||||
if (!batch.length) break;
|
||||
for (const entry of batch) {
|
||||
cursor = String(entry.id);
|
||||
await connection.beginTransaction();
|
||||
try {
|
||||
const [[row]] = await connection.execute(`SELECT * FROM ${journal} WHERE id = ? AND run_id = ? FOR UPDATE`, [cursor, runId]);
|
||||
if (!row || Number(row.rolled_back) === 1) { await connection.commit(); continue; }
|
||||
const before = restoreSnapshot(row.before_json);
|
||||
const after = restoreSnapshot(row.after_json);
|
||||
const pkValue = unpack(JSON.parse(row.pk_value));
|
||||
const metadata = await tableMetadata(connection, row.schema_name, row.table_name);
|
||||
assertFields(metadata, row.pk_name, before, after);
|
||||
const target = qualified(row.schema_name, row.table_name);
|
||||
const expectedAfter = condition({ ...before, ...after });
|
||||
const [matchesAfter] = await connection.execute(
|
||||
`SELECT 1 AS matched FROM ${target} WHERE ${quote(row.pk_name)} = ? AND ${expectedAfter.sql} FOR UPDATE`,
|
||||
[parameter(pkValue), ...expectedAfter.params]);
|
||||
let alreadyRestored = false;
|
||||
if (!matchesAfter.length) {
|
||||
const expectedBefore = condition(before);
|
||||
const [matchesBefore] = await connection.execute(
|
||||
`SELECT 1 AS matched FROM ${target} WHERE ${quote(row.pk_name)} = ? AND ${expectedBefore.sql} FOR UPDATE`,
|
||||
[parameter(pkValue), ...expectedBefore.params]);
|
||||
if (!matchesBefore.length) {
|
||||
await connection.rollback();
|
||||
result.conflicts.push({ id: cursor, schema: row.schema_name, table: row.table_name, pkValue: typeof pkValue === 'bigint' ? pkValue.toString() : pkValue });
|
||||
continue;
|
||||
}
|
||||
alreadyRestored = true;
|
||||
} else {
|
||||
const values = Object.fromEntries(Object.keys(after).map(name => [name, before[name]]));
|
||||
const update = assignments(values, metadata);
|
||||
const [updated] = await connection.execute(
|
||||
`UPDATE ${target} SET ${update.sql} WHERE ${quote(row.pk_name)} = ? AND ${expectedAfter.sql}`,
|
||||
[...update.params, parameter(pkValue), ...expectedAfter.params]);
|
||||
if (updated.affectedRows !== 1) throw new Error('已锁定的回滚记录发生变化，事务已终止');
|
||||
if (updated.warningStatus > 0) throw new Error('数据库回滚产生警告，事务已终止');
|
||||
}
|
||||
await connection.execute(`UPDATE ${journal} SET rolled_back = 1 WHERE id = ? AND run_id = ?`, [cursor, runId]);
|
||||
await connection.commit();
|
||||
if (alreadyRestored) result.alreadyRestored++;
|
||||
else result.restored++;
|
||||
} catch (error) {
|
||||
await connection.rollback();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
/**
|
||||
* Keep database waits and long scans visible without logging every record.
|
||||
* update() also checks elapsed time when a synchronous loop delays timers.
|
||||
*/
|
||||
export function createProgress({ log = console.log, intervalMs = 5000, now = Date.now } = {}) {
|
||||
if (!Number.isSafeInteger(intervalMs) || intervalMs < 1) throw new Error('进度输出间隔必须为正整数毫秒');
|
||||
let message = '';
|
||||
let startedAt;
|
||||
let lastOutput;
|
||||
let closed = false;
|
||||
|
||||
function output(force = false) {
|
||||
if (closed || !message) return;
|
||||
const current = now();
|
||||
if (!force && lastOutput !== undefined && current - lastOutput < intervalMs) return;
|
||||
lastOutput = current;
|
||||
const seconds = Math.max(0, Math.floor((current - startedAt) / 1000));
|
||||
log(message + '（已用时 ' + seconds + ' 秒）');
|
||||
}
|
||||
|
||||
const timer = setInterval(() => output(), intervalMs);
|
||||
timer.unref?.();
|
||||
|
||||
return {
|
||||
stage(value) {
|
||||
if (closed) return;
|
||||
message = String(value ?? '').trim();
|
||||
startedAt = now();
|
||||
output(true);
|
||||
},
|
||||
update(value) {
|
||||
if (closed) return;
|
||||
message = String(value ?? '').trim();
|
||||
if (startedAt === undefined) startedAt = now();
|
||||
output();
|
||||
},
|
||||
close() {
|
||||
if (closed) return;
|
||||
closed = true;
|
||||
clearInterval(timer);
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { resolve } from 'node:path';
|
||||
import { isIP } from 'node:net';
|
||||
import { parseAllDocuments } from 'yaml';
|
||||
|
||||
const own = (object, key) => Object.prototype.hasOwnProperty.call(object, key);
|
||||
const record = value => value !== null && typeof value === 'object' && !Array.isArray(value);
|
||||
const canonical = key => key.replace(/[-_]/g, '').toLowerCase();
|
||||
const environmentName = path => path.replace(/([a-z0-9])([A-Z])/g, '$1_$2').replace(/[.-]/g, '_').toUpperCase();
|
||||
|
||||
function merge(target, source) {
|
||||
const result = Object.assign(Object.create(null), target);
|
||||
for (const [key, value] of Object.entries(source)) {
|
||||
result[key] = record(value) && record(result[key]) ? merge(result[key], value) : value;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function property(root, path) {
|
||||
const parts = path.split('.');
|
||||
let current = root;
|
||||
for (let index = 0; index < parts.length; index++) {
|
||||
if (!record(current)) return undefined;
|
||||
const remainder = parts.slice(index).join('.');
|
||||
if (own(current, remainder)) return current[remainder];
|
||||
const part = parts[index];
|
||||
let key = part;
|
||||
if (!own(current, key)) {
|
||||
const matches = Object.keys(current).filter(candidate => canonical(candidate) === canonical(part));
|
||||
if (matches.length > 1) throw new Error('配置存在多个等价属性名，无法自动选择；请使用明确的迁移配置');
|
||||
if (!matches.length) return undefined;
|
||||
key = matches[0];
|
||||
}
|
||||
current = current[key];
|
||||
}
|
||||
return current;
|
||||
}
|
||||
|
||||
function resolver(configuration, env) {
|
||||
const raw = name => {
|
||||
if (own(env, name) && env[name] !== undefined) return env[name];
|
||||
const key = environmentName(name);
|
||||
if (own(env, key) && env[key] !== undefined) return env[key];
|
||||
return property(configuration, name);
|
||||
};
|
||||
const scalar = value => {
|
||||
if (!['string', 'number', 'bigint', 'boolean'].includes(typeof value)) {
|
||||
throw new Error('自动配置所需字段必须为单个值，请检查环境配置');
|
||||
}
|
||||
return String(value);
|
||||
};
|
||||
function expand(value, trail = []) {
|
||||
const text = scalar(value);
|
||||
let output = '', cursor = 0;
|
||||
while (cursor < text.length) {
|
||||
const start = text.indexOf('${', cursor);
|
||||
if (start < 0) return output + text.slice(cursor);
|
||||
output += text.slice(cursor, start);
|
||||
let depth = 1, end = start + 2;
|
||||
for (; end < text.length && depth; end++) {
|
||||
if (text.startsWith('${', end)) { depth++; end++; }
|
||||
else if (text[end] === '}') depth--;
|
||||
}
|
||||
if (depth) throw new Error('自动配置所需字段包含未闭合的占位符，请检查环境配置');
|
||||
const expression = text.slice(start + 2, end - 1);
|
||||
let separator = -1, nested = 0;
|
||||
for (let index = 0; index < expression.length; index++) {
|
||||
if (expression.startsWith('${', index)) { nested++; index++; }
|
||||
else if (expression[index] === '}') nested--;
|
||||
else if (expression[index] === ':' && nested === 0) { separator = index; break; }
|
||||
}
|
||||
const name = (separator < 0 ? expression : expression.slice(0, separator)).trim();
|
||||
if (!name || name.includes('${')) throw new Error('自动配置不支持动态占位符名称，请使用明确的迁移配置');
|
||||
if (trail.includes(name) || trail.length >= 32) throw new Error('自动配置所需字段存在循环占位符引用，请检查环境配置');
|
||||
let replacement = raw(name);
|
||||
if (replacement === undefined || replacement === null) {
|
||||
if (separator < 0) throw new Error('自动配置所需占位符未提供，请设置对应环境变量或使用明确的迁移配置');
|
||||
replacement = expression.slice(separator + 1);
|
||||
}
|
||||
output += expand(replacement, [...trail, name]);
|
||||
cursor = end;
|
||||
}
|
||||
return output;
|
||||
}
|
||||
return {
|
||||
raw,
|
||||
field(name, fallback) {
|
||||
const value = raw(name);
|
||||
if (value === undefined || value === null) {
|
||||
if (fallback !== undefined) return expand(fallback);
|
||||
throw new Error('项目配置缺少迁移所需字段，请检查主数据源配置或使用明确的迁移配置');
|
||||
}
|
||||
return expand(value, [name]);
|
||||
},
|
||||
expand,
|
||||
};
|
||||
}
|
||||
|
||||
async function yamlFile(path) {
|
||||
let text;
|
||||
try { text = await readFile(path, 'utf8'); }
|
||||
catch { throw new Error('无法读取基础或所选环境 YAML，请检查项目路径和 --profile'); }
|
||||
try {
|
||||
const documents = parseAllDocuments(text, { uniqueKeys: true, intAsBigInt: true });
|
||||
let result = Object.create(null);
|
||||
for (const document of documents) {
|
||||
if (document.errors.length || document.warnings.length) throw new Error();
|
||||
const value = document.toJS({ maxAliasCount: 100 });
|
||||
if (value === null) continue;
|
||||
if (!record(value)) throw new Error();
|
||||
if (property(value, 'spring.config.activate.on-profile') !== undefined
|
||||
|| typeof property(value, 'spring.profiles') === 'string'
|
||||
|| Array.isArray(property(value, 'spring.profiles'))) {
|
||||
throw Object.assign(new Error('自动配置不支持带 profile 条件的 YAML 文档，请使用明确的迁移配置'), { unsupported: true });
|
||||
}
|
||||
result = merge(result, value);
|
||||
}
|
||||
return result;
|
||||
} catch (error) {
|
||||
if (error.unsupported) throw error;
|
||||
throw new Error('YAML 无法安全解析，请检查缩进、重复键和文档结构');
|
||||
}
|
||||
}
|
||||
|
||||
function singleProfile(value) {
|
||||
if (typeof value !== 'string' || !/^[a-zA-Z0-9][a-zA-Z0-9._-]*$/.test(value.trim())) {
|
||||
throw new Error('自动配置仅支持单个 profile，请使用 --profile dev、local 或 prod 明确选择');
|
||||
}
|
||||
return value.trim();
|
||||
}
|
||||
|
||||
function assertSelectionSupported(configuration, profile, env) {
|
||||
const resolved = resolver(configuration, env);
|
||||
const imports = resolved.raw('spring.config.import');
|
||||
const includes = resolved.raw('spring.profiles.include');
|
||||
if (imports !== undefined && imports !== null && imports !== '' && (!Array.isArray(imports) || imports.length)) {
|
||||
throw new Error('自动配置不支持 spring.config.import，请使用明确的迁移配置');
|
||||
}
|
||||
if (includes !== undefined && includes !== null && includes !== '' && (!Array.isArray(includes) || includes.length)) {
|
||||
throw new Error('自动配置不支持附加 profiles.include，请使用明确的迁移配置');
|
||||
}
|
||||
const groups = property(configuration, 'spring.profiles.group');
|
||||
if ((record(groups) && own(groups, profile))
|
||||
|| resolved.raw('spring.profiles.group.' + profile) !== undefined) {
|
||||
throw new Error('所选 profile 是配置组，无法唯一确定数据源；请使用具体环境或明确的迁移配置');
|
||||
}
|
||||
}
|
||||
|
||||
function port(value, fallbackMessage) {
|
||||
if (!/^\d+$/.test(value) || Number(value) < 1 || Number(value) > 65535) throw new Error(fallbackMessage);
|
||||
return Number(value);
|
||||
}
|
||||
|
||||
function mysqlAddress(value) {
|
||||
if (!value.startsWith('jdbc:mysql://')) throw new Error('自动配置仅支持单主机 jdbc:mysql:// URL，请使用明确的迁移配置');
|
||||
try {
|
||||
const url = new URL(value.slice(5));
|
||||
const host = url.hostname.replace(/^\[|\]$/g, '');
|
||||
if (!host || (!isIP(host) && !/^[a-zA-Z0-9.-]+$/.test(host)) || url.username || url.password || url.hash) throw new Error();
|
||||
const parts = url.pathname.split('/');
|
||||
if (parts.length !== 2 || !parts[1]) throw new Error();
|
||||
const database = decodeURIComponent(parts[1]);
|
||||
if (!database || /[/\\\u0000]/.test(database)) throw new Error();
|
||||
return { host, port: url.port ? port(url.port, '') : 3306, database };
|
||||
} catch {
|
||||
throw new Error('MySQL URL 必须指定单个主机、有效端口和数据库，不能含账号或多主机地址');
|
||||
}
|
||||
}
|
||||
|
||||
function route(value, allowEmpty) {
|
||||
if (value === '' && allowEmpty) return '';
|
||||
if (!value.startsWith('/') || /[?#\\]/.test(value)
|
||||
|| value.split('/').some(part => part === '.' || part === '..')) {
|
||||
throw new Error('HTTP context-path 或 API 前缀无效，请使用明确的迁移下载地址');
|
||||
}
|
||||
return value.replace(/\/+$/, '');
|
||||
}
|
||||
|
||||
/**
|
||||
* Read only the selected project's YAML. This deliberately supports a narrow
|
||||
* Spring configuration subset and never logs credentials or parser source text.
|
||||
*/
|
||||
export async function loadProjectConfig({ projectRoot, profile, env = process.env }) {
|
||||
if (!projectRoot || typeof projectRoot !== 'string') throw new Error('请指定后端项目根目录');
|
||||
const resourceRoot = resolve(projectRoot, 'yudao-server/src/main/resources');
|
||||
const baseFile = resolve(resourceRoot, 'application.yaml');
|
||||
const base = await yamlFile(baseFile);
|
||||
const baseResolver = resolver(base, env);
|
||||
let selected;
|
||||
if (profile !== undefined) selected = profile;
|
||||
else if (own(env, 'SPRING_PROFILES_ACTIVE') && env.SPRING_PROFILES_ACTIVE !== undefined) selected = env.SPRING_PROFILES_ACTIVE;
|
||||
else selected = baseResolver.raw('spring.profiles.active');
|
||||
if (typeof selected === 'string') selected = baseResolver.expand(selected);
|
||||
const selectedProfile = singleProfile(selected);
|
||||
assertSelectionSupported(base, selectedProfile, env);
|
||||
const selectedFile = resolve(resourceRoot, 'application-' + selectedProfile + '.yaml');
|
||||
const selectedConfiguration = await yamlFile(selectedFile);
|
||||
if (property(selectedConfiguration, 'spring.profiles.active') !== undefined) {
|
||||
throw new Error('所选环境文件再次声明 active profile，自动选择存在歧义；请使用明确的迁移配置');
|
||||
}
|
||||
const configuration = merge(base, selectedConfiguration);
|
||||
assertSelectionSupported(configuration, selectedProfile, env);
|
||||
const selectedResolver = resolver(configuration, env);
|
||||
const primary = selectedResolver.field('spring.datasource.dynamic.primary', 'master');
|
||||
if (!/^[a-zA-Z0-9_-]+$/.test(primary)) throw new Error('主数据源名称无效，请检查 dynamic.primary');
|
||||
const datasource = 'spring.datasource.dynamic.datasource.' + primary + '.';
|
||||
const address = mysqlAddress(selectedResolver.field(datasource + 'url'));
|
||||
const user = selectedResolver.field(datasource + 'username');
|
||||
const password = selectedResolver.field(datasource + 'password');
|
||||
if (!user.trim() || !password.trim()) throw new Error('主数据库用户名或密码缺失，请配置凭据后重试');
|
||||
if (/^\s*ENC\s*\(/i.test(password)) throw new Error('数据库密码仍为 ENC 加密内容，请通过环境变量提供已解密密码');
|
||||
const serverPort = port(selectedResolver.field('server.port', '8080'), 'HTTP 端口无效，无法推导下载地址');
|
||||
const context = route(selectedResolver.field('server.servlet.context-path', ''), true);
|
||||
const prefix = route(selectedResolver.field('yudao.web.admin-api.prefix', '/admin-api'), false);
|
||||
const ssl = selectedResolver.field('server.ssl.enabled', 'false');
|
||||
if (!['true', 'false'].includes(ssl.toLowerCase())) throw new Error('HTTP SSL 配置无效，无法推导下载地址');
|
||||
return {
|
||||
profile: selectedProfile,
|
||||
mysql: { ...address, user, password },
|
||||
downloadBaseUrl: (ssl.toLowerCase() === 'true' ? 'https' : 'http') + '://127.0.0.1:' + serverPort + context + prefix + '/infra/file',
|
||||
sourceFiles: [baseFile, selectedFile],
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
const DEFAULT_EXCLUDED_TABLES = new Set([
|
||||
'infra_file', 'infra_file_content', 'infra_file_config',
|
||||
'infra_api_access_log', 'infra_api_error_log', 'infra_job_log',
|
||||
'system_operate_log', 'system_login_log',
|
||||
]);
|
||||
const MIGRATION_TABLE_PREFIX = /^(?:_+)?(?:migration(?:_|$)|migrate(?:_|$)|(?:infra_file_oss|infra_file|file|oss)_migration(?:_|$))/i;
|
||||
const URL_RUN = /[a-z][a-z0-9+.-]*:\/\/[^\s"'`<>]+/gi;
|
||||
const NEXT_URL = /,(?=[a-z][a-z0-9+.-]*:\/\/)/gi;
|
||||
const TRAILING_DELIMITERS = /[,\])}]+$/;
|
||||
|
||||
function lookup(value, replacements) {
|
||||
const replacement = replacements.get(value);
|
||||
return typeof replacement === 'string' && replacement !== value
|
||||
? { value: replacement, changed: true, replacements: 1 }
|
||||
: { value, changed: false, replacements: 0 };
|
||||
}
|
||||
|
||||
function rewriteUrlPart(part, replacements) {
|
||||
const exact = lookup(part, replacements);
|
||||
if (exact.changed || replacements.has(part)) return exact;
|
||||
// Closing prose/Markdown punctuation is separate from a URL. A dictionary
|
||||
// entry that includes that punctuation always takes precedence.
|
||||
const delimiters = part.match(TRAILING_DELIMITERS)?.[0] ?? '';
|
||||
for (let count = 1; count <= delimiters.length; count += 1) {
|
||||
const prefix = part.slice(0, -count);
|
||||
const result = lookup(prefix, replacements);
|
||||
if (result.changed) return { ...result, value: result.value + part.slice(-count) };
|
||||
}
|
||||
return exact;
|
||||
}
|
||||
|
||||
function rewriteUrlRun(run, replacements) {
|
||||
const exact = rewriteUrlPart(run, replacements);
|
||||
if (exact.changed || replacements.has(run)) return exact;
|
||||
const parts = run.split(NEXT_URL);
|
||||
if (parts.length === 1) return exact;
|
||||
let count = 0;
|
||||
const rewritten = [];
|
||||
for (let index = 0; index < parts.length; index += 1) {
|
||||
const result = rewriteUrlPart(parts[index], replacements);
|
||||
rewritten.push(result.value);
|
||||
count += result.replacements;
|
||||
// An unknown query/fragment can contain a literal URL. Do not treat its
|
||||
// comma-delimited contents as independent business references.
|
||||
if (!result.changed && !replacements.has(parts[index]) && /[?#]/.test(parts[index])) {
|
||||
rewritten.push(...parts.slice(index + 1));
|
||||
break;
|
||||
}
|
||||
}
|
||||
return { value: rewritten.join(','), changed: count > 0, replacements: count };
|
||||
}
|
||||
|
||||
function rewriteText(value, replacements) {
|
||||
const exact = lookup(value, replacements);
|
||||
if (exact.changed || replacements.has(value)) return exact;
|
||||
|
||||
// Validate JSON first, then edit only string-value tokens. This preserves
|
||||
// whitespace, key spelling and untouched escapes in serialized JSON.
|
||||
if (/^[\s]*[\[{\"]/.test(value)) {
|
||||
let isJson = false;
|
||||
try {
|
||||
const parsed = JSON.parse(value);
|
||||
isJson = parsed !== null && (typeof parsed === 'object' || typeof parsed === 'string');
|
||||
} catch {
|
||||
// A regular text or HTML field may also begin with a JSON-like character.
|
||||
}
|
||||
if (isJson) {
|
||||
let count = 0;
|
||||
const rewritten = value.replace(/"(?:\\[\s\S]|[^"\\])*"/g, (token, offset) => {
|
||||
if (/^\s*:/.test(value.slice(offset + token.length))) return token;
|
||||
const result = rewriteText(JSON.parse(token), replacements);
|
||||
count += result.replacements;
|
||||
return result.changed ? JSON.stringify(result.value) : token;
|
||||
});
|
||||
return { value: rewritten, changed: count > 0, replacements: count };
|
||||
}
|
||||
}
|
||||
|
||||
let count = 0;
|
||||
const rewritten = value.replace(URL_RUN, (run) => {
|
||||
const result = rewriteUrlRun(run, replacements);
|
||||
count += result.replacements;
|
||||
return result.value;
|
||||
});
|
||||
return { value: rewritten, changed: count > 0, replacements: count };
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace complete absolute URL references, without URL decoding, prefix
|
||||
* substitution, or query/fragment normalization. JSON values returned as
|
||||
* objects by mysql2 are supported; object keys are never rewritten.
|
||||
* The input is not mutated and replacement values are not processed again.
|
||||
*/
|
||||
export function rewriteReferences(value, replacements) {
|
||||
if (!(replacements instanceof Map)) throw new TypeError('replacements must be a Map');
|
||||
const active = new WeakSet();
|
||||
function visit(input) {
|
||||
if (typeof input === 'string') return rewriteText(input, replacements);
|
||||
if (input === null || typeof input !== 'object' || input instanceof Date || ArrayBuffer.isView(input)) {
|
||||
return { value: input, changed: false, replacements: 0 };
|
||||
}
|
||||
if (active.has(input)) throw new TypeError('Reference values must not contain cycles');
|
||||
active.add(input);
|
||||
const output = Array.isArray(input) ? [] : {};
|
||||
let count = 0;
|
||||
for (const [key, item] of Object.entries(input)) {
|
||||
const result = visit(item);
|
||||
// defineProperty also handles JSON keys such as "__proto__" safely.
|
||||
Object.defineProperty(output, key, { value: result.value, enumerable: true, writable: true, configurable: true });
|
||||
count += result.replacements;
|
||||
}
|
||||
active.delete(input);
|
||||
return { value: count > 0 ? output : input, changed: count > 0, replacements: count };
|
||||
}
|
||||
return visit(value);
|
||||
}
|
||||
|
||||
/** Detect remaining legacy URLs in text/JSON values, including nested JSON. */
|
||||
export function containsLegacyReference(value, sourceConfigIds) {
|
||||
const markers = [...sourceConfigIds].map((id) => `/infra/file/${String(id)}/get/`);
|
||||
if (!markers.length) return false;
|
||||
const visited = new WeakSet();
|
||||
function visit(input) {
|
||||
if (typeof input === 'string') {
|
||||
if (/^[\s]*[\[{\"]/.test(input)) {
|
||||
try {
|
||||
const parsed = JSON.parse(input);
|
||||
// Parse JSON before examining its text so keys do not count as
|
||||
// references, and \\u002f / nested JSON escapes are decoded properly.
|
||||
return visit(parsed);
|
||||
} catch {
|
||||
// Rich text or malformed JSON still receives the literal text check.
|
||||
}
|
||||
}
|
||||
const normalized = input.replaceAll('\\/', '/');
|
||||
return markers.some((marker) => normalized.includes(marker));
|
||||
}
|
||||
if (input === null || typeof input !== 'object' || input instanceof Date || ArrayBuffer.isView(input)) return false;
|
||||
if (visited.has(input)) return false;
|
||||
visited.add(input);
|
||||
return Object.values(input).some(visit);
|
||||
}
|
||||
return visit(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Discover writable text/JSON columns with one primary-key column. All schema
|
||||
* names are bound values, never SQL identifiers. excludedTables accepts table
|
||||
* names or schema.table names in an Array/Set (case-insensitive exclusion).
|
||||
*/
|
||||
export async function discoverReferenceColumns(connection, schemas, excludedTables = []) {
|
||||
const schemaNames = [...new Set(schemas)];
|
||||
if (schemaNames.some((name) => typeof name !== 'string' || !name.trim())) {
|
||||
throw new TypeError('schemas must contain non-empty schema names');
|
||||
}
|
||||
if (schemaNames.length === 0) return { columns: [], skipped: [] };
|
||||
const excluded = new Set([...excludedTables].map((name) => String(name).toLowerCase()));
|
||||
const placeholders = schemaNames.map(() => '?').join(', ');
|
||||
const [rows] = await connection.execute(`
|
||||
SELECT c.TABLE_SCHEMA AS schemaName, c.TABLE_NAME AS tableName,
|
||||
c.COLUMN_NAME AS columnName, c.DATA_TYPE AS dataType,
|
||||
c.EXTRA AS extra, c.GENERATION_EXPRESSION AS generationExpression,
|
||||
pk.primaryKey, pk.primaryKeyCount
|
||||
FROM information_schema.COLUMNS c
|
||||
JOIN information_schema.TABLES t
|
||||
ON t.TABLE_SCHEMA = c.TABLE_SCHEMA AND t.TABLE_NAME = c.TABLE_NAME
|
||||
LEFT JOIN (
|
||||
SELECT TABLE_SCHEMA, TABLE_NAME, MIN(COLUMN_NAME) AS primaryKey,
|
||||
COUNT(*) AS primaryKeyCount
|
||||
FROM information_schema.KEY_COLUMN_USAGE
|
||||
WHERE CONSTRAINT_NAME = 'PRIMARY' AND TABLE_SCHEMA IN (${placeholders})
|
||||
GROUP BY TABLE_SCHEMA, TABLE_NAME
|
||||
) pk ON pk.TABLE_SCHEMA = c.TABLE_SCHEMA AND pk.TABLE_NAME = c.TABLE_NAME
|
||||
WHERE c.TABLE_SCHEMA IN (${placeholders}) AND t.TABLE_TYPE = 'BASE TABLE'
|
||||
AND c.DATA_TYPE IN ('char', 'varchar', 'tinytext', 'text', 'mediumtext', 'longtext', 'json')
|
||||
ORDER BY c.TABLE_SCHEMA, c.TABLE_NAME, c.ORDINAL_POSITION
|
||||
`, [...schemaNames, ...schemaNames]);
|
||||
|
||||
const columns = [];
|
||||
const skipped = [];
|
||||
const skippedTables = new Set();
|
||||
for (const row of rows) {
|
||||
const schema = row.schemaName;
|
||||
const table = row.tableName;
|
||||
const lowerTable = table.toLowerCase();
|
||||
const fullName = `${schema}.${table}`;
|
||||
let reason;
|
||||
if (DEFAULT_EXCLUDED_TABLES.has(lowerTable) || MIGRATION_TABLE_PREFIX.test(table)
|
||||
|| excluded.has(lowerTable) || excluded.has(fullName.toLowerCase())) {
|
||||
reason = 'excluded table';
|
||||
} else if (Number(row.primaryKeyCount) !== 1) {
|
||||
reason = Number(row.primaryKeyCount) > 1 ? 'composite primary key' : 'no primary key';
|
||||
}
|
||||
if (reason) {
|
||||
if (!skippedTables.has(fullName)) {
|
||||
skipped.push({ schema, table, reason });
|
||||
skippedTables.add(fullName);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const extra = row.extra ?? '';
|
||||
if (row.columnName === row.primaryKey || /\bauto_increment\b/i.test(extra)
|
||||
|| row.generationExpression || /\b(?:VIRTUAL|STORED)\s+GENERATED\b/i.test(extra)) continue;
|
||||
columns.push({ schema, table, column: row.columnName, primaryKey: row.primaryKey, dataType: row.dataType.toLowerCase() });
|
||||
}
|
||||
return { columns, skipped };
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
import { id, qualified } from './core.mjs';
|
||||
|
||||
function sourceChanged() {
|
||||
return Object.assign(new Error('源图片在任务建立后发生变化或已删除，请检查该批文件'), {
|
||||
code: 'MIGRATION_SOURCE_CHANGED'
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate a bounded batch immediately after the caller begins an InnoDB
|
||||
* REPEATABLE READ transaction, BEFORE any consistent read or explicitly created
|
||||
* consistent snapshot. The plain metadata read must establish its snapshot
|
||||
* only after the primary-key range has been locked.
|
||||
*
|
||||
* The first locking read scans from the minimum expected ID through the upper
|
||||
* key range, including deleted rows and potential newer versions. This retains
|
||||
* range/gap protection until commit/rollback, including against undeleting or
|
||||
* changing another row into a newer matching version. Do not restrict this
|
||||
* range lock to only expected IDs, active rows, or matching paths.
|
||||
*
|
||||
* The caller owns transaction boundaries, rollback on failure and batch size.
|
||||
*/
|
||||
export async function verifySourceBatch(db, config, items) {
|
||||
if (!Array.isArray(items) || items.length === 0) throw new TypeError('源文件校验批次不能为空');
|
||||
if (!config?.mysql?.database) throw new TypeError('源文件校验需要数据库名称');
|
||||
const expected = new Map();
|
||||
for (const item of items) {
|
||||
const contentId = id(item?.contentId);
|
||||
const configId = id(item?.configId);
|
||||
if (expected.has(contentId)) throw new TypeError('源文件校验批次不能包含重复的内容编号');
|
||||
if (typeof item.path !== 'string' || !item.path.length) throw new TypeError('源文件路径必须为非空字符串');
|
||||
if (!Number.isSafeInteger(item.size) || item.size < 0 || typeof item.sha256 !== 'string' || !item.sha256.length) {
|
||||
throw new TypeError('源文件校验需要有效的预期大小和 SHA-256');
|
||||
}
|
||||
expected.set(contentId, { ...item, contentId, configId });
|
||||
}
|
||||
const entries = [...expected.values()];
|
||||
const minimumId = entries.reduce((minimum, item) => BigInt(item.contentId) < BigInt(minimum) ? item.contentId : minimum, entries[0].contentId);
|
||||
const sourceTable = qualified(config.mysql.database, 'infra_file_content');
|
||||
const cases = entries.map(() => 'WHEN config_id=? AND path=? THEN ?').join(' ');
|
||||
const predicates = entries.map(() => '(config_id=? AND path=? AND id>=?)').join(' OR ');
|
||||
const caseParams = entries.flatMap(item => [item.configId, item.path, item.contentId]);
|
||||
const predicateParams = entries.flatMap(item => [item.configId, item.path, item.contentId]);
|
||||
|
||||
// A primary-key-only locking scan avoids the large-row access path triggered
|
||||
// by locking reads that evaluate config/path predicates. Do not filter out
|
||||
// deleted rows: their undelete could otherwise create a new active version.
|
||||
await db.execute(
|
||||
`SELECT id FROM ${sourceTable} WHERE id>=? ORDER BY id FOR UPDATE`,
|
||||
[minimumId]
|
||||
);
|
||||
|
||||
// CASE is evaluated by MySQL using the path column's collation. The returned
|
||||
// expected ID is the only grouping key used in JavaScript. This consistent
|
||||
// read establishes a fresh snapshot while the whole relevant range is held.
|
||||
const [latestRows] = await db.execute(
|
||||
`SELECT id,CASE ${cases} END AS expected_content_id FROM ${sourceTable}
|
||||
WHERE deleted=0 AND id>=? AND (${predicates}) ORDER BY id DESC`,
|
||||
[...caseParams, minimumId, ...predicateParams]
|
||||
);
|
||||
const latest = new Map();
|
||||
for (const row of latestRows) {
|
||||
const expectedId = String(row.expected_content_id);
|
||||
if (!expected.has(expectedId)) throw sourceChanged();
|
||||
const actualId = id(row.id);
|
||||
if (!latest.has(expectedId) || BigInt(actualId) > BigInt(latest.get(expectedId))) latest.set(expectedId, actualId);
|
||||
}
|
||||
for (const contentId of expected.keys()) {
|
||||
if (latest.get(contentId) !== contentId) throw sourceChanged();
|
||||
}
|
||||
|
||||
// The range is already locked and each config/path version is validated.
|
||||
// Digest work now touches only the selected primary keys, not a sorted scan.
|
||||
const [contentRows] = await db.execute(
|
||||
`SELECT id,config_id,path,OCTET_LENGTH(content) AS size,LOWER(SHA2(content,256)) AS sha256
|
||||
FROM ${sourceTable} WHERE deleted=0 AND id IN (${entries.map(() => '?').join(',')}) FOR UPDATE`,
|
||||
entries.map(item => item.contentId)
|
||||
);
|
||||
const verified = new Set();
|
||||
for (const row of contentRows) {
|
||||
const contentId = id(row.id);
|
||||
const item = expected.get(contentId);
|
||||
if (!item || verified.has(contentId) || Number(row.size) !== item.size || row.sha256 !== item.sha256) throw sourceChanged();
|
||||
verified.add(contentId);
|
||||
}
|
||||
if (verified.size !== expected.size) throw sourceChanged();
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
import { id, qualified } from './core.mjs';
|
||||
|
||||
const compareIds = (left, right) => BigInt(left) < BigInt(right) ? -1 : BigInt(left) > BigInt(right) ? 1 : 0;
|
||||
|
||||
function boundary(value) {
|
||||
if ((typeof value === 'number' && !Number.isSafeInteger(value)) || !/^\d+$/.test(String(value))) {
|
||||
throw new Error('文件清单上界必须是非负整数；大整数请使用字符串');
|
||||
}
|
||||
return BigInt(value).toString();
|
||||
}
|
||||
|
||||
function pathText(value) {
|
||||
if (Buffer.isBuffer(value)) return value.toString('utf8');
|
||||
if (typeof value !== 'string') throw new Error('文件路径不是有效的文本值');
|
||||
return value;
|
||||
}
|
||||
|
||||
function fileMetadata(row) {
|
||||
return {
|
||||
id: id(row.id), config_id: id(row.config_id), path: pathText(row.path),
|
||||
url: row.url, type: row.type, size: row.size
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a metadata-only index in three SELECTs. MySQL performs all path
|
||||
* grouping/comparisons with the columns' collation; JavaScript only associates
|
||||
* records by their latest numeric ID. No file bytes or digests are selected.
|
||||
*/
|
||||
export async function loadSourceIndex(db, config, { maxId, progress } = {}) {
|
||||
if (!config?.mysql?.database || !Array.isArray(config.sourceConfigIds) || !config.sourceConfigIds.length) {
|
||||
throw new Error('文件清单需要数据库名称和至少一个源配置编号');
|
||||
}
|
||||
const sourceIds = [...new Set(config.sourceConfigIds.map(id))].sort(compareIds);
|
||||
const upperBound = maxId === undefined ? undefined : boundary(maxId);
|
||||
const contentTable = qualified(config.mysql.database, 'infra_file_content');
|
||||
const fileTable = qualified(config.mysql.database, 'infra_file');
|
||||
const markers = sourceIds.map(() => '?').join(',');
|
||||
// Keep the bound outside this grouping: a newer active row above the bound
|
||||
// must suppress the whole group, not make an older duplicate current again.
|
||||
const latest = `SELECT config_id,path,MAX(id) AS id FROM ${contentTable}
|
||||
WHERE deleted=0 AND config_id IN (${markers}) GROUP BY config_id,path`;
|
||||
const boundClause = upperBound === undefined ? '' : ' WHERE g.id<=?';
|
||||
const boundParams = upperBound === undefined ? [] : [upperBound];
|
||||
|
||||
progress?.stage?.('读取数据库文件分组（只读取元数据，不计算图片哈希）');
|
||||
const [latestRows] = await db.execute(
|
||||
`SELECT c.id AS content_id,c.config_id,c.path FROM ${contentTable} c
|
||||
JOIN (${latest}) g ON g.id=c.id${boundClause} ORDER BY c.id`,
|
||||
[...sourceIds, ...boundParams]
|
||||
);
|
||||
const byContentId = new Map();
|
||||
let observedMaximum = '0';
|
||||
for (const row of latestRows) {
|
||||
const contentId = id(row.content_id);
|
||||
if (upperBound !== undefined && compareIds(contentId, upperBound) > 0) continue;
|
||||
const group = {
|
||||
contentId, configId: id(row.config_id), path: pathText(row.path), files: [], spellings: []
|
||||
};
|
||||
byContentId.set(contentId, group);
|
||||
if (compareIds(contentId, observedMaximum) > 0) observedMaximum = contentId;
|
||||
}
|
||||
|
||||
progress?.stage?.('批量匹配文件记录并检查缺失内容');
|
||||
const [fileRows] = await db.execute(
|
||||
`SELECT f.id,f.config_id,f.path,f.url,f.type,f.size,g.id AS content_id
|
||||
FROM ${fileTable} f LEFT JOIN (${latest}) g
|
||||
ON g.config_id=f.config_id AND g.path=f.path
|
||||
WHERE f.deleted=0 AND f.config_id IN (${markers}) ORDER BY f.id`,
|
||||
[...sourceIds, ...sourceIds]
|
||||
);
|
||||
const missingFiles = [];
|
||||
for (const row of fileRows) {
|
||||
const file = fileMetadata(row);
|
||||
if (row.content_id === null || row.content_id === undefined) {
|
||||
missingFiles.push(file);
|
||||
continue;
|
||||
}
|
||||
// The join deliberately has no upper bound: a file whose latest content
|
||||
// lies beyond this batch is not a missing-content file.
|
||||
byContentId.get(id(row.content_id))?.files.push(file);
|
||||
}
|
||||
|
||||
progress?.stage?.('批量收集原路径拼写，保留历史图片地址兼容性');
|
||||
const [spellingRows] = await db.execute(
|
||||
`SELECT DISTINCT g.id AS content_id,BINARY c.path AS path
|
||||
FROM ${contentTable} c JOIN (${latest}) g
|
||||
ON g.config_id=c.config_id AND g.path=c.path
|
||||
WHERE c.deleted=0${upperBound === undefined ? '' : ' AND g.id<=?'} ORDER BY content_id`,
|
||||
[...sourceIds, ...boundParams]
|
||||
);
|
||||
const spellingSets = new Map();
|
||||
for (const row of spellingRows) {
|
||||
const contentId = id(row.content_id);
|
||||
if (!byContentId.has(contentId)) continue;
|
||||
if (!spellingSets.has(contentId)) spellingSets.set(contentId, new Set());
|
||||
spellingSets.get(contentId).add(pathText(row.path));
|
||||
}
|
||||
const groups = [...byContentId.values()].sort((left, right) => compareIds(left.contentId, right.contentId));
|
||||
for (const group of groups) {
|
||||
group.files.sort((left, right) => compareIds(left.id, right.id));
|
||||
group.spellings = [...(spellingSets.get(group.contentId) || [])]
|
||||
.sort((left, right) => Buffer.compare(Buffer.from(left, 'utf8'), Buffer.from(right, 'utf8')));
|
||||
}
|
||||
missingFiles.sort((left, right) => compareIds(left.id, right.id));
|
||||
return { maxId: upperBound ?? observedMaximum, groups, missingFiles };
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
import { performance } from 'node:perf_hooks';
|
||||
import { qualified, sha256 } from './core.mjs';
|
||||
import { applyChanges } from './journal.mjs';
|
||||
import { verifySourceBatch } from './source-batch.mjs';
|
||||
|
||||
/**
|
||||
* Switch a verified batch in one caller-session REPEATABLE READ transaction.
|
||||
* Source validation must be the first operation after BEGIN: it establishes
|
||||
* the source range lock before the transaction's consistent-read snapshot.
|
||||
*/
|
||||
export async function switchFiles(db, config, items) {
|
||||
if (!Array.isArray(items)) throw new TypeError('文件切换批次必须为数组');
|
||||
const timings = { sourceMs: 0, writeMs: 0, statusMs: 0, commitMs: 0 };
|
||||
if (!items.length) return timings;
|
||||
const changes = items.flatMap(item => {
|
||||
if (!Array.isArray(item?.files)) throw new TypeError('文件切换项必须包含 files 数组（无文件记录时使用空数组）');
|
||||
return item.files.map(file => ({
|
||||
schema: config.mysql.database, table: 'infra_file', primaryKey: 'id', pkValue: file.id,
|
||||
before: {
|
||||
config_id: file.config_id, path: file.path, url: file.url,
|
||||
type: file.type, size: file.size, deleted: Buffer.from([0])
|
||||
},
|
||||
after: {
|
||||
config_id: config.targetConfigId, path: item.key, url: item.url,
|
||||
type: item.type, size: item.size
|
||||
}
|
||||
}));
|
||||
});
|
||||
const keys = items.map(item => sha256(`${item.configId}\0${item.path}`));
|
||||
if (new Set(keys).size !== keys.length) throw new TypeError('文件切换批次不能包含重复源路径');
|
||||
if (new Set(changes.map(change => String(change.pkValue))).size !== changes.length) {
|
||||
throw new TypeError('文件切换批次不能包含重复文件记录');
|
||||
}
|
||||
const measure = async (name, action) => {
|
||||
const started = performance.now();
|
||||
try { return await action(); }
|
||||
finally { timings[name] += performance.now() - started; }
|
||||
};
|
||||
|
||||
await db.beginTransaction();
|
||||
let committing = false;
|
||||
try {
|
||||
// Do not insert metadata lookups or other consistent reads before this.
|
||||
await measure('sourceMs', () => verifySourceBatch(db, config, items));
|
||||
await measure('writeMs', async () => {
|
||||
// One source path may have many infra_file rows. Bound prepared-statement
|
||||
// size while retaining the same source locks and transaction for all rows.
|
||||
for (let offset = 0; offset < changes.length; offset += 100) {
|
||||
await applyChanges(db, config.mysql.database, config.runId, changes.slice(offset, offset + 100));
|
||||
}
|
||||
});
|
||||
await measure('statusMs', async () => {
|
||||
const [result] = await db.execute(
|
||||
`UPDATE ${qualified(config.mysql.database, 'infra_file_oss_migration_items')}
|
||||
SET status='switched',error=NULL,updated_at=CURRENT_TIMESTAMP
|
||||
WHERE run_id=? AND status='verified' AND item_key IN (${keys.map(() => '?').join(',')})`,
|
||||
[config.runId, ...keys]);
|
||||
if (result.affectedRows !== items.length) {
|
||||
throw Object.assign(new Error('部分迁移项已变化或不再处于 verified 状态，必须回滚整批事务'), {
|
||||
code: 'MIGRATION_CAS_CONFLICT'
|
||||
});
|
||||
}
|
||||
if (result.warningStatus > 0) {
|
||||
throw Object.assign(new Error('批量迁移状态更新产生数据库警告，必须回滚整批事务'), {
|
||||
code: 'MIGRATION_SQL_WARNING'
|
||||
});
|
||||
}
|
||||
});
|
||||
committing = true;
|
||||
await measure('commitMs', () => db.commit());
|
||||
return timings;
|
||||
} catch (error) {
|
||||
try {
|
||||
await db.rollback();
|
||||
} catch (rollbackError) {
|
||||
throw Object.assign(new Error('事务回滚失败，连接状态不确定，禁止降级重试', { cause: error }), {
|
||||
code: 'MIGRATION_ROLLBACK_FAILED', fatal: true, noFallback: true, rollbackError
|
||||
});
|
||||
}
|
||||
if (committing) {
|
||||
// A failed COMMIT acknowledgement does not prove the server rolled back.
|
||||
throw Object.assign(new Error('事务提交结果不确定，禁止降级重试，请重新核对迁移状态', { cause: error }), {
|
||||
code: error.code || 'MIGRATION_COMMIT_FAILED', fatal: true, noFallback: true
|
||||
});
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,582 @@
|
||||
#!/usr/bin/env node
|
||||
import { readFile, mkdir, writeFile } from 'node:fs/promises';
|
||||
import { resolve, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { isDeepStrictEqual } from 'node:util';
|
||||
import { aliases, bit, downloadUrl, encodePath, ensureObject, errorText, mimeType, normalizeConfig, objectKey, ossSettings, parseStorage, qualified, quote, readHttp, retry, sha256, verifyContent } from './lib/core.mjs';
|
||||
import { containsLegacyReference, discoverReferenceColumns, rewriteReferences } from './lib/references.mjs';
|
||||
import { ensureJournal, applyChange, rollbackChanges } from './lib/journal.mjs';
|
||||
import { loadProjectConfig } from './lib/project-config.mjs';
|
||||
import { discoverStorageConfig } from './lib/auto-storage.mjs';
|
||||
import { loadSourceIndex } from './lib/source-index.mjs';
|
||||
import { createProgress } from './lib/progress.mjs';
|
||||
import { switchFiles } from './lib/transition.mjs';
|
||||
import { mapBounded } from './lib/concurrency.mjs';
|
||||
|
||||
const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url));
|
||||
const PROJECT_ROOT = resolve(SCRIPT_DIR, '../..');
|
||||
const TRANSITION_BATCH_SIZE = 50;
|
||||
const VALIDATION_CONCURRENCY = 4;
|
||||
const seconds = milliseconds => (milliseconds / 1000).toFixed(2);
|
||||
|
||||
const HELP = `数据库图片迁移到阿里云 OSS（Node.js 20.19+）
|
||||
node migrate.mjs 自动读取项目配置，只读预检
|
||||
node migrate.mjs init 自动生成本机配置，只查询存储配置及已有批次
|
||||
node migrate.mjs apply 执行迁移；可加 --limit 50 先迁移少量
|
||||
node migrate.mjs rollback 回滚当前批次的数据库修改
|
||||
可选：--profile dev | --config 文件路径 | --run-id 新批次名称
|
||||
默认复用项目当前环境的主数据库和默认 OSS，不需要填写数据库密码或存储编号。
|
||||
保留原 BLOB、旧配置和 OSS 对象；rollback 只撤销本批次数据库修改。`;
|
||||
|
||||
export function parseArgs(argv) {
|
||||
if (argv.includes('--help') || argv.includes('-h')) return { help: true };
|
||||
const hasCommand = argv.length && !argv[0].startsWith('--');
|
||||
const command = hasCommand ? argv[0] : 'plan';
|
||||
const rest = hasCommand ? argv.slice(1) : argv;
|
||||
if (!['init', 'plan', 'apply', 'rollback'].includes(command)) throw new Error('命令只能为 init、plan、apply 或 rollback');
|
||||
const result = { command, limit: Infinity, configPath: resolve(SCRIPT_DIR, 'config.local.json') };
|
||||
for (let i = 0; i < rest.length; i += 2) {
|
||||
if (!['--config', '--limit', '--profile', '--run-id'].includes(rest[i]) || !rest[i + 1]) throw new Error('未知参数或缺少参数值');
|
||||
if (rest[i] === '--config') { result.configPath = resolve(rest[i + 1]); result.explicitConfig = true; }
|
||||
else if (rest[i] === '--profile') result.profile = rest[i + 1];
|
||||
else if (rest[i] === '--run-id') result.runId = rest[i + 1];
|
||||
else result.limit = Number(rest[i + 1]);
|
||||
}
|
||||
if (result.limit !== Infinity && (!Number.isSafeInteger(result.limit) || result.limit < 1 || command !== 'apply')) throw new Error('--limit 仅用于 apply，且必须为正整数');
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function prepareInput(args, projectRoot = PROJECT_ROOT) {
|
||||
let raw = {}, exists = false;
|
||||
try { raw = JSON.parse(await readFile(args.configPath, 'utf8')); exists = true; }
|
||||
catch (error) { if (error.code !== 'ENOENT' || args.explicitConfig) throw error; }
|
||||
const placeholder = raw.mysql?.database === 'YOUR_DATABASE';
|
||||
const projectManaged = !raw.mysql || placeholder || raw._autoDiscovery === true;
|
||||
let input;
|
||||
if (projectManaged) {
|
||||
const project = await loadProjectConfig({ projectRoot, profile: args.profile || raw.projectProfile });
|
||||
input = { ...raw, projectProfile: project.profile, mysql: project.mysql,
|
||||
_projectDownloadBaseUrl: project.downloadBaseUrl, _autoDiscovery: true };
|
||||
} else {
|
||||
if (args.profile) throw new Error('手动 mysql 配置不能同时使用 --profile；删除 mysql 项即可复用项目配置');
|
||||
input = { ...raw, mysql: { ...raw.mysql } };
|
||||
}
|
||||
if (args.runId) input.runId = args.runId;
|
||||
if (input.targetConfigId === null) delete input.targetConfigId;
|
||||
if (Array.isArray(input.sourceConfigIds) && input.sourceConfigIds.length === 0) delete input.sourceConfigIds;
|
||||
const password = input.mysql.passwordEnv ? process.env[input.mysql.passwordEnv] : input.mysql.password;
|
||||
if (password === undefined) throw new Error('数据库密码未提供；自动模式无需填写 mysql，手动模式请设置指定环境变量');
|
||||
input.mysql.password = password;
|
||||
return { input, exists, projectManaged, raw };
|
||||
}
|
||||
|
||||
export async function persistGeneratedConfig(config, args, prepared) {
|
||||
if (!prepared.projectManaged || (prepared.exists && prepared.raw._autoDiscovery !== true)) return;
|
||||
const saved = {
|
||||
projectProfile: config.projectProfile, runId: config.runId,
|
||||
sourceConfigIds: config.sourceConfigIds, targetConfigId: config.targetConfigId,
|
||||
sourceMode: config.sourceMode, oldDomains: config.oldDomains,
|
||||
_autoDiscovery: true, _databaseIdentity: config._databaseIdentity
|
||||
};
|
||||
// Persist discovery/batch choices, but read credentials from the project's
|
||||
// YAML on each invocation. No database or OSS secret is copied here.
|
||||
const defaults = normalizeConfig({ runId: config.runId, mysql: config.mysql, sourceConfigIds: config.sourceConfigIds, targetConfigId: config.targetConfigId, sourceMode: 'database' });
|
||||
for (const key of ['targetUrlMode', 'targetPublicDomain', 'objectPrefix', 'referenceSchemas', 'excludeReferenceTables', 'imagesOnly', 'batchSize', 'timeoutMs', 'maxFileBytes', 'retries']) {
|
||||
if (config[key] !== undefined && JSON.stringify(config[key]) !== JSON.stringify(defaults[key])) saved[key] = config[key];
|
||||
}
|
||||
if (config.sourceMode === 'http' || config.targetUrlMode === 'backend') saved.downloadBaseUrl = config.downloadBaseUrl;
|
||||
const serialized = JSON.stringify(saved, null, 2) + '\n';
|
||||
if (prepared.exists && JSON.stringify(prepared.raw) === JSON.stringify(saved)) return;
|
||||
// Never overwrite a pre-existing manual file; generated files may be
|
||||
// refreshed to pin an explicitly requested new batch.
|
||||
await writeFile(args.configPath, serialized, { mode: 0o600, flag: prepared.exists ? 'w' : 'wx' });
|
||||
}
|
||||
|
||||
const table = (config, name) => qualified(config.mysql.database, name);
|
||||
const runs = config => table(config, 'infra_file_oss_migration_runs');
|
||||
const items = config => table(config, 'infra_file_oss_migration_items');
|
||||
const parseJson = value => typeof value === 'string' ? JSON.parse(value) : value;
|
||||
const itemKey = item => sha256(`${item.configId}\0${item.path}`);
|
||||
const samePlannedItem = (left, right) => {
|
||||
const canonical = item => ({ ...item, aliases: [...item.aliases].sort(), files: [...item.files].sort((a, b) => BigInt(a.id) < BigInt(b.id) ? -1 : BigInt(a.id) > BigInt(b.id) ? 1 : 0) });
|
||||
return isDeepStrictEqual(canonical(left), canonical(right));
|
||||
};
|
||||
|
||||
export async function loadSettings(db, config) {
|
||||
const ids = [...config.sourceConfigIds, config.targetConfigId];
|
||||
const [rows] = await db.execute(`SELECT id, storage, master, config FROM ${table(config, 'infra_file_config')} WHERE deleted=0 AND id IN (${ids.map(() => '?').join(',')})`, ids);
|
||||
const byId = new Map(rows.map(row => [String(row.id), row]));
|
||||
for (const sourceId of config.sourceConfigIds) {
|
||||
const row = byId.get(sourceId);
|
||||
if (!row || Number(row.storage) !== 1) throw new Error(`源配置 ${sourceId} 不存在或不是数据库存储`);
|
||||
if (bit(row.master)) throw new Error(`源配置 ${sourceId} 仍是默认上传配置，请先切换默认配置到 OSS`);
|
||||
}
|
||||
const targetRow = byId.get(config.targetConfigId);
|
||||
if (!targetRow) throw new Error('目标存储配置不存在');
|
||||
const oss = ossSettings(targetRow, config);
|
||||
const [identity] = await db.query('SELECT @@server_uuid AS server_uuid');
|
||||
const safe = {
|
||||
version: 1, server: identity[0].server_uuid, schema: config.mysql.database,
|
||||
sourceConfigIds: config.sourceConfigIds, targetConfigId: config.targetConfigId,
|
||||
sourceDomains: config.sourceConfigIds.map(sourceId => parseStorage(byId.get(sourceId)).domain || ''),
|
||||
sourceMode: config.sourceMode, downloadBaseUrl: config.downloadBaseUrl || '',
|
||||
oldDomains: config.oldDomains, targetUrlMode: config.targetUrlMode,
|
||||
endpoint: oss.endpoint, bucket: oss.bucket, publicDomain: oss.publicDomain,
|
||||
objectPrefix: config.objectPrefix, imagesOnly: config.imagesOnly,
|
||||
referenceSchemas: config.referenceSchemas, excludeReferenceTables: config.excludeReferenceTables
|
||||
};
|
||||
return { byId, oss, safe };
|
||||
}
|
||||
|
||||
const progressOf = config => config._progress || { stage: message => console.log(message), update() {} };
|
||||
|
||||
function sourceItem(group, config, settings) {
|
||||
const type = mimeType(group.path, group.files);
|
||||
if (config.imagesOnly && !type.toLowerCase().startsWith('image/')) return null;
|
||||
encodePath(group.path);
|
||||
const item = { contentId: String(group.contentId), configId: String(group.configId), path: group.path, type, files: group.files };
|
||||
const aliasRows = [...group.files, ...group.spellings.map(path => ({ path }))];
|
||||
item.aliases = aliases(item, aliasRows, parseStorage(settings.byId.get(item.configId)).domain, config);
|
||||
return item;
|
||||
}
|
||||
|
||||
const missingCount = (index, config) => index.missingFiles.filter(file => !config.imagesOnly || mimeType(file.path, [file]).toLowerCase().startsWith('image/')).length;
|
||||
|
||||
export async function sourceCurrent(db, config, item, lock = false) {
|
||||
// Keep large content out of the unindexed path/order lookup. On the actual
|
||||
// database the combined lookup + BLOB projection took seconds per image;
|
||||
// a metadata lookup followed by a primary-key hash read takes milliseconds.
|
||||
// With lock=true the first SELECT retains the original range-lock semantics.
|
||||
const [[latest]] = await db.execute(`SELECT id FROM ${table(config, 'infra_file_content')} WHERE deleted=0 AND config_id=? AND path=? ORDER BY id DESC LIMIT 1${lock ? ' FOR UPDATE' : ''}`, [item.configId, item.path]);
|
||||
if (!latest || String(latest.id) !== String(item.contentId)) throw new Error('源图片在任务建立后发生变化或已删除，请检查该文件');
|
||||
const [[current]] = await db.execute(`SELECT id,OCTET_LENGTH(content) AS size,LOWER(SHA2(content,256)) AS sha256 FROM ${table(config, 'infra_file_content')} WHERE id=? AND config_id=? AND path=? AND deleted=0${lock ? ' FOR UPDATE' : ''}`, [item.contentId, item.configId, item.path]);
|
||||
if (!current || String(current.id) !== String(item.contentId) || current.sha256 !== item.sha256 || Number(current.size) !== item.size) throw new Error('源图片在任务建立后发生变化或已删除，请检查该文件');
|
||||
}
|
||||
|
||||
export async function* inventory(db, config, settings, maxId, summary, sourceIndex) {
|
||||
const progress = progressOf(config);
|
||||
const index = sourceIndex || await loadSourceIndex(db, config, { maxId, progress });
|
||||
const candidates = [];
|
||||
for (const group of index.groups) {
|
||||
summary.sourceGroups++;
|
||||
const item = sourceItem(group, config, settings);
|
||||
if (item) candidates.push(item); else summary.skippedNonImages++;
|
||||
}
|
||||
progress.stage(`建立迁移清单：按主键读取并校验 ${candidates.length} 个文件的内容`);
|
||||
for (let start = 0; start < candidates.length; start += config.batchSize) {
|
||||
const batch = candidates.slice(start, start + config.batchSize);
|
||||
progress.update(`建立迁移清单：已校验 ${start}/${candidates.length} 个文件，正在读取下一批`);
|
||||
const [rows] = await db.execute(`SELECT id,config_id,path,OCTET_LENGTH(content) AS size,LOWER(SHA2(content,256)) AS sha256 FROM ${table(config, 'infra_file_content')} WHERE deleted=0 AND id IN (${batch.map(() => '?').join(',')})`, batch.map(item => item.contentId));
|
||||
const byId = new Map(rows.map(row => [String(row.id), row]));
|
||||
for (const item of batch) {
|
||||
const row = byId.get(item.contentId);
|
||||
if (!row || String(row.config_id) !== item.configId || row.path !== item.path) throw new Error(`内容记录 ${item.contentId} 在建立清单期间发生变化，请检查后重试`);
|
||||
item.size = Number(row.size);
|
||||
item.sha256 = row.sha256;
|
||||
if (!item.size || item.size > config.maxFileBytes) throw new Error(`内容记录 ${item.contentId} 为空或超过 maxFileBytes`);
|
||||
item.key = objectKey(config, item);
|
||||
item.url = config.targetUrlMode === 'backend' ? downloadUrl(config.downloadBaseUrl, config.targetConfigId, item.key) : `${settings.oss.publicDomain}/${encodePath(item.key)}`;
|
||||
if (item.key.length > 512 || item.url.length > 1024) throw new Error('目标路径或访问地址超出 infra_file 字段长度');
|
||||
summary.candidates++;
|
||||
summary.bytes += item.size;
|
||||
if (!item.files.length) summary.withoutFileRecord++;
|
||||
yield item;
|
||||
}
|
||||
}
|
||||
progress.stage(`迁移清单已建立：${summary.candidates} 个文件，内容校验完成`);
|
||||
}
|
||||
|
||||
function addMappings(map, item) {
|
||||
for (const oldUrl of item.aliases) {
|
||||
if (map.has(oldUrl) && map.get(oldUrl) !== item.url) throw new Error('同一旧地址对应多个目标，必须先解决数据冲突');
|
||||
map.set(oldUrl, item.url);
|
||||
}
|
||||
}
|
||||
|
||||
const emptyInventory = () => ({ sourceGroups: 0, candidates: 0, bytes: 0, skippedNonImages: 0, withoutFileRecord: 0 });
|
||||
|
||||
async function scanReferences(db, config, map, { apply = false } = {}) {
|
||||
const progress = progressOf(config);
|
||||
progress.stage('检查需要扫描的业务表和图片地址字段');
|
||||
const discovery = await discoverReferenceColumns(db, config.referenceSchemas, config.excludeReferenceTables);
|
||||
const groups = new Map();
|
||||
for (const column of discovery.columns) {
|
||||
const key = JSON.stringify([column.schema, column.table]);
|
||||
if (!groups.has(key)) groups.set(key, { ...column, columns: [] });
|
||||
groups.get(key).columns.push(column.column);
|
||||
}
|
||||
const result = { scannedRows: 0, changedRows: 0, replacements: 0, unresolvedRows: 0, conflicts: 0, skippedTables: discovery.skipped.filter(entry => entry.reason !== 'excluded table'), excludedTables: discovery.skipped.filter(entry => entry.reason === 'excluded table'), tables: [] };
|
||||
let tableNumber = 0;
|
||||
const readBatchSize = Math.max(config.batchSize, 1000);
|
||||
for (const group of groups.values()) {
|
||||
tableNumber++;
|
||||
let scanned = 0;
|
||||
progress.stage(`引用扫描 ${tableNumber}/${groups.size}：${group.schema}.${group.table}`);
|
||||
const stat = { schema: group.schema, table: group.table, changedRows: 0, unresolvedRows: 0 };
|
||||
let cursor;
|
||||
for (;;) {
|
||||
const columns = [group.primaryKey, ...group.columns];
|
||||
const [rows] = await db.query(`SELECT ${columns.map(quote).join(',')} FROM ${qualified(group.schema, group.table)} ${cursor === undefined ? '' : `WHERE ${quote(group.primaryKey)}>?`} ORDER BY ${quote(group.primaryKey)} LIMIT ${readBatchSize}`, cursor === undefined ? [] : [cursor]);
|
||||
if (!rows.length) break;
|
||||
for (const row of rows) {
|
||||
cursor = row[group.primaryKey];
|
||||
result.scannedRows++;
|
||||
scanned++;
|
||||
progress.update(`引用扫描 ${tableNumber}/${groups.size}：${group.schema}.${group.table}，已检查 ${scanned} 行，匹配 ${stat.changedRows} 行`);
|
||||
const before = {}, after = {};
|
||||
let unresolved = false, replacements = 0;
|
||||
for (const column of group.columns) {
|
||||
const rewritten = rewriteReferences(row[column], map);
|
||||
if (rewritten.changed) { before[column] = row[column]; after[column] = rewritten.value; replacements += rewritten.replacements; }
|
||||
if (containsLegacyReference(rewritten.value, config.sourceConfigIds)) unresolved = true;
|
||||
}
|
||||
if (unresolved) { result.unresolvedRows++; stat.unresolvedRows++; }
|
||||
if (!Object.keys(after).length) continue;
|
||||
if (apply) {
|
||||
await db.beginTransaction();
|
||||
try {
|
||||
await applyChange(db, config.mysql.database, config.runId, { schema: group.schema, table: group.table, primaryKey: group.primaryKey, pkValue: cursor, before, after });
|
||||
await db.commit();
|
||||
} catch (error) {
|
||||
await db.rollback();
|
||||
result.conflicts++;
|
||||
console.error(`引用更新未完成 ${group.schema}.${group.table}：${errorText(error)}`);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
result.changedRows++; stat.changedRows++; result.replacements += replacements;
|
||||
}
|
||||
}
|
||||
if (stat.changedRows || stat.unresolvedRows) result.tables.push(stat);
|
||||
console.log(`引用扫描 ${group.schema}.${group.table}：匹配 ${stat.changedRows} 行，仍含旧地址 ${stat.unresolvedRows} 行`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async function setItem(db, config, item, status, error = null) {
|
||||
await db.execute(`UPDATE ${items(config)} SET status=?,error=?,updated_at=CURRENT_TIMESTAMP WHERE run_id=? AND item_key=?`, [status, error, config.runId, itemKey(item)]);
|
||||
}
|
||||
|
||||
async function* savedItems(db, config, statuses = null) {
|
||||
let cursor = '';
|
||||
for (;;) {
|
||||
const [rows] = await db.execute(`SELECT item_key,data,status FROM ${items(config)} WHERE run_id=? AND item_key>? ${statuses ? `AND status IN (${statuses.map(() => '?').join(',')})` : ''} ORDER BY item_key LIMIT ${config.batchSize}`, [config.runId, cursor, ...(statuses || [])]);
|
||||
if (!rows.length) return;
|
||||
for (const row of rows) { cursor = row.item_key; yield { ...parseJson(row.data), status: row.status }; }
|
||||
}
|
||||
}
|
||||
|
||||
async function migrateItem(db, config, oss, item) {
|
||||
const timings = { sourceMs: 0, ossMs: 0, publicMs: 0, recordMs: 0 };
|
||||
const measure = async (key, label, action) => {
|
||||
progressOf(config).update(`文件 ${item.contentId}：${label}`);
|
||||
const started = performance.now();
|
||||
try { return await action(); } finally { timings[key] += performance.now() - started; }
|
||||
};
|
||||
await measure('sourceMs', '复核源文件版本', () => sourceCurrent(db, config, item));
|
||||
let bytes;
|
||||
if (config.sourceMode === 'http') {
|
||||
bytes = await measure('sourceMs', '读取原图片', () => readHttp(downloadUrl(config.downloadBaseUrl, item.configId, item.path), config, item.size));
|
||||
} else {
|
||||
const [[row]] = await measure('sourceMs', '读取数据库原图片', () => db.execute(`SELECT content FROM ${table(config, 'infra_file_content')} WHERE id=? AND deleted=0`, [item.contentId]));
|
||||
if (!row) throw new Error('源内容已删除');
|
||||
bytes = row.content;
|
||||
}
|
||||
verifyContent(bytes, item);
|
||||
await measure('ossMs', '上传 OSS 并读取校验', () => ensureObject(oss, item.key, bytes, item));
|
||||
// SDK 可读不代表页面可读：按最终将写入业务表的地址匿名访问并再次校验。
|
||||
verifyContent(await measure('publicMs', '校验最终图片访问地址', () => readHttp(item.url, config, item.size)), item);
|
||||
await measure('sourceMs', '复核源图片未发生变化', () => sourceCurrent(db, config, item));
|
||||
await measure('recordMs', '保存迁移进度', () => setItem(db, config, item, 'verified'));
|
||||
return timings;
|
||||
}
|
||||
|
||||
export async function verifyCheckpoint(oss, config, item) {
|
||||
// A verified checkpoint needs no source download or PUT. Still check the
|
||||
// authenticated object and its final public URL before trusting it again.
|
||||
const head = await oss.head(item.key);
|
||||
if (Number(head.res.headers['content-length']) !== item.size || item.size > config.maxFileBytes) {
|
||||
throw new Error('OSS 文件长度不一致或超过限制');
|
||||
}
|
||||
verifyContent((await oss.get(item.key)).content, item);
|
||||
verifyContent(await readHttp(item.url, config, item.size), item);
|
||||
}
|
||||
|
||||
export async function verifySwitchedBatch(db, config, batch) {
|
||||
// A resumed run must not reuse a mapping whose object/metadata was deleted
|
||||
// or changed after the earlier successful batch. Read metadata once; only
|
||||
// HTTP validation is concurrent, never transactions on the shared connection.
|
||||
const ids = [...new Set(batch.flatMap(item => item.files.map(file => String(file.id))))];
|
||||
const byId = new Map();
|
||||
for (let start = 0; start < ids.length; start += 1000) {
|
||||
const chunk = ids.slice(start, start + 1000);
|
||||
const [rows] = await db.execute(`SELECT id,config_id,path,url,deleted FROM ${table(config, 'infra_file')} WHERE id IN (${chunk.map(() => '?').join(',')})`, chunk);
|
||||
for (const row of rows) byId.set(String(row.id), row);
|
||||
}
|
||||
return mapBounded(batch, VALIDATION_CONCURRENCY, async item => {
|
||||
try {
|
||||
for (const original of item.files) {
|
||||
const current = byId.get(String(original.id));
|
||||
if (!current || bit(current.deleted) || String(current.config_id) !== config.targetConfigId || current.path !== item.key || current.url !== item.url) {
|
||||
throw new Error('已迁移文件记录后来发生变化或已删除，停止复用该地址映射');
|
||||
}
|
||||
}
|
||||
await retry(async () => verifyContent(await readHttp(item.url, config, item.size), item), config.retries);
|
||||
return { item };
|
||||
} catch (error) {
|
||||
return { item, error };
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async function* itemBatches(iterable, size) {
|
||||
let batch = [];
|
||||
for await (const item of iterable) {
|
||||
batch.push(item);
|
||||
if (batch.length === size) { yield batch; batch = []; }
|
||||
}
|
||||
if (batch.length) yield batch;
|
||||
}
|
||||
|
||||
export async function runPlan(db, config, settings) {
|
||||
const progress = progressOf(config);
|
||||
progress.stage('开始只读预检：只读取文件信息，不读取图片内容或计算 SHA-256');
|
||||
const index = await loadSourceIndex(db, config, { progress });
|
||||
const summary = { command: 'plan', ...emptyInventory(), bytes: null, estimatedBytes: 0, sizePending: 0, contentVerified: false, missingContent: missingCount(index, config) };
|
||||
const mappings = new Map();
|
||||
for (const group of index.groups) {
|
||||
summary.sourceGroups++;
|
||||
const item = sourceItem(group, config, settings);
|
||||
if (!item) { summary.skippedNonImages++; continue; }
|
||||
summary.candidates++;
|
||||
const sizes = item.files.map(file => Number(file.size)).filter(size => Number.isSafeInteger(size) && size >= 0);
|
||||
if (sizes.length) summary.estimatedBytes += Math.max(...sizes); else summary.sizePending++;
|
||||
if (!item.files.length) summary.withoutFileRecord++;
|
||||
// Preview markers exist only in memory. Actual object keys require the
|
||||
// verified content hash and are created exclusively during apply.
|
||||
item.url = `migration-preview://${item.configId}/${item.contentId}`;
|
||||
addMappings(mappings, item);
|
||||
}
|
||||
progress.stage(`已识别 ${summary.candidates} 个图片文件；开始检查业务引用（实际内容在迁移时校验）`);
|
||||
summary.references = await scanReferences(db, config, mappings);
|
||||
summary.status = summary.missingContent || summary.references.unresolvedRows || summary.references.skippedTables.length ? 'needs-review' : 'planned';
|
||||
progress.stage('只读预检完成');
|
||||
console.log('预检未读取图片二进制内容、未计算 SHA-256；容量仅为文件表估算。未验证 HTTP/OSS 连通性，也未创建迁移表、上传或修改数据。');
|
||||
return summary;
|
||||
}
|
||||
|
||||
export async function runApply(db, config, settings, limit = Infinity, createOss = null) {
|
||||
const progress = progressOf(config);
|
||||
progress.stage('读取迁移批次和断点记录');
|
||||
// Batch source checks depend on InnoDB next-key locks and a snapshot first
|
||||
// established AFTER their range lock, even if the server defaults to RC.
|
||||
await db.query('SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ');
|
||||
const [engines] = await db.execute(`SELECT TABLE_NAME AS name,ENGINE AS engine FROM information_schema.TABLES WHERE TABLE_SCHEMA=? AND TABLE_NAME IN ('infra_file_content','infra_file')`, [config.mysql.database]);
|
||||
if (engines.length !== 2 || engines.some(row => String(row.engine).toLowerCase() !== 'innodb')) {
|
||||
throw new Error('infra_file_content 和 infra_file 必须为 InnoDB，才能保证源校验与记录更新的事务保护');
|
||||
}
|
||||
await ensureJournal(db, config.mysql.database);
|
||||
const [[saved]] = await db.execute(`SELECT settings,status FROM ${runs(config)} WHERE run_id=?`, [config.runId]);
|
||||
let runSettings, sourceIndex;
|
||||
if (saved) {
|
||||
runSettings = parseJson(saved.settings);
|
||||
if (JSON.stringify(runSettings.binding) !== JSON.stringify(settings.safe)) throw new Error('同一 runId 的数据库、源/目标或引用范围发生变化，请恢复原配置或使用新 runId');
|
||||
if (saved.status.startsWith('rollback') || saved.status === 'rolled_back') throw new Error('本批次已开始回滚，不能继续迁移，请继续回滚或使用新 runId');
|
||||
} else {
|
||||
sourceIndex = await loadSourceIndex(db, config, { progress });
|
||||
const missing = missingCount(sourceIndex, config);
|
||||
if (missing) throw new Error(`发现 ${missing} 条图片记录没有数据库内容，请先执行 plan 并处理`);
|
||||
runSettings = { binding: settings.safe, maxId: sourceIndex.maxId };
|
||||
await db.execute(`INSERT INTO ${runs(config)} (run_id,settings,status) VALUES (?,?,'inventory')`, [config.runId, JSON.stringify(runSettings)]);
|
||||
}
|
||||
const summary = { command: 'apply', ...emptyInventory(), ...runSettings.inventory, uploaded: 0, revalidated: 0, switched: 0, failed: 0, limited: false };
|
||||
if (!saved || saved.status === 'inventory') {
|
||||
for await (const item of inventory(db, config, settings, runSettings.maxId, summary, sourceIndex)) {
|
||||
const [[existing]] = await db.execute(`SELECT data FROM ${items(config)} WHERE run_id=? AND item_key=?`, [config.runId, itemKey(item)]);
|
||||
if (existing) {
|
||||
if (!samePlannedItem(parseJson(existing.data), item)) throw new Error('建立清单期间源数据发生变化，请检查后建立新批次');
|
||||
} else {
|
||||
await db.execute(`INSERT INTO ${items(config)} (run_id,item_key,data,status) VALUES (?,?,?,'planned')`, [config.runId, itemKey(item), JSON.stringify(item)]);
|
||||
}
|
||||
}
|
||||
runSettings.inventory = Object.fromEntries(Object.keys(emptyInventory()).map(key => [key, summary[key]]));
|
||||
await db.execute(`UPDATE ${runs(config)} SET settings=?,status='active',updated_at=CURRENT_TIMESTAMP WHERE run_id=?`, [JSON.stringify(runSettings), config.runId]);
|
||||
}
|
||||
// 先验证所有映射没有歧义，再发生 OSS 写入。
|
||||
const allMappings = new Map();
|
||||
let initialSwitched = 0, totalItems = 0;
|
||||
progress.stage('检查迁移地址映射是否存在冲突');
|
||||
for await (const item of savedItems(db, config)) {
|
||||
addMappings(allMappings, item);
|
||||
totalItems++;
|
||||
if (item.status === 'switched') initialSwitched++;
|
||||
}
|
||||
const { publicDomain, ...ossOptions } = settings.oss;
|
||||
const oss = createOss ? createOss() : new (await import('ali-oss')).default({ ...ossOptions, timeout: config.timeoutMs });
|
||||
let processed = 0;
|
||||
const selected = [], checkpoints = [];
|
||||
progress.stage('开始上传并校验 OSS 文件');
|
||||
for await (const item of savedItems(db, config, ['planned', 'failed', 'verified'])) {
|
||||
if (processed >= limit) { summary.limited = true; break; }
|
||||
processed++;
|
||||
if (item.status === 'verified') { checkpoints.push(item); continue; }
|
||||
try {
|
||||
const started = performance.now();
|
||||
const timings = await retry(() => migrateItem(db, config, oss, item), config.retries);
|
||||
summary.uploaded++;
|
||||
selected.push(item);
|
||||
console.log(`上传校验通过：内容记录 ${item.contentId}（本次 ${processed}，总计 ${seconds(performance.now() - started)} 秒；源读取/复核 ${seconds(timings.sourceMs)} 秒，OSS 上传/校验 ${seconds(timings.ossMs)} 秒，访问校验 ${seconds(timings.publicMs)} 秒）`);
|
||||
} catch (error) {
|
||||
await setItem(db, config, item, 'failed', errorText(error));
|
||||
summary.failed++;
|
||||
console.error(`上传未完成：内容记录 ${item.contentId}，${errorText(error)}`);
|
||||
}
|
||||
}
|
||||
if (checkpoints.length) {
|
||||
let checked = 0;
|
||||
progress.stage(`复核已上传断点：0/${checkpoints.length}，最多 ${VALIDATION_CONCURRENCY} 张同时校验`);
|
||||
const results = await mapBounded(checkpoints, VALIDATION_CONCURRENCY, async item => {
|
||||
try {
|
||||
await retry(() => verifyCheckpoint(oss, config, item), config.retries);
|
||||
return { item };
|
||||
} catch (error) {
|
||||
return { item, error };
|
||||
} finally {
|
||||
progress.update(`复核已上传断点：${++checked}/${checkpoints.length}`);
|
||||
}
|
||||
});
|
||||
// All network workers have settled before using the database again.
|
||||
for (const { item, error } of results) {
|
||||
if (error) {
|
||||
await setItem(db, config, item, 'failed', errorText(error));
|
||||
summary.failed++;
|
||||
console.error(`已上传断点校验失败：内容记录 ${item.contentId}，${errorText(error)}`);
|
||||
} else { selected.push(item); summary.revalidated++; }
|
||||
}
|
||||
}
|
||||
if (summary.failed) {
|
||||
summary.status = 'partial';
|
||||
const [counts] = await db.execute(`SELECT status,COUNT(*) AS count FROM ${items(config)} WHERE run_id=? GROUP BY status`, [config.runId]);
|
||||
summary.items = Object.fromEntries(counts.map(row => [row.status, Number(row.count)]));
|
||||
await db.execute(`UPDATE ${runs(config)} SET status='partial',updated_at=CURRENT_TIMESTAMP WHERE run_id=?`, [config.runId]);
|
||||
return summary;
|
||||
}
|
||||
const switchProgress = () => `更新已校验文件记录：本次完成 ${summary.switched}/${selected.length}，累计 ${initialSwitched + summary.switched}/${totalItems}`;
|
||||
progress.stage(switchProgress());
|
||||
for (let start = 0; start < selected.length; start += TRANSITION_BATCH_SIZE) {
|
||||
const batch = selected.slice(start, start + TRANSITION_BATCH_SIZE);
|
||||
const started = performance.now();
|
||||
try {
|
||||
const timing = await switchFiles(db, config, batch);
|
||||
summary.switched += batch.length;
|
||||
console.log(`文件记录批次完成：${batch.length} 张，${seconds(performance.now() - started)} 秒（源校验 ${seconds(timing.sourceMs)} 秒，文件及回滚日志 ${seconds(timing.writeMs)} 秒，进度 ${seconds(timing.statusMs)} 秒，提交 ${seconds(timing.commitMs)} 秒）；累计 ${initialSwitched + summary.switched}/${totalItems}`);
|
||||
} catch (error) {
|
||||
// Isolate known data conflicts after the whole batch rolled back. Never
|
||||
// replay a transaction with an ambiguous commit or broken connection.
|
||||
if (error.fatal || !['MIGRATION_CAS_CONFLICT', 'MIGRATION_SOURCE_CHANGED'].includes(error.code)) throw error;
|
||||
console.error(`该批文件存在修改冲突，事务已回滚；逐项定位：${errorText(error)}`);
|
||||
for (const item of batch) {
|
||||
try { await switchFiles(db, config, [item]); summary.switched++; }
|
||||
catch (itemError) {
|
||||
if (itemError.fatal || !['MIGRATION_CAS_CONFLICT', 'MIGRATION_SOURCE_CHANGED'].includes(itemError.code)) throw itemError;
|
||||
summary.failed++;
|
||||
console.error(`文件记录未切换：内容记录 ${item.contentId}，${errorText(itemError)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
progress.update(switchProgress());
|
||||
}
|
||||
const mappings = new Map();
|
||||
let checkedSwitched = 0;
|
||||
progress.stage(`复核已迁移文件的访问地址：0/${initialSwitched + summary.switched}`);
|
||||
for await (const batch of itemBatches(savedItems(db, config, ['switched']), TRANSITION_BATCH_SIZE)) {
|
||||
for (const { item, error } of await verifySwitchedBatch(db, config, batch)) {
|
||||
if (error) {
|
||||
summary.failed++;
|
||||
console.error(`目标地址不可复用：内容记录 ${item.contentId}，${errorText(error)}`);
|
||||
} else addMappings(mappings, item);
|
||||
}
|
||||
checkedSwitched += batch.length;
|
||||
progress.update(`复核已迁移文件的访问地址：${checkedSwitched}/${initialSwitched + summary.switched}`);
|
||||
}
|
||||
summary.references = await scanReferences(db, config, mappings, { apply: true });
|
||||
const [counts] = await db.execute(`SELECT status,COUNT(*) AS count FROM ${items(config)} WHERE run_id=? GROUP BY status`, [config.runId]);
|
||||
summary.items = Object.fromEntries(counts.map(row => [row.status, Number(row.count)]));
|
||||
const unfinished = counts.some(row => row.status !== 'switched');
|
||||
summary.status = unfinished || summary.failed || summary.references.conflicts ? 'partial'
|
||||
: summary.references.unresolvedRows || summary.references.skippedTables.length ? 'needs-review' : 'complete';
|
||||
await db.execute(`UPDATE ${runs(config)} SET status=?,updated_at=CURRENT_TIMESTAMP WHERE run_id=?`, [summary.status, config.runId]);
|
||||
return summary;
|
||||
}
|
||||
|
||||
export async function runRollback(db, config) {
|
||||
const [[run]] = await db.execute(`SELECT settings,status FROM ${runs(config)} WHERE run_id=?`, [config.runId]);
|
||||
if (!run) throw new Error('找不到此 runId 的迁移日志');
|
||||
const [[identity]] = await db.query('SELECT @@server_uuid AS server_uuid');
|
||||
const binding = parseJson(run.settings).binding;
|
||||
if (binding.schema !== config.mysql.database || binding.server !== identity.server_uuid) throw new Error('回滚数据库与迁移时不一致');
|
||||
await db.execute(`UPDATE ${runs(config)} SET status='rollback',updated_at=CURRENT_TIMESTAMP WHERE run_id=?`, [config.runId]);
|
||||
const result = await rollbackChanges(db, config.mysql.database, config.runId);
|
||||
const status = result.conflicts.length ? 'rollback_partial' : 'rolled_back';
|
||||
await db.execute(`UPDATE ${runs(config)} SET status=?,updated_at=CURRENT_TIMESTAMP WHERE run_id=?`, [status, config.runId]);
|
||||
return { command: 'rollback', status, ...result };
|
||||
}
|
||||
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
const args = parseArgs(argv);
|
||||
if (args.help) { console.log(HELP); return; }
|
||||
const prepared = await prepareInput(args);
|
||||
const { default: mysql } = await import('mysql2/promise');
|
||||
const { passwordEnv, ...mysqlOptions } = prepared.input.mysql;
|
||||
console.log(`读取${prepared.input.projectProfile ? `项目 ${prepared.input.projectProfile} 环境` : '指定配置'}：MySQL ${mysqlOptions.host || 'localhost'}:${mysqlOptions.port || 3306}/${mysqlOptions.database}`);
|
||||
const db = await mysql.createConnection({ connectTimeout: 10000, ...mysqlOptions, multipleStatements: false, supportBigNumbers: true, bigNumberStrings: true, jsonStrings: true, dateStrings: true });
|
||||
const progress = createProgress();
|
||||
let locked = false;
|
||||
const lockName = `oss-file-migration:${sha256(mysqlOptions.database).slice(0, 40)}`;
|
||||
try {
|
||||
progress.stage('识别旧存储、默认 OSS 和迁移批次');
|
||||
await db.query("SET SESSION sql_mode = CONCAT_WS(',', NULLIF(@@SESSION.sql_mode,''), 'STRICT_ALL_TABLES')");
|
||||
if (['apply', 'rollback'].includes(args.command)) {
|
||||
const [[lock]] = await db.execute('SELECT GET_LOCK(?,0) AS acquired', [lockName]);
|
||||
if (Number(lock.acquired) !== 1) throw new Error('已有文件迁移或回滚任务运行，请等待其结束');
|
||||
locked = true;
|
||||
}
|
||||
const auto = prepared.projectManaged || !prepared.input.runId || !prepared.input.sourceConfigIds?.length || !prepared.input.targetConfigId;
|
||||
const config = normalizeConfig(auto ? await discoverStorageConfig(db, prepared.input, { command: args.command }) : prepared.input);
|
||||
config._progress = progress;
|
||||
await db.query(`SET SESSION max_execution_time=${config.timeoutMs}`);
|
||||
await persistGeneratedConfig(config, args, prepared);
|
||||
if (args.command === 'init') {
|
||||
await loadSettings(db, config);
|
||||
progress.close();
|
||||
console.log(`配置已就绪：旧数据库存储 ${config.sourceConfigIds.join(',')} → 默认 OSS ${config.targetConfigId}；批次 ${config.runId}`);
|
||||
console.log(`配置文件：${args.configPath}；数据库密码和 OSS 密钥未复制到此文件。`);
|
||||
console.log('只读取了项目和数据库配置，未上传图片或修改数据库。');
|
||||
return { status: 'configured', configPath: args.configPath };
|
||||
}
|
||||
let summary;
|
||||
if (args.command === 'rollback') summary = await runRollback(db, config);
|
||||
else {
|
||||
const settings = await loadSettings(db, config);
|
||||
console.log(`源配置 ${config.sourceConfigIds.join(',')} → OSS 配置 ${config.targetConfigId}；批次 ${config.runId}`);
|
||||
summary = args.command === 'plan' ? await runPlan(db, config, settings) : await runApply(db, config, settings, args.limit);
|
||||
}
|
||||
const reportDir = resolve(dirname(args.configPath), 'reports');
|
||||
await mkdir(reportDir, { recursive: true, mode: 0o700 });
|
||||
const reportPath = resolve(reportDir, `${config.runId}-${args.command}-${Date.now()}.json`);
|
||||
await writeFile(reportPath, JSON.stringify(summary, null, 2), { mode: 0o600 });
|
||||
progress.close();
|
||||
console.log(JSON.stringify(summary, null, 2));
|
||||
console.log(`报告：${reportPath}`);
|
||||
if (['partial', 'needs-review', 'rollback_partial'].includes(summary.status)) process.exitCode = 2;
|
||||
return summary;
|
||||
} finally {
|
||||
progress.close();
|
||||
try { if (locked) await db.execute('SELECT RELEASE_LOCK(?)', [lockName]); }
|
||||
finally { await db.end(); }
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
main().catch(error => { console.error(`执行停止：${errorText(error)}`); process.exitCode = 1; });
|
||||
}
|
||||
+1101
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"name": "friendship-db-files-to-oss",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"engines": {
|
||||
"node": ">=20.19"
|
||||
},
|
||||
"scripts": {
|
||||
"test": "node --test test/*.test.mjs",
|
||||
"init": "node migrate.mjs init",
|
||||
"plan": "node migrate.mjs",
|
||||
"apply": "node migrate.mjs apply",
|
||||
"rollback": "node migrate.mjs rollback"
|
||||
},
|
||||
"dependencies": {
|
||||
"ali-oss": "6.23.0",
|
||||
"mysql2": "3.15.3",
|
||||
"yaml": "2.8.2"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { discoverStorageConfig } from '../lib/auto-storage.mjs';
|
||||
|
||||
const database = 'local_app';
|
||||
const server = 'test-server-uuid';
|
||||
const input = () => ({ mysql: { database, user: 'local-test', password: 'db-test-only' }, projectProfile: 'local', _projectDownloadBaseUrl: 'http://localhost:48080/admin-api/infra/file' });
|
||||
const source = value => ({ id: value, storage: 1, master: Buffer.from([0]), config: JSON.stringify({ domain: 'https://old.example' }) });
|
||||
const target = (value = '30', master = 1) => ({ id: value, storage: 20, master: Buffer.from([master]), config: JSON.stringify({ endpoint: 'oss-cn-hangzhou.aliyuncs.com', bucket: 'migration-test', accessKey: 'oss-test-access', accessSecret: 'oss-secret-must-not-escape', domain: 'https://media.example' }) });
|
||||
function binding(overrides = {}) {
|
||||
return { server, schema: database, sourceConfigIds: ['5'], targetConfigId: '30', sourceMode: 'database', downloadBaseUrl: 'https://old.example/admin-api/infra/file', oldDomains: ['https://historic.example'], objectPrefix: 'migration/original', imagesOnly: true, referenceSchemas: [database, 'chat'], excludeReferenceTables: ['audit'], targetUrlMode: 'direct', publicDomain: 'https://original-media.example', ...overrides };
|
||||
}
|
||||
const run = (runId = 'original-run', status = 'partial', overrides = {}) => ({ run_id: runId, status, settings: JSON.stringify({ binding: binding(overrides), maxId: '9007199254740993' }) });
|
||||
|
||||
class FakeDb {
|
||||
constructor(options = {}) {
|
||||
this.server = server;
|
||||
this.configs = [source('5'), target()];
|
||||
this.sourceIds = ['5'];
|
||||
this.runs = [];
|
||||
this.domains = ['https://old.example'];
|
||||
this.maximum = '9007199254740993';
|
||||
this.calls = [];
|
||||
Object.assign(this, options);
|
||||
}
|
||||
async execute(sql, params = []) {
|
||||
assert.match(sql.trim(), /^SELECT\b/i, 'discovery must only issue SELECT');
|
||||
this.calls.push({ sql, params });
|
||||
if (sql.includes('@@server_uuid')) return [[{ server_uuid: this.server }]];
|
||||
if (sql.includes('information_schema.TABLES')) {
|
||||
assert.deepEqual(params, [database, 'infra_file_oss_migration_runs']);
|
||||
return [this.runs.length ? [{ present: 1 }] : []];
|
||||
}
|
||||
if (sql.includes('infra_file_oss_migration_runs')) return [this.runs];
|
||||
if (sql.includes('infra_file_config')) return [this.configs];
|
||||
if (sql.includes('MAX(id)')) return [[{ max_id: this.maximum }]];
|
||||
if (sql.includes('infra_file_content')) {
|
||||
assert.match(sql, /deleted=0/);
|
||||
return [this.sourceIds.filter(value => params.includes(value)).map(config_id => ({ config_id }))];
|
||||
}
|
||||
if (sql.includes('SUBSTRING_INDEX')) return [this.domains.map(domain => ({ domain }))];
|
||||
throw new Error(`Unexpected test SQL: ${sql}`);
|
||||
}
|
||||
}
|
||||
|
||||
test('discovers source BLOB configurations and the unique OSS default using SELECT only', async () => {
|
||||
const db = new FakeDb({ configs: [source('9007199254740995'), target(), source('12'), source('2'), source('999')], sourceIds: ['12', '9007199254740995', '2', '2'] });
|
||||
const result = await discoverStorageConfig(db, input());
|
||||
assert.deepEqual(result.sourceConfigIds, ['2', '12', '9007199254740995']);
|
||||
assert.equal(result.targetConfigId, '30');
|
||||
assert.equal(result.sourceMode, 'database');
|
||||
assert.equal(result.downloadBaseUrl, input()._projectDownloadBaseUrl);
|
||||
assert.equal(result.projectProfile, 'local');
|
||||
assert.equal(result._autoDiscovery, true);
|
||||
assert.deepEqual(result._databaseIdentity, { server, schema: database });
|
||||
assert.match(result.runId, /^db-images-[a-f0-9]{16}$/);
|
||||
assert.equal(JSON.stringify(result).includes('oss-secret-must-not-escape'), false);
|
||||
assert.ok(db.calls.some(call => call.sql.includes('SELECT DISTINCT config_id FROM')));
|
||||
});
|
||||
|
||||
test('generated runId is independent of source ordering, credentials, domains and remaining file metadata', async () => {
|
||||
const first = await discoverStorageConfig(new FakeDb({ configs: [source('7'), target(), source('5')], sourceIds: ['7', '5'] }), input());
|
||||
const secondInput = input();
|
||||
secondInput.mysql.password = 'rotated-password';
|
||||
const second = await discoverStorageConfig(new FakeDb({ configs: [source('5'), source('7'), target()], sourceIds: ['5', '7'], domains: [] }), secondInput);
|
||||
assert.equal(second.runId, first.runId);
|
||||
assert.deepEqual(second.sourceConfigIds, first.sourceConfigIds);
|
||||
});
|
||||
|
||||
test('only valid HTTP base domains without credentials are inferred and sorted', async () => {
|
||||
const result = await discoverStorageConfig(new FakeDb({ domains: ['https://z.example/', 'https://a.example/prefix', 'https://z.example', 'https://user:secret@example.com', 'ftp://file.example', 'https://x.example?secret=1', 'not a URL'] }), input());
|
||||
assert.deepEqual(result.oldDomains, ['https://a.example/prefix', 'https://z.example']);
|
||||
});
|
||||
|
||||
test('requires a unique default and refuses non-Aliyun defaults instead of selecting a different OSS', async () => {
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb({ configs: [source('5'), target('30', 0)] }), input()), /默认.*唯一/);
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb({ configs: [source('5'), target(), target('31')] }), input()), /默认.*唯一/);
|
||||
const invalidDefault = target();
|
||||
invalidDefault.config = JSON.stringify({ endpoint: 'minio.example', accessSecret: 'dont-leak' });
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb({ configs: [source('5'), invalidDefault, target('31', 0)] }), input()), error => /阿里云/.test(error.message) && !error.message.includes('dont-leak'));
|
||||
const malformedDefault = target();
|
||||
malformedDefault.config = '{"secret":"dont-leak", broken';
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb({ configs: [source('5'), malformedDefault] }), input()), error => !error.message.includes('dont-leak'));
|
||||
});
|
||||
|
||||
test('restores an unfinished run before current metadata can shrink its original scope', async () => {
|
||||
const original = run('original-run', 'partial', { sourceConfigIds: ['7', '5'] });
|
||||
const db = new FakeDb({ runs: [original], configs: [source('5'), source('7'), target()], sourceIds: [], domains: [] });
|
||||
const result = await discoverStorageConfig(db, input(), { command: 'apply' });
|
||||
assert.equal(result.runId, 'original-run');
|
||||
assert.deepEqual(result.sourceConfigIds, ['7', '5'], 'historical order is part of the existing fingerprint');
|
||||
assert.deepEqual(result.oldDomains, ['https://historic.example']);
|
||||
assert.equal(result.objectPrefix, 'migration/original');
|
||||
assert.equal(result.downloadBaseUrl, 'https://old.example/admin-api/infra/file');
|
||||
assert.equal(result.targetPublicDomain, 'https://original-media.example');
|
||||
assert.deepEqual(result.referenceSchemas, [database, 'chat']);
|
||||
assert.ok(!db.calls.some(call => call.sql.includes('SELECT DISTINCT config_id')));
|
||||
assert.ok(!db.calls.some(call => call.sql.includes('SUBSTRING_INDEX')));
|
||||
});
|
||||
|
||||
test('explicit input fields override restored fields for downstream fingerprint validation', async () => {
|
||||
const result = await discoverStorageConfig(new FakeDb({ runs: [run()] }), {
|
||||
...input(), oldDomains: [], objectPrefix: 'migration/explicit', imagesOnly: false,
|
||||
targetPublicDomain: 'https://explicit.example', sourceMode: 'http', downloadBaseUrl: 'https://explicit.example/admin-api/infra/file'
|
||||
});
|
||||
assert.deepEqual(result.oldDomains, []);
|
||||
assert.equal(result.objectPrefix, 'migration/explicit');
|
||||
assert.equal(result.imagesOnly, false);
|
||||
assert.equal(result.targetPublicDomain, 'https://explicit.example');
|
||||
assert.equal(result.sourceMode, 'http');
|
||||
});
|
||||
|
||||
test('automatic resumed inputs and historical runs reject a changed default target', async () => {
|
||||
const options = { configs: [source('5'), target('30', 0), target('31')] };
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb(options), { ...input(), targetConfigId: '30', _autoDiscovery: true }), /原迁移目标不同/);
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb({ ...options, runs: [run()] }), input()), /原迁移目标不同/);
|
||||
});
|
||||
|
||||
test('rejects cross-server/schema inputs and restored history before accessing storage credentials', async () => {
|
||||
for (const wrong of [{ server: 'other', schema: database }, { server, schema: 'other' }]) {
|
||||
const db = new FakeDb();
|
||||
await assert.rejects(discoverStorageConfig(db, { ...input(), _databaseIdentity: wrong }), /身份.*不一致/);
|
||||
assert.equal(db.calls.length, 1);
|
||||
}
|
||||
const db = new FakeDb({ runs: [run('foreign', 'partial', { server: 'other' })] });
|
||||
await assert.rejects(discoverStorageConfig(db, input()), /身份.*不一致/);
|
||||
assert.ok(!db.calls.some(call => call.sql.includes('infra_file_config')));
|
||||
});
|
||||
|
||||
test('historical selection refuses ambiguity but honors an explicit existing or new runId', async () => {
|
||||
const unfinished = [run('one'), run('two')];
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb({ runs: unfinished }), input()), /多个未完成.*--run-id/);
|
||||
const chosen = await discoverStorageConfig(new FakeDb({ runs: unfinished }), { ...input(), runId: 'two' });
|
||||
assert.equal(chosen.runId, 'two');
|
||||
const fresh = await discoverStorageConfig(new FakeDb({ runs: unfinished }), { ...input(), runId: 'new-run' });
|
||||
assert.equal(fresh.runId, 'new-run');
|
||||
assert.equal(fresh.objectPrefix, 'migration/db-images');
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb({ runs: [run('one', 'complete'), run('two', 'rolled_back')] }), input()), /多个历史.*--run-id/);
|
||||
const single = await discoverStorageConfig(new FakeDb({ runs: [run('complete-run', 'complete')] }), input());
|
||||
assert.equal(single.runId, 'complete-run');
|
||||
const active = await discoverStorageConfig(new FakeDb({ runs: [run('old', 'complete'), run('active', 'needs-review')] }), input());
|
||||
assert.equal(active.runId, 'active');
|
||||
});
|
||||
|
||||
test('apply cannot resume a rollback while rollback does not require current storage configs or secrets', async () => {
|
||||
for (const status of ['rollback', 'rollback_partial', 'rolled_back']) {
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb({ runs: [run('old', status)] }), input(), { command: 'apply' }), /回滚.*--run-id/);
|
||||
}
|
||||
const db = new FakeDb({ runs: [run('old', 'rollback_partial')], configs: [] });
|
||||
const result = await discoverStorageConfig(db, input(), { command: 'rollback' });
|
||||
assert.equal(result.runId, 'old');
|
||||
assert.equal(result.targetConfigId, '30');
|
||||
assert.ok(!db.calls.some(call => call.sql.includes('infra_file_config')));
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb(), input(), { command: 'rollback' }), /未找到可回滚/);
|
||||
});
|
||||
|
||||
test('new content beyond a non-inventory run boundary blocks apply, including bigint boundaries', async () => {
|
||||
const options = { runs: [run()], maximum: '9007199254740994' };
|
||||
await assert.rejects(discoverStorageConfig(new FakeDb(options), input(), { command: 'apply' }), /新增内容/);
|
||||
const planned = await discoverStorageConfig(new FakeDb(options), input(), { command: 'plan' });
|
||||
assert.equal(planned.runId, 'original-run');
|
||||
const inventoryDb = new FakeDb({ ...options, runs: [run('still-building', 'inventory')] });
|
||||
const inventory = await discoverStorageConfig(inventoryDb, input(), { command: 'apply' });
|
||||
assert.equal(inventory.runId, 'still-building');
|
||||
assert.ok(!inventoryDb.calls.some(call => call.sql.includes('MAX(id)')));
|
||||
});
|
||||
|
||||
test('missing source contents produce a clear no-migration result without creating a run', async () => {
|
||||
const db = new FakeDb({ sourceIds: [] });
|
||||
await assert.rejects(discoverStorageConfig(db, input()), /无需迁移/);
|
||||
assert.ok(db.calls.every(call => /^SELECT\b/i.test(call.sql.trim())));
|
||||
});
|
||||
@@ -0,0 +1,146 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { setImmediate } from 'node:timers/promises';
|
||||
import { mapBounded } from '../lib/concurrency.mjs';
|
||||
|
||||
function deferred() {
|
||||
let resolve, reject;
|
||||
const promise = new Promise((res, rej) => { resolve = res; reject = rej; });
|
||||
return { promise, resolve, reject };
|
||||
}
|
||||
|
||||
test('bounded workers keep input order when tasks finish out of order', { timeout: 2000 }, async () => {
|
||||
const gates = Array.from({ length: 5 }, deferred);
|
||||
const started = [];
|
||||
let active = 0, maximum = 0;
|
||||
const pending = mapBounded(['a', 'b', 'c', 'd', 'e'], 2, async (value, index) => {
|
||||
started.push(index);
|
||||
maximum = Math.max(maximum, ++active);
|
||||
try { await gates[index].promise; return value.toUpperCase(); }
|
||||
finally { active--; }
|
||||
});
|
||||
assert.deepEqual(started, [0, 1]);
|
||||
gates[1].resolve();
|
||||
await setImmediate();
|
||||
assert.deepEqual(started, [0, 1, 2]);
|
||||
gates[2].resolve();
|
||||
await setImmediate();
|
||||
assert.deepEqual(started, [0, 1, 2, 3]);
|
||||
gates[0].resolve();
|
||||
await setImmediate();
|
||||
assert.deepEqual(started, [0, 1, 2, 3, 4]);
|
||||
gates[4].resolve();
|
||||
gates[3].resolve();
|
||||
assert.deepEqual(await pending, ['A', 'B', 'C', 'D', 'E']);
|
||||
assert.equal(maximum, 2);
|
||||
assert.equal(active, 0);
|
||||
});
|
||||
|
||||
test('failure stops new work and waits for all running workers before rejecting', { timeout: 2000 }, async () => {
|
||||
const gates = Array.from({ length: 5 }, deferred);
|
||||
const started = [];
|
||||
const finished = [];
|
||||
const failure = new Error('first verification failed');
|
||||
let settled = false;
|
||||
const pending = mapBounded([0, 1, 2, 3, 4], 3, async index => {
|
||||
started.push(index);
|
||||
try { await gates[index].promise; return index; }
|
||||
finally { finished.push(index); }
|
||||
});
|
||||
const outcome = pending.then(
|
||||
value => { settled = true; return { value }; },
|
||||
error => { settled = true; return { error }; },
|
||||
);
|
||||
assert.deepEqual(started, [0, 1, 2]);
|
||||
gates[1].reject(failure);
|
||||
await setImmediate();
|
||||
assert.deepEqual(started, [0, 1, 2]);
|
||||
assert.equal(settled, false);
|
||||
gates[0].resolve();
|
||||
await setImmediate();
|
||||
assert.deepEqual(started, [0, 1, 2]);
|
||||
assert.equal(settled, false);
|
||||
gates[2].resolve();
|
||||
assert.equal((await outcome).error, failure);
|
||||
assert.equal(settled, true);
|
||||
assert.deepEqual(finished.sort(), [0, 1, 2]);
|
||||
});
|
||||
|
||||
test('additional in-flight rejections are drained while preserving the first failure', { timeout: 2000 }, async () => {
|
||||
const gates = Array.from({ length: 3 }, deferred);
|
||||
const first = new Error('first');
|
||||
const second = new Error('second');
|
||||
let calls = 0, settled = false;
|
||||
const outcome = mapBounded([0, 1, 2, 3], 3, async index => {
|
||||
calls++;
|
||||
await gates[index].promise;
|
||||
return index;
|
||||
}).then(
|
||||
() => { settled = true; assert.fail('must reject'); },
|
||||
error => { settled = true; return error; },
|
||||
);
|
||||
gates[0].reject(first);
|
||||
await setImmediate();
|
||||
gates[1].reject(second);
|
||||
await setImmediate();
|
||||
assert.equal(settled, false);
|
||||
assert.equal(calls, 3);
|
||||
gates[2].resolve();
|
||||
assert.equal(await outcome, first);
|
||||
});
|
||||
|
||||
test('a synchronous failure stops scheduling immediately', async () => {
|
||||
const failure = new Error('synchronous failure');
|
||||
const started = [];
|
||||
await assert.rejects(mapBounded([0, 1, 2], 3, value => {
|
||||
started.push(value);
|
||||
throw failure;
|
||||
}), error => error === failure);
|
||||
assert.deepEqual(started, [0]);
|
||||
});
|
||||
|
||||
test('falsy rejection values still reject and never get treated as success', async () => {
|
||||
for (const failure of [undefined, null, false, 0, '']) {
|
||||
let rejected = false;
|
||||
await mapBounded([1], 1, () => Promise.reject(failure)).then(
|
||||
() => assert.fail('must reject'),
|
||||
error => { rejected = true; assert.equal(error, failure); },
|
||||
);
|
||||
assert.equal(rejected, true);
|
||||
}
|
||||
});
|
||||
|
||||
test('limit one executes serially and accepts synchronous return values', async () => {
|
||||
const order = [];
|
||||
assert.deepEqual(await mapBounded([2, 4, 6], 1, (value, index) => {
|
||||
order.push(index);
|
||||
return value * 2;
|
||||
}), [4, 8, 12]);
|
||||
assert.deepEqual(order, [0, 1, 2]);
|
||||
});
|
||||
|
||||
test('limits larger than the item count only start existing items', async () => {
|
||||
let calls = 0;
|
||||
const result = await mapBounded(['x', 'y'], 1000, async value => { calls++; return value; });
|
||||
assert.deepEqual(result, ['x', 'y']);
|
||||
assert.equal(calls, 2);
|
||||
});
|
||||
|
||||
test('empty input returns an empty array without invoking the worker', async () => {
|
||||
assert.deepEqual(await mapBounded([], 4, () => assert.fail('must not run')), []);
|
||||
});
|
||||
|
||||
test('invalid limits reject even for an empty input', async () => {
|
||||
for (const limit of [0, -1, 1.5, Infinity, NaN, '4', null, undefined, Number.MAX_SAFE_INTEGER + 1]) {
|
||||
await assert.rejects(mapBounded([], limit, () => {}), /正整数/);
|
||||
}
|
||||
});
|
||||
|
||||
test('input values are snapshotted before workers mutate the original collection', async () => {
|
||||
const input = ['first', 'second', 'third'];
|
||||
const result = await mapBounded(input, 1, async (value, index) => {
|
||||
if (index === 0) input.splice(1, 2, 'changed');
|
||||
return value;
|
||||
});
|
||||
assert.deepEqual(result, ['first', 'second', 'third']);
|
||||
});
|
||||
@@ -0,0 +1,428 @@
|
||||
import test, { mock } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createServer } from 'node:http';
|
||||
import { once } from 'node:events';
|
||||
import {
|
||||
aliases, baseUrl, bit, downloadUrl, encodePath, ensureObject, errorText, id,
|
||||
mimeType, normalizeConfig, objectKey, ossSettings, parseStorage, qualified,
|
||||
quote, readHttp, retry, sha256, verifyContent,
|
||||
} from '../lib/core.mjs';
|
||||
|
||||
const IMAGE = Buffer.from('89504e470d0a1a0a0001020304050607', 'hex');
|
||||
const ITEM = {
|
||||
configId: '9007199254740993', path: '头像/张 三+100%.PNG',
|
||||
size: IMAGE.length, sha256: sha256(IMAGE), type: 'image/png',
|
||||
};
|
||||
const HTTP_OPTIONS = { timeoutMs: 1000, maxFileBytes: 1024 };
|
||||
const sourceBase = 'http://127.0.0.1:48080/admin-api/infra/file';
|
||||
const baseConfig = overrides => ({
|
||||
runId: 'unit-test', mysql: { database: 'unit_test', user: 'test-only' },
|
||||
sourceConfigIds: ['1'], targetConfigId: '20', downloadBaseUrl: sourceBase,
|
||||
...overrides,
|
||||
});
|
||||
const storageRow = overrides => ({
|
||||
storage: 20,
|
||||
config: {
|
||||
endpoint: 'oss-cn-hangzhou.aliyuncs.com', bucket: 'test-only-bucket',
|
||||
accessKey: 'test-only-access-key', accessSecret: 'test-only-access-secret',
|
||||
...overrides,
|
||||
},
|
||||
});
|
||||
const noSuchKey = () => Object.assign(new Error('missing'), { code: 'NoSuchKey', status: 404 });
|
||||
const head = bytes => ({ res: { headers: { 'content-length': String(bytes.length) } } });
|
||||
const storedObject = bytes => ({
|
||||
head: mock.fn(async () => head(bytes)),
|
||||
get: mock.fn(async () => ({ content: bytes })),
|
||||
put: mock.fn(async () => { throw new Error('unexpected write'); }),
|
||||
});
|
||||
|
||||
async function localServer(t, handler) {
|
||||
const server = createServer(handler);
|
||||
server.listen(0, '127.0.0.1');
|
||||
await once(server, 'listening');
|
||||
t.after(async () => {
|
||||
const closed = new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve()));
|
||||
server.closeAllConnections();
|
||||
await closed;
|
||||
});
|
||||
return 'http://127.0.0.1:' + server.address().port;
|
||||
}
|
||||
|
||||
test('download URL preserves Chinese, space, plus, percent, question mark and fragment characters', () => {
|
||||
const path = '头像/张 三+100%?#.png';
|
||||
const encoded = '%E5%A4%B4%E5%83%8F/%E5%BC%A0%20%E4%B8%89%2B100%25%3F%23.png';
|
||||
assert.equal(encodePath(path), encoded);
|
||||
const url = downloadUrl(sourceBase + '///', ITEM.configId, path);
|
||||
assert.equal(url, sourceBase + '/' + ITEM.configId + '/get/' + encoded);
|
||||
assert.equal(decodeURIComponent(new URL(url).pathname.split('/get/')[1]), path);
|
||||
assert.equal(new URL(url).search, '');
|
||||
assert.equal(new URL(url).hash, '');
|
||||
});
|
||||
|
||||
test('encoded-looking path text is encoded exactly once without decoding the source path', () => {
|
||||
assert.equal(encodePath('%2F/%20+%.png'), '%252F/%2520%2B%25.png');
|
||||
assert.equal(decodeURIComponent(encodePath('%2F/%20+%.png')), '%2F/%20+%.png');
|
||||
});
|
||||
|
||||
test('ambiguous or normalized path segments are rejected', () => {
|
||||
for (const path of ['', '/image.png', 'a//b.png', 'a/', './a.png', 'a/../b.png', 'a/./b.png', 'a\\b.png']) {
|
||||
assert.throws(() => encodePath(path), /路径/, path);
|
||||
}
|
||||
});
|
||||
|
||||
test('URL base validates scheme and rejects credentials, queries and fragments', () => {
|
||||
assert.equal(baseUrl('https://images.example.test/prefix///'), 'https://images.example.test/prefix');
|
||||
for (const url of ['ftp://example.test', 'file:///tmp/image', 'https://user:pass@example.test', 'https://example.test/?key=value', 'https://example.test/#part']) {
|
||||
assert.throws(() => baseUrl(url), /HTTP/);
|
||||
}
|
||||
});
|
||||
|
||||
test('bigint IDs remain exact strings and unsafe numeric IDs are rejected', () => {
|
||||
assert.equal(id('9007199254740993'), '9007199254740993');
|
||||
assert.equal(id(9007199254740993n), '9007199254740993');
|
||||
assert.equal(id(42), '42');
|
||||
assert.throws(() => id(Number('9007199254740993')), /安全整数/);
|
||||
for (const value of [0, -1, 1.5, '01', '1e3', ' 1', '', null, undefined, true]) {
|
||||
assert.throws(() => id(value), /正整数|安全整数/, String(value));
|
||||
}
|
||||
});
|
||||
|
||||
test('MySQL BIT values distinguish zero buffers from true values', () => {
|
||||
for (const value of [Buffer.from([0]), Buffer.alloc(0), false, 0, '0', null, undefined]) assert.equal(bit(value), false);
|
||||
for (const value of [Buffer.from([1]), true, 1, '1']) assert.equal(bit(value), true);
|
||||
});
|
||||
|
||||
test('configuration normalizes IDs and schemas without changing bigint precision', () => {
|
||||
const result = normalizeConfig(baseConfig({
|
||||
sourceConfigIds: [1, '1', '9007199254740993'],
|
||||
referenceSchemas: ['chat', 'unit_test', 'chat'],
|
||||
oldDomains: ['https://old.example.test///'],
|
||||
}));
|
||||
assert.deepEqual(result.sourceConfigIds, ['1', '9007199254740993']);
|
||||
assert.deepEqual(result.referenceSchemas, ['unit_test', 'chat']);
|
||||
assert.deepEqual(result.oldDomains, ['https://old.example.test']);
|
||||
assert.equal(result.downloadBaseUrl, sourceBase);
|
||||
});
|
||||
|
||||
test('configuration rejects same source and target, invalid modes and unsafe batch limits', () => {
|
||||
for (const overrides of [
|
||||
{ targetConfigId: '1' }, { sourceMode: 'unknown' }, { targetUrlMode: 'unknown' },
|
||||
{ batchSize: 0 }, { batchSize: 1001 }, { retries: 0 }, { retries: 11 },
|
||||
{ maxFileBytes: 1.5 }, { timeoutMs: 0 }, { objectPrefix: '../unsafe' },
|
||||
]) assert.throws(() => normalizeConfig(baseConfig(overrides)));
|
||||
});
|
||||
|
||||
test('database source and direct target do not require a download address', () => {
|
||||
const result = normalizeConfig(baseConfig({ sourceMode: 'database', downloadBaseUrl: undefined }));
|
||||
assert.equal(result.sourceMode, 'database');
|
||||
});
|
||||
|
||||
test('OSS settings use the actual Java configuration field names and support JSON storage', () => {
|
||||
const row = storageRow();
|
||||
row.config = JSON.stringify(row.config);
|
||||
assert.equal(parseStorage(row).accessKey, 'test-only-access-key');
|
||||
assert.deepEqual(ossSettings(row, {}), {
|
||||
endpoint: 'https://oss-cn-hangzhou.aliyuncs.com', bucket: 'test-only-bucket',
|
||||
accessKeyId: 'test-only-access-key', accessKeySecret: 'test-only-access-secret',
|
||||
publicDomain: 'https://test-only-bucket.oss-cn-hangzhou.aliyuncs.com',
|
||||
});
|
||||
});
|
||||
|
||||
test('OSS internal endpoint gets a public fallback domain and explicit domain takes precedence', () => {
|
||||
const row = storageRow({ endpoint: 'https://oss-cn-hangzhou-internal.aliyuncs.com' });
|
||||
const result = ossSettings(row, {});
|
||||
assert.equal(result.endpoint, 'https://oss-cn-hangzhou-internal.aliyuncs.com');
|
||||
assert.equal(result.publicDomain, 'https://test-only-bucket.oss-cn-hangzhou.aliyuncs.com');
|
||||
assert.equal(ossSettings(storageRow({ domain: 'https://cdn.example.test/' }), {}).publicDomain, 'https://cdn.example.test');
|
||||
assert.equal(ossSettings(row, { targetPublicDomain: 'https://override.example.test/assets/' }).publicDomain, 'https://override.example.test/assets');
|
||||
});
|
||||
|
||||
test('OSS settings reject wrong storage type, non-OSS services, bucket endpoints and missing fields', () => {
|
||||
assert.throws(() => ossSettings({ ...storageRow(), storage: 1 }, {}), /storage=20/);
|
||||
for (const endpoint of [
|
||||
'http://127.0.0.1:9000', 's3.amazonaws.com', 'bucket.oss-cn-hangzhou.aliyuncs.com',
|
||||
'oss-cn-hangzhou.aliyuncs.com.evil.test', 'https://oss-cn-hangzhou.aliyuncs.com/path',
|
||||
'https://user:pass@oss-cn-hangzhou.aliyuncs.com', 'https://oss-cn-hangzhou.aliyuncs.com/?x=1',
|
||||
]) assert.throws(() => ossSettings(storageRow({ endpoint }), {}), /endpoint/);
|
||||
for (const field of ['bucket', 'accessKey', 'accessSecret']) {
|
||||
assert.throws(() => ossSettings(storageRow({ [field]: '' }), {}), /缺少/);
|
||||
}
|
||||
});
|
||||
|
||||
test('object keys separate config, source path and content while keeping a safe extension', () => {
|
||||
const config = { objectPrefix: 'migration/test' };
|
||||
const key = objectKey(config, ITEM);
|
||||
assert.equal(key, 'migration/test/' + ITEM.configId + '/' + sha256(ITEM.path) + '/' + ITEM.sha256 + '.png');
|
||||
assert.notEqual(key, objectKey(config, { ...ITEM, configId: '2' }));
|
||||
assert.notEqual(key, objectKey(config, { ...ITEM, path: 'other.PNG' }));
|
||||
assert.notEqual(key, objectKey(config, { ...ITEM, sha256: 'b'.repeat(64) }));
|
||||
});
|
||||
|
||||
test('aliases retain unrelated stored URLs and only route variants resolving to the source path', () => {
|
||||
const row = { path: ITEM.path, url: 'https://special.example.test/persisted-image' };
|
||||
const result = aliases(ITEM, [row, row], 'https://old.example.test', {
|
||||
oldDomains: ['https://older.example.test'], downloadBaseUrl: sourceBase,
|
||||
});
|
||||
assert.equal(result.filter(value => value === row.url).length, 1);
|
||||
assert.ok(!result.includes('https://old.example.test/admin-api/infra/file/' + ITEM.configId + '/get/' + ITEM.path));
|
||||
assert.ok(result.includes('https://older.example.test/admin-api/infra/file/' + ITEM.configId + '/get/' + encodePath(ITEM.path)));
|
||||
assert.ok(result.includes(downloadUrl(sourceBase, ITEM.configId, ITEM.path)));
|
||||
});
|
||||
|
||||
test('plus and space source files cannot claim each other through raw plus aliases', () => {
|
||||
const domain = 'https://old.example.test';
|
||||
const prefix = domain + '/admin-api/infra/file/1/get/';
|
||||
const config = { oldDomains: [], downloadBaseUrl: sourceBase };
|
||||
const plus = aliases({ configId: '1', path: 'a+b.png' }, [
|
||||
{ path: 'a+b.png', url: prefix + 'a+b.png' },
|
||||
{ path: 'a+b.png', url: prefix + 'a%2Bb.png' },
|
||||
], domain, config);
|
||||
const space = aliases({ configId: '1', path: 'a b.png' }, [
|
||||
{ path: 'a b.png', url: prefix + 'a+b.png' },
|
||||
], domain, config);
|
||||
assert.ok(!plus.includes(prefix + 'a+b.png'));
|
||||
assert.ok(plus.includes(prefix + 'a%2Bb.png'));
|
||||
assert.ok(space.includes(prefix + 'a+b.png'));
|
||||
assert.ok(space.includes(prefix + 'a b.png'));
|
||||
assert.ok(space.includes(prefix + 'a%20b.png'));
|
||||
assert.ok(!space.includes(prefix + 'a%2Bb.png'));
|
||||
assert.ok(space.includes(sourceBase + '/1/get/a+b.png'));
|
||||
assert.deepEqual(plus.filter(url => space.includes(url)), []);
|
||||
});
|
||||
|
||||
test('percent-encoded-looking filenames keep their meaning after exactly one route decode', () => {
|
||||
const domain = 'https://old.example.test';
|
||||
const prefix = domain + '/admin-api/infra/file/1/get/';
|
||||
const config = { oldDomains: [] };
|
||||
const literal = aliases({ configId: '1', path: 'a%2Fb.png' }, [
|
||||
{ path: 'a%2Fb.png', url: prefix + 'a%2Fb.png' },
|
||||
], domain, config);
|
||||
const directory = aliases({ configId: '1', path: 'a/b.png' }, [
|
||||
{ path: 'a/b.png', url: prefix + 'a%2Fb.png' },
|
||||
], domain, config);
|
||||
assert.ok(!literal.includes(prefix + 'a%2Fb.png'));
|
||||
assert.ok(literal.includes(prefix + 'a%252Fb.png'));
|
||||
assert.ok(directory.includes(prefix + 'a%2Fb.png'));
|
||||
assert.ok(directory.includes(prefix + 'a/b.png'));
|
||||
assert.deepEqual(literal.filter(url => directory.includes(url)), []);
|
||||
});
|
||||
|
||||
test('raw query, fragment and unverifiable percent paths are excluded from source aliases', () => {
|
||||
const domain = 'https://old.example.test';
|
||||
const prefix = domain + '/admin-api/infra/file/1/get/';
|
||||
for (const path of ['question?.png', 'fragment#.png', 'literal%.png', 'a%20b.png']) {
|
||||
const result = aliases({ configId: '1', path }, [{ path, url: prefix + path }], domain, { oldDomains: [] });
|
||||
assert.ok(!result.includes(prefix + path), path);
|
||||
assert.ok(result.includes(prefix + encodePath(path)), path);
|
||||
}
|
||||
});
|
||||
|
||||
test('route validation accepts SQL-equivalent spellings and ignores genuine query parameters', () => {
|
||||
const domain = 'https://old.example.test';
|
||||
const prefix = domain + '/admin-api/infra/file/1/get/';
|
||||
const alternate = 'photo a.JPG';
|
||||
const result = aliases({ configId: '1', path: 'Photo A.jpg' }, [
|
||||
{ path: alternate, url: prefix + 'photo+a.JPG?download=1#preview' },
|
||||
{ path: alternate, url: domain + '/admin-api/infra/file/2/get/photo+a.JPG' },
|
||||
{ path: alternate, url: prefix + 'unrelated.png' },
|
||||
], domain, { oldDomains: [] });
|
||||
assert.ok(result.includes(prefix + 'photo+a.JPG?download=1#preview'));
|
||||
assert.ok(result.includes(prefix + 'photo+a.JPG'));
|
||||
assert.ok(result.includes(prefix + 'Photo+A.jpg'));
|
||||
assert.ok(!result.includes(domain + '/admin-api/infra/file/2/get/photo+a.JPG'));
|
||||
assert.ok(!result.includes(prefix + 'unrelated.png'));
|
||||
});
|
||||
|
||||
test('MIME detection prefers stored image type, then extension, then other stored type', () => {
|
||||
assert.equal(mimeType('renamed.jpg', [{ type: 'image/png' }]), 'image/png');
|
||||
assert.equal(mimeType('image.WEBP', []), 'image/webp');
|
||||
assert.equal(mimeType('no-extension', [{ type: 'application/pdf' }]), 'application/pdf');
|
||||
assert.equal(mimeType('no-extension', []), 'application/octet-stream');
|
||||
});
|
||||
|
||||
test('SQL identifiers are quoted and error URLs are redacted', () => {
|
||||
assert.equal(quote('strange\u0060name'), '\u0060strange\u0060\u0060name\u0060');
|
||||
assert.equal(qualified('schema', 'table'), '\u0060schema\u0060.\u0060table\u0060');
|
||||
const redacted = errorText(new Error('failed https://example.test/?secret=private afterward'));
|
||||
assert.equal(redacted, 'failed [URL] afterward');
|
||||
assert.ok(errorText(new Error('x'.repeat(500))).length <= 300);
|
||||
});
|
||||
|
||||
test('HTTP download sends the intended encoded route and preserves binary bytes', async t => {
|
||||
let observedUrl;
|
||||
const base = await localServer(t, (request, response) => {
|
||||
observedUrl = request.url;
|
||||
response.writeHead(200, { 'Content-Type': 'image/png', 'Content-Length': IMAGE.length });
|
||||
response.end(IMAGE);
|
||||
});
|
||||
const route = downloadUrl(base + '/admin-api/infra/file', ITEM.configId, ITEM.path);
|
||||
const bytes = await readHttp(route, HTTP_OPTIONS, IMAGE.length);
|
||||
assert.deepEqual(bytes, IMAGE);
|
||||
assert.equal(observedUrl, '/admin-api/infra/file/' + ITEM.configId + '/get/' + encodePath(ITEM.path));
|
||||
assert.doesNotThrow(() => verifyContent(bytes, ITEM));
|
||||
});
|
||||
|
||||
test('HTTP errors and redirects are rejected rather than uploaded', async t => {
|
||||
const base = await localServer(t, (request, response) => {
|
||||
if (request.url === '/redirect') response.writeHead(302, { Location: '/image' });
|
||||
else response.writeHead(404);
|
||||
response.end();
|
||||
});
|
||||
await assert.rejects(readHttp(base + '/missing', HTTP_OPTIONS), /HTTP 404/);
|
||||
await assert.rejects(readHttp(base + '/redirect', HTTP_OPTIONS));
|
||||
});
|
||||
|
||||
test('HTTP 200 HTML or JSON with the same byte count fails hash verification', async t => {
|
||||
const base = await localServer(t, (request, response) => {
|
||||
const fake = Buffer.alloc(IMAGE.length, ' ');
|
||||
fake.write(request.url === '/html' ? '<html>error' : '{"error":true}');
|
||||
response.writeHead(200, { 'Content-Type': request.url === '/html' ? 'text/html' : 'application/json' });
|
||||
response.end(fake);
|
||||
});
|
||||
for (const path of ['/html', '/json']) {
|
||||
const bytes = await readHttp(base + path, HTTP_OPTIONS, IMAGE.length);
|
||||
assert.equal(bytes.length, IMAGE.length);
|
||||
assert.throws(() => verifyContent(bytes, ITEM), /SHA-256/);
|
||||
}
|
||||
});
|
||||
|
||||
test('short downloads fail expected length validation', async t => {
|
||||
const base = await localServer(t, (_request, response) => response.end(IMAGE.subarray(0, 4)));
|
||||
const bytes = await readHttp(base, HTTP_OPTIONS, IMAGE.length);
|
||||
assert.throws(() => verifyContent(bytes, ITEM), /长度/);
|
||||
});
|
||||
|
||||
test('HTTP declared length exceeding either cap is rejected', async t => {
|
||||
const base = await localServer(t, (_request, response) => {
|
||||
response.writeHead(200, { 'Content-Length': IMAGE.length });
|
||||
response.end(IMAGE);
|
||||
});
|
||||
await assert.rejects(readHttp(base, { ...HTTP_OPTIONS, maxFileBytes: 4 }), /长度超过限制/);
|
||||
await assert.rejects(readHttp(base, HTTP_OPTIONS, 4), /长度超过限制/);
|
||||
});
|
||||
|
||||
test('chunked HTTP bodies cannot bypass actual byte limits', async t => {
|
||||
const base = await localServer(t, (_request, response) => {
|
||||
response.writeHead(200, { 'Transfer-Encoding': 'chunked' });
|
||||
response.write(IMAGE.subarray(0, 8));
|
||||
response.end(IMAGE.subarray(8));
|
||||
});
|
||||
await assert.rejects(readHttp(base, { ...HTTP_OPTIONS, maxFileBytes: 4 }), /长度超过限制/);
|
||||
await assert.rejects(readHttp(base, HTTP_OPTIONS, 4), /长度超过限制/);
|
||||
});
|
||||
|
||||
test('HTTP requests time out when the source never responds', async t => {
|
||||
const base = await localServer(t, () => {});
|
||||
await assert.rejects(readHttp(base, { ...HTTP_OPTIONS, timeoutMs: 30 }));
|
||||
});
|
||||
|
||||
test('content verification checks both length and hash', () => {
|
||||
assert.doesNotThrow(() => verifyContent(IMAGE, { ...ITEM, size: String(IMAGE.length) }));
|
||||
assert.throws(() => verifyContent(IMAGE, { ...ITEM, size: IMAGE.length + 1 }), /长度/);
|
||||
assert.throws(() => verifyContent(IMAGE, { ...ITEM, sha256: '0'.repeat(64) }), /SHA-256/);
|
||||
});
|
||||
|
||||
test('existing identical OSS objects are read and verified without upload', async () => {
|
||||
const oss = storedObject(IMAGE);
|
||||
await ensureObject(oss, 'target/key.png', IMAGE, ITEM);
|
||||
assert.equal(oss.put.mock.callCount(), 0);
|
||||
assert.equal(oss.get.mock.callCount(), 1);
|
||||
assert.equal(oss.get.mock.calls[0].arguments[0], 'target/key.png');
|
||||
});
|
||||
|
||||
test('existing OSS objects with a different length are not overwritten', async () => {
|
||||
const oss = storedObject(Buffer.from('wrong length'));
|
||||
await assert.rejects(ensureObject(oss, 'target/key.png', IMAGE, ITEM), /禁止覆盖/);
|
||||
assert.equal(oss.put.mock.callCount(), 0);
|
||||
assert.equal(oss.get.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
test('existing same-length OSS objects with different bytes are not overwritten', async () => {
|
||||
const oss = storedObject(Buffer.alloc(IMAGE.length, 1));
|
||||
await assert.rejects(ensureObject(oss, 'target/key.png', IMAGE, ITEM), /SHA-256/);
|
||||
assert.equal(oss.put.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
test('missing OSS objects upload exact bytes with MIME and overwrite prevention', async () => {
|
||||
let saved;
|
||||
const oss = {
|
||||
head: mock.fn(async () => { if (!saved) throw noSuchKey(); return head(saved); }),
|
||||
put: mock.fn(async (_key, bytes) => { saved = Buffer.from(bytes); }),
|
||||
get: mock.fn(async () => ({ content: saved })),
|
||||
};
|
||||
await ensureObject(oss, 'target/key.png', IMAGE, ITEM);
|
||||
assert.equal(oss.put.mock.callCount(), 1);
|
||||
assert.deepEqual(oss.put.mock.calls[0].arguments, [
|
||||
'target/key.png', IMAGE, { headers: { 'Content-Type': 'image/png', 'x-oss-forbid-overwrite': 'true' } },
|
||||
]);
|
||||
assert.deepEqual(saved, IMAGE);
|
||||
});
|
||||
|
||||
test('OSS HEAD authorization failures do not cause a speculative upload', async () => {
|
||||
const denied = Object.assign(new Error('denied'), { code: 'AccessDenied', status: 403 });
|
||||
const oss = { head: mock.fn(async () => { throw denied; }), put: mock.fn(), get: mock.fn() };
|
||||
await assert.rejects(ensureObject(oss, 'key', IMAGE, ITEM), error => error === denied);
|
||||
assert.equal(oss.put.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
test('upload competition is accepted only after verifying the winner content', async () => {
|
||||
for (const sameContent of [true, false]) {
|
||||
const winner = sameContent ? IMAGE : Buffer.alloc(IMAGE.length, 2);
|
||||
let appeared = false;
|
||||
const oss = {
|
||||
head: mock.fn(async () => { if (!appeared) throw noSuchKey(); return head(winner); }),
|
||||
put: mock.fn(async () => {
|
||||
appeared = true;
|
||||
throw Object.assign(new Error('another upload won'), { code: 'FileAlreadyExists', status: 409 });
|
||||
}),
|
||||
get: mock.fn(async () => ({ content: winner })),
|
||||
};
|
||||
if (sameContent) await ensureObject(oss, 'key', IMAGE, ITEM);
|
||||
else await assert.rejects(ensureObject(oss, 'key', IMAGE, ITEM), /SHA-256/);
|
||||
assert.equal(oss.put.mock.callCount(), 1);
|
||||
assert.equal(oss.get.mock.callCount(), 1);
|
||||
}
|
||||
});
|
||||
|
||||
test('failed or corrupted OSS uploads never report success', async () => {
|
||||
const offline = Object.assign(new Error('offline'), { code: 'ConnectionTimeout' });
|
||||
const failed = {
|
||||
head: mock.fn(async () => { throw noSuchKey(); }),
|
||||
put: mock.fn(async () => { throw offline; }), get: mock.fn(),
|
||||
};
|
||||
await assert.rejects(ensureObject(failed, 'key', IMAGE, ITEM), error => error === offline);
|
||||
assert.equal(failed.get.mock.callCount(), 0);
|
||||
let uploaded = false;
|
||||
const corrupt = {
|
||||
head: mock.fn(async () => { if (!uploaded) throw noSuchKey(); return head(IMAGE); }),
|
||||
put: mock.fn(async () => { uploaded = true; }),
|
||||
get: mock.fn(async () => ({ content: Buffer.alloc(IMAGE.length) })),
|
||||
};
|
||||
await assert.rejects(ensureObject(corrupt, 'key', IMAGE, ITEM), /SHA-256/);
|
||||
});
|
||||
|
||||
test('retry stops on success and uses capped exponential waits', async () => {
|
||||
let calls = 0;
|
||||
const waits = [];
|
||||
const value = await retry(async () => {
|
||||
if (++calls < 7) throw new Error('transient');
|
||||
return 'done';
|
||||
}, 7, async ms => { waits.push(ms); });
|
||||
assert.equal(value, 'done');
|
||||
assert.equal(calls, 7);
|
||||
assert.deepEqual(waits, [1000, 2000, 4000, 5000, 5000, 5000]);
|
||||
});
|
||||
|
||||
test('retry preserves the last failure and never exceeds the configured attempt count', async () => {
|
||||
let calls = 0;
|
||||
const waits = [];
|
||||
const failure = new Error('permanent');
|
||||
await assert.rejects(retry(async () => { calls++; throw failure; }, 3, async ms => waits.push(ms)), error => error === failure);
|
||||
assert.equal(calls, 3);
|
||||
assert.deepEqual(waits, [1000, 2000]);
|
||||
const first = mock.fn(async () => 'first');
|
||||
const wait = mock.fn();
|
||||
assert.equal(await retry(first, 3, wait), 'first');
|
||||
assert.equal(first.mock.callCount(), 1);
|
||||
assert.equal(wait.mock.callCount(), 0);
|
||||
});
|
||||
@@ -0,0 +1,241 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createServer } from 'node:http';
|
||||
import { once } from 'node:events';
|
||||
import { encodePath, normalizeConfig } from '../lib/core.mjs';
|
||||
import { applyChange } from '../lib/journal.mjs';
|
||||
import { discoverStorageConfig } from '../lib/auto-storage.mjs';
|
||||
import { loadSourceIndex } from '../lib/source-index.mjs';
|
||||
import { loadSettings, runApply, runPlan, runRollback } from '../migrate.mjs';
|
||||
|
||||
// Only point this at the disposable test MySQL container: this suite recreates
|
||||
// its dedicated codex_oss_migration_test schema, never the application database.
|
||||
test('MySQL migration, resume, exact reference replacement and rollback', {
|
||||
skip: !process.env.MIGRATION_TEST_DB_PORT, timeout: 60000
|
||||
}, async t => {
|
||||
const { default: mysql } = await import('mysql2/promise');
|
||||
const db = await mysql.createConnection({ host: '127.0.0.1', port: Number(process.env.MIGRATION_TEST_DB_PORT), user: 'root', password: '', supportBigNumbers: true, bigNumberStrings: true, jsonStrings: true, dateStrings: true });
|
||||
const objects = new Map();
|
||||
let putCount = 0, failPut = false, sourceDownloads = 0;
|
||||
const oss = {
|
||||
async head(key) { if (!objects.has(key)) throw Object.assign(new Error('missing'), { code: 'NoSuchKey', status: 404 }); return { res: { headers: { 'content-length': String(objects.get(key).length) } } }; },
|
||||
async get(key) { return { content: Buffer.from(objects.get(key)) }; },
|
||||
async put(key, bytes, options) {
|
||||
assert.equal(options.headers['x-oss-forbid-overwrite'], 'true');
|
||||
if (failPut) throw Object.assign(new Error('test failure'), { code: 'TestOssUnavailable' });
|
||||
assert.ok(!objects.has(key)); objects.set(key, Buffer.from(bytes)); putCount++;
|
||||
}
|
||||
};
|
||||
const server = createServer(async (request, response) => {
|
||||
try {
|
||||
const url = new URL(request.url, 'http://localhost');
|
||||
if (url.pathname.startsWith('/objects/')) {
|
||||
const bytes = objects.get(decodeURIComponent(url.pathname.slice('/objects/'.length)));
|
||||
if (!bytes) { response.writeHead(404).end(); return; }
|
||||
response.end(bytes); return;
|
||||
}
|
||||
const prefix = '/admin-api/infra/file/5/get/';
|
||||
if (url.pathname.startsWith(prefix)) {
|
||||
sourceDownloads++;
|
||||
const path = decodeURIComponent(url.pathname.slice(prefix.length).replaceAll('+', ' '));
|
||||
const [[row]] = await db.execute('SELECT content FROM infra_file_content WHERE config_id=5 AND path=? AND deleted=0 ORDER BY id DESC LIMIT 1', [path]);
|
||||
if (row) response.end(row.content); else response.writeHead(404).end();
|
||||
return;
|
||||
}
|
||||
response.writeHead(404).end();
|
||||
} catch { response.writeHead(500).end(); }
|
||||
});
|
||||
server.listen(0, '127.0.0.1'); await once(server, 'listening');
|
||||
const origin = `http://127.0.0.1:${server.address().port}`;
|
||||
const schema = 'codex_oss_migration_test';
|
||||
let config;
|
||||
const firstPath = '头像/张 三+100%.png';
|
||||
const firstUrl = `${origin}/admin-api/infra/file/5/get/${encodePath(firstPath)}`;
|
||||
const orphanUrl = `${origin}/admin-api/infra/file/5/get/orphan.png`;
|
||||
const caseUrl = `${origin}/admin-api/infra/file/5/get/Case.png`;
|
||||
const originals = [Buffer.from('first current image'), Buffer.from('second orphan image'), Buffer.from('case latest image')];
|
||||
try {
|
||||
await db.query(`DROP DATABASE IF EXISTS ${schema}`);
|
||||
await db.query(`CREATE DATABASE ${schema} CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci`);
|
||||
await db.changeUser({ database: schema });
|
||||
await db.query('CREATE TABLE infra_file_config (id BIGINT PRIMARY KEY,storage INT,master BIT(1),config TEXT,deleted BIT(1) DEFAULT 0) ENGINE=InnoDB');
|
||||
await db.query('CREATE TABLE infra_file_content (id BIGINT PRIMARY KEY,config_id BIGINT,path VARCHAR(512),content MEDIUMBLOB,deleted BIT(1) DEFAULT 0) ENGINE=InnoDB');
|
||||
await db.query('CREATE TABLE infra_file (id BIGINT PRIMARY KEY,config_id BIGINT,path VARCHAR(512),url VARCHAR(1024),type VARCHAR(128),size INT,deleted BIT(1) DEFAULT 0,update_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP) ENGINE=InnoDB');
|
||||
await db.query('CREATE TABLE member_user (id BIGINT PRIMARY KEY,avatar VARCHAR(1024),wechat_qr_code VARCHAR(255),extra_json JSON,update_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP) ENGINE=InnoDB');
|
||||
await db.query('CREATE TABLE notes (id INT PRIMARY KEY,body TEXT) ENGINE=InnoDB');
|
||||
await db.execute('INSERT INTO infra_file_config VALUES (5,1,0,?,0),(30,20,1,?,0)', [JSON.stringify({ domain: origin }), JSON.stringify({ endpoint: 'oss-cn-hangzhou.aliyuncs.com', bucket: 'test', accessKey: 'test-key', accessSecret: 'test-secret', domain: `${origin}/objects` })]);
|
||||
const records = [
|
||||
['9007199254740992', 'Case.png', Buffer.from('case old image'), 0],
|
||||
['9007199254740993', firstPath, Buffer.from('first old image'), 0],
|
||||
['9007199254740994', firstPath, originals[0], 0],
|
||||
['9007199254740995', firstPath, Buffer.from('deleted newest'), 1],
|
||||
['9007199254740996', 'orphan.png', originals[1], 0],
|
||||
['9007199254740997', 'case.PNG', originals[2], 0]
|
||||
];
|
||||
for (const [contentId, path, content, deleted] of records) await db.execute('INSERT INTO infra_file_content VALUES (?,5,?,?,?)', [contentId, path, content, deleted]);
|
||||
for (const [fileId, path, url] of [['9007199254741001', firstPath, firstUrl], ['9007199254741002', 'Case.png', caseUrl], ['9007199254741003', 'case.PNG', caseUrl.replace('Case.png', 'case.PNG')]]) {
|
||||
await db.execute("INSERT INTO infra_file (id,config_id,path,url,type,size,update_time) VALUES (?,5,?,?,'image/png',1,'2020-01-01 00:00:00')", [fileId, path, url]);
|
||||
}
|
||||
const payload = JSON.stringify({ a: [firstUrl, orphanUrl], nested: { image: caseUrl }, outside: 'https://outside.example/image.png' });
|
||||
await db.execute("INSERT INTO member_user VALUES (9007199254741101,?,?,?,'2020-01-01 00:00:00')", [firstUrl, orphanUrl, payload]);
|
||||
await db.execute('INSERT INTO notes VALUES (1,?)', [`<img src="${caseUrl}">${orphanUrl},${caseUrl}`]);
|
||||
config = normalizeConfig({ runId: 'integration', mysql: { database: schema, user: 'root' }, sourceConfigIds: ['5'], targetConfigId: '30', downloadBaseUrl: `${origin}/admin-api/infra/file`, batchSize: 1, retries: 1 });
|
||||
const settings = await loadSettings(db, config);
|
||||
|
||||
await t.test('automatic discovery uses the existing database and unique OSS without configuration writes', async () => {
|
||||
const automatic = await discoverStorageConfig(db, { mysql: config.mysql, _autoDiscovery: true, _projectDownloadBaseUrl: config.downloadBaseUrl });
|
||||
assert.deepEqual(automatic.sourceConfigIds, ['5']); assert.equal(automatic.targetConfigId, '30');
|
||||
assert.equal(automatic.sourceMode, 'database'); assert.ok(automatic.runId.startsWith('db-images-'));
|
||||
assert.ok(automatic.oldDomains.includes(origin)); assert.equal(objects.size, 0);
|
||||
const [tables] = await db.query("SHOW TABLES LIKE 'infra_file_oss_migration_%'"); assert.equal(tables.length, 0);
|
||||
});
|
||||
|
||||
await t.test('plan is read-only and uses latest active content with bigint/collation semantics', async () => {
|
||||
const statements = [];
|
||||
const readOnlyDb = {
|
||||
execute(sql, params) { statements.push(sql); return db.execute(sql, params); },
|
||||
query(sql, params) { statements.push(sql); return db.query(sql, params); }
|
||||
};
|
||||
const plan = await runPlan(readOnlyDb, config, settings);
|
||||
assert.equal(plan.status, 'planned'); assert.equal(plan.candidates, 3); assert.equal(plan.withoutFileRecord, 1);
|
||||
assert.equal(plan.bytes, null); assert.equal(plan.estimatedBytes, 2); assert.equal(plan.sizePending, 1); assert.equal(plan.contentVerified, false);
|
||||
assert.ok(statements.every(sql => /^\s*SELECT/i.test(sql)));
|
||||
assert.ok(statements.every(sql => !/\b(?:SHA2|OCTET_LENGTH|content)\b|NOT EXISTS/i.test(sql)), 'plan must never scan image bytes or use per-file correlated subqueries');
|
||||
assert.equal(plan.references.changedRows, 2); assert.equal(objects.size, 0);
|
||||
const [rows] = await db.query("SHOW TABLES LIKE 'infra_file_oss_migration_%'"); assert.equal(rows.length, 0);
|
||||
});
|
||||
|
||||
await t.test('metadata upper bound excludes newer versions without misclassifying their files as missing', async () => {
|
||||
const index = await loadSourceIndex(db, config, { maxId: '9007199254740996' });
|
||||
assert.equal(index.groups.length, 2); assert.equal(index.missingFiles.length, 0);
|
||||
assert.equal(index.groups[0].path, firstPath); assert.equal(index.groups[1].path, 'orphan.png');
|
||||
const full = await loadSourceIndex(db, config);
|
||||
assert.deepEqual(full.groups[2].spellings.slice().sort(), ['Case.png', 'case.PNG'].sort());
|
||||
assert.equal(full.groups[2].files.length, 2);
|
||||
});
|
||||
|
||||
await t.test('limited trial, resume and repeated apply preserve originals and journals', async () => {
|
||||
const trial = await runApply(db, config, settings, 1, () => oss);
|
||||
assert.equal(trial.status, 'partial'); assert.equal(trial.items.switched, 1); assert.equal(objects.size, 1);
|
||||
const resumedConfig = await discoverStorageConfig(db, { mysql: config.mysql, _autoDiscovery: true }, { command: 'apply' });
|
||||
assert.equal(resumedConfig.runId, 'integration'); assert.deepEqual(resumedConfig.sourceConfigIds, ['5']);
|
||||
assert.equal(resumedConfig.sourceMode, 'http'); assert.equal(resumedConfig.downloadBaseUrl, config.downloadBaseUrl);
|
||||
assert.deepEqual((await loadSettings(db, resumedConfig)).safe, settings.safe);
|
||||
// Simulate interruption after all remaining uploads reached verified but
|
||||
// before their first file transaction. Resume must reuse the checkpoint.
|
||||
const interruptedDb = {
|
||||
execute: (...args) => db.execute(...args), query: (...args) => db.query(...args),
|
||||
beginTransaction: async () => { throw new Error('simulated interruption before transition'); }
|
||||
};
|
||||
await assert.rejects(runApply(interruptedDb, config, settings, Infinity, () => oss), /simulated interruption/);
|
||||
const [[checkpoints]] = await db.query("SELECT COUNT(*) AS n FROM infra_file_oss_migration_items WHERE run_id='integration' AND status='verified'");
|
||||
assert.equal(Number(checkpoints.n), 2);
|
||||
const downloadsBeforeResume = sourceDownloads;
|
||||
const putsBeforeResume = putCount;
|
||||
const done = await runApply(db, config, settings, Infinity, () => oss);
|
||||
assert.equal(done.status, 'complete'); assert.equal(done.items.switched, 3); assert.equal(objects.size, 3); assert.equal(putCount, 3);
|
||||
assert.equal(done.revalidated, 2); assert.equal(done.uploaded, 0);
|
||||
assert.equal(sourceDownloads, downloadsBeforeResume); assert.equal(putCount, putsBeforeResume);
|
||||
const [[member]] = await db.query('SELECT * FROM member_user');
|
||||
assert.ok(member.avatar.startsWith(`${origin}/objects/`)); assert.ok(member.wechat_qr_code.startsWith(`${origin}/objects/`));
|
||||
assert.deepEqual(Object.values(JSON.parse(member.extra_json).nested).map(url => url.startsWith(`${origin}/objects/`)), [true]);
|
||||
assert.equal(JSON.parse(member.extra_json).outside, 'https://outside.example/image.png');
|
||||
const [[counts]] = await db.query('SELECT COUNT(*) AS n FROM infra_file_content'); assert.equal(Number(counts.n), 6);
|
||||
const [files] = await db.query('SELECT * FROM infra_file ORDER BY id');
|
||||
assert.ok(files.every(file => file.config_id === '30'));
|
||||
assert.equal(files[1].path, files[2].path); // case/accent-insensitive source lookup remains equivalent
|
||||
assert.equal(files[0].size, originals[0].length); // stale old metadata is repaired
|
||||
assert.ok(files.every(file => file.update_time === '2020-01-01 00:00:00'));
|
||||
const [[before]] = await db.query('SELECT COUNT(*) AS n FROM infra_file_oss_migration_changes');
|
||||
const repeated = await runApply(db, config, settings, Infinity, () => oss); assert.equal(repeated.status, 'complete');
|
||||
const [[after]] = await db.query('SELECT COUNT(*) AS n FROM infra_file_oss_migration_changes'); assert.equal(before.n, after.n); assert.equal(putCount, 3);
|
||||
const [[entry]] = await db.query("SELECT data FROM infra_file_oss_migration_items WHERE JSON_UNQUOTE(JSON_EXTRACT(data,'$.path')) LIKE '%100%.png'");
|
||||
assert.ok(JSON.parse(entry.data).aliases.includes(firstUrl)); // logs must never rewrite themselves
|
||||
});
|
||||
|
||||
await t.test('resume does not rewrite new references using a deleted OSS object', async () => {
|
||||
const [[entry]] = await db.query("SELECT data FROM infra_file_oss_migration_items WHERE JSON_UNQUOTE(JSON_EXTRACT(data,'$.path')) LIKE '%100%.png'");
|
||||
const item = JSON.parse(entry.data), bytes = objects.get(item.key);
|
||||
objects.delete(item.key);
|
||||
await db.execute('INSERT INTO notes VALUES (2,?)', [firstUrl]);
|
||||
const result = await runApply(db, config, settings, Infinity, () => oss);
|
||||
assert.equal(result.status, 'partial'); assert.equal(result.failed, 1);
|
||||
const [[note]] = await db.query('SELECT body FROM notes WHERE id=2'); assert.equal(note.body, firstUrl);
|
||||
objects.set(item.key, bytes); await db.query('DELETE FROM notes WHERE id=2');
|
||||
});
|
||||
|
||||
await t.test('rollback preserves later user edits and can resume after conflict resolution', async () => {
|
||||
const [[migrated]] = await db.query('SELECT avatar FROM member_user');
|
||||
await db.execute('UPDATE member_user SET avatar=?', ['https://user-new.example/avatar.png']);
|
||||
const rollback = await runRollback(db, config); assert.equal(rollback.status, 'rollback_partial'); assert.ok(rollback.conflicts.length > 0);
|
||||
const [[current]] = await db.query('SELECT avatar FROM member_user'); assert.equal(current.avatar, 'https://user-new.example/avatar.png');
|
||||
await db.execute('UPDATE member_user SET avatar=?', [migrated.avatar]);
|
||||
const resumed = await runRollback(db, config); assert.equal(resumed.status, 'rolled_back');
|
||||
const [[member]] = await db.query('SELECT * FROM member_user'); assert.equal(member.avatar, firstUrl); assert.equal(member.wechat_qr_code, orphanUrl); assert.deepEqual(JSON.parse(member.extra_json), JSON.parse(payload));
|
||||
const [files] = await db.query('SELECT * FROM infra_file'); assert.ok(files.every(file => file.config_id === '5')); assert.equal(objects.size, 3);
|
||||
await assert.rejects(runApply(db, config, settings, Infinity, () => oss), /回滚/);
|
||||
});
|
||||
|
||||
await t.test('failed uploads persist a partial state and never switch database addresses', async () => {
|
||||
const failureConfig = { ...config, runId: 'upload-failure', objectPrefix: 'migration/failure' };
|
||||
const failureSettings = await loadSettings(db, failureConfig); failPut = true;
|
||||
const result = await runApply(db, failureConfig, failureSettings, Infinity, () => oss); failPut = false;
|
||||
assert.equal(result.status, 'partial'); assert.equal(result.items.failed, 3);
|
||||
const [[run]] = await db.query("SELECT status FROM infra_file_oss_migration_runs WHERE run_id='upload-failure'"); assert.equal(run.status, 'partial');
|
||||
const [[member]] = await db.query('SELECT avatar FROM member_user'); assert.equal(member.avatar, firstUrl);
|
||||
});
|
||||
|
||||
await t.test('non-strict MySQL truncation warnings roll back both data and journal', async () => {
|
||||
await db.query('CREATE TABLE short_url (id INT PRIMARY KEY,url VARCHAR(8)) ENGINE=InnoDB');
|
||||
await db.query("INSERT INTO short_url VALUES (1,'old')");
|
||||
const [[mode]] = await db.query('SELECT @@SESSION.sql_mode AS value');
|
||||
await db.query("SET SESSION sql_mode=''"); await db.beginTransaction();
|
||||
await assert.rejects(applyChange(db, schema, 'warning', { schema, table: 'short_url', primaryKey: 'id', pkValue: 1, before: { url: 'old' }, after: { url: 'this-would-be-truncated' } }), /警告/);
|
||||
await db.rollback(); await db.execute('SET SESSION sql_mode=?', [mode.value]);
|
||||
const [[row]] = await db.query('SELECT url FROM short_url'); assert.equal(row.url, 'old');
|
||||
const [[count]] = await db.query("SELECT COUNT(*) AS n FROM infra_file_oss_migration_changes WHERE run_id='warning'"); assert.equal(Number(count.n), 0);
|
||||
});
|
||||
|
||||
await t.test('batch CAS conflict isolates the edited file and commits unrelated verified items', async () => {
|
||||
const conflictConfig = { ...config, runId: 'batch-conflict', objectPrefix: 'migration/conflict' };
|
||||
const conflictSettings = await loadSettings(db, conflictConfig);
|
||||
let changed = false, rollbacks = 0;
|
||||
const concurrentEditDb = {
|
||||
query: (...args) => db.query(...args), execute: (...args) => db.execute(...args),
|
||||
async beginTransaction() {
|
||||
if (!changed) {
|
||||
changed = true;
|
||||
await db.execute('UPDATE infra_file SET url=? WHERE id=?', ['https://user-change.example/new.png', '9007199254741001']);
|
||||
}
|
||||
return db.beginTransaction();
|
||||
},
|
||||
commit: () => db.commit(),
|
||||
rollback() { rollbacks++; return db.rollback(); }
|
||||
};
|
||||
const result = await runApply(concurrentEditDb, conflictConfig, conflictSettings, Infinity, () => oss);
|
||||
assert.equal(result.status, 'partial'); assert.equal(result.failed, 1);
|
||||
assert.equal(result.items.switched, 2); assert.equal(result.items.verified, 1);
|
||||
assert.ok(rollbacks >= 2, 'the batch and the conflicted single item are rolled back');
|
||||
const [[edited]] = await db.query('SELECT config_id,url FROM infra_file WHERE id=9007199254741001');
|
||||
assert.equal(edited.config_id, '5'); assert.equal(edited.url, 'https://user-change.example/new.png');
|
||||
const [[member]] = await db.query('SELECT avatar FROM member_user'); assert.equal(member.avatar, firstUrl);
|
||||
const rollback = await runRollback(db, conflictConfig); assert.equal(rollback.status, 'rolled_back');
|
||||
await db.execute('UPDATE infra_file SET url=? WHERE id=9007199254741001', [firstUrl]);
|
||||
});
|
||||
|
||||
await t.test('database-source mode works and ambiguous raw-plus links require review', async () => {
|
||||
const rawUrl = `${origin}/admin-api/infra/file/5/get/${firstPath}`;
|
||||
await db.execute('INSERT INTO notes VALUES (3,?)', [rawUrl]);
|
||||
const directConfig = { ...config, runId: 'database-mode', sourceMode: 'database' };
|
||||
const directSettings = await loadSettings(db, directConfig);
|
||||
const result = await runApply(db, directConfig, directSettings, Infinity, () => oss);
|
||||
assert.equal(result.status, 'needs-review'); assert.equal(result.items.switched, 3);
|
||||
const [[note]] = await db.query('SELECT body FROM notes WHERE id=3'); assert.equal(note.body, rawUrl);
|
||||
assert.ok(result.references.unresolvedRows > 0);
|
||||
});
|
||||
} finally {
|
||||
server.close(); server.closeAllConnections();
|
||||
await db.query(`DROP DATABASE IF EXISTS ${schema}`);
|
||||
await db.end();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,311 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { applyChanges } from '../lib/journal.mjs';
|
||||
|
||||
const clone = value => Buffer.isBuffer(value) ? Buffer.from(value)
|
||||
: Array.isArray(value) ? value.map(clone)
|
||||
: value && typeof value === 'object' ? Object.fromEntries(Object.entries(value).map(([key, item]) => [key, clone(item)])) : value;
|
||||
const key = value => Buffer.isBuffer(value) ? `buffer:${value.toString('hex')}` : String(value);
|
||||
const same = (left, right) => Buffer.isBuffer(left) || Buffer.isBuffer(right)
|
||||
? Buffer.isBuffer(left) && Buffer.isBuffer(right) && left.equals(right)
|
||||
: left === null || right === null ? left === right : String(left) === String(right);
|
||||
|
||||
const makeRow = (id, suffix) => ({
|
||||
id, config_id: '4', path: `${suffix}.png`, url: `https://old/${suffix}.png`, type: 'image/png', size: 5,
|
||||
deleted: Buffer.from([0]), update_time: 'unchanged', touched: 'unchanged'
|
||||
});
|
||||
|
||||
// This small SQL test double evaluates the CASE dispatch and every WHERE guard
|
||||
// independently, so mixed per-row values and partially matched batches surface.
|
||||
class BatchConnection {
|
||||
constructor(rows = [makeRow('9007199254740993', 'a'), makeRow('9007199254740994', 'b')]) {
|
||||
this.rows = Object.fromEntries(rows.map(row => [key(row.id), clone(row)]));
|
||||
this.calls = [];
|
||||
this.journals = [];
|
||||
this.warningStatus = 0;
|
||||
this.failJournal = false;
|
||||
this.journalWarningStatus = 0;
|
||||
this.journalAffectedRows = null;
|
||||
this.engine = 'InnoDB';
|
||||
this.normalize = null;
|
||||
this.omitReadback = false;
|
||||
this.reverseReadback = false;
|
||||
this.nextId = 9007199254741993n;
|
||||
}
|
||||
|
||||
async beginTransaction() {
|
||||
this.calls.push({ transaction: 'begin' });
|
||||
this.saved = { rows: clone(this.rows), journals: clone(this.journals) };
|
||||
}
|
||||
async commit() { this.calls.push({ transaction: 'commit' }); this.saved = undefined; }
|
||||
async rollback() {
|
||||
this.calls.push({ transaction: 'rollback' });
|
||||
if (this.saved) { this.rows = this.saved.rows; this.journals = this.saved.journals; this.saved = undefined; }
|
||||
}
|
||||
|
||||
async execute(sql, params = []) {
|
||||
this.calls.push({ sql, params });
|
||||
if (sql.includes('FROM information_schema.COLUMNS')) {
|
||||
return [[...['id', 'config_id', 'path', 'url', 'type', 'size', 'deleted', 'json_data', 'counter', 'update_time', 'touched']
|
||||
.map(name => ({ name, engine: this.engine, extra: ['update_time', 'touched'].includes(name) ? 'on update CURRENT_TIMESTAMP' : '' }))]];
|
||||
}
|
||||
if (sql.startsWith('UPDATE')) {
|
||||
const [setSql, whereSql] = sql.split(/\bWHERE\b/);
|
||||
const cases = [...setSql.matchAll(/`([^`]+)`\s*=\s*CASE\s+`id`([\s\S]*?)\bEND/g)];
|
||||
assert.ok(cases.length > 0, 'the batch must use a CASE expression for changed columns');
|
||||
let offset = 0;
|
||||
const values = new Map();
|
||||
for (const [, column, branches] of cases) {
|
||||
const count = [...branches.matchAll(/\bWHEN\s+\?\s+THEN\s+\?/g)].length;
|
||||
assert.ok(count > 0);
|
||||
assert.match(branches, new RegExp('ELSE\\s+`' + column + '`'));
|
||||
const entries = new Map();
|
||||
for (let index = 0; index < count; index++) entries.set(key(params[offset++]), params[offset++]);
|
||||
values.set(column, entries);
|
||||
}
|
||||
let affectedRows = 0;
|
||||
const groups = whereSql.trim().replace(/^\(/, '').replace(/\)$/, '').split(/\)\s+OR\s+\(/);
|
||||
for (const group of groups) {
|
||||
assert.match(group, /`id`\s*=\s*\?/);
|
||||
const pk = key(params[offset++]);
|
||||
const guards = [...group.matchAll(/CAST\(`([^`]+)` AS BINARY\)\s*<=>\s*CAST\(\? AS BINARY\)/g)].map(match => match[1]);
|
||||
const expected = guards.map(() => params[offset++]);
|
||||
const row = this.rows[pk];
|
||||
if (!row || !guards.every((name, index) => same(row[name], expected[index]))) continue;
|
||||
for (const [column, entries] of values) if (entries.has(pk)) row[column] = entries.get(pk);
|
||||
if (this.normalize) this.normalize(row, [...values.keys()]);
|
||||
if (!setSql.includes('`update_time` = `update_time`')) row.update_time = 'changed';
|
||||
if (!setSql.includes('`touched` = `touched`')) row.touched = 'changed';
|
||||
affectedRows++;
|
||||
}
|
||||
assert.equal(offset, params.length, 'all prepared UPDATE values must have a matching SQL placeholder');
|
||||
return [{ affectedRows, warningStatus: this.warningStatus }];
|
||||
}
|
||||
if (sql.startsWith('SELECT `')) {
|
||||
assert.match(sql, /\bIN\s*\(/);
|
||||
assert.match(sql, /FOR UPDATE/);
|
||||
const selected = [...sql.split(/\bFROM\b/)[0].matchAll(/`([^`]+)`/g)].map(match => match[1]);
|
||||
const rows = params.map(pk => this.rows[key(pk)]).filter(Boolean)
|
||||
.map(row => Object.fromEntries(selected.map(column => [column, row[column]])));
|
||||
if (this.reverseReadback) rows.reverse();
|
||||
return [this.omitReadback ? rows.slice(1) : rows];
|
||||
}
|
||||
if (sql.startsWith('INSERT INTO')) {
|
||||
if (this.failJournal) throw Object.assign(new Error('injected journal INSERT failure'), { code: 'TEST_JOURNAL_FAILURE' });
|
||||
assert.equal(params.length % 7, 0);
|
||||
const firstId = this.nextId.toString();
|
||||
for (let offset = 0; offset < params.length; offset += 7) {
|
||||
const [run_id, schema_name, table_name, pk_name, pk_value, before_json, after_json] = params.slice(offset, offset + 7);
|
||||
this.journals.push({ id: (this.nextId++).toString(), run_id, schema_name, table_name, pk_name, pk_value, before_json, after_json, rolled_back: 0 });
|
||||
}
|
||||
return [{ affectedRows: this.journalAffectedRows ?? params.length / 7, warningStatus: this.journalWarningStatus, insertId: firstId }];
|
||||
}
|
||||
throw new Error(`Unexpected test SQL: ${sql}`);
|
||||
}
|
||||
}
|
||||
|
||||
function change(row, suffix = 'migrated') {
|
||||
const { config_id, path, url, type, size, deleted } = row;
|
||||
return {
|
||||
schema: 'app', table: 'infra_file', primaryKey: 'id', pkValue: row.id,
|
||||
before: { config_id, path, url, type, size, deleted },
|
||||
after: { config_id: '20', path: `${suffix}/${path}`, url: `https://oss/${suffix}/${path}`, type, size }
|
||||
};
|
||||
}
|
||||
|
||||
const changesFor = connection => Object.values(connection.rows).map(row => change(row));
|
||||
const dml = connection => connection.calls.filter(call => call.sql && !call.sql.includes('information_schema'));
|
||||
|
||||
async function commitBatch(connection, changes) {
|
||||
await connection.beginTransaction();
|
||||
try {
|
||||
const result = await applyChanges(connection, 'app', 'run', changes);
|
||||
await connection.commit();
|
||||
return result;
|
||||
} catch (error) { await connection.rollback(); throw error; }
|
||||
}
|
||||
|
||||
test('batch uses one CASE update, one readback and one journal insert for distinct bigint rows', async () => {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
assert.deepEqual(await commitBatch(connection, changes), { changed: 2, skipped: 0 });
|
||||
assert.deepEqual(dml(connection).map(call => call.sql.split(/\s/)[0]), ['UPDATE', 'SELECT', 'INSERT']);
|
||||
assert.equal(connection.rows['9007199254740993'].url, 'https://oss/migrated/a.png');
|
||||
assert.equal(connection.rows['9007199254740994'].url, 'https://oss/migrated/b.png');
|
||||
assert.equal(connection.rows['9007199254740993'].update_time, 'unchanged');
|
||||
assert.equal(connection.rows['9007199254740994'].touched, 'unchanged');
|
||||
assert.equal(connection.journals.length, 2);
|
||||
assert.deepEqual(connection.journals.map(row => JSON.parse(row.pk_value).value), ['9007199254740993', '9007199254740994']);
|
||||
const [update] = dml(connection);
|
||||
assert.match(update.sql, /CAST\(`deleted` AS BINARY\) <=> CAST\(\? AS BINARY\)/);
|
||||
assert.ok(update.params.some(value => Buffer.isBuffer(value) && value.equals(Buffer.from([0]))));
|
||||
assert.ok(!update.sql.includes('https://oss/'), 'URLs must be bound, not interpolated');
|
||||
assert.ok(!update.sql.includes('`deleted` = CASE'), 'before-only deletion guards must not become assignments');
|
||||
});
|
||||
|
||||
test('table metadata is cached across batches on the same connection', async () => {
|
||||
const connection = new BatchConnection();
|
||||
await commitBatch(connection, changesFor(connection));
|
||||
await commitBatch(connection, Object.values(connection.rows).map(row => change(row, 'second')));
|
||||
assert.equal(connection.calls.filter(call => call.sql?.includes('information_schema')).length, 1);
|
||||
assert.equal(dml(connection).length, 6);
|
||||
});
|
||||
|
||||
test('empty batches and unchanged rows create no updates or journal entries', async () => {
|
||||
const connection = new BatchConnection();
|
||||
assert.deepEqual(await commitBatch(connection, []), { changed: 0, skipped: 0 });
|
||||
const changes = changesFor(connection);
|
||||
changes.forEach(item => { item.after = Object.fromEntries(Object.keys(item.after).map(name => [name, item.before[name]])); });
|
||||
assert.deepEqual(await commitBatch(connection, changes), { changed: 0, skipped: 2 });
|
||||
assert.equal(dml(connection).length, 0);
|
||||
assert.equal(connection.journals.length, 0);
|
||||
});
|
||||
|
||||
test('a no-op row is skipped while a different row receives its own values', async () => {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
changes[1].after = Object.fromEntries(Object.keys(changes[1].after).map(name => [name, changes[1].before[name]]));
|
||||
assert.deepEqual(await commitBatch(connection, changes), { changed: 1, skipped: 1 });
|
||||
assert.equal(connection.rows['9007199254740993'].url, 'https://oss/migrated/a.png');
|
||||
assert.equal(connection.rows['9007199254740994'].url, 'https://old/b.png');
|
||||
assert.equal(connection.journals.length, 1);
|
||||
});
|
||||
|
||||
test('after field order may differ and row-specific unchanged fields stay unchanged', async () => {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
changes[0].after.type = 'image/webp';
|
||||
changes[1].after = Object.fromEntries(Object.entries(changes[1].after).reverse());
|
||||
await commitBatch(connection, changes);
|
||||
assert.equal(connection.rows['9007199254740993'].type, 'image/webp');
|
||||
assert.equal(connection.rows['9007199254740994'].type, 'image/png');
|
||||
});
|
||||
|
||||
test('different schema, table, primary key, or after-field groups are rejected before updates', async () => {
|
||||
for (const mutate of [
|
||||
item => { item.schema = 'other'; },
|
||||
item => { item.table = 'member_user'; },
|
||||
item => { item.primaryKey = 'other_id'; },
|
||||
item => { delete item.after.size; }
|
||||
]) {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
mutate(changes[1]);
|
||||
await assert.rejects(commitBatch(connection, changes));
|
||||
assert.equal(dml(connection).length, 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('duplicate primary keys are rejected even across string and bigint representations', async () => {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
changes[1].pkValue = BigInt(changes[0].pkValue);
|
||||
await assert.rejects(commitBatch(connection, changes), /主键|重复/);
|
||||
assert.equal(dml(connection).length, 0);
|
||||
});
|
||||
|
||||
test('primary key updates and missing before snapshots are rejected', async () => {
|
||||
for (const mutate of [
|
||||
item => { item.after.id = '1'; },
|
||||
item => { delete item.before.url; }
|
||||
]) {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
changes.forEach(mutate);
|
||||
await assert.rejects(commitBatch(connection, changes));
|
||||
assert.equal(dml(connection).length, 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('a concurrent case-only edit leaves the entire batch unchanged after caller rollback', async () => {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
connection.rows['9007199254740994'].url = 'https://old/B.png';
|
||||
const before = clone(connection.rows);
|
||||
await assert.rejects(commitBatch(connection, changes), { code: 'MIGRATION_CAS_CONFLICT' });
|
||||
assert.deepEqual(connection.rows, before);
|
||||
assert.equal(connection.journals.length, 0);
|
||||
assert.equal(dml(connection).length, 1);
|
||||
assert.equal(connection.calls.at(-1).transaction, 'rollback');
|
||||
});
|
||||
|
||||
test('a deletion guard conflict rolls back otherwise matching rows', async () => {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
connection.rows['9007199254740994'].deleted = Buffer.from([1]);
|
||||
const before = clone(connection.rows);
|
||||
await assert.rejects(commitBatch(connection, changes), { code: 'MIGRATION_CAS_CONFLICT' });
|
||||
assert.deepEqual(connection.rows, before);
|
||||
});
|
||||
|
||||
test('journal INSERT failure rolls back all successfully updated rows', async () => {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
const before = clone(connection.rows);
|
||||
connection.failJournal = true;
|
||||
await assert.rejects(commitBatch(connection, changes), { code: 'TEST_JOURNAL_FAILURE' });
|
||||
assert.deepEqual(connection.rows, before);
|
||||
assert.equal(connection.journals.length, 0);
|
||||
assert.equal(dml(connection).length, 3);
|
||||
});
|
||||
|
||||
test('MySQL JSON canonicalization and numeric conversion are read back into each journal entry', async () => {
|
||||
const connection = new BatchConnection();
|
||||
connection.reverseReadback = true;
|
||||
for (const row of Object.values(connection.rows)) {
|
||||
row.json_data = JSON.stringify({ url: row.url }).replaceAll('\":', '\": ');
|
||||
row.counter = 1;
|
||||
}
|
||||
const changes = Object.values(connection.rows).map(row => ({
|
||||
schema: 'app', table: 'infra_file', primaryKey: 'id', pkValue: row.id,
|
||||
before: { json_data: row.json_data, counter: 1 },
|
||||
after: { json_data: JSON.stringify({ url: `https://oss/${row.path}` }), counter: '2' }
|
||||
}));
|
||||
connection.normalize = (row, fields) => {
|
||||
if (fields.includes('json_data')) row.json_data = JSON.stringify(JSON.parse(row.json_data)).replaceAll('\":', '\": ');
|
||||
if (fields.includes('counter')) row.counter = Number(row.counter);
|
||||
};
|
||||
await commitBatch(connection, changes);
|
||||
const snapshots = connection.journals.map(row => JSON.parse(row.after_json));
|
||||
assert.equal(snapshots[0].json_data.value, '{"url": "https://oss/a.png"}');
|
||||
assert.equal(snapshots[1].json_data.value, '{"url": "https://oss/b.png"}');
|
||||
assert.equal(snapshots[0].counter.value, 2);
|
||||
assert.equal(snapshots[1].counter.value, 2);
|
||||
});
|
||||
|
||||
test('UPDATE warnings and incomplete readback each abort the entire batch', async () => {
|
||||
for (const prepare of [
|
||||
connection => { connection.warningStatus = 1; },
|
||||
connection => { connection.omitReadback = true; }
|
||||
]) {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
const before = clone(connection.rows);
|
||||
prepare(connection);
|
||||
await assert.rejects(commitBatch(connection, changes));
|
||||
assert.deepEqual(connection.rows, before);
|
||||
assert.equal(connection.journals.length, 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('incomplete or warned batch journal inserts roll back both updated rows and inserted logs', async () => {
|
||||
for (const prepare of [
|
||||
connection => { connection.journalAffectedRows = 1; },
|
||||
connection => { connection.journalWarningStatus = 1; }
|
||||
]) {
|
||||
const connection = new BatchConnection();
|
||||
const changes = changesFor(connection);
|
||||
const before = clone(connection.rows);
|
||||
prepare(connection);
|
||||
await assert.rejects(commitBatch(connection, changes), /回滚日志/);
|
||||
assert.deepEqual(connection.rows, before);
|
||||
assert.equal(connection.journals.length, 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('non-InnoDB targets are rejected before mutation', async () => {
|
||||
const connection = new BatchConnection();
|
||||
connection.engine = 'MyISAM';
|
||||
await assert.rejects(commitBatch(connection, changesFor(connection)), /InnoDB/);
|
||||
assert.equal(dml(connection).length, 0);
|
||||
});
|
||||
@@ -0,0 +1,238 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { ensureJournal, applyChange, rollbackChanges } from '../lib/journal.mjs';
|
||||
|
||||
const clone = value => Buffer.isBuffer(value) ? Buffer.from(value)
|
||||
: Array.isArray(value) ? value.map(clone)
|
||||
: value && typeof value === 'object' ? Object.fromEntries(Object.entries(value).map(([key, item]) => [key, clone(item)])) : value;
|
||||
const same = (a, b) => Buffer.isBuffer(a) && Buffer.isBuffer(b) ? a.equals(b) : a === b || (a !== null && b !== null && String(a) === String(b));
|
||||
const names = sql => [...sql.matchAll(/CAST\(`((?:``|[^`])+)` AS BINARY\)/g)].map(match => match[1].replaceAll('``', '`'));
|
||||
|
||||
class MemoryConnection {
|
||||
constructor() {
|
||||
this.calls = [];
|
||||
this.rows = { '9007199254740993': { id: '9007199254740993', avatar: 'https://old/a', nullable: null, deleted: Buffer.from([0]), update_time: 'unchanged', touched: 'unchanged' } };
|
||||
this.journals = [];
|
||||
this.nextId = 9007199254740993n;
|
||||
this.engine = 'InnoDB';
|
||||
this.failJournal = false;
|
||||
this.normalizeUpdate = null;
|
||||
this.warningStatus = 0;
|
||||
}
|
||||
|
||||
async beginTransaction() { this.calls.push({ transaction: 'begin' }); this.saved = { rows: clone(this.rows), journals: clone(this.journals) }; }
|
||||
async commit() { this.calls.push({ transaction: 'commit' }); this.saved = undefined; }
|
||||
async rollback() { this.calls.push({ transaction: 'rollback' }); if (this.saved) { this.rows = this.saved.rows; this.journals = this.saved.journals; this.saved = undefined; } }
|
||||
|
||||
async execute(sql, params = []) {
|
||||
this.calls.push({ sql, params });
|
||||
if (sql.startsWith('CREATE TABLE')) return [{ affectedRows: 0 }];
|
||||
if (sql.includes('FROM information_schema.COLUMNS')) {
|
||||
return [[...['id', 'avatar', 'nullable', 'deleted', 'update_time', 'touched', 'json_data', 'counter'].map(name => ({ name, engine: this.engine, extra: ['update_time', 'touched'].includes(name) ? 'on update CURRENT_TIMESTAMP' : '' }))]];
|
||||
}
|
||||
if (sql.startsWith('INSERT INTO')) {
|
||||
if (this.failJournal) throw Object.assign(new Error('journal unavailable'), { code: 'TEST_JOURNAL_FAILURE' });
|
||||
const [run_id, schema_name, table_name, pk_name, pk_value, before_json, after_json] = params;
|
||||
const id = (this.nextId++).toString();
|
||||
this.journals.push({ id, run_id, schema_name, table_name, pk_name, pk_value, before_json, after_json, rolled_back: 0 });
|
||||
return [{ insertId: id }];
|
||||
}
|
||||
if (sql.startsWith('SELECT CAST(id AS CHAR)')) {
|
||||
const [runId, cursor] = params;
|
||||
return [this.journals.filter(row => row.run_id === runId && row.rolled_back === 0 && (cursor === undefined || BigInt(row.id) < BigInt(cursor)))
|
||||
.sort((a, b) => BigInt(a.id) > BigInt(b.id) ? -1 : 1).slice(0, 100).map(row => ({ id: row.id }))];
|
||||
}
|
||||
if (sql.startsWith('SELECT *')) return [[this.journals.find(row => row.id === params[0] && row.run_id === params[1])].filter(Boolean)];
|
||||
if (sql.startsWith('SELECT 1 AS matched')) {
|
||||
const row = this.rows[String(params[0])];
|
||||
return [row && names(sql).every((name, index) => same(row[name], params[index + 1])) ? [{ matched: 1 }] : []];
|
||||
}
|
||||
if (sql.startsWith('SELECT `')) {
|
||||
const row = this.rows[String(params[0])];
|
||||
const selected = [...sql.split(' FROM ')[0].matchAll(/`((?:``|[^`])+)`/g)].map(match => match[1]);
|
||||
return [row ? [Object.fromEntries(selected.map(name => [name, row[name]]))] : []];
|
||||
}
|
||||
if (sql.startsWith('UPDATE') && sql.includes('SET rolled_back = 1')) {
|
||||
const row = this.journals.find(row => row.id === params[0] && row.run_id === params[1]);
|
||||
if (row) row.rolled_back = 1;
|
||||
return [{ affectedRows: row ? 1 : 0 }];
|
||||
}
|
||||
if (sql.startsWith('UPDATE')) {
|
||||
const set = sql.split(' SET ')[1].split('WHERE')[0];
|
||||
const changed = [...set.matchAll(/`((?:``|[^`])+)` = \?/g)].map(match => match[1]);
|
||||
const row = this.rows[String(params[changed.length])];
|
||||
if (!row || !names(sql).every((name, index) => same(row[name], params[changed.length + 1 + index]))) return [{ affectedRows: 0 }];
|
||||
changed.forEach((name, index) => { row[name] = params[index]; });
|
||||
if (this.normalizeUpdate) this.normalizeUpdate(row, changed);
|
||||
if (!set.includes('`update_time` = `update_time`')) row.update_time = 'changed';
|
||||
if (!set.includes('`touched` = `touched`')) row.touched = 'changed';
|
||||
return [{ affectedRows: 1, warningStatus: this.warningStatus }];
|
||||
}
|
||||
throw new Error(`Unexpected test query: ${sql}`);
|
||||
}
|
||||
}
|
||||
|
||||
const change = (before = 'https://old/a', after = 'https://oss/a') => ({
|
||||
schema: 'app', table: 'member_user', primaryKey: 'id', pkValue: '9007199254740993',
|
||||
before: { avatar: before, deleted: Buffer.from([0]), nullable: null }, after: { avatar: after }
|
||||
});
|
||||
|
||||
async function commitChange(connection, payload = change(), runId = 'run') {
|
||||
await connection.beginTransaction();
|
||||
try {
|
||||
const result = await applyChange(connection, 'app', runId, payload);
|
||||
await connection.commit();
|
||||
return result;
|
||||
} catch (error) { await connection.rollback(); throw error; }
|
||||
}
|
||||
|
||||
test('journal schema uses InnoDB and independent run/item/change records', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
await ensureJournal(connection, 'app');
|
||||
const creates = connection.calls.filter(call => call.sql?.startsWith('CREATE TABLE'));
|
||||
assert.equal(creates.length, 3);
|
||||
assert.ok(creates.every(call => call.sql.includes('ENGINE=InnoDB')));
|
||||
assert.match(creates[1].sql, /PRIMARY KEY \(run_id, item_key\)/);
|
||||
assert.match(creates[2].sql, /KEY ix_run_id_id \(run_id, id\)/);
|
||||
});
|
||||
|
||||
test('apply logs original values with binary null-safe guards and lossless bigint IDs', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
const result = await commitChange(connection);
|
||||
assert.equal(result.journalId, '9007199254740993');
|
||||
assert.equal(connection.rows['9007199254740993'].avatar, 'https://oss/a');
|
||||
assert.equal(connection.rows['9007199254740993'].update_time, 'unchanged');
|
||||
assert.equal(connection.rows['9007199254740993'].touched, 'unchanged');
|
||||
const update = connection.calls.find(call => call.sql?.startsWith('UPDATE'));
|
||||
assert.match(update.sql, /CAST\(`avatar` AS BINARY\) <=> CAST\(\? AS BINARY\)/);
|
||||
assert.match(update.sql, /CAST\(`nullable` AS BINARY\) <=> CAST\(\? AS BINARY\)/);
|
||||
assert.equal(update.params[1], '9007199254740993');
|
||||
assert.deepEqual(update.params[3], Buffer.from([0]));
|
||||
assert.equal(update.params[4], null);
|
||||
assert.equal(connection.journals.length, 1);
|
||||
});
|
||||
|
||||
test('no-op changes do not update rows or create logs', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
assert.deepEqual(await commitChange(connection, change('https://old/a', 'https://old/a')), { changed: false });
|
||||
assert.equal(connection.journals.length, 0);
|
||||
assert.equal(connection.calls.filter(call => call.sql?.startsWith('UPDATE')).length, 0);
|
||||
});
|
||||
|
||||
test('primary-key changes and missing snapshots are rejected before mutation', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
await assert.rejects(commitChange(connection, { ...change(), after: { id: '1' } }), /不能修改主键/);
|
||||
await assert.rejects(commitChange(connection, { ...change(), before: { deleted: Buffer.from([0]) } }), /缺少原值快照/);
|
||||
assert.equal(connection.journals.length, 0);
|
||||
});
|
||||
|
||||
test('concurrent edits including case-only changes fail CAS and preserve the row', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
connection.rows['9007199254740993'].avatar = 'https://old/A';
|
||||
await assert.rejects(commitChange(connection), { code: 'MIGRATION_CAS_CONFLICT' });
|
||||
assert.equal(connection.rows['9007199254740993'].avatar, 'https://old/A');
|
||||
assert.equal(connection.journals.length, 0);
|
||||
});
|
||||
|
||||
test('a journal INSERT failure is rolled back with the row by the caller transaction', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
connection.failJournal = true;
|
||||
await assert.rejects(commitChange(connection), { code: 'TEST_JOURNAL_FAILURE' });
|
||||
assert.equal(connection.rows['9007199254740993'].avatar, 'https://old/a');
|
||||
assert.equal(connection.journals.length, 0);
|
||||
assert.equal(connection.calls.at(-1).transaction, 'rollback');
|
||||
});
|
||||
|
||||
test('journal reads back native JSON canonicalization and numeric conversions before recording rollback values', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
connection.rows['9007199254740993'].json_data = '{"avatar": "https://old/a"}';
|
||||
connection.rows['9007199254740993'].counter = 1;
|
||||
connection.normalizeUpdate = (row, changed) => {
|
||||
if (changed.includes('json_data')) row.json_data = JSON.stringify(JSON.parse(row.json_data)).replaceAll('\":', '\": ');
|
||||
if (changed.includes('counter')) row.counter = Number(row.counter);
|
||||
};
|
||||
await commitChange(connection, {
|
||||
...change(),
|
||||
before: { json_data: '{"avatar": "https://old/a"}', counter: 1 },
|
||||
after: { json_data: '{"avatar":"https://oss/a"}', counter: '2' }
|
||||
});
|
||||
const after = JSON.parse(connection.journals[0].after_json);
|
||||
assert.equal(after.json_data.value, '{"avatar": "https://oss/a"}');
|
||||
assert.equal(after.counter.value, 2);
|
||||
const result = await rollbackChanges(connection, 'app', 'run');
|
||||
assert.deepEqual(result, { restored: 1, alreadyRestored: 0, conflicts: [] });
|
||||
assert.equal(connection.rows['9007199254740993'].json_data, '{"avatar": "https://old/a"}');
|
||||
assert.equal(connection.rows['9007199254740993'].counter, 1);
|
||||
});
|
||||
|
||||
test('silent SQL truncation warnings abort the transaction before recording success', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
connection.warningStatus = 1;
|
||||
await assert.rejects(commitChange(connection), /数据库更新产生警告/);
|
||||
assert.equal(connection.rows['9007199254740993'].avatar, 'https://old/a');
|
||||
assert.equal(connection.journals.length, 0);
|
||||
});
|
||||
|
||||
test('rollback restores consecutive edits in reverse journal order without losing bigint precision', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
await commitChange(connection);
|
||||
await commitChange(connection, change('https://oss/a', 'https://oss/b'));
|
||||
const result = await rollbackChanges(connection, 'app', 'run');
|
||||
assert.deepEqual(result, { restored: 2, alreadyRestored: 0, conflicts: [] });
|
||||
assert.equal(connection.rows['9007199254740993'].avatar, 'https://old/a');
|
||||
assert.equal(connection.rows['9007199254740993'].update_time, 'unchanged');
|
||||
const reads = connection.calls.filter(call => call.sql?.startsWith('SELECT *'));
|
||||
assert.deepEqual(reads.map(call => call.params[0]), ['9007199254740994', '9007199254740993']);
|
||||
assert.ok(connection.journals.every(row => row.rolled_back === 1));
|
||||
assert.deepEqual(await rollbackChanges(connection, 'app', 'run'), { restored: 0, alreadyRestored: 0, conflicts: [] });
|
||||
});
|
||||
|
||||
test('rollback detects an already-restored row and only marks the journal', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
await commitChange(connection);
|
||||
connection.rows['9007199254740993'].avatar = 'https://old/a';
|
||||
const result = await rollbackChanges(connection, 'app', 'run');
|
||||
assert.deepEqual(result, { restored: 0, alreadyRestored: 1, conflicts: [] });
|
||||
assert.equal(connection.journals[0].rolled_back, 1);
|
||||
});
|
||||
|
||||
test('rollback SQL warnings do not mark a truncated restoration as complete', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
await commitChange(connection);
|
||||
connection.warningStatus = 1;
|
||||
await assert.rejects(rollbackChanges(connection, 'app', 'run'), /数据库回滚产生警告/);
|
||||
assert.equal(connection.rows['9007199254740993'].avatar, 'https://oss/a');
|
||||
assert.equal(connection.journals[0].rolled_back, 0);
|
||||
});
|
||||
|
||||
test('rollback leaves later user edits and deletion guards unchanged', async () => {
|
||||
for (const update of [{ avatar: 'https://user/new' }, { deleted: Buffer.from([1]) }]) {
|
||||
const connection = new MemoryConnection();
|
||||
await commitChange(connection);
|
||||
Object.assign(connection.rows['9007199254740993'], update);
|
||||
const original = clone(connection.rows['9007199254740993']);
|
||||
const result = await rollbackChanges(connection, 'app', 'run');
|
||||
assert.equal(result.restored, 0);
|
||||
assert.equal(result.conflicts.length, 1);
|
||||
assert.equal(result.conflicts[0].pkValue, '9007199254740993');
|
||||
assert.deepEqual(connection.rows['9007199254740993'], original);
|
||||
assert.equal(connection.journals[0].rolled_back, 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('rollback does not resurrect missing rows or touch another run', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
await commitChange(connection);
|
||||
delete connection.rows['9007199254740993'];
|
||||
assert.deepEqual(await rollbackChanges(connection, 'app', 'another-run'), { restored: 0, alreadyRestored: 0, conflicts: [] });
|
||||
const result = await rollbackChanges(connection, 'app', 'run');
|
||||
assert.equal(result.conflicts.length, 1);
|
||||
assert.equal(Object.keys(connection.rows).length, 0);
|
||||
});
|
||||
|
||||
test('non-transactional tables are rejected before any update', async () => {
|
||||
const connection = new MemoryConnection();
|
||||
connection.engine = 'MyISAM';
|
||||
await assert.rejects(commitChange(connection), /必须为 InnoDB/);
|
||||
assert.equal(connection.calls.filter(call => call.sql?.startsWith('UPDATE')).length, 0);
|
||||
});
|
||||
@@ -0,0 +1,142 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createProgress } from '../lib/progress.mjs';
|
||||
|
||||
function fixture(t, intervalMs = 5000) {
|
||||
t.mock.timers.enable({ apis: ['setInterval'] });
|
||||
let time = 0;
|
||||
const lines = [];
|
||||
const progress = createProgress({ log: line => lines.push(line), intervalMs, now: () => time });
|
||||
t.after(() => progress.close());
|
||||
return {
|
||||
progress, lines,
|
||||
setTime(value) { time = value; },
|
||||
tick(value) { time += value; t.mock.timers.tick(value); },
|
||||
};
|
||||
}
|
||||
|
||||
test('construction stays silent until a stage is available', t => {
|
||||
const { progress, lines, tick } = fixture(t);
|
||||
tick(15000);
|
||||
assert.deepEqual(lines, []);
|
||||
progress.stage('正在等待数据库连接');
|
||||
assert.deepEqual(lines, ['正在等待数据库连接（已用时 0 秒）']);
|
||||
});
|
||||
|
||||
test('the timer reports a waiting stage at the configured interval', t => {
|
||||
const { progress, lines, tick } = fixture(t);
|
||||
progress.stage('正在统计历史图片');
|
||||
tick(5000);
|
||||
tick(5000);
|
||||
assert.deepEqual(lines, [
|
||||
'正在统计历史图片（已用时 0 秒）',
|
||||
'正在统计历史图片（已用时 5 秒）',
|
||||
'正在统计历史图片（已用时 10 秒）',
|
||||
]);
|
||||
});
|
||||
|
||||
test('frequent updates preserve the newest message without printing every record', t => {
|
||||
const { progress, lines, setTime } = fixture(t);
|
||||
progress.stage('正在扫描引用');
|
||||
for (let count = 1; count <= 1000; count++) {
|
||||
setTime(count * 4);
|
||||
progress.update('正在扫描引用，已检查 ' + count + ' 行');
|
||||
}
|
||||
assert.equal(lines.length, 1);
|
||||
setTime(5000);
|
||||
progress.update('正在扫描引用，已检查 1200 行');
|
||||
assert.equal(lines.at(-1), '正在扫描引用，已检查 1200 行（已用时 5 秒）');
|
||||
assert.equal(lines.length, 2);
|
||||
});
|
||||
|
||||
test('timer output uses the last update message without resetting stage elapsed time', t => {
|
||||
const { progress, lines, tick } = fixture(t);
|
||||
progress.stage('正在扫描引用');
|
||||
tick(2000);
|
||||
progress.update('正在扫描引用，已检查 100 行');
|
||||
tick(3000);
|
||||
assert.deepEqual(lines, [
|
||||
'正在扫描引用（已用时 0 秒）',
|
||||
'正在扫描引用，已检查 100 行（已用时 5 秒）',
|
||||
]);
|
||||
});
|
||||
|
||||
test('synchronous updates emit progress even when interval callbacks cannot run', t => {
|
||||
const { progress, lines, setTime } = fixture(t);
|
||||
progress.stage('正在处理清单');
|
||||
setTime(4999);
|
||||
progress.update('正在处理清单 1');
|
||||
assert.equal(lines.length, 1);
|
||||
setTime(5000);
|
||||
progress.update('正在处理清单 2');
|
||||
setTime(10000);
|
||||
progress.update('正在处理清单 3');
|
||||
assert.deepEqual(lines, [
|
||||
'正在处理清单（已用时 0 秒）',
|
||||
'正在处理清单 2（已用时 5 秒）',
|
||||
'正在处理清单 3（已用时 10 秒）',
|
||||
]);
|
||||
});
|
||||
|
||||
test('a timer callback immediately after an update does not duplicate output', t => {
|
||||
const { progress, lines, setTime } = fixture(t);
|
||||
progress.stage('正在上传');
|
||||
setTime(5000);
|
||||
progress.update('正在上传，已处理 50 张');
|
||||
t.mock.timers.tick(5000);
|
||||
assert.equal(lines.length, 2);
|
||||
});
|
||||
|
||||
test('changing stage prints immediately and restarts its elapsed time', t => {
|
||||
const { progress, lines, tick } = fixture(t);
|
||||
progress.stage('正在统计');
|
||||
tick(5000);
|
||||
progress.stage('正在上传');
|
||||
assert.equal(lines.at(-1), '正在上传（已用时 0 秒）');
|
||||
tick(5000);
|
||||
assert.equal(lines.at(-1), '正在上传（已用时 5 秒）');
|
||||
});
|
||||
|
||||
test('close is idempotent and prevents timer, stage and update output', t => {
|
||||
const { progress, lines, tick } = fixture(t);
|
||||
progress.stage('正在扫描');
|
||||
progress.close();
|
||||
progress.close();
|
||||
tick(15000);
|
||||
progress.stage('已关闭');
|
||||
progress.update('不应输出');
|
||||
assert.deepEqual(lines, ['正在扫描（已用时 0 秒）']);
|
||||
});
|
||||
|
||||
test('an update can initialize progress when no explicit stage was set', t => {
|
||||
const { progress, lines, tick } = fixture(t);
|
||||
progress.update('正在处理');
|
||||
tick(5000);
|
||||
assert.deepEqual(lines, ['正在处理（已用时 0 秒）', '正在处理（已用时 5 秒）']);
|
||||
});
|
||||
|
||||
test('the interval is unrefed and its exact handle is cleared', t => {
|
||||
let callback;
|
||||
const handle = { unref: t.mock.fn() };
|
||||
const interval = t.mock.method(globalThis, 'setInterval', (fn, ms) => {
|
||||
callback = fn;
|
||||
assert.equal(ms, 5000);
|
||||
return handle;
|
||||
});
|
||||
const clear = t.mock.method(globalThis, 'clearInterval', () => {});
|
||||
const log = t.mock.fn();
|
||||
const progress = createProgress({ log, now: () => 0 });
|
||||
assert.equal(interval.mock.callCount(), 1);
|
||||
assert.equal(handle.unref.mock.callCount(), 1);
|
||||
assert.equal(log.mock.callCount(), 0);
|
||||
progress.close();
|
||||
assert.deepEqual(clear.mock.calls[0].arguments, [handle]);
|
||||
callback();
|
||||
assert.equal(log.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
test('invalid intervals fail before starting a timer', () => {
|
||||
for (const intervalMs of [0, -1, 1.5, NaN, Infinity]) {
|
||||
assert.throws(() => createProgress({ intervalMs }), /正整数/);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,309 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { stringify } from 'yaml';
|
||||
import { loadProjectConfig } from '../lib/project-config.mjs';
|
||||
|
||||
const defaultBase = () => ({ spring: { profiles: { active: 'dev' } } });
|
||||
const datasource = overrides => ({
|
||||
url: 'jdbc:mysql://db.example.test:3307/fixture_db?useSSL=false&serverTimezone=Asia%2FShanghai',
|
||||
username: 'fixture-user', password: 'fixture-password', ...overrides,
|
||||
});
|
||||
const profileDocument = (overrides = {}) => ({
|
||||
spring: { datasource: { dynamic: { datasource: { master: datasource(overrides) } } } },
|
||||
server: { port: 48083 },
|
||||
});
|
||||
const documents = value => (Array.isArray(value) ? value : [value]).map(document => stringify(document)).join('---\n');
|
||||
|
||||
async function fixture(t, { base = defaultBase(), profiles = { dev: profileDocument() } } = {}) {
|
||||
const root = await mkdtemp(join(tmpdir(), 'oss-project-config-test-'));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
const resources = join(root, 'yudao-server/src/main/resources');
|
||||
await mkdir(resources, { recursive: true });
|
||||
await writeFile(join(resources, 'application.yaml'), typeof base === 'string' ? base : documents(base));
|
||||
for (const [name, content] of Object.entries(profiles)) {
|
||||
await writeFile(join(resources, 'application-' + name + '.yaml'), typeof content === 'string' ? content : documents(content));
|
||||
}
|
||||
return root;
|
||||
}
|
||||
|
||||
test('default profile, master credentials and actual server port come from the project YAML', async t => {
|
||||
const projectRoot = await fixture(t);
|
||||
const result = await loadProjectConfig({ projectRoot, env: {} });
|
||||
assert.equal(result.profile, 'dev');
|
||||
assert.deepEqual(result.mysql, {
|
||||
host: 'db.example.test', port: 3307, database: 'fixture_db',
|
||||
user: 'fixture-user', password: 'fixture-password',
|
||||
});
|
||||
assert.equal(result.downloadBaseUrl, 'http://127.0.0.1:48083/admin-api/infra/file');
|
||||
assert.deepEqual(result.sourceFiles, [
|
||||
resolve(projectRoot, 'yudao-server/src/main/resources/application.yaml'),
|
||||
resolve(projectRoot, 'yudao-server/src/main/resources/application-dev.yaml'),
|
||||
]);
|
||||
});
|
||||
|
||||
test('explicit profile overrides environment, which overrides the base active profile', async t => {
|
||||
const projectRoot = await fixture(t, { profiles: {
|
||||
dev: profileDocument({ url: 'jdbc:mysql://dev.example.test/dev_db' }),
|
||||
local: profileDocument({ url: 'jdbc:mysql://local.example.test/local_db' }),
|
||||
prod: profileDocument({ url: 'jdbc:mysql://prod.example.test/prod_db' }),
|
||||
} });
|
||||
const explicit = await loadProjectConfig({ projectRoot, profile: 'local', env: { SPRING_PROFILES_ACTIVE: 'prod' } });
|
||||
const environment = await loadProjectConfig({ projectRoot, env: { SPRING_PROFILES_ACTIVE: 'prod' } });
|
||||
const base = await loadProjectConfig({ projectRoot, env: {} });
|
||||
assert.equal(explicit.mysql.database, 'local_db');
|
||||
assert.equal(environment.mysql.database, 'prod_db');
|
||||
assert.equal(base.mysql.database, 'dev_db');
|
||||
});
|
||||
|
||||
test('single-profile selection resolves a base property or environment placeholder with a default', async t => {
|
||||
const projectRoot = await fixture(t, {
|
||||
base: { selection: { profile: '${APP_PROFILE:dev}' }, spring: { profiles: { active: '${selection.profile}' } } },
|
||||
profiles: { dev: profileDocument(), local: profileDocument() },
|
||||
});
|
||||
assert.equal((await loadProjectConfig({ projectRoot, env: {} })).profile, 'dev');
|
||||
assert.equal((await loadProjectConfig({ projectRoot, env: { APP_PROFILE: 'local' } })).profile, 'local');
|
||||
});
|
||||
|
||||
test('explicit profile works even when the base active profile is absent or unusable', async t => {
|
||||
const projectRoot = await fixture(t, { base: { spring: { profiles: { active: ['unused', 'group'] } } } });
|
||||
const result = await loadProjectConfig({ projectRoot, profile: 'dev', env: {} });
|
||||
assert.equal(result.profile, 'dev');
|
||||
});
|
||||
|
||||
test('multiple, missing and path-traversal profiles fail with actionable selection errors', async t => {
|
||||
for (const active of [['dev', 'prod'], 'dev,prod', 'dev prod', '', '../prod', null, undefined]) {
|
||||
const projectRoot = await fixture(t, { base: { spring: { profiles: { active } } } });
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: {} }), /单个 profile.*--profile/);
|
||||
}
|
||||
const projectRoot = await fixture(t);
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: { SPRING_PROFILES_ACTIVE: 'dev,prod' } }), /单个 profile/);
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, profile: '../dev', env: {} }), /单个 profile/);
|
||||
});
|
||||
|
||||
test('multiple unconditional documents merge recursively across base and selected profile', async t => {
|
||||
const projectRoot = await fixture(t, {
|
||||
base: [
|
||||
{ spring: { profiles: { active: 'dev' }, datasource: { dynamic: { primary: 'main', datasource: { main: { username: 'base-user', password: 'base-password' } } } } } },
|
||||
{ spring: { datasource: { dynamic: { datasource: { main: { url: 'jdbc:mysql://base.example.test/base_db' } } } } }, server: { port: 8081 } },
|
||||
],
|
||||
profiles: { dev: [
|
||||
{ spring: { datasource: { dynamic: { datasource: { main: { url: 'jdbc:mysql://selected.example.test:3308/selected_db' } } } } } },
|
||||
{ spring: { datasource: { dynamic: { datasource: { main: { password: 'selected-password' } } } } }, server: { port: 48083 } },
|
||||
] },
|
||||
});
|
||||
const result = await loadProjectConfig({ projectRoot, env: {} });
|
||||
assert.deepEqual(result.mysql, { host: 'selected.example.test', port: 3308, database: 'selected_db', user: 'base-user', password: 'selected-password' });
|
||||
assert.equal(result.downloadBaseUrl, 'http://127.0.0.1:48083/admin-api/infra/file');
|
||||
});
|
||||
|
||||
test('environment directly overrides master URL, username and password', async t => {
|
||||
const projectRoot = await fixture(t);
|
||||
const result = await loadProjectConfig({ projectRoot, env: {
|
||||
SPRING_DATASOURCE_DYNAMIC_DATASOURCE_MASTER_URL: 'jdbc:mysql://override.example.test:3310/env_db',
|
||||
SPRING_DATASOURCE_DYNAMIC_DATASOURCE_MASTER_USERNAME: 'env-user',
|
||||
SPRING_DATASOURCE_DYNAMIC_DATASOURCE_MASTER_PASSWORD: ' env:pass#word ',
|
||||
} });
|
||||
assert.deepEqual(result.mysql, { host: 'override.example.test', port: 3310, database: 'env_db', user: 'env-user', password: ' env:pass#word ' });
|
||||
});
|
||||
|
||||
test('dynamic primary environment override selects the named source instead of master or slave', async t => {
|
||||
const document = profileDocument();
|
||||
document.spring.datasource.dynamic.datasource.reporting = datasource({ url: 'jdbc:mysql://reporting.example.test/reporting_db' });
|
||||
document.spring.datasource.dynamic.datasource.slave = datasource({ url: 'jdbc:mysql://slave.example.test/slave_db' });
|
||||
const projectRoot = await fixture(t, { profiles: { dev: document } });
|
||||
const result = await loadProjectConfig({ projectRoot, env: {
|
||||
SPRING_DATASOURCE_DYNAMIC_PRIMARY: 'reporting',
|
||||
SPRING_DATASOURCE_DYNAMIC_DATASOURCE_REPORTING_PASSWORD: 'reporting-env-password',
|
||||
} });
|
||||
assert.equal(result.mysql.database, 'reporting_db');
|
||||
assert.equal(result.mysql.password, 'reporting-env-password');
|
||||
});
|
||||
|
||||
test('required fields support property references and nested environment defaults', async t => {
|
||||
const document = profileDocument({
|
||||
url: 'jdbc:mysql://${DB_HOST:${database.host}}:${DB_PORT:${database.port:3306}}/${database.name}',
|
||||
username: '${DB_USER:${database.user}}',
|
||||
password: '${DB_PASSWORD:${database.password}}',
|
||||
});
|
||||
document.database = { host: 'property.example.test', name: 'property_db', user: 'property-user', password: 'property-password' };
|
||||
const projectRoot = await fixture(t, { profiles: { dev: document } });
|
||||
const base = await loadProjectConfig({ projectRoot, env: {} });
|
||||
assert.deepEqual(base.mysql, { host: 'property.example.test', port: 3306, database: 'property_db', user: 'property-user', password: 'property-password' });
|
||||
const overridden = await loadProjectConfig({ projectRoot, env: { DB_HOST: 'env.example.test', DB_PORT: '3309', DB_PASSWORD: 'env-password' } });
|
||||
assert.equal(overridden.mysql.host, 'env.example.test');
|
||||
assert.equal(overridden.mysql.port, 3309);
|
||||
assert.equal(overridden.mysql.password, 'env-password');
|
||||
});
|
||||
|
||||
test('unrelated unresolved placeholders, service credentials and cycles are never resolved', async t => {
|
||||
const document = profileDocument();
|
||||
document.unrelated = { secret: '${MISSING_SERVICE_SECRET}', a: '${unrelated.b}', b: '${unrelated.a}', encrypted: 'ENC(unrelated-service-secret)' };
|
||||
const projectRoot = await fixture(t, { profiles: { dev: document } });
|
||||
assert.equal((await loadProjectConfig({ projectRoot, env: {} })).mysql.database, 'fixture_db');
|
||||
});
|
||||
|
||||
test('HTTP context path, admin prefix and SSL settings determine the download URL', async t => {
|
||||
const document = profileDocument();
|
||||
document.server = { port: 8443, servlet: { contextPath: '/backend/' }, ssl: { enabled: true } };
|
||||
document.yudao = { web: { adminApi: { prefix: '/management/' } } };
|
||||
const projectRoot = await fixture(t, { profiles: { dev: document } });
|
||||
const result = await loadProjectConfig({ projectRoot, env: {} });
|
||||
assert.equal(result.downloadBaseUrl, 'https://127.0.0.1:8443/backend/management/infra/file');
|
||||
const overridden = await loadProjectConfig({ projectRoot, env: {
|
||||
SERVER_PORT: '9080', SERVER_SERVLET_CONTEXT_PATH: '/overridden',
|
||||
YUDAO_WEB_ADMIN_API_PREFIX: '/admin', SERVER_SSL_ENABLED: 'false',
|
||||
} });
|
||||
assert.equal(overridden.downloadBaseUrl, 'http://127.0.0.1:9080/overridden/admin/infra/file');
|
||||
});
|
||||
|
||||
test('Spring defaults apply to missing primary, HTTP port and API prefix', async t => {
|
||||
const document = profileDocument();
|
||||
delete document.server;
|
||||
const projectRoot = await fixture(t, { profiles: { dev: document } });
|
||||
assert.equal((await loadProjectConfig({ projectRoot, env: {} })).downloadBaseUrl, 'http://127.0.0.1:8080/admin-api/infra/file');
|
||||
});
|
||||
|
||||
test('an explicit root context path does not produce duplicate slashes', async t => {
|
||||
const document = profileDocument();
|
||||
document.server.servlet = { 'context-path': '/' };
|
||||
document.yudao = { web: { 'admin-api': { prefix: '/' } } };
|
||||
const projectRoot = await fixture(t, { profiles: { dev: document } });
|
||||
assert.equal((await loadProjectConfig({ projectRoot, env: {} })).downloadBaseUrl, 'http://127.0.0.1:48083/infra/file');
|
||||
});
|
||||
|
||||
test('single-host MySQL URL supports omitted ports and IPv6 without changing database names', async t => {
|
||||
for (const [url, expected] of [
|
||||
['jdbc:mysql://localhost/test_db', { host: 'localhost', port: 3306, database: 'test_db' }],
|
||||
['jdbc:mysql://[::1]:3309/test_db', { host: '::1', port: 3309, database: 'test_db' }],
|
||||
['jdbc:mysql://db.example.test/test%20db', { host: 'db.example.test', port: 3306, database: 'test db' }],
|
||||
]) {
|
||||
const projectRoot = await fixture(t, { profiles: { dev: profileDocument({ url }) } });
|
||||
const result = await loadProjectConfig({ projectRoot, env: {} });
|
||||
assert.deepEqual({ host: result.mysql.host, port: result.mysql.port, database: result.mysql.database }, expected);
|
||||
}
|
||||
});
|
||||
|
||||
test('ambiguous, unsupported and incomplete JDBC URLs are rejected without echoing their values', async t => {
|
||||
for (const url of [
|
||||
'jdbc:mysql:loadbalance://first,second/db',
|
||||
'jdbc:mysql://first,second/db',
|
||||
'jdbc:mysql://first:3306,second:3306/db',
|
||||
'jdbc:postgresql://secret-host/secret-database',
|
||||
'jdbc:mysql://db.example.test',
|
||||
'jdbc:mysql://db.example.test/a/b',
|
||||
'jdbc:mysql://db.example.test:65536/db',
|
||||
'jdbc:mysql://db.example.test:0/db',
|
||||
'jdbc:mysql://user:sensitive-url-password@db.example.test/db',
|
||||
'jdbc:mysql://db.example.test/a%2Fb',
|
||||
]) {
|
||||
const projectRoot = await fixture(t, { profiles: { dev: profileDocument({ url }) } });
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: {} }), error => {
|
||||
assert.match(error.message, /MySQL|jdbc:mysql/);
|
||||
assert.ok(!error.message.includes(url));
|
||||
assert.ok(!error.message.includes('sensitive-url-password'));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test('missing or still-encrypted passwords are rejected, with direct environment override available', async t => {
|
||||
for (const password of [null, undefined, '', 'ENC(sensitive-encrypted-password)']) {
|
||||
const projectRoot = await fixture(t, { profiles: { dev: profileDocument({ password }) } });
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: {} }), error => {
|
||||
assert.match(error.message, /缺少|缺失|ENC/);
|
||||
assert.ok(!error.message.includes('sensitive-encrypted-password'));
|
||||
return true;
|
||||
});
|
||||
assert.equal((await loadProjectConfig({ projectRoot, env: { SPRING_DATASOURCE_DYNAMIC_DATASOURCE_MASTER_PASSWORD: 'provided-password' } })).mysql.password, 'provided-password');
|
||||
}
|
||||
});
|
||||
|
||||
test('required unresolved, malformed or cyclic placeholders stop safely without revealing secrets', async t => {
|
||||
for (const password of [
|
||||
'sensitive-prefix-${MISSING_PASSWORD}',
|
||||
'sensitive-prefix-${MISSING_PASSWORD',
|
||||
'${spring.datasource.dynamic.datasource.master.password}',
|
||||
]) {
|
||||
const projectRoot = await fixture(t, { profiles: { dev: profileDocument({ password }) } });
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: {} }), error => {
|
||||
assert.match(error.message, /占位符/);
|
||||
assert.ok(!error.message.includes('sensitive-prefix'));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test('numeric YAML scalar credentials retain bigint precision', async t => {
|
||||
const document = profileDocument({ password: 900719925474099312345n });
|
||||
const projectRoot = await fixture(t, { profiles: { dev: document } });
|
||||
assert.equal((await loadProjectConfig({ projectRoot, env: {} })).mysql.password, '900719925474099312345');
|
||||
});
|
||||
|
||||
test('config imports, included profiles and a selected profile group require explicit configuration', async t => {
|
||||
for (const base of [
|
||||
{ spring: { profiles: { active: 'dev' }, config: { import: 'optional:file:sensitive-location' } } },
|
||||
{ spring: { profiles: { active: 'dev', include: ['common'] } } },
|
||||
{ spring: { profiles: { active: 'dev', group: { dev: ['common', 'database'] } } } },
|
||||
]) {
|
||||
const projectRoot = await fixture(t, { base });
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: {} }), error => {
|
||||
assert.match(error.message, /import|include|配置组/);
|
||||
assert.ok(!error.message.includes('sensitive-location'));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
const projectRoot = await fixture(t);
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: { SPRING_CONFIG_IMPORT: 'optional:file:sensitive-location' } }), /import/);
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: { SPRING_PROFILES_GROUP_DEV: 'common,database' } }), /配置组/);
|
||||
});
|
||||
|
||||
test('an unrelated profile group does not affect an explicitly selected concrete profile', async t => {
|
||||
const projectRoot = await fixture(t, { base: { spring: { profiles: { active: 'dev', group: { production: ['prod', 'shared'] } } } } });
|
||||
assert.equal((await loadProjectConfig({ projectRoot, env: {} })).profile, 'dev');
|
||||
});
|
||||
|
||||
test('conditional YAML documents and active-profile redefinition are rejected', async t => {
|
||||
for (const content of [
|
||||
[profileDocument(), { spring: { config: { activate: { 'on-profile': 'dev' } } }, server: { port: 8080 } }],
|
||||
[profileDocument(), { spring: { profiles: 'dev' }, server: { port: 8080 } }],
|
||||
[profileDocument(), { spring: { profiles: { active: 'prod' } } }],
|
||||
]) {
|
||||
const projectRoot = await fixture(t, { profiles: { dev: content } });
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: {} }), /条件|歧义/);
|
||||
}
|
||||
});
|
||||
|
||||
test('missing files and malformed YAML errors do not expose parser source or secrets', async t => {
|
||||
const projectRoot = await fixture(t, { base: 'spring: [sensitive-password' });
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: {} }), error => {
|
||||
assert.match(error.message, /YAML/);
|
||||
assert.ok(!error.message.includes('sensitive-password'));
|
||||
return true;
|
||||
});
|
||||
const validRoot = await fixture(t);
|
||||
await assert.rejects(loadProjectConfig({ projectRoot: validRoot, profile: 'missing', env: {} }), /--profile/);
|
||||
await assert.rejects(loadProjectConfig({ projectRoot: '', env: {} }), /项目根目录/);
|
||||
});
|
||||
|
||||
test('duplicate keys inside one YAML document cannot silently choose credentials', async t => {
|
||||
const projectRoot = await fixture(t, { base: 'spring:\n profiles:\n active: dev\nspring:\n password: sensitive-password\n' });
|
||||
await assert.rejects(loadProjectConfig({ projectRoot, env: {} }), error => {
|
||||
assert.match(error.message, /YAML/);
|
||||
assert.ok(!error.message.includes('sensitive-password'));
|
||||
return true;
|
||||
});
|
||||
});
|
||||
|
||||
test('invalid HTTP ports and route prefixes fail instead of generating unusable URLs', async t => {
|
||||
const projectRoot = await fixture(t);
|
||||
for (const env of [
|
||||
{ SERVER_PORT: '0' }, { SERVER_PORT: '65536' }, { SERVER_PORT: 'not-a-port' },
|
||||
{ SERVER_SERVLET_CONTEXT_PATH: 'missing-leading-slash' },
|
||||
{ YUDAO_WEB_ADMIN_API_PREFIX: '/admin?query' },
|
||||
{ SERVER_SERVLET_CONTEXT_PATH: '/a/../b' },
|
||||
{ SERVER_SSL_ENABLED: 'unknown' },
|
||||
]) await assert.rejects(loadProjectConfig({ projectRoot, env }), /HTTP/);
|
||||
});
|
||||
@@ -0,0 +1,171 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { containsLegacyReference, discoverReferenceColumns, rewriteReferences } from '../lib/references.mjs';
|
||||
|
||||
const oldUrl = 'https://old.example/admin-api/infra/file/1/get/a.png';
|
||||
const newUrl = 'https://bucket.example/migrated/a.png';
|
||||
const otherUrl = 'https://elsewhere.example/a.png';
|
||||
const replacements = new Map([[oldUrl, newUrl]]);
|
||||
|
||||
test('replaces an exact URL once and never cascades a destination mapping', () => {
|
||||
assert.deepEqual(rewriteReferences(oldUrl, new Map([[oldUrl, newUrl], [newUrl, otherUrl]])), {
|
||||
value: newUrl, changed: true, replacements: 1
|
||||
});
|
||||
assert.deepEqual(rewriteReferences(null, replacements), { value: null, changed: false, replacements: 0 });
|
||||
});
|
||||
|
||||
test('replaces comma lists, repeated values and rich text while preserving surrounding text', () => {
|
||||
const input = `${oldUrl},${otherUrl}, ${oldUrl}\n<a href="${oldUrl}"><img src=${oldUrl}></a>`;
|
||||
const expected = `${newUrl},${otherUrl}, ${newUrl}\n<a href="${newUrl}"><img src=${newUrl}></a>`;
|
||||
assert.deepEqual(rewriteReferences(input, replacements), { value: expected, changed: true, replacements: 4 });
|
||||
assert.equal(rewriteReferences(`[image](${oldUrl}),`, replacements).value, `[image](${newUrl}),`);
|
||||
});
|
||||
|
||||
test('never matches URL prefixes, query extensions, fragments or external embedded URLs', () => {
|
||||
const unchanged = [
|
||||
`${oldUrl}.bak`, `${oldUrl}/thumbnail`, `${oldUrl}?width=200`, `${oldUrl}#preview`,
|
||||
`${oldUrl}%20other`, `${oldUrl}&x=1`, `${oldUrl};v=1`, otherUrl,
|
||||
`https://external.example/?redirect=${oldUrl}`,
|
||||
`https://external.example/?redirect=x,${oldUrl}`,
|
||||
`${oldUrl}?url=x,${oldUrl}`
|
||||
];
|
||||
for (const value of unchanged) {
|
||||
assert.deepEqual(rewriteReferences(value, replacements), { value, changed: false, replacements: 0 }, value);
|
||||
}
|
||||
});
|
||||
|
||||
test('only rewrites queries and fragments explicitly present in the dictionary', () => {
|
||||
const url = `${oldUrl}?v=2#original`;
|
||||
assert.equal(rewriteReferences(`<img src="${url}">`, new Map([[url, newUrl]])).value, `<img src="${newUrl}">`);
|
||||
assert.equal(rewriteReferences(url, replacements).value, url);
|
||||
});
|
||||
|
||||
test('keeps URL encoding literal and supports Unicode paths', () => {
|
||||
const encoded = oldUrl.replace('a.png', '%E5%A4%B4%E5%83%8F%20a.png');
|
||||
const unicode = oldUrl.replace('a.png', '头像.png');
|
||||
const mapping = new Map([[encoded, newUrl], [unicode, `${newUrl}?kind=unicode`]]);
|
||||
assert.equal(rewriteReferences(`<img src="${encoded}"> ${unicode}`, mapping).replacements, 2);
|
||||
assert.equal(rewriteReferences(encoded.replace('%20', ' '), mapping).changed, false);
|
||||
assert.equal(rewriteReferences(encoded.toLowerCase(), mapping).changed, false);
|
||||
});
|
||||
|
||||
test('JSON text retains formatting and untouched escapes; keys remain unchanged', () => {
|
||||
const escapedUrl = oldUrl.replaceAll('/', '\\/');
|
||||
const input = `{\n "${oldUrl}": "key stays", "avatar" : "${escapedUrl}",\n "nested": [{"pic":"${oldUrl}"}], "untouched":"\\u4e2d"\n}`;
|
||||
const expected = `{\n "${oldUrl}": "key stays", "avatar" : "${newUrl}",\n "nested": [{"pic":"${newUrl}"}], "untouched":"\\u4e2d"\n}`;
|
||||
assert.deepEqual(rewriteReferences(input, replacements), { value: expected, changed: true, replacements: 2 });
|
||||
assert.equal(rewriteReferences(JSON.stringify(JSON.stringify({ avatar: oldUrl })), replacements).value,
|
||||
JSON.stringify(JSON.stringify({ avatar: newUrl })));
|
||||
});
|
||||
|
||||
test('parsed JSON is recursively rewritten without changing input or prototype', () => {
|
||||
const input = JSON.parse(`{"__proto__":{"url":"${oldUrl}"},"photos":["${oldUrl}",null,4],"url":"${otherUrl}"}`);
|
||||
const result = rewriteReferences(input, replacements);
|
||||
assert.equal(result.replacements, 2);
|
||||
assert.equal(input.photos[0], oldUrl);
|
||||
assert.equal(result.value.photos[0], newUrl);
|
||||
assert.equal(result.value.__proto__.url, newUrl);
|
||||
assert.equal(Object.getPrototypeOf(result.value), Object.prototype);
|
||||
assert.equal(rewriteReferences({ url: otherUrl }, replacements).changed, false);
|
||||
});
|
||||
|
||||
test('malformed JSON safely uses literal URL replacement without changing escapes', () => {
|
||||
const input = `{"avatar":"${oldUrl}",}`;
|
||||
assert.equal(rewriteReferences(input, replacements).value, `{"avatar":"${newUrl}",}`);
|
||||
});
|
||||
|
||||
test('remaining-reference detection finds Unicode-escaped JSON URLs with unknown queries', () => {
|
||||
const withQuery = `${oldUrl}?unmapped=1#preview`;
|
||||
const unicodeJson = JSON.stringify({ avatar: withQuery }).replaceAll('/', '\\u002f');
|
||||
const rewritten = rewriteReferences(unicodeJson, replacements);
|
||||
assert.equal(rewritten.changed, false);
|
||||
assert.equal(containsLegacyReference(rewritten.value, ['1']), true);
|
||||
assert.equal(containsLegacyReference(rewritten.value, ['2']), false);
|
||||
});
|
||||
|
||||
test('remaining-reference detection recursively handles nested JSON strings and parsed values', () => {
|
||||
const nested = JSON.stringify({ data: JSON.stringify({ photos: [`${oldUrl}?v=2`] }).replaceAll('/', '\\u002f') });
|
||||
assert.equal(containsLegacyReference(nested, new Set([1])), true);
|
||||
assert.equal(containsLegacyReference({ data: [null, { image: oldUrl }] }, [1]), true);
|
||||
assert.equal(containsLegacyReference(JSON.stringify(JSON.stringify(`${oldUrl}#unknown`)), ['1']), true);
|
||||
assert.equal(containsLegacyReference(rewriteReferences(JSON.stringify({ image: oldUrl }), replacements).value, [1]), false);
|
||||
});
|
||||
|
||||
test('remaining-reference detection handles escaped plain text, exact config IDs, and excludes JSON keys', () => {
|
||||
assert.equal(containsLegacyReference(`<img src="${oldUrl.replaceAll('/', '\\/')}?size=2">`, [1]), true);
|
||||
assert.equal(containsLegacyReference(`${oldUrl}?query=2`, [11]), false);
|
||||
assert.equal(containsLegacyReference(oldUrl.replace('/file/1/', '/file/11/'), [1]), false);
|
||||
assert.equal(containsLegacyReference(JSON.stringify({ [oldUrl]: 'not a URL value' }), [1]), false);
|
||||
assert.equal(containsLegacyReference({ [oldUrl]: 'not a URL value' }, [1]), false);
|
||||
assert.equal(containsLegacyReference(null, [1]), false);
|
||||
assert.equal(containsLegacyReference(oldUrl, []), false);
|
||||
const cycle = {};
|
||||
cycle.self = cycle;
|
||||
assert.equal(containsLegacyReference(cycle, [1]), false);
|
||||
});
|
||||
|
||||
test('discovery binds schema names and excludes unsafe or non-content columns', async () => {
|
||||
const row = (tableName, columnName, extra = {}) => ({
|
||||
schemaName: 'app', tableName, columnName, dataType: 'varchar', extra: '',
|
||||
generationExpression: '', primaryKey: 'id', primaryKeyCount: 1, ...extra
|
||||
});
|
||||
const rows = [
|
||||
row('member_user', 'avatar'), row('member_user', 'id'),
|
||||
row('member_user', 'generated', { generationExpression: 'avatar', extra: 'VIRTUAL GENERATED' }),
|
||||
row('member_user', 'stored', { extra: 'STORED GENERATED' }),
|
||||
row('member_user', 'seq', { extra: 'auto_increment' }),
|
||||
row('member_user', 'defaulted', { extra: 'DEFAULT_GENERATED' }),
|
||||
row('member_com_auth', 'auth_extra_json', { dataType: 'json' }),
|
||||
row('infra_file', 'url'), row('infra_file_content', 'path'), row('infra_file_config', 'config'),
|
||||
row('migration_files', 'old_url'), row('infra_file_migration_journal', 'old_url'),
|
||||
row('infra_file_oss_migration_runs', 'settings'), row('infra_file_oss_migration_items', 'data'),
|
||||
row('infra_file_oss_migration_changes', 'before_json'), row('INFRA_FILE_OSS_MIGRATION_CHANGES', 'after_json'),
|
||||
row('no_key', 'a', { primaryKeyCount: null, primaryKey: null }),
|
||||
row('no_key', 'b', { primaryKeyCount: null, primaryKey: null }),
|
||||
row('composite', 'a', { primaryKeyCount: '2' }),
|
||||
row('skip_me', 'a'), row('skip_qualified', 'a'),
|
||||
];
|
||||
const schemas = ['app', 'odd` schema; DROP TABLE x'];
|
||||
const connection = { async execute(sql, params) {
|
||||
assert.deepEqual(params, [...schemas, ...schemas]);
|
||||
assert.ok(!sql.includes(schemas[1]));
|
||||
assert.match(sql, /t\.TABLE_TYPE = 'BASE TABLE'/);
|
||||
assert.match(sql, /c\.DATA_TYPE IN \('char', 'varchar', 'tinytext', 'text', 'mediumtext', 'longtext', 'json'\)/);
|
||||
return [rows];
|
||||
} };
|
||||
const result = await discoverReferenceColumns(connection, schemas, new Set(['skip_me', 'APP.skip_qualified']));
|
||||
assert.deepEqual(result.columns, [
|
||||
{ schema: 'app', table: 'member_user', column: 'avatar', primaryKey: 'id', dataType: 'varchar' },
|
||||
{ schema: 'app', table: 'member_user', column: 'defaulted', primaryKey: 'id', dataType: 'varchar' },
|
||||
{ schema: 'app', table: 'member_com_auth', column: 'auth_extra_json', primaryKey: 'id', dataType: 'json' }
|
||||
]);
|
||||
assert.equal(result.skipped.length, 13);
|
||||
for (const table of ['infra_file_oss_migration_runs', 'infra_file_oss_migration_items', 'infra_file_oss_migration_changes', 'INFRA_FILE_OSS_MIGRATION_CHANGES']) {
|
||||
assert.equal(result.skipped.find((item) => item.table === table).reason, 'excluded table');
|
||||
assert.ok(!result.columns.some((item) => item.table === table));
|
||||
}
|
||||
assert.equal(result.skipped.find((item) => item.table === 'no_key').reason, 'no primary key');
|
||||
assert.equal(result.skipped.find((item) => item.table === 'composite').reason, 'composite primary key');
|
||||
});
|
||||
|
||||
test('discovery makes no connection for an empty schema list and validates names', async () => {
|
||||
assert.deepEqual(await discoverReferenceColumns({}, []), { columns: [], skipped: [] });
|
||||
await assert.rejects(discoverReferenceColumns({}, ['']), /schema names/);
|
||||
});
|
||||
|
||||
test('discovery explicitly excludes known technical and audit logs while retaining business log tables', async () => {
|
||||
const excluded = [
|
||||
'infra_api_access_log', 'infra_api_error_log', 'infra_job_log',
|
||||
'system_operate_log', 'system_login_log', 'SYSTEM_LOGIN_LOG',
|
||||
];
|
||||
const retained = ['member_activity_log', 'chat_message_log', 'custom_audit_log', 'infra_api_access_log_archive'];
|
||||
const rows = [...excluded, ...retained].flatMap(tableName => ['content', 'details'].map(columnName => ({
|
||||
schemaName: 'app', tableName, columnName, dataType: 'text', extra: '',
|
||||
generationExpression: '', primaryKey: 'id', primaryKeyCount: 1,
|
||||
})));
|
||||
const connection = { async execute() { return [rows]; } };
|
||||
const result = await discoverReferenceColumns(connection, ['app']);
|
||||
assert.deepEqual(result.skipped, excluded.map(table => ({ schema: 'app', table, reason: 'excluded table' })));
|
||||
assert.deepEqual([...new Set(result.columns.map(column => column.table))], retained);
|
||||
assert.equal(result.columns.length, retained.length * 2);
|
||||
});
|
||||
@@ -0,0 +1,56 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtemp, mkdir, readFile, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { normalizeConfig } from '../lib/core.mjs';
|
||||
import { parseArgs, prepareInput, persistGeneratedConfig } from '../migrate.mjs';
|
||||
|
||||
test('zero arguments default to read-only plan, with optional profile or explicit actions', () => {
|
||||
const args = parseArgs([]);
|
||||
assert.equal(args.command, 'plan'); assert.ok(args.configPath.endsWith('config.local.json'));
|
||||
assert.equal(parseArgs(['--profile', 'dev']).command, 'plan');
|
||||
assert.equal(parseArgs(['init']).command, 'init');
|
||||
assert.equal(parseArgs(['apply', '--limit', '50']).limit, 50);
|
||||
assert.equal(parseArgs(['apply', '--run-id', 'new-batch']).runId, 'new-batch');
|
||||
assert.equal(parseArgs(['--help']).help, true);
|
||||
assert.throws(() => parseArgs(['plan', '--limit', '50']));
|
||||
assert.throws(() => parseArgs(['delete']));
|
||||
});
|
||||
|
||||
test('automatic configuration inherits project credentials without exporting a password variable', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'oss-runtime-'));
|
||||
try {
|
||||
const resources = join(root, 'yudao-server/src/main/resources'); await mkdir(resources, { recursive: true });
|
||||
await writeFile(join(resources, 'application.yaml'), 'spring:\n profiles:\n active: dev\n');
|
||||
await writeFile(join(resources, 'application-dev.yaml'), `server:\n port: 48083\nspring:\n datasource:\n dynamic:\n primary: master\n datasource:\n master:\n url: jdbc:mysql://127.0.0.1:3306/fixture_db\n username: fixture_user\n password: fixture-secret\n`);
|
||||
const args = { command: 'init', configPath: join(root, 'config.local.json') };
|
||||
const prepared = await prepareInput(args, root);
|
||||
assert.equal(prepared.projectManaged, true); assert.equal(prepared.exists, false);
|
||||
assert.equal(prepared.input.mysql.database, 'fixture_db'); assert.equal(prepared.input.mysql.password, 'fixture-secret');
|
||||
assert.equal(prepared.input.sourceMode, undefined); // defaults must not overwrite a resumed run
|
||||
assert.equal(prepared.input._projectDownloadBaseUrl, 'http://127.0.0.1:48083/admin-api/infra/file');
|
||||
const config = normalizeConfig({ ...prepared.input, sourceMode: 'database', sourceConfigIds: ['5'], targetConfigId: '30', runId: 'auto-test', _databaseIdentity: { server: 'test-uuid', schema: 'fixture_db' } });
|
||||
await persistGeneratedConfig(config, args, prepared);
|
||||
const text = await readFile(args.configPath, 'utf8');
|
||||
assert.ok(!text.includes('fixture-secret')); assert.ok(!text.includes('"mysql"'));
|
||||
const saved = JSON.parse(text); assert.equal(saved.runId, 'auto-test'); assert.equal(saved.projectProfile, 'dev'); assert.equal(saved.targetConfigId, '30');
|
||||
const reloaded = await prepareInput(args, root);
|
||||
assert.equal(reloaded.input.mysql.password, 'fixture-secret'); assert.equal(reloaded.input.runId, 'auto-test');
|
||||
} finally { await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('manual connection remains supported and is never overwritten with project settings', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'oss-manual-'));
|
||||
try {
|
||||
const path = join(root, 'manual.json');
|
||||
const raw = { mysql: { host: 'manual.example', database: 'manual_db', user: 'manual', password: 'manual-secret' }, runId: 'manual-run', sourceConfigIds: ['5'], targetConfigId: '30', sourceMode: 'database' };
|
||||
await writeFile(path, JSON.stringify(raw));
|
||||
const args = { command: 'init', configPath: path, explicitConfig: true };
|
||||
const prepared = await prepareInput(args, root);
|
||||
assert.equal(prepared.projectManaged, false); assert.equal(prepared.input.mysql.host, 'manual.example');
|
||||
await persistGeneratedConfig(normalizeConfig(prepared.input), args, prepared);
|
||||
assert.deepEqual(JSON.parse(await readFile(path, 'utf8')), raw);
|
||||
await assert.rejects(prepareInput({ ...args, profile: 'prod' }, root), /不能同时使用/);
|
||||
} finally { await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
@@ -0,0 +1,139 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { verifySourceBatch } from '../lib/source-batch.mjs';
|
||||
|
||||
const config = { mysql: { database: 'source_batch_test' } };
|
||||
const first = { contentId: '9007199254740993', configId: '5', path: '相册/Résumé.PNG', size: 1234, sha256: 'a'.repeat(64) };
|
||||
const second = { contentId: '9007199254740995', configId: '9007199254740999', path: '头像.png', size: 4321, sha256: 'b'.repeat(64) };
|
||||
const latest = item => ({ id: item.contentId, expected_content_id: item.contentId });
|
||||
const locked = (...items) => items.map(item => ({ id: item.contentId }));
|
||||
const content = (item, overrides = {}) => ({ id: item.contentId, config_id: item.configId, path: item.path, size: String(item.size), sha256: item.sha256, ...overrides });
|
||||
const changed = error => error.code === 'MIGRATION_SOURCE_CHANGED';
|
||||
|
||||
class FakeDb {
|
||||
constructor(responses) {
|
||||
this.responses = responses;
|
||||
this.calls = [];
|
||||
}
|
||||
async execute(sql, params) {
|
||||
assert.match(sql.trim(), /^SELECT\b/);
|
||||
if (this.calls.length === 1) assert.doesNotMatch(sql, /FOR UPDATE|LOCK IN SHARE MODE/i);
|
||||
else assert.match(sql, /FOR UPDATE$/);
|
||||
assert.doesNotMatch(sql, /\b(?:START TRANSACTION|BEGIN|COMMIT|ROLLBACK)\b/i);
|
||||
this.calls.push({ sql, params });
|
||||
if (this.calls.length > this.responses.length) throw new Error('Unexpected database query');
|
||||
const result = this.responses[this.calls.length - 1];
|
||||
if (result instanceof Error) throw result;
|
||||
return [result];
|
||||
}
|
||||
}
|
||||
|
||||
test('validates in three SELECTs with a primary-key-only range lock and a separate digest read', async () => {
|
||||
const db = new FakeDb([locked(first, second), [latest(second), latest(first)], [content(first), content(second)]]);
|
||||
await verifySourceBatch(db, config, [second, first]);
|
||||
assert.equal(db.calls.length, 3);
|
||||
const [range, metadata, hashes] = db.calls;
|
||||
assert.match(range.sql, /^SELECT id FROM /);
|
||||
assert.match(range.sql, /WHERE id>=\? ORDER BY id FOR UPDATE$/);
|
||||
assert.doesNotMatch(range.sql, /\b(?:content|deleted|config_id|path|CASE|LIMIT)\b|SHA2|OCTET_LENGTH|id\s+IN\s*\(/i,
|
||||
'lock every row and gap above the minimum ID, including deleted and unrelated-path rows');
|
||||
assert.deepEqual(range.params, [first.contentId]);
|
||||
assert.match(metadata.sql, /^SELECT id,CASE WHEN config_id=\? AND path=\? THEN \?/);
|
||||
assert.match(metadata.sql, /END AS expected_content_id/);
|
||||
assert.match(metadata.sql, /WHERE deleted=0 AND id>=\? AND \(\(config_id=\? AND path=\? AND id>=\?\) OR/);
|
||||
assert.match(metadata.sql, /ORDER BY id DESC$/);
|
||||
assert.doesNotMatch(metadata.sql, /\bcontent\b|OCTET_LENGTH|SHA2|\bLIMIT\b|id\s+IN\s*\(/i,
|
||||
'metadata lookup must examine newer IDs without projecting BLOB expressions');
|
||||
assert.deepEqual(metadata.params, [
|
||||
second.configId, second.path, second.contentId,
|
||||
first.configId, first.path, first.contentId,
|
||||
first.contentId,
|
||||
second.configId, second.path, second.contentId,
|
||||
first.configId, first.path, first.contentId
|
||||
]);
|
||||
assert.match(hashes.sql, /OCTET_LENGTH\(content\) AS size,LOWER\(SHA2\(content,256\)\) AS sha256/);
|
||||
assert.match(hashes.sql, /WHERE deleted=0 AND id IN \(\?,\?\) FOR UPDATE$/);
|
||||
assert.doesNotMatch(hashes.sql, /ORDER BY|\bCASE\b|\bJOIN\b|\bLIMIT\b/i);
|
||||
assert.deepEqual(hashes.params, [second.contentId, first.contentId]);
|
||||
});
|
||||
|
||||
test('uses server-provided grouping IDs and accepts collation-equivalent case/accent spellings', async () => {
|
||||
const db = new FakeDb([locked(first), [latest(first)], [content(first, { path: '相册/RESUME.png' })]]);
|
||||
await verifySourceBatch(db, config, [first]);
|
||||
assert.deepEqual(db.calls[1].params, [first.configId, first.path, first.contentId, first.contentId, first.configId, first.path, first.contentId]);
|
||||
assert.doesNotMatch(db.calls[1].sql, /BINARY\s+path|LOWER\s*\(path\)/i);
|
||||
});
|
||||
|
||||
test('rejects a newer active version before performing the batch digest read', async () => {
|
||||
const db = new FakeDb([locked(first, second), [
|
||||
latest(second), latest(first), { id: '9007199254740994', expected_content_id: first.contentId }
|
||||
]]);
|
||||
await assert.rejects(verifySourceBatch(db, config, [first, second]), changed);
|
||||
assert.equal(db.calls.length, 2);
|
||||
});
|
||||
|
||||
test('rejects a deleted/missing group or an unexpected server grouping ID before reading hashes', async t => {
|
||||
for (const [label, rows] of [
|
||||
['missing group', [latest(first)]],
|
||||
['all deleted', []],
|
||||
['unexpected group', [latest(first), { id: second.contentId, expected_content_id: '88' }]],
|
||||
['null CASE result', [latest(first), { id: second.contentId, expected_content_id: null }]]
|
||||
]) {
|
||||
await t.test(label, async () => {
|
||||
const db = new FakeDb([locked(first, second), rows]);
|
||||
await assert.rejects(verifySourceBatch(db, config, [first, second]), changed);
|
||||
assert.equal(db.calls.length, 2);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test('rejects disappearance, size changes, hash changes and wrong IDs in the digest result', async t => {
|
||||
for (const [label, rows] of [
|
||||
['missing content', [content(first)]],
|
||||
['wrong size', [content(first), content(second, { size: second.size + 1 })]],
|
||||
['same size different hash', [content(first), content(second, { sha256: 'c'.repeat(64) })]],
|
||||
['wrong ID', [content(first), content(second, { id: '9007199254740996' })]],
|
||||
['duplicate ID', [content(first), content(first)]]
|
||||
]) {
|
||||
await t.test(label, async () => {
|
||||
const db = new FakeDb([locked(first, second), [latest(first), latest(second)], rows]);
|
||||
await assert.rejects(verifySourceBatch(db, config, [first, second]), changed);
|
||||
assert.equal(db.calls.length, 3);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test('preserves unsigned bigint IDs and binds all paths instead of interpolating them', async () => {
|
||||
const large = { ...first, contentId: '18446744073709551615', path: "x.png' OR '1'='1" };
|
||||
const db = new FakeDb([locked(large), [latest(large)], [content(large)]]);
|
||||
await verifySourceBatch(db, { mysql: { database: 'app`db' } }, [large]);
|
||||
for (const call of db.calls) {
|
||||
assert.match(call.sql, /`app``db`\.`infra_file_content`/);
|
||||
assert.ok(!call.sql.includes(large.path));
|
||||
assert.ok(!call.sql.includes(large.contentId));
|
||||
}
|
||||
assert.deepEqual(db.calls[2].params, [large.contentId]);
|
||||
});
|
||||
|
||||
test('rejects empty batches, duplicate IDs and unsafe numeric IDs before any SQL', async t => {
|
||||
for (const [label, batch, expectedError] of [
|
||||
['empty batch', [], /不能为空/],
|
||||
['duplicate content ID', [first, { ...first }], /重复/],
|
||||
['unsafe content ID', [{ ...first, contentId: Number.MAX_SAFE_INTEGER + 1 }], /安全整数/],
|
||||
['unsafe config ID', [{ ...first, configId: Number.MAX_SAFE_INTEGER + 1 }], /安全整数/],
|
||||
['invalid expected size', [{ ...first, size: Number.MAX_SAFE_INTEGER + 1 }], /预期大小/]
|
||||
]) {
|
||||
await t.test(label, async () => {
|
||||
const db = new FakeDb([]);
|
||||
await assert.rejects(verifySourceBatch(db, config, batch), expectedError);
|
||||
assert.equal(db.calls.length, 0);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test('propagates database lock/read errors and leaves rollback to the transaction owner', async () => {
|
||||
const lockFailure = Object.assign(new Error('lock wait timed out'), { code: 'ER_LOCK_WAIT_TIMEOUT' });
|
||||
const db = new FakeDb([lockFailure]);
|
||||
await assert.rejects(verifySourceBatch(db, config, [first]), error => error === lockFailure);
|
||||
assert.equal(db.calls.length, 1);
|
||||
});
|
||||
@@ -0,0 +1,131 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { sourceCurrent } from '../migrate.mjs';
|
||||
|
||||
const config = { mysql: { database: 'source_current_test' } };
|
||||
const item = {
|
||||
contentId: '9007199254740993', configId: '9007199254740995',
|
||||
path: '头像/Résumé + 100%.png', size: 1234, sha256: 'a'.repeat(64)
|
||||
};
|
||||
const current = overrides => ({ id: item.contentId, size: String(item.size), sha256: item.sha256, ...overrides });
|
||||
|
||||
class FakeDb {
|
||||
constructor(responses) {
|
||||
this.responses = responses;
|
||||
this.calls = [];
|
||||
}
|
||||
async execute(sql, params) {
|
||||
assert.match(sql, /^SELECT\b/, 'source validation must not write or manage the caller transaction');
|
||||
this.calls.push({ sql, params });
|
||||
if (this.calls.length > this.responses.length) throw new Error('Unexpected extra database query');
|
||||
const response = this.responses[this.calls.length - 1];
|
||||
if (response instanceof Error) throw response;
|
||||
return [response];
|
||||
}
|
||||
}
|
||||
|
||||
function assertMetadataLookup(call, expectedItem, locked = false) {
|
||||
assert.match(call.sql, /^SELECT id FROM /);
|
||||
assert.doesNotMatch(call.sql, /\bcontent\b|SHA2\s*\(|OCTET_LENGTH\s*\(/i,
|
||||
'the unindexed path/order lookup must not project file bytes or content expressions');
|
||||
assert.match(call.sql, /WHERE deleted=0 AND config_id=\? AND path=\? ORDER BY id DESC LIMIT 1/);
|
||||
assert.doesNotMatch(call.sql, /BINARY\s+path|LOWER\s*\(path\)/i, 'preserve native MySQL path collation');
|
||||
assert.deepEqual(call.params, [expectedItem.configId, expectedItem.path]);
|
||||
if (locked) assert.match(call.sql, / FOR UPDATE$/);
|
||||
else assert.doesNotMatch(call.sql, /FOR UPDATE/i);
|
||||
}
|
||||
|
||||
function assertPrimaryKeyHashRead(call, expectedItem, locked = false) {
|
||||
assert.match(call.sql, /OCTET_LENGTH\(content\) AS size/);
|
||||
assert.match(call.sql, /LOWER\(SHA2\(content,256\)\) AS sha256/);
|
||||
assert.match(call.sql, /WHERE id=\? AND config_id=\? AND path=\? AND deleted=0/);
|
||||
assert.doesNotMatch(call.sql, /ORDER BY|LIMIT|JOIN/i, 'read the digest through a single primary-key lookup');
|
||||
assert.deepEqual(call.params, [expectedItem.contentId, expectedItem.configId, expectedItem.path]);
|
||||
if (locked) assert.match(call.sql, / FOR UPDATE$/);
|
||||
else assert.doesNotMatch(call.sql, /FOR UPDATE/i);
|
||||
}
|
||||
|
||||
test('looks up the latest ID without BLOB expressions, then validates size and SHA by primary key', async () => {
|
||||
const db = new FakeDb([[{ id: item.contentId }], [current()]]);
|
||||
await sourceCurrent(db, config, item);
|
||||
assert.equal(db.calls.length, 2);
|
||||
assertMetadataLookup(db.calls[0], item);
|
||||
assertPrimaryKeyHashRead(db.calls[1], item);
|
||||
});
|
||||
|
||||
test('rejects a newer active version before any content/hash query, without rounding bigint IDs', async () => {
|
||||
const db = new FakeDb([[{ id: '9007199254740994' }]]);
|
||||
await assert.rejects(sourceCurrent(db, config, item), /源图片.*变化或已删除/);
|
||||
assert.equal(db.calls.length, 1);
|
||||
assertMetadataLookup(db.calls[0], item);
|
||||
});
|
||||
|
||||
test('rejects a missing latest record without issuing a content/hash query', async () => {
|
||||
const db = new FakeDb([[]]);
|
||||
await assert.rejects(sourceCurrent(db, config, item), /源图片.*变化或已删除/);
|
||||
assert.equal(db.calls.length, 1);
|
||||
assertMetadataLookup(db.calls[0], item);
|
||||
});
|
||||
|
||||
test('rejects content disappearing between the metadata and primary-key reads', async () => {
|
||||
const db = new FakeDb([[{ id: item.contentId }], []]);
|
||||
await assert.rejects(sourceCurrent(db, config, item), /源图片.*变化或已删除/);
|
||||
assert.equal(db.calls.length, 2);
|
||||
assertPrimaryKeyHashRead(db.calls[1], item);
|
||||
});
|
||||
|
||||
test('rejects changed length, changed hash, or an unexpected primary-key row', async t => {
|
||||
for (const [name, row] of [
|
||||
['changed size', current({ size: item.size + 1 })],
|
||||
['changed hash with the same size', current({ sha256: 'b'.repeat(64) })],
|
||||
['wrong content ID', current({ id: '9007199254740994' })],
|
||||
['missing hash', current({ sha256: null })]
|
||||
]) {
|
||||
await t.test(name, async () => {
|
||||
const db = new FakeDb([[{ id: item.contentId }], [row]]);
|
||||
await assert.rejects(sourceCurrent(db, config, item), /源图片.*变化或已删除/);
|
||||
assert.equal(db.calls.length, 2);
|
||||
assertPrimaryKeyHashRead(db.calls[1], item);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test('retains FOR UPDATE on both validation queries when the caller owns a transaction', async () => {
|
||||
const db = new FakeDb([[{ id: item.contentId }], [current()]]);
|
||||
await sourceCurrent(db, config, item, true);
|
||||
assert.equal(db.calls.length, 2);
|
||||
assertMetadataLookup(db.calls[0], item, true);
|
||||
assertPrimaryKeyHashRead(db.calls[1], item, true);
|
||||
});
|
||||
|
||||
test('locked validation still stops at the metadata lookup if the latest version changed', async () => {
|
||||
const db = new FakeDb([[{ id: '9007199254740994' }]]);
|
||||
await assert.rejects(sourceCurrent(db, config, item, true), /源图片.*变化或已删除/);
|
||||
assert.equal(db.calls.length, 1);
|
||||
assertMetadataLookup(db.calls[0], item, true);
|
||||
});
|
||||
|
||||
test('supports unsigned-bigint-sized IDs and binds paths without interpolation or normalization', async () => {
|
||||
const largeItem = { ...item, contentId: '18446744073709551615', path: "case/IMAGE.png' OR 'x'='x" };
|
||||
const db = new FakeDb([[{ id: largeItem.contentId }], [current({ id: largeItem.contentId })]]);
|
||||
await sourceCurrent(db, { mysql: { database: 'app`db' } }, largeItem, false);
|
||||
assertMetadataLookup(db.calls[0], largeItem);
|
||||
assertPrimaryKeyHashRead(db.calls[1], largeItem);
|
||||
for (const call of db.calls) {
|
||||
assert.match(call.sql, /`app``db`\.`infra_file_content`/);
|
||||
assert.ok(!call.sql.includes(largeItem.contentId));
|
||||
assert.ok(!call.sql.includes(largeItem.path));
|
||||
}
|
||||
});
|
||||
|
||||
test('propagates database failures instead of accepting an unverified source', async () => {
|
||||
for (const responses of [
|
||||
[Object.assign(new Error('metadata lookup timed out'), { code: 'TEST_TIMEOUT' })],
|
||||
[[{ id: item.contentId }], Object.assign(new Error('hash read failed'), { code: 'TEST_READ_FAILURE' })]
|
||||
]) {
|
||||
const db = new FakeDb(responses);
|
||||
const expected = responses[responses.length - 1];
|
||||
await assert.rejects(sourceCurrent(db, config, item), error => error === expected);
|
||||
assert.equal(db.calls.length, responses.length);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,127 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { loadSourceIndex } from '../lib/source-index.mjs';
|
||||
|
||||
const config = { mysql: { database: 'source_index_test' }, sourceConfigIds: ['9007199254740999', '5'] };
|
||||
const makeFile = (value, contentId, overrides = {}) => ({
|
||||
id: value, config_id: '5', path: 'image.png', url: 'https://old.example/image.png',
|
||||
type: 'image/png', size: 3, content_id: contentId, ...overrides
|
||||
});
|
||||
|
||||
class FakeDb {
|
||||
constructor(batches, stageEvents) {
|
||||
this.batches = batches;
|
||||
this.stageEvents = stageEvents;
|
||||
this.calls = [];
|
||||
}
|
||||
async execute(sql, params) {
|
||||
assert.match(sql.trim(), /^SELECT\b/);
|
||||
assert.doesNotMatch(sql, /\b(?:OCTET_LENGTH|SHA2)\s*\(/i);
|
||||
assert.doesNotMatch(sql, /\bcontent\b|SELECT\s+\*/i, 'only metadata columns may be selected');
|
||||
assert.doesNotMatch(sql, /\b(?:NOT EXISTS|CREATE|INSERT|UPDATE|DELETE|ALTER)\b/i);
|
||||
assert.match(sql, /GROUP BY config_id,path/);
|
||||
assert.match(sql, /MAX\(id\) AS id/);
|
||||
assert.match(sql, /WHERE deleted=0 AND config_id IN \(\?,\?\)/);
|
||||
if (this.stageEvents) assert.equal(this.stageEvents.length, this.calls.length + 1, 'announce each phase before executing it');
|
||||
this.calls.push({ sql, params });
|
||||
if (this.calls.length > 3) throw new Error('Unexpected per-file query');
|
||||
return [this.batches[this.calls.length - 1]];
|
||||
}
|
||||
}
|
||||
|
||||
test('loads all metadata in exactly three SELECTs and preserves bigint IDs and raw spellings', async () => {
|
||||
const stages = [];
|
||||
const db = new FakeDb([
|
||||
[
|
||||
{ content_id: '9007199254740995', config_id: '9007199254740999', path: '头像.png' },
|
||||
{ content_id: '9', config_id: '5', path: 'IMAGE.png' }
|
||||
],
|
||||
[makeFile('9007199254740996', '9'), makeFile('2', '9', { path: 'IMAGE.png' }), makeFile('10', null)],
|
||||
[
|
||||
{ content_id: '9', path: Buffer.from('image.png') },
|
||||
{ content_id: '9', path: Buffer.from('IMAGE.png') },
|
||||
{ content_id: '9', path: Buffer.from('image.png ') },
|
||||
{ content_id: '9', path: Buffer.from('image.png') },
|
||||
{ content_id: '9007199254740995', path: Buffer.from('头像.png') }
|
||||
]
|
||||
], stages);
|
||||
const result = await loadSourceIndex(db, config, { progress: { stage: message => stages.push(message) } });
|
||||
assert.equal(db.calls.length, 3);
|
||||
assert.equal(stages.length, 3);
|
||||
assert.equal(result.maxId, '9007199254740995');
|
||||
assert.deepEqual(result.groups.map(group => group.contentId), ['9', '9007199254740995']);
|
||||
assert.equal(result.groups[0].path, 'IMAGE.png', 'the latest row supplies the canonical raw path');
|
||||
assert.deepEqual(result.groups[0].files.map(file => file.id), ['2', '9007199254740996']);
|
||||
assert.deepEqual(result.groups[0].spellings, ['IMAGE.png', 'image.png', 'image.png ']);
|
||||
assert.equal(result.groups[1].configId, '9007199254740999');
|
||||
assert.deepEqual(result.groups[1].files, [], 'orphan BLOBs remain candidates');
|
||||
assert.deepEqual(result.groups[1].spellings, ['头像.png']);
|
||||
assert.deepEqual(result.missingFiles.map(file => file.id), ['10']);
|
||||
assert.equal(Object.hasOwn(result.missingFiles[0], 'content_id'), false);
|
||||
assert.deepEqual(db.calls[0].params, ['5', '9007199254740999']);
|
||||
assert.deepEqual(db.calls[1].params, ['5', '9007199254740999', '5', '9007199254740999']);
|
||||
});
|
||||
|
||||
test('upper bound applies after grouping and files with newer content are never marked missing', async () => {
|
||||
const bound = '9007199254740993';
|
||||
const db = new FakeDb([
|
||||
[{ content_id: '8', config_id: '5', path: 'image.png' }],
|
||||
[makeFile('1', '8'), makeFile('2', '9007199254740994'), makeFile('3', null)],
|
||||
[{ content_id: '8', path: 'image.png' }, { content_id: '9007199254740994', path: 'newer.png' }]
|
||||
]);
|
||||
const result = await loadSourceIndex(db, config, { maxId: bound });
|
||||
assert.equal(result.maxId, bound);
|
||||
assert.deepEqual(result.groups.map(group => group.contentId), ['8']);
|
||||
assert.deepEqual(result.groups[0].files.map(file => file.id), ['1']);
|
||||
assert.deepEqual(result.missingFiles.map(file => file.id), ['3']);
|
||||
assert.match(db.calls[0].sql, /GROUP BY config_id,path\) g ON g.id=c.id WHERE g.id<=\?/);
|
||||
assert.doesNotMatch(db.calls[0].sql, /WHERE deleted=0[^)]*id\s*[<>=]/i);
|
||||
assert.doesNotMatch(db.calls[1].sql, /g\.id<=|id<=/);
|
||||
assert.match(db.calls[2].sql, /WHERE c.deleted=0 AND g.id<=\?/);
|
||||
assert.deepEqual(db.calls[0].params, ['5', '9007199254740999', bound]);
|
||||
assert.deepEqual(db.calls[2].params, ['5', '9007199254740999', bound]);
|
||||
});
|
||||
|
||||
test('all path associations use server-generated content IDs, never JavaScript case folding', async () => {
|
||||
const db = new FakeDb([
|
||||
[{ content_id: '7', config_id: '5', path: 'Résumé.png' }, { content_id: '8', config_id: '5', path: 'resume.png' }],
|
||||
[makeFile('1', '7', { path: 'RESUME.png' }), makeFile('2', '8', { path: 'resume.png' })],
|
||||
[{ content_id: '7', path: Buffer.from('Résumé.png') }, { content_id: '7', path: Buffer.from('RESUME.png') }, { content_id: '8', path: Buffer.from('resume.png') }]
|
||||
]);
|
||||
const result = await loadSourceIndex(db, config);
|
||||
assert.equal(result.groups.length, 2);
|
||||
assert.deepEqual(result.groups[0].files.map(file => file.id), ['1']);
|
||||
assert.deepEqual(result.groups[1].files.map(file => file.id), ['2']);
|
||||
assert.match(db.calls[1].sql, /g\.config_id=f\.config_id AND g\.path=f\.path/);
|
||||
assert.match(db.calls[2].sql, /SELECT DISTINCT g.id AS content_id,BINARY c.path AS path/);
|
||||
assert.match(db.calls[2].sql, /g\.config_id=c\.config_id AND g\.path=c\.path/);
|
||||
assert.match(db.calls[1].sql, /WHERE f.deleted=0/);
|
||||
assert.match(db.calls[2].sql, /WHERE c.deleted=0/);
|
||||
});
|
||||
|
||||
test('empty content results still report missing files and use a zero upper bound', async () => {
|
||||
const db = new FakeDb([[], [makeFile('4', null)], []]);
|
||||
const result = await loadSourceIndex(db, config);
|
||||
assert.equal(db.calls.length, 3);
|
||||
assert.equal(result.maxId, '0');
|
||||
assert.deepEqual(result.groups, []);
|
||||
assert.deepEqual(result.missingFiles.map(file => file.id), ['4']);
|
||||
const bounded = await loadSourceIndex(new FakeDb([[], [], []]), config, { maxId: '0' });
|
||||
assert.equal(bounded.maxId, '0');
|
||||
});
|
||||
|
||||
test('source IDs are deduplicated and database identifiers are safely quoted', async () => {
|
||||
const db = new FakeDb([[], [], []]);
|
||||
const unusual = { mysql: { database: 'app`db' }, sourceConfigIds: ['9007199254740999', '5', '5'] };
|
||||
await loadSourceIndex(db, unusual);
|
||||
assert.match(db.calls[0].sql, /`app``db`\.`infra_file_content`/);
|
||||
assert.deepEqual(db.calls[0].params, ['5', '9007199254740999']);
|
||||
});
|
||||
|
||||
test('rejects unsafe numeric boundaries and invalid source configurations without SQL', async () => {
|
||||
const db = new FakeDb([[], [], []]);
|
||||
await assert.rejects(loadSourceIndex(db, config, { maxId: Number.MAX_SAFE_INTEGER + 1 }), /大整数/);
|
||||
await assert.rejects(loadSourceIndex(db, config, { maxId: '-1' }), /非负整数/);
|
||||
await assert.rejects(loadSourceIndex(db, { ...config, sourceConfigIds: [] }), /至少一个/);
|
||||
assert.equal(db.calls.length, 0);
|
||||
});
|
||||
@@ -0,0 +1,245 @@
|
||||
import test, { mock } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createServer } from 'node:http';
|
||||
import { once } from 'node:events';
|
||||
import { setImmediate } from 'node:timers/promises';
|
||||
import { sha256 } from '../lib/core.mjs';
|
||||
import { verifyCheckpoint, verifySwitchedBatch } from '../migrate.mjs';
|
||||
|
||||
const CONTENT = Buffer.from('89504e470d0a1a0a0011223344556677', 'hex');
|
||||
const CORRUPT = Buffer.alloc(CONTENT.length, 1);
|
||||
const CONFIG = { mysql: { database: 'target_test' }, targetConfigId: '20', timeoutMs: 2000, maxFileBytes: 1024, retries: 1 };
|
||||
const item = (url, files = [{ id: '9007199254740993' }]) => ({
|
||||
contentId: '9007199254740995', configId: '1', path: 'source.png',
|
||||
key: 'migration/object.png', url, size: CONTENT.length, sha256: sha256(CONTENT), files,
|
||||
});
|
||||
const stored = entry => ({
|
||||
id: entry.files[0].id, config_id: '20', path: entry.key,
|
||||
url: entry.url, deleted: Buffer.from([0]),
|
||||
});
|
||||
const oss = (bytes = CONTENT) => ({
|
||||
head: mock.fn(async () => ({ res: { headers: { 'content-length': String(bytes.length) } } })),
|
||||
get: mock.fn(async () => ({ content: bytes })),
|
||||
put: mock.fn(async () => { throw new Error('validation must never upload'); }),
|
||||
});
|
||||
|
||||
async function server(t, handler) {
|
||||
const instance = createServer(handler);
|
||||
instance.listen(0, '127.0.0.1');
|
||||
await once(instance, 'listening');
|
||||
t.after(async () => {
|
||||
const closing = new Promise((resolve, reject) => instance.close(error => error ? reject(error) : resolve()));
|
||||
instance.closeAllConnections();
|
||||
await closing;
|
||||
});
|
||||
return 'http://127.0.0.1:' + instance.address().port;
|
||||
}
|
||||
|
||||
test('verified checkpoint validates authenticated and public bytes without PUT or source access', async t => {
|
||||
let publicReads = 0;
|
||||
const url = await server(t, (_request, response) => { publicReads++; response.end(CONTENT); });
|
||||
const entry = item(url);
|
||||
const client = oss();
|
||||
await verifyCheckpoint(client, CONFIG, entry);
|
||||
assert.equal(client.head.mock.callCount(), 1);
|
||||
assert.equal(client.get.mock.callCount(), 1);
|
||||
assert.equal(client.put.mock.callCount(), 0);
|
||||
assert.deepEqual(client.head.mock.calls[0].arguments, [entry.key]);
|
||||
assert.deepEqual(client.get.mock.calls[0].arguments, [entry.key]);
|
||||
assert.equal(publicReads, 1);
|
||||
});
|
||||
|
||||
test('missing checkpoint object is rejected without downloading or creating another object', async () => {
|
||||
const client = oss();
|
||||
const missing = Object.assign(new Error('missing object'), { code: 'NoSuchKey', status: 404 });
|
||||
client.head = mock.fn(async () => { throw missing; });
|
||||
await assert.rejects(verifyCheckpoint(client, CONFIG, item('http://127.0.0.1:1/unused')), error => error === missing);
|
||||
assert.equal(client.get.mock.callCount(), 0);
|
||||
assert.equal(client.put.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
test('checkpoint HEAD rejects mismatched sizes or configured size limits before GET', async () => {
|
||||
for (const tooLarge of [false, true]) {
|
||||
const client = oss();
|
||||
if (!tooLarge) client.head = mock.fn(async () => ({ res: { headers: { 'content-length': String(CONTENT.length + 1) } } }));
|
||||
await assert.rejects(verifyCheckpoint(client, { ...CONFIG, maxFileBytes: tooLarge ? 1 : CONFIG.maxFileBytes }, item('http://127.0.0.1:1/unused')), /长度/);
|
||||
assert.equal(client.get.mock.callCount(), 0);
|
||||
assert.equal(client.put.mock.callCount(), 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('same-length authenticated corruption rejects before public validation', async t => {
|
||||
let publicReads = 0;
|
||||
const url = await server(t, (_request, response) => { publicReads++; response.end(CONTENT); });
|
||||
const client = oss(CORRUPT);
|
||||
await assert.rejects(verifyCheckpoint(client, CONFIG, item(url)), /SHA-256/);
|
||||
assert.equal(publicReads, 0);
|
||||
assert.equal(client.put.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
test('public corruption or access denial prevents reusing an otherwise valid checkpoint', async t => {
|
||||
const url = await server(t, (request, response) => {
|
||||
if (request.url === '/denied') { response.writeHead(403); response.end(); }
|
||||
else response.end(CORRUPT);
|
||||
});
|
||||
for (const [path, message] of [['/corrupt', /SHA-256/], ['/denied', /HTTP 403/]]) {
|
||||
const client = oss();
|
||||
await assert.rejects(verifyCheckpoint(client, CONFIG, item(url + path)), message);
|
||||
assert.equal(client.put.mock.callCount(), 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('switched batch reads all metadata once with exact bigint IDs and deduplicates repeated records', async t => {
|
||||
const url = await server(t, (_request, response) => response.end(CONTENT));
|
||||
const first = item(url + '/one');
|
||||
const second = { ...item(url + '/two', [{ id: '9007199254740997' }, { id: '9007199254740997' }]), key: 'migration/two.png' };
|
||||
const db = { execute: mock.fn(async (sql, params) => {
|
||||
assert.match(sql, /SELECT id,config_id,path,url,deleted/);
|
||||
assert.match(sql, /WHERE id IN \(\?,\?\)/);
|
||||
assert.deepEqual(params, ['9007199254740993', '9007199254740997']);
|
||||
return [[stored(first), stored(second)]];
|
||||
}) };
|
||||
const results = await verifySwitchedBatch(db, CONFIG, [first, second]);
|
||||
assert.equal(db.execute.mock.callCount(), 1);
|
||||
assert.deepEqual(results, [{ item: first }, { item: second }]);
|
||||
});
|
||||
|
||||
test('orphaned source content with no file metadata still validates its public target', async t => {
|
||||
let publicReads = 0;
|
||||
const url = await server(t, (_request, response) => { publicReads++; response.end(CONTENT); });
|
||||
const entry = item(url, []);
|
||||
const db = { execute: mock.fn(async () => { throw new Error('orphan must not query file metadata'); }) };
|
||||
assert.deepEqual(await verifySwitchedBatch(db, CONFIG, [entry]), [{ item: entry }]);
|
||||
assert.equal(db.execute.mock.callCount(), 0);
|
||||
assert.equal(publicReads, 1);
|
||||
});
|
||||
|
||||
test('mutated, deleted and missing file records are returned as errors before public access', async t => {
|
||||
let publicReads = 0;
|
||||
const url = await server(t, (_request, response) => { publicReads++; response.end(CONTENT); });
|
||||
const entry = item(url);
|
||||
for (const changed of [
|
||||
null,
|
||||
{ ...stored(entry), config_id: '21' },
|
||||
{ ...stored(entry), path: 'some/other/object.png' },
|
||||
{ ...stored(entry), url: url + '/edited' },
|
||||
{ ...stored(entry), deleted: Buffer.from([1]) },
|
||||
]) {
|
||||
const db = { execute: mock.fn(async () => [changed ? [changed] : []]) };
|
||||
const results = await verifySwitchedBatch(db, CONFIG, [entry]);
|
||||
assert.equal(results.length, 1);
|
||||
assert.equal(results[0].item, entry);
|
||||
assert.match(results[0].error.message, /发生变化|已删除/);
|
||||
assert.deepEqual(results.filter(result => !result.error), []);
|
||||
}
|
||||
assert.equal(publicReads, 0);
|
||||
});
|
||||
|
||||
test('a single missing metadata record rejects an item that has multiple original records', async t => {
|
||||
let publicReads = 0;
|
||||
const url = await server(t, (_request, response) => { publicReads++; response.end(CONTENT); });
|
||||
const entry = item(url, [{ id: '9007199254740993' }, { id: '9007199254740997' }]);
|
||||
const db = { execute: mock.fn(async () => [[stored(entry)]]) };
|
||||
const [result] = await verifySwitchedBatch(db, CONFIG, [entry]);
|
||||
assert.ok(result.error);
|
||||
assert.equal(publicReads, 0);
|
||||
});
|
||||
|
||||
test('switched public corruption excludes only its own mapping and keeps input order', async t => {
|
||||
const url = await server(t, (request, response) => response.end(request.url === '/bad' ? CORRUPT : CONTENT));
|
||||
const bad = item(url + '/bad', []);
|
||||
const good = item(url + '/good', []);
|
||||
const results = await verifySwitchedBatch({}, CONFIG, [bad, good]);
|
||||
assert.equal(results[0].item, bad);
|
||||
assert.match(results[0].error.message, /SHA-256/);
|
||||
assert.deepEqual(results[1], { item: good });
|
||||
assert.deepEqual(results.filter(result => !result.error).map(result => result.item.url), [good.url]);
|
||||
});
|
||||
|
||||
test('metadata query errors reject before network validation starts', async t => {
|
||||
let publicReads = 0;
|
||||
const url = await server(t, (_request, response) => { publicReads++; response.end(CONTENT); });
|
||||
const failure = new Error('metadata unavailable');
|
||||
const db = { execute: mock.fn(async () => { throw failure; }) };
|
||||
await assert.rejects(verifySwitchedBatch(db, CONFIG, [item(url)]), error => error === failure);
|
||||
assert.equal(publicReads, 0);
|
||||
});
|
||||
|
||||
test('final validation is capped at four active requests and settles only after every started request', { timeout: 4000 }, async t => {
|
||||
const responses = new Map();
|
||||
let active = 0, maximum = 0, settled = false;
|
||||
const url = await server(t, (request, response) => {
|
||||
active++;
|
||||
maximum = Math.max(maximum, active);
|
||||
response.once('finish', () => { active--; });
|
||||
responses.set(Number(request.url.slice(1)), response);
|
||||
});
|
||||
const batch = Array.from({ length: 6 }, (_, index) => item(url + '/' + index, []));
|
||||
const pending = verifySwitchedBatch({}, CONFIG, batch).then(results => { settled = true; return results; });
|
||||
const until = async predicate => {
|
||||
const deadline = Date.now() + 1500;
|
||||
while (!predicate()) {
|
||||
if (Date.now() > deadline) assert.fail('validation did not reach the expected request state');
|
||||
await setImmediate();
|
||||
}
|
||||
};
|
||||
await until(() => responses.size === 4);
|
||||
assert.equal(active, 4);
|
||||
assert.equal(settled, false);
|
||||
// Per-item failures are collected, so remaining items must still be checked.
|
||||
responses.get(0).end(CORRUPT);
|
||||
responses.get(1).end(CONTENT);
|
||||
await until(() => responses.size === 6);
|
||||
assert.equal(active, 4);
|
||||
assert.equal(settled, false);
|
||||
responses.get(2).end(CONTENT);
|
||||
responses.get(3).end(CONTENT);
|
||||
responses.get(4).end(CONTENT);
|
||||
await setImmediate();
|
||||
assert.equal(settled, false);
|
||||
responses.get(5).end(CONTENT);
|
||||
const results = await pending;
|
||||
assert.equal(maximum, 4);
|
||||
assert.equal(active, 0);
|
||||
assert.equal(settled, true);
|
||||
assert.equal(results.length, 6);
|
||||
assert.match(results[0].error.message, /SHA-256/);
|
||||
assert.deepEqual(results.slice(1), batch.slice(1).map(entry => ({ item: entry })));
|
||||
});
|
||||
|
||||
test('empty switched batches perform no database or network work', async () => {
|
||||
const db = { execute: mock.fn(async () => { throw new Error('must not query'); }) };
|
||||
assert.deepEqual(await verifySwitchedBatch(db, CONFIG, []), []);
|
||||
assert.equal(db.execute.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
test('more than 1000 file records are loaded in bounded sequential queries before public validation', async t => {
|
||||
let completedQueries = 0, activeQueries = 0, maximumQueries = 0, publicReads = 0;
|
||||
const url = await server(t, (_request, response) => {
|
||||
publicReads++;
|
||||
assert.equal(completedQueries, 2);
|
||||
assert.equal(activeQueries, 0);
|
||||
response.end(CONTENT);
|
||||
});
|
||||
const files = Array.from({ length: 1001 }, (_, index) => ({ id: String(9007199254740993n + BigInt(index)) }));
|
||||
const entry = item(url, files);
|
||||
const seenIds = [];
|
||||
const querySizes = [];
|
||||
const db = { execute: mock.fn(async (_sql, params) => {
|
||||
maximumQueries = Math.max(maximumQueries, ++activeQueries);
|
||||
assert.ok(params.length <= 1000);
|
||||
assert.equal(publicReads, 0);
|
||||
querySizes.push(params.length);
|
||||
seenIds.push(...params);
|
||||
await setImmediate();
|
||||
completedQueries++;
|
||||
activeQueries--;
|
||||
return [params.map(id => ({ ...stored(entry), id })).reverse()];
|
||||
}) };
|
||||
assert.deepEqual(await verifySwitchedBatch(db, CONFIG, [entry]), [{ item: entry }]);
|
||||
assert.deepEqual(querySizes, [1000, 1]);
|
||||
assert.deepEqual(seenIds, files.map(file => file.id));
|
||||
assert.equal(maximumQueries, 1);
|
||||
assert.equal(db.execute.mock.callCount(), 2);
|
||||
assert.equal(publicReads, 1);
|
||||
});
|
||||
@@ -0,0 +1,213 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { encodePath, sha256 } from '../lib/core.mjs';
|
||||
import { ensureJournal, rollbackChanges } from '../lib/journal.mjs';
|
||||
import { verifySourceBatch } from '../lib/source-batch.mjs';
|
||||
import { switchFiles } from '../lib/transition.mjs';
|
||||
|
||||
// Opt in only against the disposable localhost test server. No host, database,
|
||||
// credentials or application configuration is read from the project.
|
||||
const SCHEMA = 'codex_oss_transition_test';
|
||||
const PORT = process.env.MIGRATION_TEST_DB_PORT;
|
||||
const IDS = {
|
||||
older: '9007199254741001', first: '9007199254741003', second: '9007199254741005',
|
||||
orphan: '9007199254741007', deletedNewer: '9007199254741009', newVersion: '9007199254741011'
|
||||
};
|
||||
const FIRST_PATH = '相册/Résumé.PNG';
|
||||
const EQUIVALENT_PATH = '相册/RESUME.png';
|
||||
const imageBytes = [Buffer.from([1, 2, 3, 4, 5]), Buffer.from([6, 7, 8, 9]), Buffer.from([10, 11, 12])];
|
||||
const changedSource = error => error.code === 'MIGRATION_SOURCE_CHANGED';
|
||||
const changedRow = error => error.code === 'MIGRATION_CAS_CONFLICT';
|
||||
|
||||
test('MySQL batch transitions preserve source locks, atomic journals and rollback', {
|
||||
skip: !PORT, timeout: 45000
|
||||
}, async t => {
|
||||
const port = Number(PORT);
|
||||
assert.ok(Number.isInteger(port) && port > 0 && port <= 65535, 'MIGRATION_TEST_DB_PORT must be a valid disposable localhost MySQL port');
|
||||
const { default: mysql } = await import('mysql2/promise');
|
||||
const connectionOptions = {
|
||||
host: '127.0.0.1', port, user: 'root', password: '', connectTimeout: 3000,
|
||||
supportBigNumbers: true, bigNumberStrings: true, jsonStrings: true, dateStrings: true
|
||||
};
|
||||
const db = await mysql.createConnection(connectionOptions);
|
||||
let contender, observer;
|
||||
try {
|
||||
await db.query(`DROP DATABASE IF EXISTS ${SCHEMA}`);
|
||||
await db.query(`CREATE DATABASE ${SCHEMA} CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci`);
|
||||
await db.changeUser({ database: SCHEMA });
|
||||
await db.query('SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ');
|
||||
await db.query("SET SESSION sql_mode=CONCAT_WS(',',NULLIF(@@SESSION.sql_mode,''),'STRICT_ALL_TABLES')");
|
||||
await db.query('SET SESSION innodb_lock_wait_timeout=5');
|
||||
await db.query(`CREATE TABLE infra_file_content (
|
||||
id BIGINT UNSIGNED NOT NULL PRIMARY KEY,config_id BIGINT UNSIGNED NOT NULL,
|
||||
path VARCHAR(512) NOT NULL,content MEDIUMBLOB NOT NULL,deleted BIT(1) NOT NULL DEFAULT 0
|
||||
) ENGINE=InnoDB`);
|
||||
await db.query(`CREATE TABLE infra_file (
|
||||
id BIGINT UNSIGNED NOT NULL PRIMARY KEY,config_id BIGINT UNSIGNED NOT NULL,
|
||||
path VARCHAR(512) NOT NULL,url VARCHAR(1024) NOT NULL,type VARCHAR(128),size INT,
|
||||
deleted BIT(1) NOT NULL DEFAULT 0,
|
||||
update_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
) ENGINE=InnoDB`);
|
||||
await ensureJournal(db, SCHEMA);
|
||||
|
||||
contender = await mysql.createConnection({ ...connectionOptions, database: SCHEMA });
|
||||
observer = await mysql.createConnection({ ...connectionOptions, database: SCHEMA });
|
||||
await contender.query('SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ');
|
||||
await contender.query('SET SESSION innodb_lock_wait_timeout=5');
|
||||
const [[thread]] = await contender.query('SELECT CONNECTION_ID() AS id');
|
||||
|
||||
const readFiles = async () => {
|
||||
const [rows] = await db.query('SELECT id,config_id,path,url,type,size,deleted,update_time FROM infra_file ORDER BY id');
|
||||
return rows.map(row => ({ ...row }));
|
||||
};
|
||||
const readStatuses = async runId => {
|
||||
const [rows] = await db.execute('SELECT status FROM infra_file_oss_migration_items WHERE run_id=? ORDER BY item_key', [runId]);
|
||||
return rows.map(row => row.status);
|
||||
};
|
||||
const journalCount = async runId => {
|
||||
const [[row]] = await db.execute('SELECT COUNT(*) AS n FROM infra_file_oss_migration_changes WHERE run_id=?', [runId]);
|
||||
return Number(row.n);
|
||||
};
|
||||
|
||||
async function seed(runId) {
|
||||
await db.rollback();
|
||||
for (const table of ['infra_file_oss_migration_changes', 'infra_file_oss_migration_items', 'infra_file_oss_migration_runs', 'infra_file', 'infra_file_content']) {
|
||||
await db.query(`DELETE FROM ${table}`);
|
||||
}
|
||||
const sources = [
|
||||
[IDS.older, EQUIVALENT_PATH, Buffer.from('older version'), 0],
|
||||
[IDS.first, FIRST_PATH, imageBytes[0], 0],
|
||||
[IDS.second, 'second.png', imageBytes[1], 0],
|
||||
[IDS.orphan, 'orphan.png', imageBytes[2], 0],
|
||||
[IDS.deletedNewer, EQUIVALENT_PATH, Buffer.from('deleted newer version'), 1]
|
||||
];
|
||||
await db.execute(`INSERT INTO infra_file_content(id,config_id,path,content,deleted) VALUES ${sources.map(() => '(?,5,?,?,?)').join(',')}`, sources.flat());
|
||||
const files = [
|
||||
['9007199254741101', FIRST_PATH], ['9007199254741102', EQUIVALENT_PATH], ['9007199254741103', 'second.png']
|
||||
];
|
||||
await db.execute(`INSERT INTO infra_file(id,config_id,path,url,type,size,update_time) VALUES ${files.map(() => "(?,5,?,?,'image/png',1,'2020-01-01 00:00:00')").join(',')}`,
|
||||
files.flatMap(([fileId, path]) => [fileId, path, `https://legacy.test/admin-api/infra/file/5/get/${encodePath(path)}`]));
|
||||
const items = [];
|
||||
for (const [index, [contentId, path]] of [[IDS.first, FIRST_PATH], [IDS.second, 'second.png'], [IDS.orphan, 'orphan.png']].entries()) {
|
||||
const [records] = await db.execute('SELECT id,config_id,path,url,type,size FROM infra_file WHERE config_id=5 AND path=? AND deleted=0 ORDER BY id', [path]);
|
||||
const key = `migration/test/${contentId}.png`;
|
||||
items.push({ contentId, configId: '5', path, size: imageBytes[index].length, sha256: sha256(imageBytes[index]), key, url: `https://oss.test/${key}`, type: 'image/png', files: records.map(row => ({ ...row })) });
|
||||
}
|
||||
await db.execute("INSERT INTO infra_file_oss_migration_runs(run_id,settings,status) VALUES (?,?,'active')", [runId, JSON.stringify({ test: true })]);
|
||||
for (const item of items) {
|
||||
await db.execute("INSERT INTO infra_file_oss_migration_items(run_id,item_key,data,status) VALUES (?,?,?,'verified')", [runId, sha256(`${item.configId}\0${item.path}`), JSON.stringify(item)]);
|
||||
}
|
||||
return { config: { mysql: { database: SCHEMA }, runId, targetConfigId: '30' }, items, originals: await readFiles() };
|
||||
}
|
||||
|
||||
await t.test('batch success switches duplicate metadata and orphan progress; journals restore original values', async () => {
|
||||
const fixture = await seed('batch-success');
|
||||
assert.equal(fixture.items[0].files.length, 2, 'native collation includes the case/accent-equivalent metadata');
|
||||
assert.equal(fixture.items[2].files.length, 0, 'orphan content remains a migration item');
|
||||
await switchFiles(db, fixture.config, fixture.items);
|
||||
const files = await readFiles();
|
||||
for (const item of fixture.items) for (const original of item.files) {
|
||||
const stored = files.find(file => file.id === original.id);
|
||||
assert.equal(stored.config_id, '30');
|
||||
assert.equal(stored.path, item.key);
|
||||
assert.equal(stored.url, item.url);
|
||||
assert.equal(stored.size, item.size);
|
||||
assert.equal(stored.update_time, '2020-01-01 00:00:00');
|
||||
}
|
||||
assert.deepEqual(await readStatuses(fixture.config.runId), ['switched', 'switched', 'switched']);
|
||||
assert.equal(await journalCount(fixture.config.runId), 3);
|
||||
const result = await rollbackChanges(db, SCHEMA, fixture.config.runId);
|
||||
assert.equal(result.restored, 3);
|
||||
assert.deepEqual(result.conflicts, []);
|
||||
assert.deepEqual(await readFiles(), fixture.originals);
|
||||
const [[source]] = await db.execute('SELECT content FROM infra_file_content WHERE id=?', [IDS.first]);
|
||||
assert.deepEqual(source.content, imageBytes[0]);
|
||||
});
|
||||
|
||||
await t.test('a business-row CAS conflict rolls back the entire batch and preserves concurrent edits', async () => {
|
||||
const fixture = await seed('cas-conflict');
|
||||
await db.execute('UPDATE infra_file SET url=? WHERE id=?', ['https://user-edit.test/new.png', fixture.items[1].files[0].id]);
|
||||
const beforeAttempt = await readFiles();
|
||||
await assert.rejects(switchFiles(db, fixture.config, fixture.items), changedRow);
|
||||
assert.deepEqual(await readFiles(), beforeAttempt);
|
||||
assert.deepEqual(await readStatuses(fixture.config.runId), ['verified', 'verified', 'verified']);
|
||||
assert.equal(await journalCount(fixture.config.runId), 0);
|
||||
});
|
||||
|
||||
await t.test('a final progress-state CAS conflict rolls back already-written files and journals', async () => {
|
||||
const fixture = await seed('status-conflict');
|
||||
const orphan = fixture.items[2];
|
||||
await db.execute("UPDATE infra_file_oss_migration_items SET status='failed' WHERE run_id=? AND item_key=?", [fixture.config.runId, sha256(`${orphan.configId}\0${orphan.path}`)]);
|
||||
await assert.rejects(switchFiles(db, fixture.config, fixture.items), changedRow);
|
||||
assert.deepEqual(await readFiles(), fixture.originals);
|
||||
assert.deepEqual((await readStatuses(fixture.config.runId)).sort(), ['failed', 'verified', 'verified']);
|
||||
assert.equal(await journalCount(fixture.config.runId), 0);
|
||||
});
|
||||
|
||||
await t.test('source mutations and a newer collation-equivalent version prevent every database switch', async t2 => {
|
||||
for (const kind of ['same-size-hash-change', 'deleted-source', 'newer-equivalent-path']) {
|
||||
await t2.test(kind, async () => {
|
||||
const fixture = await seed(kind);
|
||||
if (kind === 'same-size-hash-change') await db.execute('UPDATE infra_file_content SET content=? WHERE id=?', [Buffer.from([9, 9, 9, 9, 9]), IDS.first]);
|
||||
else if (kind === 'deleted-source') await db.execute('UPDATE infra_file_content SET deleted=1 WHERE id=?', [IDS.first]);
|
||||
else await db.execute('INSERT INTO infra_file_content(id,config_id,path,content) VALUES (?,5,?,?)', [IDS.newVersion, EQUIVALENT_PATH, Buffer.from('new active version')]);
|
||||
await assert.rejects(switchFiles(db, fixture.config, fixture.items), changedSource);
|
||||
assert.deepEqual(await readFiles(), fixture.originals);
|
||||
assert.deepEqual(await readStatuses(fixture.config.runId), ['verified', 'verified', 'verified']);
|
||||
assert.equal(await journalCount(fixture.config.runId), 0);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
async function proveRangeLock(kind, statement, parameters) {
|
||||
const fixture = await seed(`range-${kind}`);
|
||||
let outcome, pending;
|
||||
await db.beginTransaction();
|
||||
try {
|
||||
// Must be the first consistent read after BEGIN; the locking read
|
||||
// inside verification establishes protection before the RR snapshot.
|
||||
await verifySourceBatch(db, fixture.config, [fixture.items[0]]);
|
||||
pending = contender.execute(statement, parameters).then(
|
||||
value => { outcome = { value }; }, error => { outcome = { error }; }
|
||||
);
|
||||
const deadline = Date.now() + 3000;
|
||||
let observedWait = false;
|
||||
while (Date.now() < deadline) {
|
||||
assert.equal(outcome, undefined, 'concurrent source mutation must not complete before commit');
|
||||
const [transactions] = await observer.execute('SELECT trx_state FROM information_schema.innodb_trx WHERE trx_mysql_thread_id=?', [thread.id]);
|
||||
if (transactions.some(transaction => transaction.trx_state === 'LOCK WAIT')) { observedWait = true; break; }
|
||||
await delay(20);
|
||||
}
|
||||
assert.ok(observedWait, 'observe an actual InnoDB lock wait, not merely a delayed promise');
|
||||
assert.equal(outcome, undefined);
|
||||
await db.commit();
|
||||
await pending;
|
||||
if (outcome.error) throw outcome.error;
|
||||
assert.equal(outcome.value[0].affectedRows, 1, 'mutation completes after the source lock is released');
|
||||
} finally {
|
||||
await db.rollback();
|
||||
if (pending) await pending;
|
||||
}
|
||||
// Once the waiting writer commits, the next batch must detect the new
|
||||
// active version rather than continuing with its old snapshot.
|
||||
await db.beginTransaction();
|
||||
try {
|
||||
await assert.rejects(verifySourceBatch(db, fixture.config, [fixture.items[0]]), changedSource);
|
||||
} finally { await db.rollback(); }
|
||||
}
|
||||
|
||||
await t.test('RR range locks block insertion of a new matching version until commit', async () => {
|
||||
await proveRangeLock('insert', 'INSERT INTO infra_file_content(id,config_id,path,content) VALUES (?,5,?,?)', [IDS.newVersion, EQUIVALENT_PATH, Buffer.from('concurrent version')]);
|
||||
});
|
||||
await t.test('RR range locks include deleted rows and block undeleting a newer version until commit', async () => {
|
||||
await proveRangeLock('undelete', 'UPDATE infra_file_content SET deleted=0 WHERE id=?', [IDS.deletedNewer]);
|
||||
});
|
||||
} finally {
|
||||
await db.rollback().catch(() => {});
|
||||
if (contender) await contender.end();
|
||||
if (observer) await observer.end();
|
||||
try { await db.query(`DROP DATABASE IF EXISTS ${SCHEMA}`); }
|
||||
finally { await db.end(); }
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,251 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { switchFiles } from '../lib/transition.mjs';
|
||||
import { sha256 } from '../lib/core.mjs';
|
||||
|
||||
const config = { mysql: { database: 'transition_test' }, runId: 'run', targetConfigId: '20' };
|
||||
const itemKey = item => sha256(`${item.configId}\0${item.path}`);
|
||||
const clone = value => Buffer.isBuffer(value) ? Buffer.from(value)
|
||||
: Array.isArray(value) ? value.map(clone)
|
||||
: value && typeof value === 'object' ? Object.fromEntries(Object.entries(value).map(([key, item]) => [key, clone(item)])) : value;
|
||||
|
||||
function fixture({ orphan = false, count = 1 } = {}) {
|
||||
const item = {
|
||||
contentId: '9007199254740993', configId: '5', path: '头像/原图.png',
|
||||
size: 12, sha256: 'a'.repeat(64), key: 'migration/new.png',
|
||||
url: 'https://oss/new.png', type: 'image/png', files: []
|
||||
};
|
||||
if (!orphan) for (let index = 0; index < count; index++) item.files.push({
|
||||
id: String(9007199254741093n + BigInt(index)), config_id: '5', path: item.path,
|
||||
url: 'https://old/old.png', type: 'image/png', size: 12
|
||||
});
|
||||
return [item];
|
||||
}
|
||||
|
||||
class TransitionDb {
|
||||
constructor(items) {
|
||||
this.items = items;
|
||||
this.calls = [];
|
||||
this.files = Object.fromEntries(items.flatMap(item => item.files.map(file => [String(file.id), clone(file)])));
|
||||
this.states = Object.fromEntries(items.map(item => [itemKey(item), 'verified']));
|
||||
this.logs = [];
|
||||
this.statusWarning = 0;
|
||||
this.statusAffected = null;
|
||||
this.fileAffected = null;
|
||||
}
|
||||
async beginTransaction() {
|
||||
this.calls.push({ transaction: 'begin' });
|
||||
if (this.beginFailure) throw this.beginFailure;
|
||||
this.saved = clone({ files: this.files, states: this.states, logs: this.logs });
|
||||
}
|
||||
async commit() {
|
||||
this.calls.push({ transaction: 'commit' });
|
||||
if (this.commitFailure) throw this.commitFailure;
|
||||
this.saved = undefined;
|
||||
}
|
||||
async rollback() {
|
||||
this.calls.push({ transaction: 'rollback' });
|
||||
if (this.rollbackFailure) throw this.rollbackFailure;
|
||||
if (this.saved) Object.assign(this, this.saved);
|
||||
this.saved = undefined;
|
||||
}
|
||||
async execute(sql, params = []) {
|
||||
this.calls.push({ sql, params });
|
||||
if (sql.startsWith('SELECT id FROM')) {
|
||||
if (this.sourceFailure) throw this.sourceFailure;
|
||||
return [this.items.map(item => ({ id: item.contentId }))];
|
||||
}
|
||||
if (sql.startsWith('SELECT id,CASE')) {
|
||||
return [this.items.map(item => ({ id: this.newerSource ? String(BigInt(item.contentId) + 1n) : item.contentId, expected_content_id: item.contentId }))];
|
||||
}
|
||||
if (sql.startsWith('SELECT id,config_id,path,OCTET_LENGTH')) {
|
||||
return [this.items.map(item => ({ id: item.contentId, config_id: item.configId, path: item.path, size: item.size, sha256: item.sha256 }))];
|
||||
}
|
||||
if (sql.includes('information_schema.COLUMNS')) {
|
||||
return [[...['id', 'config_id', 'path', 'url', 'type', 'size', 'deleted', 'update_time'].map(name => ({
|
||||
name, extra: name === 'update_time' ? 'on update CURRENT_TIMESTAMP' : '', engine: 'InnoDB'
|
||||
}))]];
|
||||
}
|
||||
if (sql.startsWith('UPDATE') && sql.includes('`infra_file`')) {
|
||||
const [set, where] = sql.split(' WHERE ');
|
||||
let cursor = (set.match(/\?/g) || []).length;
|
||||
const groups = where.split(' OR ');
|
||||
for (const group of groups) {
|
||||
const pk = String(params[cursor]);
|
||||
cursor += (group.match(/\?/g) || []).length;
|
||||
const item = this.items.find(value => value.files.some(file => String(file.id) === pk));
|
||||
assert.ok(item, 'file updates must target the selected batch');
|
||||
this.files[pk] = { ...this.files[pk], config_id: config.targetConfigId, path: item.key, url: item.url, type: item.type, size: item.size };
|
||||
}
|
||||
return [{ affectedRows: this.fileAffected ?? groups.length, warningStatus: 0 }];
|
||||
}
|
||||
if (sql.startsWith('SELECT `')) {
|
||||
return [params.map(pk => this.files[String(pk)])];
|
||||
}
|
||||
if (sql.startsWith('INSERT INTO')) {
|
||||
if (this.journalFailure) throw this.journalFailure;
|
||||
for (let index = 0; index < params.length; index += 7) this.logs.push(params.slice(index, index + 7));
|
||||
return [{ affectedRows: params.length / 7, warningStatus: 0 }];
|
||||
}
|
||||
if (sql.startsWith('UPDATE') && sql.includes('`infra_file_oss_migration_items`')) {
|
||||
assert.match(sql, /WHERE run_id=\? AND status='verified' AND item_key IN \(/);
|
||||
assert.match(sql, /SET status='switched',error=NULL,updated_at=CURRENT_TIMESTAMP/);
|
||||
assert.equal(params[0], config.runId);
|
||||
let affected = 0;
|
||||
for (const key of params.slice(1)) if (this.states[key] === 'verified') { this.states[key] = 'switched'; affected++; }
|
||||
return [{ affectedRows: this.statusAffected ?? affected, warningStatus: this.statusWarning }];
|
||||
}
|
||||
throw new Error(`Unexpected query: ${sql}`);
|
||||
}
|
||||
}
|
||||
|
||||
const state = db => clone({ files: db.files, states: db.states, logs: db.logs });
|
||||
|
||||
test('source range verification is the first operation after BEGIN, then files and guarded item status commit together', async () => {
|
||||
const items = fixture();
|
||||
const db = new TransitionDb(items);
|
||||
const result = await switchFiles(db, config, items);
|
||||
assert.deepEqual(Object.keys(result), ['sourceMs', 'writeMs', 'statusMs', 'commitMs']);
|
||||
assert.ok(Object.values(result).every(value => Number.isFinite(value) && value >= 0));
|
||||
assert.equal(db.calls[0].transaction, 'begin');
|
||||
assert.match(db.calls[1].sql, /^SELECT id FROM .* WHERE id>=\? ORDER BY id FOR UPDATE$/);
|
||||
assert.match(db.calls[2].sql, /^SELECT id,CASE/);
|
||||
assert.match(db.calls[3].sql, /^SELECT id,config_id,path,OCTET_LENGTH/);
|
||||
assert.ok(db.calls[4].sql.includes('information_schema.COLUMNS'));
|
||||
assert.equal(db.calls.at(-1).transaction, 'commit');
|
||||
assert.equal(db.files[items[0].files[0].id].url, items[0].url);
|
||||
assert.equal(db.states[itemKey(items[0])], 'switched');
|
||||
assert.equal(db.logs.length, 1);
|
||||
const statusCall = db.calls.find(call => call.sql?.includes("SET status='switched'"));
|
||||
assert.deepEqual(statusCall.params, [config.runId, itemKey(items[0])]);
|
||||
});
|
||||
|
||||
test('more than 100 file records use bounded write chunks in one source-locked transaction', async () => {
|
||||
const items = fixture({ count: 101 });
|
||||
const db = new TransitionDb(items);
|
||||
await switchFiles(db, config, items);
|
||||
assert.deepEqual(db.calls.filter(call => call.transaction).map(call => call.transaction), ['begin', 'commit']);
|
||||
assert.equal(db.calls.filter(call => call.sql?.startsWith('SELECT id FROM')).length, 1);
|
||||
const writes = db.calls.filter(call => call.sql?.startsWith('UPDATE') && call.sql.includes('`infra_file`'));
|
||||
assert.equal(writes.length, 2);
|
||||
assert.equal(writes[0].sql.split(' WHERE ')[1].split(' OR ').length, 100);
|
||||
assert.equal(writes[1].sql.split(' WHERE ')[1].split(' OR ').length, 1);
|
||||
assert.equal(db.logs.length, 101);
|
||||
assert.equal(Object.values(db.files).filter(file => file.url === items[0].url).length, 101);
|
||||
});
|
||||
|
||||
test('orphan content is source-verified and status-switched even without any infra_file rows', async () => {
|
||||
const items = fixture({ orphan: true });
|
||||
const db = new TransitionDb(items);
|
||||
await switchFiles(db, config, items);
|
||||
assert.equal(db.states[itemKey(items[0])], 'switched');
|
||||
assert.equal(db.logs.length, 0);
|
||||
assert.equal(db.calls.filter(call => call.sql).length, 4);
|
||||
assert.equal(db.calls.some(call => call.sql?.includes('information_schema')), false);
|
||||
});
|
||||
|
||||
test('mixed regular and orphan items both participate in one guarded status update', async () => {
|
||||
const items = fixture();
|
||||
items.push({ ...fixture({ orphan: true })[0], contentId: '9007199254740994', path: 'orphan.png' });
|
||||
const db = new TransitionDb(items);
|
||||
await switchFiles(db, config, items);
|
||||
assert.equal(db.logs.length, 1);
|
||||
assert.deepEqual(Object.values(db.states), ['switched', 'switched']);
|
||||
});
|
||||
|
||||
test('source errors are propagated unchanged after a successful rollback', async () => {
|
||||
const items = fixture();
|
||||
const db = new TransitionDb(items);
|
||||
const failure = Object.assign(new Error('source changed'), { code: 'MIGRATION_SOURCE_CHANGED' });
|
||||
db.sourceFailure = failure;
|
||||
await assert.rejects(switchFiles(db, config, items), error => error === failure);
|
||||
assert.deepEqual(db.calls.filter(call => call.transaction).map(call => call.transaction), ['begin', 'rollback']);
|
||||
assert.equal(db.calls.some(call => call.sql?.startsWith('UPDATE')), false);
|
||||
});
|
||||
|
||||
test('a newer source version stops the transaction before any file updates', async () => {
|
||||
const items = fixture();
|
||||
const db = new TransitionDb(items);
|
||||
db.newerSource = true;
|
||||
await assert.rejects(switchFiles(db, config, items), { code: 'MIGRATION_SOURCE_CHANGED' });
|
||||
assert.equal(db.calls.some(call => call.sql?.startsWith('UPDATE')), false);
|
||||
assert.equal(db.calls.at(-1).transaction, 'rollback');
|
||||
});
|
||||
|
||||
test('file CAS mismatches roll back all file edits and preserve the conflict code', async () => {
|
||||
const items = fixture();
|
||||
const db = new TransitionDb(items);
|
||||
const before = state(db);
|
||||
db.fileAffected = 0;
|
||||
await assert.rejects(switchFiles(db, config, items), { code: 'MIGRATION_CAS_CONFLICT' });
|
||||
assert.deepEqual(state(db), before);
|
||||
assert.equal(db.calls.some(call => call.sql?.includes("SET status='switched'")), false);
|
||||
});
|
||||
|
||||
test('non-verified item status rolls back previously updated files and journal inserts', async () => {
|
||||
const items = fixture();
|
||||
const db = new TransitionDb(items);
|
||||
db.states[itemKey(items[0])] = 'planned';
|
||||
const before = state(db);
|
||||
await assert.rejects(switchFiles(db, config, items), { code: 'MIGRATION_CAS_CONFLICT' });
|
||||
assert.deepEqual(state(db), before);
|
||||
assert.ok(db.calls.some(call => call.sql?.startsWith('INSERT INTO')));
|
||||
});
|
||||
|
||||
test('item status SQL warnings roll back files, logs and item statuses', async () => {
|
||||
const items = fixture();
|
||||
const db = new TransitionDb(items);
|
||||
const before = state(db);
|
||||
db.statusWarning = 1;
|
||||
await assert.rejects(switchFiles(db, config, items), { code: 'MIGRATION_SQL_WARNING' });
|
||||
assert.deepEqual(state(db), before);
|
||||
});
|
||||
|
||||
test('rollback failure is fatal and prohibits fallback while preserving both causes', async () => {
|
||||
const items = fixture();
|
||||
const db = new TransitionDb(items);
|
||||
db.sourceFailure = Object.assign(new Error('source failed'), { code: 'MIGRATION_SOURCE_CHANGED' });
|
||||
db.rollbackFailure = new Error('connection lost');
|
||||
await assert.rejects(switchFiles(db, config, items), error => {
|
||||
assert.equal(error.code, 'MIGRATION_ROLLBACK_FAILED');
|
||||
assert.equal(error.fatal, true);
|
||||
assert.equal(error.noFallback, true);
|
||||
assert.equal(error.cause, db.sourceFailure);
|
||||
assert.equal(error.rollbackError, db.rollbackFailure);
|
||||
return true;
|
||||
});
|
||||
});
|
||||
|
||||
test('commit acknowledgement failure is fatal even if a rollback call subsequently succeeds', async () => {
|
||||
const items = fixture();
|
||||
const db = new TransitionDb(items);
|
||||
db.commitFailure = Object.assign(new Error('commit transport failure'), { code: 'PROTOCOL_CONNECTION_LOST' });
|
||||
await assert.rejects(switchFiles(db, config, items), error => {
|
||||
assert.equal(error.code, 'PROTOCOL_CONNECTION_LOST');
|
||||
assert.equal(error.fatal, true);
|
||||
assert.equal(error.noFallback, true);
|
||||
assert.equal(error.cause, db.commitFailure);
|
||||
return true;
|
||||
});
|
||||
assert.deepEqual(db.calls.filter(call => call.transaction).map(call => call.transaction), ['begin', 'commit', 'rollback']);
|
||||
});
|
||||
|
||||
test('BEGIN failure is propagated without performing any reads, writes or rollback', async () => {
|
||||
const items = fixture();
|
||||
const db = new TransitionDb(items);
|
||||
db.beginFailure = new Error('cannot begin');
|
||||
await assert.rejects(switchFiles(db, config, items), error => error === db.beginFailure);
|
||||
assert.deepEqual(db.calls, [{ transaction: 'begin' }]);
|
||||
});
|
||||
|
||||
test('empty batches are no-ops and malformed or duplicate input is rejected before BEGIN', async () => {
|
||||
const db = new TransitionDb([]);
|
||||
assert.deepEqual(await switchFiles(db, config, []), { sourceMs: 0, writeMs: 0, statusMs: 0, commitMs: 0 });
|
||||
await assert.rejects(switchFiles(db, config, null), /数组/);
|
||||
await assert.rejects(switchFiles(db, config, [{ ...fixture()[0], files: null }]), /files/);
|
||||
await assert.rejects(switchFiles(db, config, [fixture()[0], fixture()[0]]), /重复源路径/);
|
||||
const duplicateFiles = fixture();
|
||||
duplicateFiles[0].files.push({ ...duplicateFiles[0].files[0] });
|
||||
await assert.rejects(switchFiles(db, config, duplicateFiles), /重复文件记录/);
|
||||
assert.equal(db.calls.length, 0);
|
||||
});
|
||||
Reference in new issue
Block a user