Disclose partial SQLite schema records

This commit is contained in:
Afloat
2026-11-19 20:44:12 +08:00
parent e865276f84
commit 3f2aedfe7e
3 changed files with 78 additions and 6 deletions
+74 -3
View File
@@ -455,8 +455,8 @@ impl<'a> SqliteDb<'a> {
/// Walk the `sqlite_schema` b-tree (rooted at page 1) and return its objects. Columns are
/// parsed from the whole CREATE statement; the text shown is cut at [`MAX_TEXT`] bytes. The
/// flag is true when the list may be incomplete (the object cap was hit, or part of the b-tree
/// couldn't be read), and the notes say what the walk skipped or cut.
/// flag is true when the list or an object's metadata may be incomplete (an object or text
/// cap was hit, or part of the b-tree or a record couldn't be read). Notes say what was cut.
pub fn schema(&self) -> (Vec<SchemaObject>, bool, Vec<String>) {
let (mut objects, truncated, notes) = self.whole_schema();
for object in &mut objects {
@@ -475,7 +475,21 @@ impl<'a> SqliteDb<'a> {
w.text_cap = MAX_SCHEMA_TEXT;
w.text_budget = MAX_SCHEMA_TEXT_TOTAL;
self.collect_table_rows(&mut w, 1, 0);
let truncated = w.incomplete || w.out.len() >= MAX_SCHEMA_OBJECTS;
let partial = w
.out
.iter()
.filter(|row| {
row.values.len() < 5
|| row
.values
.iter()
.any(|value| matches!(value, CellValue::NotRead | CellValue::Text { truncated: true, .. } | CellValue::PartialBlob { .. }))
})
.count();
let truncated = w.incomplete || w.out.len() >= MAX_SCHEMA_OBJECTS || partial > 0;
if partial > 0 {
w.notes.push(format!("{partial} schema record(s) were read only partially; object metadata or column lists may be incomplete."));
}
for row in w.out {
// sqlite_schema columns: type, name, tbl_name, rootpage, sql.
let get_text = |i: usize| match row.values.get(i) {
@@ -2411,6 +2425,63 @@ mod tests {
}
}
#[test]
fn partial_schema_overflow_is_disclosed_without_dropping_the_object() {
use craft_anything_testkit::sqlite::{PAGE_SIZE, table_db};
let sql = format!("CREATE TABLE t(first TEXT /*{}*/, last TEXT)", "x".repeat(12_000));
let intact = table_db(&sql, &[]);
let complete = SqliteDb::parse(&intact).unwrap().overview();
assert!(!complete.schema_truncated);
assert_eq!(complete.schema[0].columns.len(), 2);
assert!(complete.schema[0].sql.len() < sql.len(), "display-only shortening is not incomplete parsing");
for next in [0, 3, u32::MAX] {
let mut data = intact.clone();
// Page 3 is the first schema overflow page: end early, loop, or leave the file.
data[2 * PAGE_SIZE..2 * PAGE_SIZE + 4].copy_from_slice(&next.to_be_bytes());
let parsed = SqliteDb::parse(&data).unwrap();
let overview = parsed.overview();
assert_eq!(overview.schema.len(), 1);
assert_eq!(overview.schema[0].name, "t");
assert!(overview.schema[0].columns.len() < complete.schema[0].columns.len());
assert!(overview.schema_truncated, "overflow next page {next}");
assert!(parsed.schema().1);
assert!(overview.integrity_notes.iter().any(|note| note.contains("schema record")));
}
}
#[test]
fn schema_text_caps_are_reported_before_display_shortening() {
use craft_anything_testkit::sqlite::table_db;
let sql = format!("CREATE TABLE t(first TEXT /*{}*/, last TEXT)", "x".repeat(MAX_SCHEMA_TEXT));
let data = table_db(&sql, &[]);
let overview = SqliteDb::parse(&data).unwrap().overview();
assert_eq!(overview.schema.len(), 1);
assert!(overview.schema_truncated);
assert!(overview.integrity_notes.iter().any(|note| note.contains("schema record")));
}
#[test]
fn exhausted_schema_text_budget_marks_later_objects_partial() {
use craft_anything_testkit::sqlite::table_db;
let sql = format!("CREATE TABLE t(first TEXT /*{}*/, last TEXT)", "x".repeat(MAX_SCHEMA_TEXT - 100));
let mut data = table_db(&sql, &[]);
let copies = MAX_SCHEMA_TEXT_TOTAL / sql.len() + 2;
let pointer = [data[108], data[109]];
assert!(108 + 2 * copies < usize::from(u16::from_be_bytes(pointer)));
data[103..105].copy_from_slice(&(copies as u16).to_be_bytes());
// Repeated cells sharing one overflow chain exercise the real aggregate text budget.
for index in 0..copies {
data[108 + 2 * index..110 + 2 * index].copy_from_slice(&pointer);
}
let overview = SqliteDb::parse(&data).unwrap().overview();
assert_eq!(overview.schema.len(), copies);
assert_eq!(overview.schema[0].columns.len(), 2);
assert!(overview.schema.last().unwrap().columns.len() < 2);
assert!(overview.schema_truncated);
assert!(overview.integrity_notes.iter().any(|note| note.contains("schema record")));
}
#[test]
fn wal_mode_is_reported() {
let db = craft_anything_testkit::sqlite::wal_mode_db();
+3 -3
View File
@@ -2524,9 +2524,9 @@ Owner: open · Status: in progress
`db_sqlite_show` adds a `schema_incomplete` limitation, partial coverage, and an unknown-table
error that says only the listed part was searched
— claimed: mansur-akhmadi, branch bugfix/r3.12-sqlite-schema-incomplete
- [ ] A schema row whose CREATE statement was read only in part (a cut or looping overflow
chain, or text cut once the schema text budget is spent) leaves `schema_truncated` false,
so its table's column list can be short with no flag on it
- [x] Schema rows read only in part (cut or looping overflow chains, unread values, or text cut
at a per-value or aggregate schema budget) set `schema_truncated` and report how many
records have incomplete metadata or column lists. Display-only SQL shortening stays separate
— claimed: Afloat16, branch bugfix/r3.12-sqlite-partial-schema-20261119
- [x] Require a complete DNS question's type and class fields before retaining it, and keep
truncated questions out of packet inventories while reporting the incomplete message
+1
View File
@@ -275,6 +275,7 @@ coverage and secrets passed. The companion CLI was built with locked dependencie
| Atomic-write temporary files | Reserve short temporary names distinct from the destination and its filename aliases, with exclusive creation and bounded collision retries. Private files have owner-only Unix permissions from creation; cleanup removes only a file reserved by the current write. | Original implementation `5f0f1ce2` passed [full Linux CI](https://github.com/storytold/craft-anything/actions/runs/41687019068). The review follow-up preserves collision, symlink, inherited-permission, retry-limit and cleanup assertions, and adds a bounded probe for a valid long name, direct creation-permission checks and deterministic destination-alias exclusion checks. `27e5d631` passed [full Linux CI](https://github.com/storytold/craft-anything/actions/runs/41792023953). |
| SQLite b-tree completeness | Null, cyclic, reused, missing or wrong-kind children leave row counts as lower bounds and row reads marked incomplete; in `sqlite_schema` they mark the schema listing truncated, with the walk's first notes among the integrity notes, a `schema_incomplete` limitation and partial coverage. Readable siblings survive, and traversal limits or skipped rows are disclosed. | Synthetic table and WITHOUT ROWID regressions cover both child positions, reused leaves, missing pages, page kinds and the depth cap; complete-tree order, pagination and empty-leaf controls. Schema regressions cover a missing and a reused child of an interior schema root and unreadable schema rows (past the note cap too), against an intact control, in the parser and through `db_sqlite_show`. |
| SQLite payload limits | Capped records retain column positions, disclose partial TEXT and BLOB values, omit incomplete BLOB hashes, and report partial coverage with read notes | Hand-built overflow-page regressions for five oversized records (including WITHOUT ROWID in declaration order) and two controls; table and index engine coverage and limitation regressions |
| SQLite schema value completeness | Partial schema records set `schema_truncated` and add a counted integrity note while keeping readable object metadata. This covers incomplete overflow chains and text caps before parsing column lists; shortening SQL only for display does not mark a fully parsed schema incomplete. | Synthetic database regressions cover early-ended, cyclic and out-of-file overflow chains, per-value schema text limits, the real aggregate text budget, and a complete long CREATE statement as a control. |
| SQLite added columns | Short records include missing trailing fields using literal column defaults, or NULL when absent; unsupported default expressions produce NULL with a row note. A default is read with its column's affinity, as SQLite reads it: a TEXT column's `DEFAULT 5` is '5', an INTEGER column's `DEFAULT '7'` is 7, and a BLOB or untyped column (or a STRICT table's ANY column) keeps the literal as written. Bare or double-quoted words are text constants, `DEFAULT -'5'` coerces a numeric string, and one underscore between digits is a digit separator. A non-integer REAL default uses SQLite's REAL text (`1e3` is '1000.0', `5.` is '5.0', infinity is 'Inf'). Stored fields and records longer than the schema retain their values. Defaults are filled before restoring rowid aliases | Hand-built parser regressions and an engine tool test cover added-column defaults, rewritten records and short records with a rowid alias, including a missing alias field; defaults of every affinity (rowid, WITHOUT ROWID and STRICT tables, UTF-8 and UTF-16le) compared with the official SQLite library on generated files |
| SQLite primary-key collations | Repeated key columns with distinct collations keep their distinct record slots. Identical collations share a slot, and later values stay aligned. | Binary row regressions checked against SQLite index_xinfo and SELECT results. |
| Parenthesized SQLite primary-key columns | WITHOUT ROWID row layouts keep the full key list and honor collations on grouped column names. | Binary row regressions checked against SQLite index_xinfo and SELECT results. |