mirror of
https://github.com/storytold/pdfcraft.git
synced 2026-11-21 20:38:48 +00:00
* Issue #2385: audit finds thick strokes, black highlights and white boxes over text Three ways of hiding text that a reader sees as redacted were reported clean while the text stayed extractable: - a thick stroke along the text (`0 G 20 w 70 705 m 400 705 l S`, marker style): the walker now tracks the stroking color, the line width and `/CA`, and a single axis-aligned `m`/`l` segment stroked opaquely counts as the rectangle it paints (half the width each side); - an opaque `/Highlight` annotation dark enough to hide the text (`/C [0 0 0]`): the annotation pass now reads highlights too (color from `/C`, `/CA`, Hidden/NoView skipped) and reports them as `covered-text`; - an opaque white rectangle after dark text (`1 g 70 695 330 20 re f`): its own kind, `hidden-under-white` (relative luminance of the paint >= 0.9; the `0.9 g` table shading stays unflagged), so callers can decide. The `doc_audit_redactions` tool description lists the new covers and kind; the module docs state the limits (slanted or bent strokes, other filled paths, shadings and patterns are still out of scope). Tests: thick_stroke_over_text_is_a_cover, black_highlight_annotation_over_ text_is_a_cover, white_rectangle_over_text_is_reported_as_hidden_under_white (the report's three cases, with the hairline, yellow, translucent, hidden and outlined negatives); graphics_state_carries_across_streams now expects the white kind for its white rectangle. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_011K1peyrh5MdZ6ttHkVq8Gh * audit: ExtGState /LW and omitted alphas, stroke width across the CTM, white first, highlight quads (#2385 review) - gs sets /LW as the line width, and a /ca or /CA the dictionary leaves out keeps its earlier value (ISO 32000-2 8.4.5) instead of resetting to opaque. - A stroke's width is measured across the segment after the CTM, so a non-uniform scale no longer widens horizontal and vertical strokes alike. - A white cover is hidden-under-white even over white text, which the contrast check had classified as covered-text. - An opaque Highlight hides what its /QuadPoints paint, not its /Rect; the /Rect stays the fallback when there are no usable quads. Tests: ext_gstate_sets_line_width_and_keeps_omitted_alphas, stroke_width_follows_a_non_uniform_ctm_across_the_segment, white_cover_over_white_text_is_hidden_under_white, highlight_covers_its_quad_points_not_its_rect * audit: document that a Highlight is judged by its dictionary, not its /AP Developed with AI assistance (Claude Code). * audit: a white Square is hidden-under-white; a hidden Redact mark is still unapplied (#2385 review) An opaque /Square with a white interior went through the dark-cover contrast test and was passed over, although it hides the text as a white rectangle in the content does: it is reported as HiddenUnderWhite, like that rectangle. The Hidden/NoView skip applied to every annotation, so an unapplied /Redact mark with either flag was no longer reported, which main reports: the flags hide the mark, not the text, so the skip is for the cover annotations only. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_011K1peyrh5MdZ6ttHkVq8Gh * audit: a stroke's round or square caps are part of the cover (#2385 review) `stroked_rect` left the caps out, so a `1 J` / `2 J` marker line (or `/LC` in an ExtGState) painted half a line width past each end that the audit did not count: text under a cap stayed extractable and unreported. The cap is tracked like the width (`J`, `/LC`, cumulative, saved by `q`), and an axis-aligned stroke is lengthened by the pen's extent along it, `w/2 · |Mᵀ d̂|` after the CTM (the across-width is `w/2 · |Mᵀ n̂|`, the same rule). A round cap counts by its box. Butt caps, the default, add nothing. Test `stroke_caps_lengthen_the_cover`: butt, round, square, back to butt, vertical, a 4× scale along the stroke, a shear (√2), `/LC` with `/LW`, `Q` restoring it, and a stroke that reaches `IBAN` only by its cap. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_011K1peyrh5MdZ6ttHkVq8Gh * audit: charge a highlight's quads against the comparison bound A highlight is compared with every line of its page once per quad, but was charged for one cover: a /Highlight with many /QuadPoints did page_cost x quads work for the price of page_cost. hidden_text now spends page_cost per area. QuadPoints is read only for highlights, and its length is checked before the numbers are collected. Developed with AI assistance (Claude Code). --------- Co-authored-by: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
eeee51ff7b
commit
f0bbce5b30
+104
-20
@@ -3,10 +3,13 @@
|
||||
//! This is the geometric half of the faux-redaction audit: a rectangle filled with an opaque
|
||||
//! color (`re` + `f`) is the classic way a "redaction" is faked. A single closed `m`/`l` subpath
|
||||
//! whose four corners form an axis-aligned rectangle in user space is the same shape and counts
|
||||
//! too (#1510). Graphics state tracked here is deliberately small: the CTM stack (`q`/`Q`/`cm`),
|
||||
//! the non-stroking color (`g`/`rg`/`k`) and the non-stroking alpha from ExtGState (`gs` reading
|
||||
//! `/ca`). Form XObjects are followed read-only with a depth cap. Anything fancier (other filled paths, shadings, patterns) is
|
||||
//! out of scope and documented as such.
|
||||
//! too (#1510). A single `m`/`l` segment along an axis, stroked (`S`) wide enough to blot the text
|
||||
//! out ("marker" style), counts as the rectangle the stroke paints (#2385). Graphics state tracked
|
||||
//! here is deliberately small: the CTM stack (`q`/`Q`/`cm`), the colors (`g`/`rg`/`k` and
|
||||
//! `G`/`RG`/`K`), the line width (`w`) and what ExtGState sets of them (`gs` reading `/ca`,
|
||||
//! `/CA` and `/LW`). Form XObjects are followed read-only with a depth cap. Anything fancier
|
||||
//! (other filled paths, slanted or bent strokes, shadings, patterns) is out of scope and
|
||||
//! documented as such.
|
||||
//!
|
||||
//! Paint order: a page's content streams are one stream in pieces (ISO 32000-2 §7.8.2), so they
|
||||
//! are parsed joined and walked once with a single graphics state — state set in an earlier
|
||||
@@ -27,12 +30,12 @@ const MAX_DEPTH: usize = 32;
|
||||
/// Hitting it fails the audit: a truncated walk must never report "clean".
|
||||
const MAX_OPS: usize = 2_000_000;
|
||||
|
||||
/// One opaque filled rectangle, in page user space.
|
||||
/// One opaque filled rectangle (or a stroke that paints one), in page user space.
|
||||
#[derive(Clone, Debug, PartialEq)]
|
||||
pub struct FilledRect {
|
||||
/// `[x0, y0, x1, y1]`, normalized so `x0 <= x1` and `y0 <= y1`.
|
||||
pub rect: [f64; 4],
|
||||
/// The fill color as sRGB-ish `[r, g, b]` (CMYK is converted approximately).
|
||||
/// The paint color as sRGB-ish `[r, g, b]` (CMYK is converted approximately).
|
||||
pub rgb: [f64; 3],
|
||||
/// Paint position: `(content-stream index, operator index in the joined stream)`.
|
||||
/// Rectangles inside a form XObject carry the `Do` operator's position.
|
||||
@@ -44,6 +47,12 @@ struct Gs {
|
||||
ctm: Matrix,
|
||||
fill: [f64; 3],
|
||||
alpha: f64,
|
||||
stroke: [f64; 3],
|
||||
stroke_alpha: f64,
|
||||
/// Line width, in the space the path is drawn in.
|
||||
width: f64,
|
||||
/// Line cap (`J`, `/LC`): 0 butt, 1 round, 2 projecting square.
|
||||
cap: i64,
|
||||
}
|
||||
|
||||
/// The path under construction, as far as the audit cares about it.
|
||||
@@ -66,6 +75,36 @@ impl Path {
|
||||
Path::Empty | Path::Other => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// The rectangle a stroke of this path paints, if the path is one segment along an axis: the
|
||||
/// segment widened by half the line width each side, and lengthened by as much at each end
|
||||
/// under a round or projecting square cap (a round cap is counted by its box). Both are
|
||||
/// measured after the CTM: the pen is a disc of the line width in the space the path is
|
||||
/// drawn in, and its extent along a page direction is that direction's length under the
|
||||
/// transposed matrix, so a non-uniform scale widens only the strokes it stretches across.
|
||||
fn stroked_rect(&self, gs: &Gs) -> Option<[f64; 4]> {
|
||||
const EPS: f64 = 1e-6;
|
||||
let Path::Lines(pts, _) = self else { return None };
|
||||
let [a, b] = <[[f64; 2]; 2]>::try_from(pts.as_slice()).ok()?;
|
||||
let [m0, m1, m2, m3, ..] = gs.ctm.0;
|
||||
let (dx, dy) = (b[0] - a[0], b[1] - a[1]);
|
||||
let len = dx.hypot(dy);
|
||||
let half = gs.width * (m3 * dx - m2 * dy).hypot(m0 * dy - m1 * dx) / len / 2.0;
|
||||
if half.is_nan() || half <= EPS {
|
||||
return None;
|
||||
}
|
||||
let cap = if gs.cap == 0 { 0.0 } else { gs.width * (m0 * dx + m1 * dy).hypot(m2 * dx + m3 * dy) / len / 2.0 };
|
||||
if cap.is_nan() {
|
||||
return None;
|
||||
}
|
||||
if (a[1] - b[1]).abs() <= EPS && (a[0] - b[0]).abs() > EPS {
|
||||
Some([a[0].min(b[0]) - cap, a[1] - half, a[0].max(b[0]) + cap, a[1] + half])
|
||||
} else if (a[0] - b[0]).abs() <= EPS && (a[1] - b[1]).abs() > EPS {
|
||||
Some([a[0] - half, a[1].min(b[1]) - cap, a[0] + half, a[1].max(b[1]) + cap])
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The bounding box of a closed polygon that is an axis-aligned rectangle, or `None`. A repeated
|
||||
@@ -100,7 +139,7 @@ fn lines_rect(mut pts: Vec<[f64; 2]>) -> Option<[f64; 4]> {
|
||||
|
||||
impl Default for Gs {
|
||||
fn default() -> Self {
|
||||
Gs { ctm: Matrix::IDENTITY, fill: [0.0, 0.0, 0.0], alpha: 1.0 }
|
||||
Gs { ctm: Matrix::IDENTITY, fill: [0.0, 0.0, 0.0], alpha: 1.0, stroke: [0.0, 0.0, 0.0], stroke_alpha: 1.0, width: 1.0, cap: 0 }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -168,31 +207,41 @@ impl<'a> Walker<'a> {
|
||||
path = Path::Other;
|
||||
}
|
||||
}
|
||||
b"g" => {
|
||||
b"g" | b"G" => {
|
||||
if let Some(v) = op.num(0).filter(|v| v.is_finite()) {
|
||||
let v = v.clamp(0.0, 1.0);
|
||||
gs.fill = [v, v, v];
|
||||
*gs.color(op.op[0] == b'G') = [v, v, v];
|
||||
}
|
||||
}
|
||||
b"rg" => {
|
||||
b"rg" | b"RG" => {
|
||||
if let Some([r, g, b]) = op.nums::<3>()
|
||||
&& [r, g, b].iter().all(|v| v.is_finite())
|
||||
{
|
||||
gs.fill = [r.clamp(0.0, 1.0), g.clamp(0.0, 1.0), b.clamp(0.0, 1.0)];
|
||||
*gs.color(op.op[0] == b'R') = [r.clamp(0.0, 1.0), g.clamp(0.0, 1.0), b.clamp(0.0, 1.0)];
|
||||
}
|
||||
}
|
||||
b"k" => {
|
||||
b"k" | b"K" => {
|
||||
if let Some([c, m, y, k]) = op.nums::<4>()
|
||||
&& [c, m, y, k].iter().all(|v| v.is_finite())
|
||||
{
|
||||
// Naive CMYK -> RGB; good enough to tell black from white.
|
||||
let (c, m, y, k) = (c.clamp(0.0, 1.0), m.clamp(0.0, 1.0), y.clamp(0.0, 1.0), k.clamp(0.0, 1.0));
|
||||
gs.fill = [(1.0 - c) * (1.0 - k), (1.0 - m) * (1.0 - k), (1.0 - y) * (1.0 - k)];
|
||||
*gs.color(op.op[0] == b'K') = [(1.0 - c) * (1.0 - k), (1.0 - m) * (1.0 - k), (1.0 - y) * (1.0 - k)];
|
||||
}
|
||||
}
|
||||
b"w" => {
|
||||
if let Some(v) = op.num(0).filter(|v| v.is_finite() && *v >= 0.0) {
|
||||
gs.width = v;
|
||||
}
|
||||
}
|
||||
b"J" => {
|
||||
if let Some(v) = op.num(0).filter(|v| [0.0, 1.0, 2.0].contains(v)) {
|
||||
gs.cap = v as i64;
|
||||
}
|
||||
}
|
||||
b"gs" => {
|
||||
if let Some(name) = op.name(0) {
|
||||
gs.alpha = ext_alpha(self.doc, resources, name);
|
||||
ext_state(self.doc, resources, name, gs);
|
||||
}
|
||||
}
|
||||
b"re" => {
|
||||
@@ -232,13 +281,27 @@ impl<'a> Walker<'a> {
|
||||
}
|
||||
// `b`/`b*` close the path before filling it, which the fill does anyway.
|
||||
b"f" | b"F" | b"f*" | b"B" | b"B*" | b"b" | b"b*" => {
|
||||
if let Some(r) = std::mem::replace(&mut path, Path::Empty).rect()
|
||||
let done = std::mem::replace(&mut path, Path::Empty);
|
||||
if matches!(op.op.as_slice(), b"B" | b"B*" | b"b" | b"b*")
|
||||
&& let Some(r) = done.stroked_rect(gs)
|
||||
&& gs.stroke_alpha >= 1.0
|
||||
{
|
||||
self.out.push(FilledRect { rect: r, rgb: gs.stroke, pos });
|
||||
}
|
||||
if let Some(r) = done.rect()
|
||||
&& gs.alpha >= 1.0
|
||||
{
|
||||
self.out.push(FilledRect { rect: r, rgb: gs.fill, pos });
|
||||
}
|
||||
}
|
||||
b"S" | b"s" | b"n" => path = Path::Empty,
|
||||
b"S" | b"s" => {
|
||||
if let Some(r) = std::mem::replace(&mut path, Path::Empty).stroked_rect(gs)
|
||||
&& gs.stroke_alpha >= 1.0
|
||||
{
|
||||
self.out.push(FilledRect { rect: r, rgb: gs.stroke, pos });
|
||||
}
|
||||
}
|
||||
b"n" => path = Path::Empty,
|
||||
b"c" | b"v" | b"y" | b"W" | b"W*" => path = Path::Other,
|
||||
b"Do" => {
|
||||
if let Some(name) = op.name(0) {
|
||||
@@ -299,13 +362,34 @@ fn user_point(gs: &Gs, op: &Op) -> Option<[f64; 2]> {
|
||||
Some([x, y])
|
||||
}
|
||||
|
||||
/// The non-stroking alpha (`/ca`) of a named ExtGState, defaulting to opaque.
|
||||
fn ext_alpha(doc: &Document, resources: &Dict, name: &[u8]) -> f64 {
|
||||
impl Gs {
|
||||
/// The stroking or the non-stroking color.
|
||||
fn color(&mut self, stroking: bool) -> &mut [f64; 3] {
|
||||
if stroking { &mut self.stroke } else { &mut self.fill }
|
||||
}
|
||||
}
|
||||
|
||||
/// Apply what a named ExtGState sets of the state tracked here: the non-stroking and stroking
|
||||
/// alphas (`/ca`, `/CA`), the line width (`/LW`) and the line cap (`/LC`). A key it leaves out
|
||||
/// keeps its value.
|
||||
fn ext_state(doc: &Document, resources: &Dict, name: &[u8], gs: &mut Gs) {
|
||||
// Borrowed, not copied: a `gs` must not cost the size of the whole ExtGState table.
|
||||
let states = resources.get(b"ExtGState").map(|o| doc.resolve(o));
|
||||
let state = states.as_deref().and_then(Object::as_dict).and_then(|d| d.get(name)).map(|o| doc.resolve(o));
|
||||
let alpha = state.as_deref().and_then(Object::as_dict).and_then(|d| d.get(b"ca").map(|o| doc.resolve(o)).and_then(|o| o.as_f64()));
|
||||
alpha.filter(|a| a.is_finite()).map(|a| a.clamp(0.0, 1.0)).unwrap_or(1.0)
|
||||
let Some(state) = state.as_deref().and_then(Object::as_dict) else { return };
|
||||
let num = |key: &[u8]| state.get(key).map(|o| doc.resolve(o)).and_then(|o| o.as_f64()).filter(|v| v.is_finite());
|
||||
if let Some(a) = num(b"ca") {
|
||||
gs.alpha = a.clamp(0.0, 1.0);
|
||||
}
|
||||
if let Some(a) = num(b"CA") {
|
||||
gs.stroke_alpha = a.clamp(0.0, 1.0);
|
||||
}
|
||||
if let Some(w) = num(b"LW").filter(|w| *w >= 0.0) {
|
||||
gs.width = w;
|
||||
}
|
||||
if let Some(c) = num(b"LC").filter(|c| [0.0, 1.0, 2.0].contains(c)) {
|
||||
gs.cap = c as i64;
|
||||
}
|
||||
}
|
||||
|
||||
/// Every opaque filled rectangle drawn by a page's content (and its form XObjects), with paint
|
||||
|
||||
+151
-79
@@ -7,16 +7,25 @@
|
||||
//! A rectangle counts as a cover only when it was painted strictly *after* the text it
|
||||
//! overlaps, and only when it is dark enough against the text's color to make the text
|
||||
//! unreadable (WCAG contrast below 3.0). Page backgrounds, table shading and highlight
|
||||
//! marks are therefore not flagged.
|
||||
//! marks are therefore not flagged. A white (or nearly white) opaque rectangle painted after
|
||||
//! dark text hides it just as well, the way forms are "whited out"; that is reported as its
|
||||
//! own kind, [`FindingKind::HiddenUnderWhite`], so callers can decide (#2385).
|
||||
//!
|
||||
//! Detection limits (documented, not hidden): `re` rectangles and single closed `m`/`l`
|
||||
//! subpaths whose four corners form an axis-aligned rectangle in user space (#1510) are
|
||||
//! considered when closed by a fill operator — other filled paths, shadings and patterns are
|
||||
//! not; a rectangle drawn with a translucent ExtGState (`/ca < 1`) is not flagged; opaque
|
||||
//! filled `/Square` annotations count as covers, but annotation rectangles are compared in
|
||||
//! unrotated user space, so pages with `/Rotate` may misalign;
|
||||
//! covers inside form XObjects are found, but only text drawn directly in the page content
|
||||
//! is checked under them, so text inside a form XObject is not audited.
|
||||
//! Covers come from the page content — `re` rectangles and single closed `m`/`l` subpaths whose
|
||||
//! four corners form an axis-aligned rectangle in user space (#1510), filled opaquely, and a
|
||||
//! thick opaque stroke along one axis (a "marker" line, its round or square caps included) — and
|
||||
//! from the annotations: an opaque `/Highlight` dark enough to hide the text under it (#2385), and
|
||||
//! an opaque filled `/Square` likewise, or white (#2854). A `/Redact` mark is reported whatever its
|
||||
//! flags: Hidden or NoView hides the mark, not the text.
|
||||
//!
|
||||
//! Detection limits (documented, not hidden): other filled paths, slanted or bent strokes,
|
||||
//! shadings and patterns are not considered; a cover drawn with a translucent ExtGState
|
||||
//! (`/ca`, `/CA < 1`) or a highlight or square with `/CA < 1` is not flagged; annotation rectangles are
|
||||
//! compared in unrotated user space, so pages with `/Rotate` may misalign; covers inside form
|
||||
//! XObjects are found, but only text drawn directly in the page content is checked under them,
|
||||
//! so text inside a form XObject is not audited; a `/Highlight` is judged by its own
|
||||
//! `/QuadPoints`, `/C` and `/CA`, never by its appearance stream (`/AP`), so a finding can differ
|
||||
//! from what a viewer paints.
|
||||
//!
|
||||
//! The audit fails closed: any content that cannot be read (an undecodable content stream,
|
||||
//! malformed annotations, form XObjects nested deeper than 32 or in a cycle) is an error, never
|
||||
@@ -35,10 +44,15 @@ mod tests;
|
||||
/// What the audit found.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum FindingKind {
|
||||
/// An opaque filled rectangle covers text that is still extractable.
|
||||
/// An opaque dark cover (a filled rectangle, a thick stroke, a highlight annotation) hides
|
||||
/// text that is still extractable.
|
||||
CoveredText,
|
||||
/// A `/Redact` annotation was placed but never applied: the text is untouched.
|
||||
UnappliedMark,
|
||||
/// A white (or nearly white) opaque rectangle painted after text, or a white filled square
|
||||
/// annotation over it, hides it: the text is invisible on the page and still extractable (a
|
||||
/// "whited out" form).
|
||||
HiddenUnderWhite,
|
||||
}
|
||||
|
||||
impl FindingKind {
|
||||
@@ -51,6 +65,7 @@ impl FindingKind {
|
||||
match self {
|
||||
FindingKind::CoveredText => "covered-text",
|
||||
FindingKind::UnappliedMark => "unapplied-mark",
|
||||
FindingKind::HiddenUnderWhite => "hidden-under-white",
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -99,6 +114,9 @@ const COVER_FRACTION: f64 = 0.4;
|
||||
const WORD_FRACTION: f64 = 0.5;
|
||||
/// WCAG contrast ratio below which the text is unreadable against the cover.
|
||||
const COVER_CONTRAST: f64 = 3.0;
|
||||
/// Relative luminance from which an opaque cover counts as white (white is 1.0; `0.9 g`, the
|
||||
/// kind of shading a table has, is 0.79): dark text under it is gone, not shaded.
|
||||
const WHITE_LUMINANCE: f64 = 0.9;
|
||||
/// Bytes of a hidden line's text read (copied or trimmed) that cost one comparison: a byte scan
|
||||
/// is far cheaper than a box comparison, but a long padded line under many covers adds up.
|
||||
const TEXT_BYTES_PER_COMPARISON: usize = 16;
|
||||
@@ -118,7 +136,8 @@ pub fn audit_redactions(doc: &Document) -> Result<Vec<AuditFinding>, AuditError>
|
||||
#[derive(Clone, Copy)]
|
||||
struct Limits {
|
||||
/// Cover-against-line comparisons per page, counting a line once per word box it has (at
|
||||
/// least once), since [`covers`] walks them.
|
||||
/// least once), since [`covers`] walks them, and a cover once per area it paints (a
|
||||
/// highlight's quads).
|
||||
comparisons: usize,
|
||||
/// Bytes of covered text the findings report.
|
||||
covered_text: usize,
|
||||
@@ -135,7 +154,7 @@ impl Limits {
|
||||
struct Budget {
|
||||
limits: Limits,
|
||||
comparisons: usize,
|
||||
/// What comparing one cover with every line of the current page costs.
|
||||
/// What comparing one area of a cover with every line of the current page costs.
|
||||
page_cost: usize,
|
||||
covered_text: usize,
|
||||
}
|
||||
@@ -149,18 +168,18 @@ impl Budget {
|
||||
self.page_cost = lines.iter().fold(0, |n, line| n.saturating_add(line.ink.len().max(line.runs().count()).max(1)));
|
||||
}
|
||||
|
||||
/// The text `cover` hides in the `lines` that `hides` ([`covered_words`]), joined with single
|
||||
/// The text `areas` hide in the `lines` that `hides` ([`covered_words`]), joined with single
|
||||
/// spaces, or `None` when they show no text. Without `copying` the text is empty and never
|
||||
/// assembled: one long line under many covers would otherwise be copied once per cover,
|
||||
/// outside both bounds.
|
||||
/// outside both bounds. Every area is compared with every line, so each one costs the page.
|
||||
fn hidden_text(
|
||||
&mut self,
|
||||
lines: &[TextLine],
|
||||
copying: bool,
|
||||
cover: &[f64; 4],
|
||||
areas: &[[f64; 4]],
|
||||
hides: impl Fn(&TextLine) -> bool,
|
||||
) -> Result<Option<String>, AuditError> {
|
||||
self.spend(self.page_cost)?;
|
||||
self.spend(self.page_cost.saturating_mul(areas.len()))?;
|
||||
if !copying {
|
||||
let mut hidden = false;
|
||||
for line in lines.iter().filter(|l| hides(l)) {
|
||||
@@ -181,7 +200,7 @@ impl Budget {
|
||||
if !covered.is_empty() {
|
||||
covered.push(' ');
|
||||
}
|
||||
covered_words(cover, line, &mut covered);
|
||||
covered_words(areas, line, &mut covered);
|
||||
}
|
||||
Ok((!covered.trim().is_empty()).then_some(covered))
|
||||
}
|
||||
@@ -217,28 +236,27 @@ fn audit(doc: &Document, limits: Limits) -> Result<Vec<AuditFinding>, AuditError
|
||||
// or substitute one must not hide text a cover sits on.
|
||||
let lines = pdfcraft_edit::glyph_text_lines(doc, page).map_err(|e| AuditError::Text(n, e.to_string()))?;
|
||||
budget.start_page(&lines);
|
||||
// 1. Opaque filled rectangles over extractable text.
|
||||
// 1. Opaque rectangles (filled, or painted by a thick stroke) over extractable text.
|
||||
for r in cover::filled_rects(doc, &page.dict)? {
|
||||
if let Some(covered) = budget.hidden_text(&lines, copying, &r.rect, |line| is_cover(&r, line))? {
|
||||
if let Some(covered) = budget.hidden_text(&lines, copying, std::slice::from_ref(&r.rect), |line| hides(&r, line))? {
|
||||
let covered_text = budget.report_text(covered, copying)?;
|
||||
out.push(AuditFinding { page: n, kind: FindingKind::CoveredText, rect: r.rect, covered_text });
|
||||
out.push(AuditFinding { page: n, kind: rect_kind(&r), rect: r.rect, covered_text });
|
||||
}
|
||||
}
|
||||
// 2. Annotation covers over extractable text.
|
||||
for annot in annots(doc, &page.dict)? {
|
||||
let cover = match &annot {
|
||||
AnnotationCover::Redact(rect) | AnnotationCover::Square { rect, .. } => *rect,
|
||||
// 2. Redact annotations that were never applied, and highlights and filled squares that
|
||||
// hide the text under them (annotations are drawn over the page's content): dark enough
|
||||
// against it, or, for a square, white, like a white rectangle in the content.
|
||||
for mark in annots(doc, &page.dict)? {
|
||||
let (kind, hides) = match mark.kind {
|
||||
Mark::Redact => (FindingKind::UnappliedMark, None),
|
||||
Mark::Square(rgb) if luminance(rgb) >= WHITE_LUMINANCE => (FindingKind::HiddenUnderWhite, None),
|
||||
Mark::Highlight(rgb) | Mark::Square(rgb) => (FindingKind::CoveredText, Some(rgb)),
|
||||
};
|
||||
if let Some(covered) = budget.hidden_text(&lines, copying, &cover, |line| match &annot {
|
||||
AnnotationCover::Redact(rect) => covers(rect, line),
|
||||
AnnotationCover::Square { rect, rgb } => covers(rect, line) && contrast(line.color, *rgb) < COVER_CONTRAST,
|
||||
if let Some(covered) = budget.hidden_text(&lines, copying, &mark.areas, |line| {
|
||||
mark.areas.iter().any(|a| covers(a, line)) && hides.is_none_or(|rgb| contrast(line.color, rgb) < COVER_CONTRAST)
|
||||
})? {
|
||||
let (kind, rect) = match annot {
|
||||
AnnotationCover::Redact(rect) => (FindingKind::UnappliedMark, rect),
|
||||
AnnotationCover::Square { rect, .. } => (FindingKind::CoveredText, rect),
|
||||
};
|
||||
let covered_text = budget.report_text(covered, copying)?;
|
||||
out.push(AuditFinding { page: n, kind, rect, covered_text });
|
||||
out.push(AuditFinding { page: n, kind, rect: mark.rect, covered_text });
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -255,6 +273,21 @@ fn is_cover(rect: &cover::FilledRect, line: &TextLine) -> bool {
|
||||
rect.pos > line.first_paint_pos() && covers(&rect.rect, line) && contrast(line.color, rect.rgb) < COVER_CONTRAST
|
||||
}
|
||||
|
||||
/// What a painted rectangle does to the text under it: white or nearly so, it whites the text out
|
||||
/// whatever its color; otherwise it is a cover, and only hides text it is dark enough against.
|
||||
fn rect_kind(rect: &cover::FilledRect) -> FindingKind {
|
||||
if luminance(rect.rgb) >= WHITE_LUMINANCE { FindingKind::HiddenUnderWhite } else { FindingKind::CoveredText }
|
||||
}
|
||||
|
||||
/// Does `rect` hide `line`: painted after the start of the line and over it, and white or a cover
|
||||
/// ([`is_cover`]). Light covers in between (table shading) are not flagged.
|
||||
fn hides(rect: &cover::FilledRect, line: &TextLine) -> bool {
|
||||
match rect_kind(rect) {
|
||||
FindingKind::HiddenUnderWhite => rect.pos > line.first_paint_pos() && covers(&rect.rect, line),
|
||||
_ => is_cover(rect, line),
|
||||
}
|
||||
}
|
||||
|
||||
/// Does `cover` hide a substantial part of what `line` draws? Three ways, any is enough:
|
||||
///
|
||||
/// - Half of one word lies under the cover. A box over one number in a long line hides a few
|
||||
@@ -284,14 +317,14 @@ fn covers(cover: &[f64; 4], line: &TextLine) -> bool {
|
||||
inked || (boxed && (line.ink.is_empty() || word_intersects()))
|
||||
}
|
||||
|
||||
/// Appends the text of `line` that `cover` lies on to `out`: the runs whose words it overlaps
|
||||
/// (#4164), so the words of the line painted beside the cover, which stay visible, are not
|
||||
/// Appends the text of `line` that `areas` lie on to `out`: the runs whose words one of them
|
||||
/// overlaps (#4164; a highlight's quads are several areas, #2385), so the words of the line painted beside the cover, which stay visible, are not
|
||||
/// reported as hidden. Neighbouring runs are joined as the line joins them, so a word split
|
||||
/// across runs stays one word; runs with others left out between them are spaced. The whole
|
||||
/// line when its runs can't say: a single run, a line whose text was reordered for reading, or
|
||||
/// a cover that overlaps the line's box but none of its words.
|
||||
fn covered_words(cover: &[f64; 4], line: &TextLine, out: &mut String) {
|
||||
let overlaps = |w: &[f64; 4]| area(&[cover[0].max(w[0]), cover[1].max(w[1]), cover[2].min(w[2]), cover[3].min(w[3])]) > 0.0;
|
||||
fn covered_words(areas: &[[f64; 4]], line: &TextLine, out: &mut String) {
|
||||
let overlaps = |w: &[f64; 4]| areas.iter().any(|c| area(&[c[0].max(w[0]), c[1].max(w[1]), c[2].min(w[2]), c[3].min(w[3])]) > 0.0);
|
||||
let mut words = String::new();
|
||||
let mut last = None;
|
||||
for (i, (text, ink, spaced)) in line.runs().enumerate() {
|
||||
@@ -333,14 +366,28 @@ fn contrast(a: [f64; 3], b: [f64; 3]) -> f64 {
|
||||
(hi + 0.05) / (lo + 0.05)
|
||||
}
|
||||
|
||||
enum AnnotationCover {
|
||||
Redact([f64; 4]),
|
||||
Square { rect: [f64; 4], rgb: [f64; 3] },
|
||||
/// An annotation the audit looks at.
|
||||
enum Mark {
|
||||
Redact,
|
||||
/// An opaque highlight, with its color.
|
||||
Highlight([f64; 3]),
|
||||
/// An opaque filled square, with its interior color (`/IC`).
|
||||
Square([f64; 3]),
|
||||
}
|
||||
|
||||
/// Opaque annotation covers on a page (unrotated user space). A missing `/Annots` means no
|
||||
/// covers; a malformed one fails the audit rather than silently passing.
|
||||
fn annots(doc: &Document, page: &Dict) -> Result<Vec<AnnotationCover>, AuditError> {
|
||||
struct Annot {
|
||||
/// `/Rect`, normalized, in unrotated user space.
|
||||
rect: [f64; 4],
|
||||
/// What the annotation paints: a highlight's `/QuadPoints`, each as its bounding box, or else
|
||||
/// the `/Rect`.
|
||||
areas: Vec<[f64; 4]>,
|
||||
kind: Mark,
|
||||
}
|
||||
|
||||
/// The `/Redact` annotations on a page, and the `/Highlight` and filled `/Square` ones shown
|
||||
/// opaque with a color (unrotated user space). A missing `/Annots` means no marks; a malformed
|
||||
/// one fails the audit rather than silently passing.
|
||||
fn annots(doc: &Document, page: &Dict) -> Result<Vec<Annot>, AuditError> {
|
||||
let Some(a) = page.get(b"Annots") else { return Ok(Vec::new()) };
|
||||
let resolved = doc.resolve(a);
|
||||
// A missing /Annots means no marks; a malformed one fails the audit.
|
||||
@@ -351,29 +398,70 @@ fn annots(doc: &Document, page: &Dict) -> Result<Vec<AnnotationCover>, AuditErro
|
||||
// copied once per listing.
|
||||
let resolved = doc.resolve(a);
|
||||
let Some(dict) = resolved.as_dict() else { continue };
|
||||
let v: Vec<f64> = match dict.get(b"Rect") {
|
||||
None => continue,
|
||||
Some(r) => doc.resolve(r).as_array().map(|arr| arr.iter().filter_map(|o| doc.resolve(o).as_f64()).collect()).unwrap_or_default(),
|
||||
let nums = |key: &[u8]| -> Vec<f64> {
|
||||
dict.get(key)
|
||||
.map(|r| doc.resolve(r))
|
||||
.and_then(|r| r.as_array().map(|arr| arr.iter().filter_map(|o| doc.resolve(o).as_f64()).collect()))
|
||||
.unwrap_or_default()
|
||||
};
|
||||
if v.len() != 4 || !v.iter().all(|x| x.is_finite()) {
|
||||
continue;
|
||||
}
|
||||
let rect = [v[0].min(v[2]), v[1].min(v[3]), v[0].max(v[2]), v[1].max(v[3])];
|
||||
match subtype(doc, dict)?.as_deref() {
|
||||
Some(b"Redact") => out.push(AnnotationCover::Redact(rect)),
|
||||
Some(b"Square") => {
|
||||
let flags = dict.get(b"F").and_then(|f| doc.resolve(f).as_int()).unwrap_or(0);
|
||||
if flags & 2 != 0 || flags & 32 != 0 {
|
||||
continue;
|
||||
}
|
||||
let alpha = dict.get(b"CA").map(|o| doc.resolve(o)).and_then(|o| o.as_f64()).unwrap_or(1.0);
|
||||
if alpha >= 1.0
|
||||
&& let Some(rgb) = annotation_color(doc, dict, b"IC")
|
||||
{
|
||||
out.push(AnnotationCover::Square { rect, rgb });
|
||||
}
|
||||
// Hidden or NoView: not drawn, so a cover hides nothing. A `/Redact` mark is audited
|
||||
// whatever its flags: hiding the mark does not apply the redaction.
|
||||
let flags = dict.get(b"F").and_then(|f| doc.resolve(f).as_int()).unwrap_or(0);
|
||||
let drawn = flags & (2 | 32) == 0;
|
||||
// The color the annotation paints opaquely: a highlight's `/C`, a square's interior `/IC`.
|
||||
// `None`: no color (drawn transparent), or `/CA` below 1.
|
||||
let opaque = |key: &[u8]| -> Option<[f64; 3]> {
|
||||
let alpha = dict.get(b"CA").and_then(|a| doc.resolve(a).as_f64()).unwrap_or(1.0);
|
||||
let rgb = match nums(key).as_slice() {
|
||||
[g] => [*g, *g, *g],
|
||||
[r, g, b] => [*r, *g, *b],
|
||||
[c, m, y, k] => [(1.0 - c) * (1.0 - k), (1.0 - m) * (1.0 - k), (1.0 - y) * (1.0 - k)],
|
||||
_ => return None,
|
||||
};
|
||||
if alpha.is_nan() || alpha < 1.0 || rgb.iter().any(|v| !v.is_finite()) {
|
||||
return None;
|
||||
}
|
||||
_ => {}
|
||||
Some(rgb.map(|v| v.clamp(0.0, 1.0)))
|
||||
};
|
||||
let kind = match subtype(doc, dict)?.as_deref() {
|
||||
Some(b"Redact") => Mark::Redact,
|
||||
Some(b"Highlight") => match opaque(b"C") {
|
||||
Some(rgb) if drawn => Mark::Highlight(rgb),
|
||||
_ => continue,
|
||||
},
|
||||
Some(b"Square") => match opaque(b"IC") {
|
||||
Some(rgb) if drawn => Mark::Square(rgb),
|
||||
_ => continue,
|
||||
},
|
||||
_ => continue,
|
||||
};
|
||||
let v = nums(b"Rect");
|
||||
if v.len() == 4 && v.iter().all(|x| x.is_finite()) {
|
||||
let rect = [v[0].min(v[2]), v[1].min(v[3]), v[0].max(v[2]), v[1].max(v[3])];
|
||||
// Only a highlight paints quads, and only a whole number of them is read: the length
|
||||
// is checked before any number is collected.
|
||||
let quads = if matches!(kind, Mark::Highlight(_)) { dict.get(b"QuadPoints").map(|q| doc.resolve(q)) } else { None };
|
||||
let q: Vec<f64> = quads
|
||||
.as_ref()
|
||||
.and_then(|q| q.as_array())
|
||||
.filter(|arr| !arr.is_empty() && arr.len() % 8 == 0)
|
||||
.and_then(|arr| arr.iter().map(|o| doc.resolve(o).as_f64()).collect())
|
||||
.unwrap_or_default();
|
||||
let areas = if !q.is_empty() && q.iter().all(|x| x.is_finite()) {
|
||||
q.chunks(8)
|
||||
.map(|c| {
|
||||
[
|
||||
c[0].min(c[2]).min(c[4]).min(c[6]),
|
||||
c[1].min(c[3]).min(c[5]).min(c[7]),
|
||||
c[0].max(c[2]).max(c[4]).max(c[6]),
|
||||
c[1].max(c[3]).max(c[5]).max(c[7]),
|
||||
]
|
||||
})
|
||||
.collect()
|
||||
} else {
|
||||
vec![rect]
|
||||
};
|
||||
out.push(Annot { rect, areas, kind });
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
@@ -388,19 +476,3 @@ fn subtype(doc: &Document, dict: &Dict) -> Result<Option<Vec<u8>>, AuditError> {
|
||||
};
|
||||
Ok(name)
|
||||
}
|
||||
|
||||
fn annotation_color(doc: &Document, dict: &Dict, key: &[u8]) -> Option<[f64; 3]> {
|
||||
let values: Vec<f64> = dict.get(key).map(|o| doc.resolve(o))?.as_array()?.iter().map(|o| doc.resolve(o).as_f64()).collect::<Option<_>>()?;
|
||||
if !values.iter().all(|v| v.is_finite()) {
|
||||
return None;
|
||||
}
|
||||
match values.as_slice() {
|
||||
[g] => Some([g.clamp(0.0, 1.0); 3]),
|
||||
[r, g, b] => Some([r.clamp(0.0, 1.0), g.clamp(0.0, 1.0), b.clamp(0.0, 1.0)]),
|
||||
[c, m, y, k] => {
|
||||
let (c, m, y, k) = (c.clamp(0.0, 1.0), m.clamp(0.0, 1.0), y.clamp(0.0, 1.0), k.clamp(0.0, 1.0));
|
||||
Some([(1.0 - c) * (1.0 - k), (1.0 - m) * (1.0 - k), (1.0 - y) * (1.0 - k)])
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
+295
-6
@@ -279,6 +279,44 @@ fn print_flag_square_annotation_is_still_a_finding() {
|
||||
assert_eq!(findings[0].kind, FindingKind::CoveredText);
|
||||
}
|
||||
|
||||
/// A white filled Square annotation hides dark text as a white rectangle in the content does:
|
||||
/// reported as hidden under white, not passed over by the contrast test.
|
||||
#[test]
|
||||
fn white_square_annotation_over_text_is_hidden_under_white() {
|
||||
let content = b"BT /F1 24 Tf 100 700 Td (SECRET DATA 123) Tj ET\n";
|
||||
let mut b = Builder::new();
|
||||
b.add("<< /Type /Catalog /Pages 2 0 R >>");
|
||||
b.add("<< /Type /Pages /Kids [3 0 R] /Count 1 >>");
|
||||
b.add("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> /Annots [6 0 R] >>");
|
||||
b.add_stream(content);
|
||||
b.add(FONT);
|
||||
b.add("<< /Type /Annot /Subtype /Square /Rect [95 685 325 725] /C [0 0 0] /IC [1 1 1] >>");
|
||||
let findings = audit_redactions(&open(b.build())).expect("audit must run");
|
||||
assert_eq!(findings.len(), 1, "{findings:?}");
|
||||
assert_eq!((findings[0].kind.clone(), findings[0].rect), (FindingKind::HiddenUnderWhite, [95.0, 685.0, 325.0, 725.0]));
|
||||
assert_eq!(findings[0].covered_text, "SECRET DATA 123");
|
||||
}
|
||||
|
||||
/// A Hidden or NoView `/Redact` mark is still an unapplied one: the flag hides the mark, the
|
||||
/// text is untouched and extractable.
|
||||
#[test]
|
||||
fn hidden_redact_mark_is_still_unapplied() {
|
||||
for flags in ["/F 2", "/F 32"] {
|
||||
let content = b"BT /F1 24 Tf 100 700 Td (SECRET DATA 123) Tj ET\n";
|
||||
let mut b = Builder::new();
|
||||
b.add("<< /Type /Catalog /Pages 2 0 R >>");
|
||||
b.add("<< /Type /Pages /Kids [3 0 R] /Count 1 >>");
|
||||
b.add("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> /Annots [6 0 R] >>");
|
||||
b.add_stream(content);
|
||||
b.add(FONT);
|
||||
b.add(&format!("<< /Type /Annot /Subtype /Redact /Rect [95 685 325 725] {flags} >>"));
|
||||
let findings = audit_redactions(&open(b.build())).expect("audit must run");
|
||||
assert_eq!(findings.len(), 1, "{flags}: {findings:?}");
|
||||
assert_eq!(findings[0].kind, FindingKind::UnappliedMark, "{flags}");
|
||||
assert_eq!(findings[0].covered_text, "SECRET DATA 123");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn background_before_text_is_not_flagged() {
|
||||
// An opaque dark rectangle painted *before* the text is a background, not a cover.
|
||||
@@ -298,12 +336,13 @@ fn light_shading_after_text_is_not_flagged() {
|
||||
#[test]
|
||||
fn graphics_state_carries_across_streams() {
|
||||
// The first stream sets a white fill after the text; the second stream's rectangle over
|
||||
// the text is therefore white, not the default black: no finding. Without the carry fix
|
||||
// the rectangle would default to black and flag.
|
||||
// the text is therefore white, not the default black: hidden under white, not a dark
|
||||
// cover. Without the carry fix the rectangle would default to black and flag as one.
|
||||
let s1 = b"BT /F1 24 Tf 100 700 Td (SECRET DATA 123) Tj ET\n1 g\n";
|
||||
let s2 = b"100 688 220 32 re f\n";
|
||||
let findings = audit_redactions(&open(two_streams(s1, s2))).expect("audit must run");
|
||||
assert!(findings.is_empty(), "{findings:?}");
|
||||
assert_eq!(findings.len(), 1, "{findings:?}");
|
||||
assert_eq!(findings[0].kind, FindingKind::HiddenUnderWhite, "white, as the first stream set, not the default black");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -481,6 +520,7 @@ fn finding_kind_ids_are_stable() {
|
||||
// formats). They must not change; add a variant instead of renaming.
|
||||
assert_eq!(FindingKind::CoveredText.id(), "covered-text");
|
||||
assert_eq!(FindingKind::UnappliedMark.id(), "unapplied-mark");
|
||||
assert_eq!(FindingKind::HiddenUnderWhite.id(), "hidden-under-white");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -593,6 +633,229 @@ fn reading_blocks_respect_stream_comments_and_split_operands() {
|
||||
}
|
||||
}
|
||||
|
||||
/// #2385: a thick black stroke along the text (a "marker" line) paints the same rectangle a
|
||||
/// filled one would, and a white one whites the text out; a hairline doesn't hide it, a
|
||||
/// translucent stroke doesn't either, and a stroked rectangle only outlines.
|
||||
#[test]
|
||||
fn thick_stroke_over_text_is_a_cover() {
|
||||
let text = &b"BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj ET\n"[..];
|
||||
let audit = |cover: &[u8]| audit_redactions(&open(one_page(&[text, cover].concat(), ""))).expect("audit must run");
|
||||
let findings = audit(b"0 G 20 w 70 705 m 400 705 l S\n");
|
||||
assert_eq!(findings.len(), 1, "{findings:?}");
|
||||
assert_eq!((findings[0].kind.clone(), findings[0].rect), (FindingKind::CoveredText, [70.0, 695.0, 400.0, 715.0]));
|
||||
assert!(findings[0].covered_text.contains("IBAN"), "{:?}", findings[0].covered_text);
|
||||
// Fill and stroke together. A vertical marker over most of the first word hides that word
|
||||
// (#2662), as a filled box there would; a narrow one over its first letter does not.
|
||||
assert_eq!(audit(b"0 G 20 w 70 705 m 400 705 l B\n").len(), 1);
|
||||
assert_eq!(audit(b"0 G 20 w 80 690 m 80 720 l S\n").len(), 1, "a vertical stroke over most of IBAN");
|
||||
assert!(audit(b"0 G 6 w 75 690 m 75 720 l S\n").is_empty(), "a vertical stroke hides only the first letter");
|
||||
for not_a_cover in [&b"0 G 1 w 70 705 m 400 705 l S\n"[..], b"0 G 20 w 70 695 330 20 re S\n"] {
|
||||
assert!(audit(not_a_cover).is_empty(), "{:?}", String::from_utf8_lossy(not_a_cover));
|
||||
}
|
||||
// A white marker whites the text out, like a white rectangle.
|
||||
let findings = audit(b"1 G 20 w 70 705 m 400 705 l S\n");
|
||||
assert_eq!(findings.len(), 1, "{findings:?}");
|
||||
assert_eq!(findings[0].kind, FindingKind::HiddenUnderWhite);
|
||||
// A translucent stroke (`/CA`) is not a cover; `/ca` does not apply to strokes.
|
||||
let content = [&b"BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj ET\n"[..], b"/GS1 gs 0 G 20 w 70 705 m 400 705 l S\n"].concat();
|
||||
for (state, found) in [("<< /Type /ExtGState /CA 0.5 >>", 0), ("<< /Type /ExtGState /ca 0.5 >>", 1)] {
|
||||
let mut b = Builder::new();
|
||||
b.add("<< /Type /Catalog /Pages 2 0 R >>");
|
||||
b.add("<< /Type /Pages /Kids [3 0 R] /Count 1 >>");
|
||||
b.add("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> /ExtGState << /GS1 6 0 R >> >> >>");
|
||||
b.add_stream(&content);
|
||||
b.add(FONT);
|
||||
b.add(state);
|
||||
assert_eq!(audit_redactions(&open(b.build())).expect("audit must run").len(), found, "{state}");
|
||||
}
|
||||
}
|
||||
|
||||
/// #2385: a black, opaque Highlight annotation over the text hides it like a cover; a yellow one
|
||||
/// or a translucent one doesn't, and a hidden one isn't drawn at all.
|
||||
#[test]
|
||||
fn black_highlight_annotation_over_text_is_a_cover() {
|
||||
let content = b"BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj ET\n";
|
||||
let audit = |extra: &str| {
|
||||
let mut b = Builder::new();
|
||||
b.add("<< /Type /Catalog /Pages 2 0 R >>");
|
||||
b.add("<< /Type /Pages /Kids [3 0 R] /Count 1 >>");
|
||||
b.add("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> /Annots [6 0 R] >>");
|
||||
b.add_stream(content);
|
||||
b.add(FONT);
|
||||
b.add(&format!("<< /Type /Annot /Subtype /Highlight /Rect [70 695 400 715] /F 4 /QuadPoints [70 715 400 715 70 695 400 695] {extra} >>"));
|
||||
audit_redactions(&open(b.build())).expect("audit must run")
|
||||
};
|
||||
let findings = audit("/C [0 0 0]");
|
||||
assert_eq!(findings.len(), 1, "{findings:?}");
|
||||
assert_eq!((findings[0].kind.clone(), findings[0].rect), (FindingKind::CoveredText, [70.0, 695.0, 400.0, 715.0]));
|
||||
assert!(findings[0].covered_text.contains("IBAN"), "{:?}", findings[0].covered_text);
|
||||
assert_eq!(audit("/C [0]").len(), 1, "a gray level");
|
||||
for not_a_cover in ["/C [1 1 0]", "/C [0 0 0] /CA 0.5", "/C [0 0 0] /F 2", ""] {
|
||||
assert!(audit(not_a_cover).is_empty(), "{not_a_cover}");
|
||||
}
|
||||
}
|
||||
|
||||
/// #2385: an opaque white rectangle painted after dark text makes it invisible, the way a form is
|
||||
/// "whited out"; that is its own kind, apart from the dark covers, so callers can decide. Light
|
||||
/// shading (`0.9 g`) stays unflagged, as before.
|
||||
#[test]
|
||||
fn white_rectangle_over_text_is_reported_as_hidden_under_white() {
|
||||
let text = &b"BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj ET\n"[..];
|
||||
let audit = |cover: &[u8]| audit_redactions(&open(one_page(&[text, cover].concat(), ""))).expect("audit must run");
|
||||
let findings = audit(b"1 g 70 695 330 20 re f\n");
|
||||
assert_eq!(findings.len(), 1, "{findings:?}");
|
||||
assert_eq!((findings[0].kind.clone(), findings[0].rect), (FindingKind::HiddenUnderWhite, [70.0, 695.0, 400.0, 715.0]));
|
||||
assert_eq!(findings[0].kind.clone().id(), "hidden-under-white");
|
||||
assert!(findings[0].covered_text.contains("IBAN"), "{:?}", findings[0].covered_text);
|
||||
assert!(audit(b"0.9 g 70 695 330 20 re f\n").is_empty(), "shading");
|
||||
assert!(
|
||||
audit(b"1 g 70 695 330 20 re f BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj ET\n").len() == 1,
|
||||
"the second copy of the text is drawn after the white"
|
||||
);
|
||||
}
|
||||
|
||||
/// One page with `content` and `/ExtGState << /GS1 6 0 R /GS2 7 0 R >>` holding `states`.
|
||||
fn page_with_states(content: &[u8], states: [&str; 2]) -> Vec<u8> {
|
||||
let mut b = Builder::new();
|
||||
b.add("<< /Type /Catalog /Pages 2 0 R >>");
|
||||
b.add("<< /Type /Pages /Kids [3 0 R] /Count 1 >>");
|
||||
b.add("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> /ExtGState << /GS1 6 0 R /GS2 7 0 R >> >> >>");
|
||||
b.add_stream(content);
|
||||
b.add(FONT);
|
||||
for s in states {
|
||||
b.add(s);
|
||||
}
|
||||
b.build()
|
||||
}
|
||||
|
||||
/// A `gs` sets only the parameters its dictionary names: `/LW` is the line width as `w` is, and
|
||||
/// an alpha it leaves out keeps the value an earlier `gs` set (ISO 32000-2 §8.4.5).
|
||||
#[test]
|
||||
fn ext_gstate_sets_line_width_and_keeps_omitted_alphas() {
|
||||
let text = &b"BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj ET\n"[..];
|
||||
let audit = |cover: &[u8], states| audit_redactions(&open(page_with_states(&[text, cover].concat(), states))).expect("audit must run");
|
||||
let findings = audit(b"/GS1 gs 0 G 70 705 m 400 705 l S\n", ["<< /LW 20 >>", "<< >>"]);
|
||||
assert_eq!(findings.len(), 1, "{findings:?}");
|
||||
assert_eq!((findings[0].kind.clone(), findings[0].rect), (FindingKind::CoveredText, [70.0, 695.0, 400.0, 715.0]));
|
||||
assert!(audit(b"/GS1 gs /GS2 gs 0 G 20 w 70 705 m 400 705 l S\n", ["<< /CA 0.5 >>", "<< /ca 0.5 >>"]).is_empty(), "/CA 0.5 still applies");
|
||||
assert!(audit(b"/GS1 gs /GS2 gs 0 g 70 695 330 20 re f\n", ["<< /ca 0.5 >>", "<< /LW 20 >>"]).is_empty(), "/ca 0.5 still applies");
|
||||
}
|
||||
|
||||
/// The line width is measured across the stroke after the CTM: under `4 0 0 1 0 0 cm` a
|
||||
/// horizontal stroke keeps its width and a vertical one is four times as wide.
|
||||
#[test]
|
||||
fn stroke_width_follows_a_non_uniform_ctm_across_the_segment() {
|
||||
let text = &b"BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj ET\n"[..];
|
||||
let audit = |cover: &[u8]| audit_redactions(&open(one_page(&[text, cover].concat(), ""))).expect("audit must run");
|
||||
let rects = |cover: &[u8]| audit(cover).into_iter().map(|f| f.rect).collect::<Vec<_>>();
|
||||
assert_eq!(rects(b"q 4 0 0 1 0 0 cm 0 G 20 w 17.5 705 m 100 705 l S Q\n"), [[70.0, 695.0, 400.0, 715.0]]);
|
||||
assert_eq!(rects(b"q 4 0 0 1 0 0 cm 0 G 5 w 20 690 m 20 720 l S Q\n"), [[70.0, 690.0, 90.0, 720.0]]);
|
||||
}
|
||||
|
||||
/// A round or projecting square cap (`1 J`, `2 J`, or `/LC` in an ExtGState) paints half the
|
||||
/// line width past each end of the stroke, so the text under a cap is hidden too; a butt cap
|
||||
/// (the default, and `0 J` again) paints none. Under a CTM the cap is the pen's extent along
|
||||
/// the stroke: four times the width under `4 0 0 1 0 0 cm`, `sqrt 2` times it under a shear.
|
||||
#[test]
|
||||
fn stroke_caps_lengthen_the_cover() {
|
||||
let text = &b"BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj ET\n"[..];
|
||||
let rects = |cover: &[u8]| {
|
||||
audit_redactions(&open(one_page(&[text, cover].concat(), ""))).expect("audit must run").into_iter().map(|f| f.rect).collect::<Vec<_>>()
|
||||
};
|
||||
assert_eq!(rects(b"0 G 20 w 80 705 m 400 705 l S\n"), [[80.0, 695.0, 400.0, 715.0]], "butt");
|
||||
assert_eq!(rects(b"0 G 1 J 20 w 80 705 m 400 705 l S\n"), [[70.0, 695.0, 410.0, 715.0]], "round");
|
||||
assert_eq!(rects(b"0 G 2 J 20 w 80 705 m 400 705 l S\n"), [[70.0, 695.0, 410.0, 715.0]], "projecting square");
|
||||
assert_eq!(rects(b"0 G 1 J 0 J 20 w 80 705 m 400 705 l S\n"), [[80.0, 695.0, 400.0, 715.0]], "butt again");
|
||||
assert_eq!(rects(b"0 G 1 J 20 w 80 700 m 80 720 l S\n"), [[70.0, 690.0, 90.0, 730.0]], "vertical");
|
||||
assert_eq!(rects(b"q 4 0 0 1 0 0 cm 0 G 1 J 20 w 20 705 m 100 705 l S Q\n"), [[40.0, 695.0, 440.0, 715.0]], "scaled along the stroke");
|
||||
let [sheared] = rects(b"q 1 0 1 1 0 0 cm 0 G 1 J 20 w -625 705 m -305 705 l S Q\n")[..] else { panic!("one cover") };
|
||||
let want = [80.0 - 10.0 * 2f64.sqrt(), 695.0, 400.0 + 10.0 * 2f64.sqrt(), 715.0];
|
||||
assert!(sheared.iter().zip(want).all(|(a, b)| (a - b).abs() < 1e-6), "sheared: {sheared:?} vs {want:?}");
|
||||
// The cap is state like the width: `/LC` sets it, a `gs` without it leaves it, `Q` restores it.
|
||||
let states = |cover: &[u8], states| {
|
||||
audit_redactions(&open(page_with_states(&[text, cover].concat(), states)))
|
||||
.expect("audit must run")
|
||||
.into_iter()
|
||||
.map(|f| f.rect)
|
||||
.collect::<Vec<_>>()
|
||||
};
|
||||
assert_eq!(states(b"/GS1 gs /GS2 gs 0 G 80 705 m 400 705 l S\n", ["<< /LC 1 /LW 20 >>", "<< >>"]), [[70.0, 695.0, 410.0, 715.0]]);
|
||||
assert_eq!(rects(b"q 1 J Q 0 G 20 w 80 705 m 400 705 l S\n"), [[80.0, 695.0, 400.0, 715.0]], "restored");
|
||||
// The cap alone can hide a word: with butt caps the stroke stops short of IBAN, the round cap reaches it.
|
||||
let hidden = |cover: &[u8]| {
|
||||
audit_redactions(&open(one_page(&[text, cover].concat(), "")))
|
||||
.expect("audit must run")
|
||||
.into_iter()
|
||||
.map(|f| f.covered_text)
|
||||
.collect::<Vec<_>>()
|
||||
};
|
||||
assert!(hidden(b"0 G 30 w 20 705 m 84 705 l S\n").is_empty(), "butt: under a third of IBAN");
|
||||
assert_eq!(hidden(b"0 G 1 J 30 w 20 705 m 84 705 l S\n"), ["IBAN IT60X0542811101000000123456"], "the cap reaches over most of IBAN");
|
||||
}
|
||||
|
||||
/// A white cover is `hidden-under-white` whatever the text's color: white text on a dark page,
|
||||
/// whited out, is not a dark cover just because text and cover have no contrast.
|
||||
#[test]
|
||||
fn white_cover_over_white_text_is_hidden_under_white() {
|
||||
let content = b"0 g 0 0 612 792 re f\n1 g BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj ET\n1 g 70 695 330 20 re f\n";
|
||||
let findings = audit_redactions(&open(one_page(content, ""))).expect("audit must run");
|
||||
assert_eq!(findings.len(), 1, "{findings:?}");
|
||||
assert_eq!((findings[0].kind.clone(), findings[0].rect), (FindingKind::HiddenUnderWhite, [70.0, 695.0, 400.0, 715.0]));
|
||||
}
|
||||
|
||||
/// A highlight paints its `/QuadPoints`, not its `/Rect`: a line inside the rectangle but
|
||||
/// outside every quadrilateral is not hidden. Without `/QuadPoints` the `/Rect` is used.
|
||||
#[test]
|
||||
fn highlight_covers_its_quad_points_not_its_rect() {
|
||||
let content = b"BT /F1 14 Tf 72 700 Td (IBAN IT60X0542811101000000123456) Tj 0 -30 Td (Account holder Jane Doe) Tj ET\n";
|
||||
let audit = |quads: &str| {
|
||||
let mut b = Builder::new();
|
||||
b.add("<< /Type /Catalog /Pages 2 0 R >>");
|
||||
b.add("<< /Type /Pages /Kids [3 0 R] /Count 1 >>");
|
||||
b.add("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> /Annots [6 0 R] >>");
|
||||
b.add_stream(content);
|
||||
b.add(FONT);
|
||||
b.add(&format!("<< /Type /Annot /Subtype /Highlight /Rect [70 660 400 715] /C [0 0 0] {quads} >>"));
|
||||
audit_redactions(&open(b.build())).expect("audit must run")
|
||||
};
|
||||
let findings = audit("/QuadPoints [70 715 400 715 70 695 400 695]");
|
||||
assert_eq!(findings.len(), 1, "{findings:?}");
|
||||
assert_eq!(findings[0].covered_text, "IBAN IT60X0542811101000000123456");
|
||||
assert_eq!(audit("")[0].covered_text, "IBAN IT60X0542811101000000123456 Account holder Jane Doe");
|
||||
}
|
||||
|
||||
/// A highlight reports the words under its quadrilaterals, not the words its `/Rect` spans: a
|
||||
/// `/Rect` over two lines with quads on part of one reports that part (#4164 for marks with
|
||||
/// several areas).
|
||||
#[test]
|
||||
fn highlight_reports_the_words_under_its_quad_points() {
|
||||
let audit = |quads: &str| {
|
||||
let mut b = Builder::new();
|
||||
b.add("<< /Type /Catalog /Pages 2 0 R >>");
|
||||
b.add("<< /Type /Pages /Kids [3 0 R] /Count 1 >>");
|
||||
b.add("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> /Annots [6 0 R] >>");
|
||||
b.add_stream(
|
||||
b"BT /F1 24 Tf 100 700 Td (SECRET) Tj ET
|
||||
BT /F1 24 Tf 200 700 Td (tail) Tj ET
|
||||
BT /F1 24 Tf 100 660 Td (Jane) Tj ET
|
||||
",
|
||||
);
|
||||
b.add(FONT);
|
||||
b.add(&format!("<< /Type /Annot /Subtype /Highlight /Rect [98 650 260 720] /C [0 0 0] /QuadPoints [{quads}] >>"));
|
||||
audit_redactions(&open(b.build())).expect("audit must run")
|
||||
};
|
||||
let secret = "98 720 192 720 98 688 192 688";
|
||||
let jane = "98 680 160 680 98 650 160 650";
|
||||
for (quads, covered) in [(secret.to_owned(), "SECRET"), (format!("{secret} {jane}"), "SECRET Jane")] {
|
||||
let findings = audit(&quads);
|
||||
assert_eq!(
|
||||
findings.iter().map(|f| (f.rect, f.covered_text.as_str())).collect::<Vec<_>>(),
|
||||
[([98.0, 650.0, 260.0, 720.0], covered)],
|
||||
"{quads}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The current path is not graphics state: a rectangle built before `Q` and filled after it
|
||||
/// paints the box, so it is a cover.
|
||||
#[test]
|
||||
@@ -704,6 +967,32 @@ fn cover_and_line_comparisons_are_bounded() {
|
||||
assert!(crate::audit(&doc, limits).expect("within the budget").is_empty());
|
||||
}
|
||||
|
||||
/// A highlight is compared with the page once per quad, so it is charged once per quad: one
|
||||
/// annotation with a few hundred thousand `/QuadPoints` must not be checked for the price of one
|
||||
/// cover (review on #2408). 20 quads against 20 two-word lines are 800; the `/Rect` alone is 40.
|
||||
#[test]
|
||||
fn a_highlights_quads_count_against_the_comparison_bound() {
|
||||
let mut content = Vec::new();
|
||||
for i in 0..20 {
|
||||
content.extend_from_slice(format!("BT /F1 12 Tf 72 {} Td (Line {i}) Tj ET\n", 700 - 14 * i).as_bytes());
|
||||
}
|
||||
let audit = |quads: usize, comparisons: usize| {
|
||||
let mut b = Builder::new();
|
||||
b.add("<< /Type /Catalog /Pages 2 0 R >>");
|
||||
b.add("<< /Type /Pages /Kids [3 0 R] /Count 1 >>");
|
||||
b.add("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> /Annots [6 0 R] >>");
|
||||
b.add_stream(&content);
|
||||
b.add(FONT);
|
||||
let quads = "0 10 10 10 0 0 10 0 ".repeat(quads);
|
||||
b.add(&format!("<< /Type /Annot /Subtype /Highlight /Rect [0 0 10 10] /C [0 0 0] /QuadPoints [{quads}] >>"));
|
||||
crate::audit(&open(b.build()), crate::Limits { comparisons, covered_text: usize::MAX })
|
||||
};
|
||||
assert!(matches!(audit(20, 799), Err(crate::AuditError::TooManyComparisons)));
|
||||
assert!(audit(20, 800).expect("within the budget").is_empty());
|
||||
assert!(matches!(audit(0, 39), Err(crate::AuditError::TooManyComparisons)));
|
||||
assert!(audit(0, 40).expect("within the budget").is_empty());
|
||||
}
|
||||
|
||||
/// A run that draws no word still costs a comparison: the covered text walks every run of a
|
||||
/// covered line, so a word followed by many `( ) Tj` must not be checked for free (review on
|
||||
/// #4198).
|
||||
@@ -774,10 +1063,10 @@ fn withheld_covered_text_is_not_assembled() {
|
||||
let lines = pdfcraft_edit::text_lines(&doc, 0).expect("text");
|
||||
let mut budget = crate::Budget { limits: crate::Limits::DEFAULT, comparisons: 0, page_cost: 0, covered_text: 0 };
|
||||
budget.start_page(&lines);
|
||||
let withheld = budget.hidden_text(&lines, false, &PAGE, |_| true).expect("within the budget");
|
||||
let withheld = budget.hidden_text(&lines, false, &[PAGE], |_| true).expect("within the budget");
|
||||
assert_eq!(withheld.map(|t| t.capacity()), Some(0), "a finding, with no text built for it");
|
||||
assert_eq!(budget.hidden_text(&lines, false, &PAGE, |_| false).expect("within the budget"), None);
|
||||
let copied = budget.hidden_text(&lines, true, &PAGE, |_| true).expect("within the budget");
|
||||
assert_eq!(budget.hidden_text(&lines, false, &[PAGE], |_| false).expect("within the budget"), None);
|
||||
let copied = budget.hidden_text(&lines, true, &[PAGE], |_| true).expect("within the budget");
|
||||
assert_eq!(copied.as_deref(), Some("SECRET DATA 123"));
|
||||
}
|
||||
|
||||
|
||||
@@ -746,7 +746,7 @@ pub fn tools() -> Vec<ToolDef> {
|
||||
t(
|
||||
"doc_audit_redactions",
|
||||
"Audit faux redactions",
|
||||
"Find faux redactions: opaque filled rectangles painted over text that is still extractable, and /Redact annotations that were marked but never applied. Paint-order and colour-aware: a rectangle counts only when painted after the text and dark enough to hide it (page backgrounds and light shading are not flagged). Each finding reports the 1-based page, the kind (covered-text or unapplied-mark), the rectangle (points from the top-left of the displayed page, like comment_list), and the hidden text. Fails closed: unreadable content is an error, never a clean bill. A properly applied redaction removes the text and never flags.",
|
||||
"Find faux redactions: opaque covers painted over text that is still extractable (filled rectangles, thick marker-style strokes, dark opaque Highlight annotations), text whited out under an opaque white rectangle, and /Redact annotations that were marked but never applied. Paint-order and colour-aware: a cover counts only when painted after the text and dark enough to hide it (page backgrounds and light shading are not flagged); a white rectangle over dark text is reported apart. Each finding reports the 1-based page, the kind (covered-text, hidden-under-white or unapplied-mark), the rectangle, and the hidden text. Fails closed: unreadable content is an error, never a clean bill. A properly applied redaction removes the text and never flags.",
|
||||
)
|
||||
.ro()
|
||||
.with(schema(json!({ "doc": doc() }), &["doc"])),
|
||||
|
||||
Reference in New Issue
Block a user