Attack-Aware Evaluation of Face De-identification: Task Preservation and Query Exposure

Face images contain several kinds of information at once: identity, head pose, gaze, and expression. This overlap makes sharing visual data difficult, because suppressing identity can also remove the information that downstream tasks require — for example, facial movement and orientation in driver-monitoring imagery. A protection score is therefore conditional on the available attacks, the utility a task needs, and the population the queries represent. Our research asks a different question from whether an output looks anonymous: how much identity linkage remains when an attacker can restore images before matching, and what utility is actually lost when methods are selected by aggregate scores?

LISA has built a unified attack-aware evaluation framework covering six fixed de-identification configurations — Blur, Pixelation, FALCO, SimSwap, TIP-IM, and FaceAnon — on VGGFace2-derived photographs and YawDD-derived driver-facing vehicle frames. The framework pairs three face recognizers (ArcFace, FaceNet, and InsightFace) with a GFPGAN restoration attack route, under frozen shared-source manifests and photograph-disjoint query/gallery assignments, so that every comparison holds the data fixed and varies only the attack set, the utility objectives, or the query exposure.

Restoration Attacks and Incremental Linkage Beyond Direct Matching

An attacker who can choose among several recognizers already holds the strongest direct attack as a baseline; a restoration route adds capability only when it beats that baseline. Expanding the attack set with GFPGAN restoration decreases TIP-IM’s strongest tested equal error rate (EER) from 14.713% to 7.352% on VGG and from 12.479% to 0.262% on vehicle frames. Restoration that visually improves the output can make retained identity features more usable to a recognizer.

VGG Blur shows why averages mislead: its mean EER improves after restoration, but the expanded attack set never exceeds the strongest direct attack, so the attacker gains nothing. FALCO’s strongest attack remains direct in both domains. The framework therefore reports both recognizer means and additional attack gain over the strongest direct attack, separating an improvement for an average recognizer from a real increase in the strongest measured linkage capability.

Task Preservation and Multi-Objective Pareto Screening

Utility is measured with eight preservation proxies spanning perceptual quality (LPIPS, NIQE, FID), geometric semantics (head pose and gaze), and prediction consistency (expression and gender). Pareto analysis of protection and utility translates non-dominated sets into measurable screening losses: on vehicle frames, aggregate screening excludes FALCO even though it achieves the highest attribute-preservation score, at a cost of about 0.101 in the attribute proxy score and 1.79 percentage points of raw expression agreement.

Candidate selection by a single aggregate utility score can therefore silently discard the method that best preserves a specific task’s objective. The framework instead reports per-task screening losses and feasible proxy-threshold intervals, so method selection can be tied to the objectives a downstream application actually requires.

Query Exposure and Attacker-Selection Rules

Risk estimates also depend on which queries an attacker targets. Under concentrated vehicle query exposure with Zipf-style frequency weights, TIP-IM shows a true acceptance rate of 89.65% for a calibration-selected fixed attacker and 95.46% for an evaluation-selected envelope — a 5.81 percentage-point gap created purely by the selection rule. With randomized frequency ranks, the expected fixed-attacker acceptance returns to its full-pool value.

A protection claim measured on a uniform query pool does not describe vulnerability under an attacker who concentrates on the most frequently seen identities. Making this distinction explicit turns query exposure into a reported estimand rather than a hidden assumption.

By combining frozen shared-source protocols, restoration-aware attack sets, multi-objective screening, and exposure-controlled analysis — all runnable through a reproducible evaluation pipeline with per-experiment provenance certificates and 247 automated checks — LISA aims to give face de-identification method selection a decision-oriented evidence base rather than a single leaderboard number.

Key Innovation: A protection score is conditional on the attack set — so we measure the linkage capability that restoration adds beyond the strongest direct attack, and translate protection-utility trade-offs into quantifiable screening losses and exposure-dependent risk for each downstream task.