PCI Compliant and Still Broken: Why Payment Trust Is a System Property

Walk into any mid-sized retailer and audit the payment stack component by component. The terminal runs a certified EMV kernel. The P2PE implementation is validated. Keys live in an HSM with documented rotation. The gateway holds current PCI DSS attestation. The fraud model was retrained last quarter and its false-positive rate looks respectable.

Every component passes. And the merchant is still eating duplicate charges on Saturday nights, still declining a recognisable slice of good customers, and still spending twenty hours a month reconciling a settlement file against a ledger that disagrees with it.

Nothing in the audit is wrong. The audit is just answering a different question than the one that matters.

The idea worth borrowing

In the November 2025 issue of IEEE Computer, Marco Vieira argues that the trustworthy-AI conversation has fixated on the model — its fairness, robustness, explainability — while ignoring that models never run alone. They sit inside data pipelines, infrastructure, human workflows and governance regimes, and trustworthiness is a property of that whole assembly, evaluated continuously, not a certificate stapled to one part of it.

His cases are instructive precisely because none of them is a model failure. Google Health’s diabetic retinopathy screener was accurate in the lab and failed in clinics because nurses couldn’t reliably capture images that met its input standard. A Waymo vehicle detected traffic cones correctly and then deadlocked because the handoff between perception and route planning couldn’t resolve what to do about them. The Epic sepsis model was miscalibrated to local populations and buried clinicians in alerts. Amazon’s recruiting tool stripped gendered terms and still encoded a biased hiring history.

The pattern: the component was fine, the seam was not.

Swap “AI model” for “payment component” and the argument lands on our industry without modification. Payments has spent two decades building certification regimes for parts — EMV Level 1/2/3, PCI DSS and PCI PIN, P2PE, scheme mandates — and comparatively little on assurance for the assembled flow. That gap is where the incidents live.

The payment stack, in layers

Vieira’s model nests the AI/ML component inside data, infrastructure, human-system and governance layers. The payment equivalent, with what actually goes wrong at each:

Component layer. Terminal firmware, EMV kernel, tokenisation vault, decision engine, gateway SDK. This is the layer we already certify well, and it’s the layer that fails least often.

Data layer. BIN tables, network token lifecycle state, card-on-file updater feeds, FX rates, MCC assignments, risk features, ledger records. Payment data rots at different speeds in different places, and stale data produces confidently wrong behaviour: routing to the wrong acquirer, mispriced interchange, a fraud model scoring against a feature distribution that shifted three months ago, a token that desynced from the issuer and now silently fails a recurring charge.

Infrastructure layer. Connectivity, store-and-forward behaviour, retry and timeout semantics, idempotency guarantees, HSM availability, orchestration health checks, batch windows. Most duplicate-charge incidents are born here, in the space between one system’s timeout and another’s commit.

Human-system layer. Cashiers who learn that tapping twice “makes it work.” Staff who bypass PIN because the queue is long. Ops analysts tightening a risk threshold on Friday and forgetting on Monday. Support agents who can’t tell a soft decline from a hard one and tell the customer to try again — six times. Customers who abandon at a 3DS challenge they don’t recognise as legitimate.

Governance layer. PCI DSS, PSD2/SCA, scheme rules and monitoring programmes, AML/KYC obligations, dispute rights, model-risk expectations for credit decisioning, and — increasingly for BNPL and underwriting — algorithmic accountability regimes. Governance failures rarely announce themselves as outages. They show up as fines, remediation programmes and consent orders.

Where payments actually breaks: five emergent patterns

Vieira’s sharpest point is that undesired behaviour emerges from interaction rather than residing in any single element. Each of these patterns requires every component to be working correctly.

1. The retry storm. A gateway call times out at the client before the authorisation commits at the issuer. Client-side retry logic fires. Idempotency keys are scoped per-attempt rather than per-intent, so the retry reads as a fresh authorisation. Now the velocity features feeding the fraud model spike, and the model — correctly, given its inputs — declines the third attempt as card testing. The customer sees two pending holds and a decline. Terminal: fine. Gateway: fine. Fraud model: fine. System: hostile.

2. The self-confirming risk model. A risk engine declines a cohort at elevated rates — thin-file customers, a prepaid BIN range, a geography. Those customers transact elsewhere. The next training set contains fewer successful transactions from that cohort, which the model reads as confirmation. This is precisely the feedback dynamic Vieira describes in LinkedIn’s recruiter tool, and it’s structurally identical in underwriting and fraud. The model is not drifting away from its objective; it’s optimising against data its own decisions shaped.

3. Dunning against issuer limits. Subscription billing retries failed charges on a schedule. The schedule was tuned for recovery rate in isolation. In aggregate it pushes the merchant’s retry volume into territory that issuers throttle and schemes monitor, which depresses the authorisation rate across all traffic including first-attempt charges from healthy customers. A local optimisation degrades a global one, and the ledger only shows you the recovered revenue, not the suppressed baseline.

4. Offline authorisation and the trust window. A terminal loses connectivity and store-and-forwards under floor limits. Goods leave the store. The batch uploads hours later and a portion declines. The system behaved exactly as configured. The loss is real, the accountability is nowhere, and no component logged an error.

5. Orchestration gray failure. A multi-PSP routing layer health-checks its providers. One provider degrades — latency climbs, decline reasons get vaguer — without failing any health check. Traffic keeps routing there because the check measures liveness, not decision quality. Same root shape as the Waymo case: correct perception, broken downstream coordination.

A failure-point matrix for payments

Vieira summarises his cases in a layer-by-incident table. That format transfers well, and it’s worth building your own from your last four postmortems. The exercise is more useful than it looks, because the pattern it exposes is always the same: the component row is clean and the seam rows are not.

If your matrix looks like this, buying a better fraud model will not help you. The governance row failing across the board is the honest finding: nobody owns the flow.

What continuous assurance looks like in a payment stack

Vieira points toward the assurance community’s posture of “trust but verify, continuously,” and toward continuous evidence collection rather than point-in-time certification. Translated into payments operations, that means instrumenting the seams:

• Authorisation rate segmented, not aggregate. By issuer, BIN range, card-on-file status, channel, first-attempt versus retry. A stable blended rate routinely conceals a collapsing segment.
• Decline reason-code drift as a monitored signal. A shift in the mix of soft versus hard declines is an early warning that something upstream changed. Almost nobody alerts on this.
• Reconciliation break rate as a reliability SLO. Authorisation, capture, settlement and ledger are four records of one event. Treat the disagreement rate as a production metric with a threshold, not as month-end busywork.
• Duplicate-charge rate per million. The cleanest single indicator of whether your idempotency contract actually holds end to end.
• Cohort false-positive review. Sample declined transactions that later succeeded elsewhere, broken out by segment. This is the only practical way to catch pattern 2 before it compounds.
• Fault injection on the flow, not the box. Vieira’s own research background is fault injection, and it’s underused here. Kill connectivity mid-authorisation. Delay the settlement file. Return a malformed token response. Let the terminal battery die between auth and capture. Certification tells you the component meets spec; injection tells you what the assembly does when a neighbour misbehaves.
• Shadow-mode deployment for risk changes. Score in parallel, don’t act, compare against realised outcomes, then promote.

None of that replaces PCI DSS or EMV certification. It sits on top, answering the question certification was never designed to answer: does the assembled system deserve to be trusted this week?

The uncomfortable version

The reason the model-centric view persists in AI is the same reason the component-centric view persists in payments: components have owners, vendors, certificates and clear boundaries. Seams have none of those things. It is genuinely easier to procure a better fraud engine than to establish that one team owns the customer’s path from tap to ledger and is measured on it.

Vieira is candid that his framework is a call to consolidate rather than a finished methodology — no metrics, no prioritisation, illustrative cases from secondary sources. In payments we’re better positioned than that, because we already have the telemetry. Every one of the measures above is derivable from data most processors and merchants already retain.

What’s usually missing isn’t instrumentation. It’s someone whose job it is to look at the whole thing.

Reference: M. Vieira, “Why We Should Trust Systems, Not Just Their AI/ML Components,” Computer, vol. 58, no. 11, pp. 84–94, Nov. 2025, doi: 10.1109/MC.2025.3604335.