Saltar a contenido

Load and infrastructure-fault report (QA simulation)

Branch test/load-and-faults. Harness and exact commands: tools/load-test/README.md. Date of the run: 2026-11-16.

Question asked: under load, and when the infrastructure fails, can the gateway lose a payment or leave a customer charged without a sale?

Short answer. The invariant checker found 0 violations in every scenario: no payment stuck, no double reversal, no customer charged for a sale the POS did not record unless the compensation was still running or had been escalated with its alert, trace and ticket numbers unique and gap-free. That held through 11 hard kills of the gateway, a database taken offline for 10, 30 and 60 seconds, 3 kills of its sessions, three 25 s table locks, 15 kinds of Prisma failure, a ramp to 50 concurrent checkouts and a 22 minute soak. A negative control proves the checker can fail (section 3). Not everything ended clean: 5 customers (all from one injected Prisma failure, F2) were still charged at the end and were escalated to a person with the COMPENSATION_FAILED alert exactly 30 minutes later, four of them after the cashier had been told the annulment was accepted. The campaign also found two real weaknesses: a duplicate-check query of the callback inbox that reads and locks the whole table (F1, High: at 50 concurrent checkouts on a 37k-row inbox the gateway as shipped did 9.4 payments/s with a BINs p95 of 2.1 s, and 21.7 payments/s with a BINs p95 of 54 ms after changing only the JDBC parameter type) and the F2 behavior itself (Medium). Details and ranked fixes in sections 5 and 6.

Totals: 20,237 payments (the whole database): 8 scenario runs, 2 warm-up runs and the A/B; 11 gateway kills, 3 database outages, 3 session kills, 3 table locks, 15 Prisma fault kinds. Counted at the POS side: 264 trxid refused and 148 trxpago left without an answer while the database or the gateway was down on purpose (plus 60 bogus "no answer" that were the harness's own port exhaustion, section 4.3).

1. Environment and limits (read before trusting any number)

Item Value
Machine one Windows 10 IoT LTSC dev box, 16 logical CPUs, 14.5 GB RAM; gateway, emulator, SQL Server 2022 (RTM 16.0.1000.6) and the load generator all share it
Software JDK 25.0.2, the gateway jar of the main checkout (main at 626af78) started by sim.ps1, profile local, one gateway node
Gateway config defaults of the repo: Hikari pool 20 / 5 s connection timeout, pos.wait.deadline 45 s, caller-read-timeout 59 s, status-query lead 15 s, compensation max-attempts 10, poll-deadline 30 m, pos-polling-window 10 m. No -FastWaits.
Database GatewayPrismaQR_SIM, freshly created by sim.ps1 up (8 MB files, 64 MB autogrow), read-committed without snapshot isolation
Wallet / Prisma prisma-emulator, not real Prisma. It never retries a callback, never loses a charge by itself, answers instantly, and keeps every request in a copy-on-write list
Network loopback only, no external traffic
Load closed loop: one worker thread per checkout (a new payment on a checkout supersedes the previous one, AC-32), 100 seeded checkouts on one merchant, one terminal each

What these numbers are not: capacity planning for production (shared CPU, emulator instead of Prisma, a database that is small and cold), nor a test of a second live node (out of scope, see 7). They are good for ranking weaknesses and for proving the invariants, not for absolute latency.

Harness artifacts found and fixed while building it (so nobody re-discovers them): the emulated wallet always picks the latest intention of a checkout, so the harness waits for the previous intention to be closed at the emulator before the next customer arrives (otherwise it pays a stale one); one HTTP socket per control call exhausted the Windows ephemeral ports at 50 checkouts (WinError 10048, which looked like "POS got no answer", 48 bogus cases in the first ramp, now using persistent connections for the emulator calls and called out below); and a pay-after-cancel scenario that cancels before the complete intention produces a charge with no trace/ticket numbers (F5).

2. Method

  • Drive (lt_driver.py): POST /api/trxid, POST /api/trxpago (blocking, like the POS), GET /api/trxes/{id}, POST /api/trxanulacion (twice, to hit AC-23). The wallet is played through the emulator scenarios. Default mix: 78 % APPROVED, 7 % REJECTED, 4 % DUPLICATE_NOTIFICATION, 3 % MISSING_NOTIFICATION (resolved by the gateway's own status query), 3 % LATE_AFTER_CANCEL (wallet pays with the plan numbers, the cashier cancels before the notification arrives, INV-01), 2 % CASHIER_CANCEL, 3 % ANNUL_AFTER_APPROVED. A POS that gets no answer polls the consulta, then cancels (as a cashier would).
  • Measure: BINs callback and every outbound Prisma call from wire_log (the gateway's own duration_ms), complete intention as the time from the BINs row in inbound_notification to complete_snapshot.sent_at, trxpago latency at the client, JVM heap and GC (jstat), OS threads and handles (wmic), DB sessions and blocked requests (the gateway exposes no actuator, so sys.dm_exec_sessions stands in for Hikari usage), row growth of ten tables.
  • Prove (lt_checker.py), after every scenario, from three sources that must agree: the database (schema from V1__baseline.sql), the emulator (/emulator/intentions/{id}: what the wallet really charged and reversed; /emulator/requests: forced reversals it accepted) and the client log (what the POS was told).

3. The invariant checker, and the proof that it can fail

Code What must hold
INV-STUCK nothing waits for the wallet past its deadline + reaper grace + sweep + hand-over; no COMPENSATION_PENDING without an active task
INV-CHARGE every wallet charge still out is a sale the POS heard as approved, or has a reversal in flight or completed, or is COMPENSATION_FAILED with its alert line in the log. An approved payment the POS never heard stays pending until the 10 min reconciliation window, then must be annulled
INV-SALE approved <=> sale: told APPROVED only for an APPROVED payment with a live charge; a recorded sale is never reversed unless the cashier annulled it; an APPROVED payment always has a live charge
INV-DOUBLE at most one money-returning reversal per charge in the DB, and at most one forced annulment/refund accepted by Prisma per intention
AC-26 per terminal: trace and ticket numbers unique, contiguous from 1, equal to the terminal counters
INV-BACKLOG compensation tasks drain; nothing overdue without a lease; COMPENSATION_FAILED has an alert
INV-INBOX no callback left unprocessed (INV-03)
INV-DEADLINE / INV-PAYSTATE RUNBOOK section 3 step 5 query; impossible state combinations

Negative control (python run.py selftest): the harness corrupts rows of the throwaway database one at a time and the checker must fail. All four expectations were met, and the same payment checks PASS after the restore:

Injected corruption Detected as
an APPROVED sale the POS recorded flipped to REJECTED INV-SALE ("told APPROVED but the payment is REJECTED") and INV-CHARGE ("CUSTOMER CHARGED, NO SALE and nothing owed")
a terminal counter one ahead of the numbers issued AC-26 ("a number was consumed without being used")
a REJECTED payment put back in INTENT_CREATED, deadline 10 minutes ago INV-STUCK

4. Results by scenario

All invariant verdicts below are the checker's, scoped to the payments of the run (payment ids given), with the global checks (AC-26, compensation backlog, inbox) over the whole database. "Pending" means money in flight or inside the reconciliation window at the time of the check; the final global strict check is in section 4.7.

4.1 Scenario 3, gateway crash mid-payment: PASS (2 runs, 11 kills)

Setup. 20 checkouts, default mix, load running. The gateway is killed with Stop-Process -Force by exact PID (gateway-ctl.ps1 first checks the PID against state.json, image java, the sim's jar command line and the start time), left down 4 s, then restarted the way sim.ps1 starts it (same jar, same environment, same database; takes 7-10 s to answer /api/ping). Targets are chosen by polling the database for a payment in the wanted state and killing at once.

Run Payments Kills (state of the trigger payment) POS told Verdict
crash (ids 376-1776) 1,448 6: INTENT_CREATED (720), COMPLETE_SENT (741, 1557), APPROVED (1078, 1214, see below), COMPENSATION_PENDING (1120) APPROVED 1,185, DENIED 216, no payment 47 (trxid refused while down) PASS, 0 violations, 0 pending
crash-window (ids 1777-2561) 804 5: four with the payment APPROVED and the POS not yet answered (1980, 2205, 2364, 2526), one COMPENSATION_PENDING (2444) APPROVED 660, DENIED 125, no payment 19 PASS, 0 violations, 0 pending

"After the notification, before the POS answer" needed a trick. In the first run the kills aimed at that window landed 180 ms too late: the window between the approval commit and the answer is about 1 ms. For the second run a test-only trigger in the throwaway database (trg_lt_widen_answer_window, dropped at the end) delays the write of pos_approval_intent_at by 2 s, so the gateway is killed while the payment is APPROVED and pos_approval_intent_at/pos_told_approved_at are both NULL. Result for all four: trxpago got a connection reset, the POS polled the consulta, the restarted gateway wrote the intent and answered APPROVED, the told marker was set, the sale stands (e.g. payment 1980: intent 14:37:52, told 14:37:54, kill at 14:37:39). Nothing was lost and nothing was annulled.

Charged but never notified. In the first run payments 741, 1120 and 1557 (MISSING_NOTIFICATION: the wallet charged, the gateway never got the notification) were in flight at the kill. The POS was denied (after its consulta polling) and each ended COMPENSATED: its cancel task found the intention paid (the intention was paid: its charge has its own reversal) and a forced annulment completed (reversal ANNULMENT COMPLETED, triggered by COMPENSATION).

Numbers. 42 + 25 payments ANNULLED (cashier or late charge), 117 + 67 COMPENSATED, 0 COMPENSATION_FAILED, all compensation tasks DONE (310 and 446). Drain after each run: waiting 0, tasks 0, reversals 0, unprocessed callbacks 0. trxpago that got no answer: 62 and 68 (all handled by the consulta/cancel path of the POS model). A hard kill never produced a 5xx or a corrupted row.

4.2 Scenario 4, database outage: PASS (ids 2562-4607, 2,232 payments)

Setup. 20 checkouts. ALTER DATABASE GatewayPrismaQR_SIM SET OFFLINE WITH ROLLBACK IMMEDIATE for 10 s, 30 s and 60 s (45 s apart), then KILL of the gateway's _SIM sessions (filtered by database_id) three times (18, 14 and 14 sessions). SQL Server and the other databases were never touched. _SIM was ONLINE after each step (checked) and at the end.

What the POS saw Count
trxid refused (500, generic message) while the database was down 186
trxpago answered APPROVED 1,723 (told)
trxpago answered denied 323
trxpago with no answer 18

Recovery: the first trxid after the database came back was served 3.2 s after the 10 s outage and 0.4 s after the 30 s and 60 s ones (Hikari revalidates its connections).

What the gateway did while it could not persist (log scan of the run): INV02_UNPERSISTED_DENIAL 14 (the documented exception to INV-02: the POS was denied with no intent stored), POS_API_UNEXPECTED_FAILURE 225, INBOX_WORK_FAILED 609, COMPENSATION_JOB_FAILED 52, WIRE_LOG_NOT_RECORDED 47, ORPHAN_WAIT_SWEEP_FAILED 3, all CannotGetJdbcConnectionException or CannotCreateTransaction, all of them while the database was off, none after. Of the 14 unpersisted denials, 11 were customers the emulated wallet had really charged: they are all MISSING_NOTIFICATION flows (the wallet pays and the notification never comes), whose wait expired while the database was down, so the POS was denied with no intent stored. After the database came back the gateway found each charge (status query / reconciliation of the unpersisted denials) and annulled it: all 11 ended COMPENSATED with an ANNULMENT COMPLETED. The other 3 were never charged. Payments 2656, 2898, 2951, 3033, 3038, 3079, 3099, 3122, 3201, 3209, 3212 are the 11.

Verdict. PASS: 0 violations, 0 pending, final states ANNULLED 62, APPROVED 1,661, CANCELLED 18, COMPENSATED 141, REJECTED 164; every compensation task DONE (719), drain to zero. No corruption, AC-26 intact. Note the cost: 41 scenarios of the emulated wallet failed and trxid returned errors for the whole outage, i.e. the gateway cannot take payments without its database (by design: nothing is approved that is not persisted).

4.3 Scenario 1, load ramp 1 -> 10 -> 25 -> 50 checkouts: invariants PASS, latency FAILS AC-07 at 50 (ids 4608-8566)

Setup. Four stages of 120 s, think time 0-0.5 s, default mix, real wait deadlines. 4,029 payments. Verdict: PASS, 0 violations, 0 pending. States: ANNULLED 126, APPROVED 3,308, CANCELLED 48, COMPENSATED 212, REJECTED 265.

Checkouts Payments / s trxpago ms p50 / p95 / p99 / max (wallet answers at once) BINs callback (wire IN) ms p50 / p95 / p99 / max complete intention after BINs ms p95 / p99 DB sessions (pool 20)
1 0.3 767 / 3,287 / 6,328 / 6,328 (37 payments, JIT warm-up) 13 / 26 / 132 / 132 2,016 / 5,104 14
10 5.8 516 / 850 / 1,782 / 3,142 40 / 206 / 526 / 1,377 396 / 742 20
25 14.3 280 / 771 / 927 / 1,790 29 / 182 / 287 / 548 266 / 427 20
50 13.2 (saturated) 1,046 / 1,995 / 2,273 / 7,080 301 / 541 / 901 / 1,328 660 / 1,228 20

Reading it. * AC-07 (BINs callback 204 in under 200 ms p95): against the SPEC's design target of 5 payments/s the relevant stage is 10 checkouts (5.8/s): p95 206 ms, borderline. 25 checkouts (14.3/s) 182 ms, 50 checkouts (13.2/s, saturated) 541 ms. At about 5 payments/s on a young database (crash and outage runs, 1-9k inbound rows) it was 14-30 ms and in the 22 minute soak (3.6/s, 27-35k rows) 75-117 ms. The numbers are not a clean function of checkouts or table size; the A/B of section 4.8 and F1 show the cause. * Complete intention p99 < 3 s: met in every stage once warm (427-1,228 ms; the 5.1 s of the 1-checkout stage is its first 35 payments after the previous run). Always far below the 27 s of v27. * Throughput: the closed loop saturates at about 13-14 payments/s (50 checkouts do not beat 25), roughly 2.7 times the design target of 5/s, on a machine that also runs the load generator, the emulator and SQL Server. * Where it degrades: the Hikari pool (20) is fully in use from 10 checkouts on (sessions plateau at 20); at 50 checkouts up to 16 requests were blocked behind locks at the same time (sys.dm_exec_requests.blocking_session_id). * Error rate: trxid 0 errors; trxpago 5xx 0; 48 "no answer" at 50 checkouts are the harness's own ephemeral-port exhaustion (the request never left the client in 2-3 ms), fixed afterwards, not a gateway failure. * Gateway JVM during the ramp: 69-84 threads, working set 373-432 MB, no growth across the four stages. (The heap column of this run was lost to a locale bug in my jstat parser, fixed afterwards: in the later runs the heap stayed between 51 and 104 MB with no full GC.)

4.4 Scenario 6, Prisma degradation under load: PASS, with the F2 cases still open at check time (ids 8567-10166, 1,600 payments)

Setup. 20 checkouts, a mix richer in compensation (55 % APPROVED, 14 % ANNUL_AFTER_APPROVED, 10 % MISSING_NOTIFICATION, 9 % LATE_AFTER_CANCEL...), one injected failure every 25 s through POST /emulator/faults:

Operation Modes injected (count) Outcome
CREATE_INTENTION STATUS 503 (6), RESET_AFTER_REQUEST (4), DELAY 2.5 s (10) POS denied, nothing charged; 4 notifications for intentions the gateway never stored ended UNCORRELATED with the alert (the wallet never got a complete intention, so nobody was charged)
COMPLETE_INTENTION STATUS 503 (6), DELAY 4 s (8), TIMEOUT 12 s (4), RESET_AFTER_REQUEST (4) the 10 unknown outcomes (COMPLETE_UNKNOWN) were compensated; the delays were absorbed
QUERY_PAYMENT STATUS 503 (8) the status query retried within the lead window
CANCEL_INTENTION STATUS 503 (6), TIMEOUT 12 s (2) retried by the job, all DONE
FORCED_ANNULMENT STATUS 503 (5), RESET_AFTER_REQUEST (3), TIMEOUT 12 s (2) RESET and TIMEOUT: resolved by querying (done at Prisma); 503: see F2
QUERY_ANNULMENT, FORCED_REFUND STATUS 503 (6, 3) retried

Verdict at the end of the drain: 0 violations, 6 pending: five customers still charged (payments 9565, 9569, 9574, 9575, 9576, see F2) and one COMPENSATION_PENDING, all inside the gateway's 30 minute poll deadline. When that deadline passed the five ended COMPENSATION_FAILED with their alert (section 4.7). The other numbers: 1,600 payments, POS told APPROVED 1,291 / DENIED 309, APPROVED flow success 98.4 %, trxpago p50/p95/p99 274/1,018/2,761 ms for prompt approvals, 162 payments approved 30-35 s after a missing notification (the gateway's status query at deadline minus 15 s: expected). Compensation drained (1,784 tasks DONE), 218 ANNULLED, 239 COMPENSATED, 0 double reversals.

4.5 Scenario 5, lock contention and pool exhaustion: PASS (2,045 payments)

Setup. 25 checkouts; three 25 s locks taken from a separate session inside a transaction that is then rolled back: (a) UPDATE terminal SET trace_seq = trace_seq (every terminal row, which every complete intention needs), (b) TABLOCKX on payment, (c) TABLOCKX on compensation_task and inbound_notification.

  • The pool reached its limit every time: 20 of 20 sessions, up to 20 requests blocked, no connection left for new work.
  • Observed effect: trxid was refused 12 times with the generic 500 during the locks (pool timeout, 5 s); no trxpago was lost or answered with a 5xx; BINs callbacks held up to 25 s (the lock time; p50 234 ms because the whole run is slow, see F1); INBOX_WORK_FAILED 203, COMPENSATION_JOB_FAILED 16, WIRE_LOG_NOT_RECORDED 27 while it lasted. No INV02_UNPERSISTED_DENIAL (the locks were shorter than the deadline).
  • Recovery: all workers resumed by themselves when the lock was released, drain to zero in seconds.
  • Verdict: PASS, 0 violations, 0 pending, 1 payment left ID_ISSUED (a trxid whose trxpago never came: not a payment).

4.6 Scenario 2, soak: PASS, no leak (ids 12676-17446, 4,771 payments in 22 minutes)

Setup. 15 checkouts, think time 1-4 s, default mix, about 3.6 payments/s for 22 minutes (a little under the design target of 5/s), no faults. Verdict: PASS, 0 violations, 0 pending; ANNULLED 127, APPROVED 4,031, COMPENSATED 247, REJECTED 366; every payment ended terminal; trxpago 0 errors, 0 no-answer, APPROVED-flow success 100 %; trxpago p50/p95/p99/max 511 / 1,040 / 1,520 / 2,025 ms for prompt approvals.

Resource (sampled every 30 s) First Last Min - max Reading
Gateway heap used (MB) 79 54 46 - 104 saw-tooth, no trend; old generation 45-52 MB flat; 0 full GCs
Gateway OS threads / Java threads 80 / 78 77 / 75 77-81 / 75-79 flat
Gateway handles 808 756 756 - 808 flat
Gateway working set (MB) 428 400 358 - 428 flat
DB sessions of the gateway (Hikari 20) 20 15 12 - 20 never above the pool
Emulator working set (MB) 363 412 358 - 412 grows, but that is the emulator keeping every request (harness, not gateway)

Table growth over the 22 minutes (rows, first -> last): payment 12,690 -> 17,446, intention 12,611 -> 17,367, wire_log 54,349 -> 74,355 (4.2 per payment), inbound_notification 25,550 -> 35,172 (2.0 per payment), complete_snapshot_method 24,456 -> 33,774, bins_snapshot 12,229 -> 16,888, wallet_charge 11,337 -> 15,639, compensation_task 2,137 -> 2,655, reversal 1,158 -> 1,429. Linear in the number of payments, no purge anywhere (F6).

Does latency follow the table size at a constant load? python trend.py buckets the BINs callback (wire IN) per 3 minutes while inbound_notification grew from 26.9k to 35.2k rows:

Minute 0 3 6 9 12 15 18 21
BINs p50 / p95 / p99 (ms) 28 / 109 / 564 28 / 92 / 226 28 / 75 / 183 28 / 92 / 137 29 / 82 / 114 31 / 108 / 177 31 / 103 / 132 32 / 117 / 215
inbound rows at the end of the bucket 26.9k 28.1k 29.4k 30.9k 32.3k 33.5k 34.8k 35.2k

No trend (p95 75-117 ms throughout, p50 28 -> 32 ms): at this load the scan of F1 costs a roughly constant ~20-30 ms and p95 stays under AC-07's 200 ms even at 35k rows. The slowdown seen in the ramp is therefore driven by concurrency (callbacks queueing behind the table lock), not by growth at constant load. See F1.

4.7 Final global check (strict): PASS, 5 payments escalated to a person

python run.py final --strict over every payment of the database since the last sim.ps1 up: 20,237 payments (18,146 wallet charges, 1,578 completed reversals, 2,989 compensation tasks DONE, 50 terminals with trace/ticket numbers), all log lines of every gateway process (the original and the restarts), after every reconciliation window had elapsed:

  • 0 violations, 0 pending. Final payment states: APPROVED 16,567, REJECTED 1,412, COMPENSATED 1,410, ANNULLED 768, CANCELLED 78, COMPENSATION_FAILED 1, ID_ISSUED 1 (a trxid never used). No payment in any waiting state; no callback unprocessed; every invariant of section 3 PASS, AC-26 PASS on all 50 terminals.
  • 5 payments escalated (not clean, not a violation of the spec's escape hatch): 9565, 9569, 9574, 9575, 9576 (F2). Each is COMPENSATION_FAILED with its COMPENSATION_FAILED payment=<id> alert in the gateway log (logged at 15:36:12Z, 30 minutes after the send). The customer is still charged in all five until a person acts. The checker treats "charged and COMPENSATION_FAILED with the alert logged" as escalated, lists it, and fails a COMPENSATION_FAILED that has no alert.
  • The gateway's own alerts over the whole campaign, as they should be: INV02_UNPERSISTED_DENIAL 14 (all from the 60 s database outage), COMPENSATION_FAILED 5, UNCORRELATED 7 (create calls whose answer was cut by an injected fault), INBOX_WORK_FAILED 851, COMPENSATION_JOB_FAILED 73, WIRE_LOG_NOT_RECORDED 149, POS_API_UNEXPECTED_FAILURE 245 (all while the database was off, a lock was held or the pool was blocked), ORPHAN_WAIT_SWEEP_FAILED 4, INBOX_ROW_FAILED 1.

4.8 A/B for F1: same load, the JDBC parameter type being the only change (ids 17447-20237)

50 checkouts, 90 s, default mix, think 0-0.5 s, back to back on the same database. A is the gateway as shipped (inbox 35,172 rows at the start). B is the same jar restarted with --spring.datasource.url=...;sendStringParametersAsUnicode=false on the command line (configuration only, no product code; inbox already 36,885 rows, so the table is larger in B). Both runs: invariants PASS, 0 violations, 0 pending.

A: as shipped B: sendStringParametersAsUnicode=false
Payments in 90 s 842 (9.4/s) 1,949 (21.7/s)
BINs callback (wire IN) p50 / p95 / p99 / max (ms) 1,239 / 2,115 / 2,708 / 3,184 7 / 54 / 13,540 (see below) / 13,633
Complete intention after BINs p50 / p95 / p99 (ms) 1,142 / 2,142 / 2,819 17 / 132 / 1,570
trxpago (wallet answers at once) p50 / p95 / p99 (ms) 3,699 / 5,254 / 6,272 271 / 2,899 / 5,037
trxid p50 / p95 (ms) 198 / 707 4 / 268
5xx to the POS 0 2

Run B's p99 of 13.5 s and its 5xx come from a single stall in its first seconds after the restart (the pool showed total=20, active=20, waiting=114, one deadlock retry, 21 BINs callbacks held 13.5 s together, WIRE_LOG_NOT_RECORDED 75), the same shape as the cold-start stall of F3, not from the inbox scan. Without it B's BINs p95 is 54 ms.

5. Findings

F1 (High, performance, no money lost): the callback inbox duplicate check scans and locks the whole table

JdbcInboundNotifications.FIND_EARLIER runs for every Prisma callback (BINs, payment, annulment, refund) before the insert: SELECT TOP 1 id FROM inbound_notification WITH (UPDLOCK, HOLDLOCK) WHERE body_hash = :hash AND type = :type AND ISNULL(prisma_intention_id,-1) = .... The JDBC driver sends String parameters as NVARCHAR(4000) (sendStringParametersAsUnicode defaults to true and the datasource URL does not change it), but body_hash is CHAR(64) and type is VARCHAR(20). SQL Server therefore converts the column (CONVERT_IMPLICIT, the plan carries PlanAffectingConvert), the predicate is not sargable, and the first delivery of a callback (the normal case: no earlier row) reads the entire table under a serializable lock, which escalates to an OBJECT X lock on the table. Every callback then queues behind the previous one.

Evidence: * The gateway's own plan cache after the load: 70 executions averaging 14,975 logical reads, max 30,794, at 21,396 rows (statement text with @P0 nvarchar(4000), PlanAffectingConvert). * Reproduction, tools/load-test/repro-inbox-scan.sql: with the types the driver sends 30,689 logical reads and an OBJECT X lock; with a CHAR(64) parameter 3 logical reads and one KEY RangeS-U lock. * Wait probe under 50 checkouts (python run.py contention): 1,384 blocked-statement samples were LCK_M_RS_U on this exact statement and 1,363 had it as the blocker. * A/B (section 4.8), the causal proof: same gateway, same load, same table, only the parameter type changed. As shipped: 9.4 payments/s, BINs p95 2,115 ms, trxpago p50 3.7 s. With CHAR parameters: 21.7 payments/s, BINs p95 54 ms, trxpago p50 271 ms. The fix is worth more than a factor of 2 in throughput and 40 in callback latency at saturation.

What it is not: a steady-state problem at the design load. At a constant 3.6 payments/s the 22 minute soak showed BINs p95 of 75-117 ms with no trend while the table grew from 26.9k to 35.2k rows (section 4.6), and at about 5/s on a young table 14-30 ms. The scan costs roughly 20 ms of serialized table lock per callback at these sizes; what matters is what happens when arrivals exceed 1 / (scan time): the lock becomes a queue and latency jumps (stage 50 of the ramp, run A). That ceiling falls as the table grows (the scan is linear, 30k pages at 21k rows), there is no retention anywhere (F6), and a burst (a branch opening, a Prisma retry storm, a recovery after a database outage when every pending callback lands at once) can reach it with no warning at all.

Recommended fix (not done here, product code is out of scope): lead with the typed parameter, a CAST(:hash AS CHAR(64)) (and CAST(:type AS VARCHAR(20))) or a Types.CHAR / Types.VARCHAR bind in that statement, plus a regression test that asserts an index seek (logical reads, or no PlanAffectingConvert) for it; audit the other statements that compare a String parameter with a CHAR/VARCHAR column. The URL flag sendStringParametersAsUnicode=false is what the A/B used to prove the cause, but it also downgrades every NVARCHAR parameter (denial_reason, merchant names, raw_body), so use it only after checking the Spanish text columns. Add a retention job for inbound_notification, wire_log and bins_snapshot.

F2 (Medium): one transient 503 on the first forced annulment ends in a manual case 30 minutes later

When POST /forcedannulments is answered 503 (the emulator's STATUS fault without executeAnyway, i.e. Prisma did not run it) the gateway classifies it "may have started" (ForcedReversalOutcome: only 4xx and failures before a byte was sent count as "never received"), marks the reversal IN_PROGRESS and never sends it again. It then asks GET /annulments?intention_id=: the answer is 404 every 10 s (read as "maybe not processed yet", question 17d) until the 30 minute poll-deadline, then the task and the reversal give up, the log gets COMPENSATION_FAILED payment=... and the customer is still charged.

Who is affected (faults run, the five injected 503s hit five payments):

Payment Flow What the POS / cashier was told End state
9565 LATE_AFTER_CANCEL (wallet charged after the cashier cancelled) "no sale" payment COMPENSATION_FAILED, charge out
9569, 9574, 9575, 9576 ANNUL_AFTER_APPROVED (cashier annuls an approved sale) AUTORIZACION_DE_ANULACION_APROBADA, "the money is being returned" payment still APPROVED, charge out, reversal IN_PROGRESS, task COMPENSATION_FAILED

Evidence: each intention stays APPROVED at the emulator (money out); each payment has one 503 FORCED_ANNULMENT row followed by about 50 QUERY_ANNULMENT rows, all 404 but the injected 503s; reversal.accepted_at is NULL; the five alerts COMPENSATION_FAILED payment=... Prisma still has not finished the annulment sent at 15:06:12-15:06:14Z and the gateway will not send it again were logged at 15:36:12Z, exactly 30 minutes after the send. This is the documented fail-closed rule (no idempotency at Prisma, never risk a double reversal), the alert fired for all five and the RUNBOOK says what a person does, so it is not an invariant violation by the checker's definition (they are counted as escalated, section 4.7). The cost is the point: 5 of 5 compensations that met a single transient 503 needed a person, 30 minutes late, and for the four cashier cases the cashier believed the customer had been refunded (a later trxanulacion would hear 409, per the RUNBOOK).

Options: ask Prisma (question 17d) whether a 404 from GET /annulments is authoritative for "never received"; if it is, allow one resend after a 404 observed a few minutes after a 5xx; otherwise page at first 5xx plus 5 minutes instead of 30, and give operators a supported "I asked Prisma, resend" command.

F3 (Medium, FIXED): the first burst after a start stalled the gateway for 9-15 s

Symptom. After a restart, 50 checkouts starting at once got no answer for 9-15 s (first trxpago answered 8.9-15.5 s after the burst began, nothing before 8 s). Seen before in the first validation run and in run B of section 4.8.

Reproduction (python run.py coldstart --checkouts 50 --secs 25: kill the gateway, start it, wait for /api/ping, fire the burst at once; --no-restart is the warm control, --probe samples sys.dm_exec_requests and sys.dm_tran_locks every 0.25 s). On the gateway as shipped, 11 cold starts: 8 stalled (first answer after 8.98, 8.99, 13.3, 14.4, 14.8, 14.85, 14.9 and 15.5 s), 3 did not (1.8, 2.5 and 2.7 s). The same burst on a warm gateway: first answer 0.32 s, BINs callback p99 137 ms, never a stall.

Root cause, with evidence. Not the JVM, the pool or the plans: it is deadlock cycles in the inbox duplicate check (JdbcInboundNotifications.FIND_EARLIER), broken by SQL Server only every 5 seconds. * Thread dumps during the stall: no thread burning CPU, the JIT compiler threads idle ("No compile task"), everything parked. * The wait probe: for the whole stall all 20 pooled connections were in LCK_M_RS_U on the statement of the duplicate check, 20 of 20 at every sample, and blocking_session_id formed a cycle (sessions 54 -> 59 -> 61 -> 54 in the captured sample). The lock listing shows every session holding a PAGE IU and a KEY RangeS-U (or waiting for one) on ix_inbound_notification_body_hash. * The gateway log: DeadlockRetry ... deadlock victim lines 5 seconds apart (23:25:41, 23:25:46 in one run, +4 s and +9 s after the burst began), and Connection is not available, request timed out after 5000ms (total=20, active=20, waiting=95) while it lasted: three cycles in a row, one victim per 5 s monitor pass, is the 13-15 s. * Why only after a start: the statement takes UPDLOCK, HOLDLOCK, a serializable range lock on the gap around the new hash. A warm gateway answers each callback in 5 ms, so the 50 callbacks of the burst barely overlap. A cold one (class loading, JIT) releases them together, they land in the same gaps of a sparse index, each takes the range lock for the check and then needs the same gap to insert (RangeI-N), and the cycles form. It needs concurrency of the callbacks, not a start: F1's burst and a recovery after an outage can produce it too. It is far less likely on a production inbox of millions of rows (the gaps are tiny), which is why the severity is Medium, not High.

Fix (JdbcInboundNotifications.FIND_EARLIER). The duplicate check now holds an application lock on the body hash (sp_getapplock, Exclusive, owner Transaction, 10 s timeout) and reads the index at plain read committed, with the same index hint and CAST. Two simultaneous deliveries of the same body still queue (the second sees the first one's row, AC-10), but deliveries of different bodies never wait for each other, so there is no range to share and no cycle. Two details found while doing it, both commented in the code: sp_getapplock takes a variable, not an expression, and the driver's implicit-transaction mode does not open a transaction on an EXEC, so a SELECT from the table precedes it (otherwise it fails with -999). Tests: InboundNotificationsIT.f3_simultaneousCallbacksOfDifferentBodiesDoNotDeadlockEachOther (4 rounds of 40 simultaneous callbacks, all under 3 s: a cycle costs 5 s or more) next to the existing simultaneous-duplicates test, and InboxDuplicateCheckPlanIT still asserts the index seek.

Before and after (same machine, same database growth, 50 checkouts at once right after the gateway answers /api/ping):

Before (11 cold starts) After (8 cold starts)
First trxpago answered after 1.8 to 15.5 s; 8 of 11 runs above 8 s 1.96 to 3.56 s, none above 4 s
BINs callback (wire IN) max up to 15.9 s (9.1 to 15.9 s in most stalled runs; the first one stalled the complete intention instead, 14.3 s) 306 to 806 ms
BINs callback p99 up to 14.98 s 163 to 549 ms
Deadlock victims / pool exhaustion in the burst yes (total=20, active=20, waiting=95) none

Invariants: every run PASS, and python run.py final over the 15,032 payments of the database: 0 violations, 0 pending, 0 escalated. The remaining 2-3.5 s for the first payment is the JVM warming up, as for any process; no warm-up is needed before readiness (RUNBOOK 2b).

F4 (Info): the pool is the first thing to saturate

Hikari (20, 5 s) is fully used from 10 checkouts, and when SQL Server blocks, all 20 connections are blocked within seconds (section 4.5). The gateway then answers trxid with a 500 and keeps trxpago alive, which is the right trade-off, and it recovers by itself. The pool is not the cause of the F1 queue (those requests were blocked inside SQL Server), but a pool of 20 gives no headroom if F1 is not fixed.

F5 (Info, question for Prisma): a charge with no trace/ticket numbers cannot be reversed

Scenario LATE_PAYMENT_AFTER_CANCEL (the cashier cancels before the complete intention is sent, then the emulated wallet pays the cancelled intention) produced 12 payments in the first smoke test (database since recreated) whose charge carries no method, establishment, terminal, trace or ticket. The gateway fails them on the first attempt (the charge cannot be named to Prisma ...), logs COMPENSATION_FAILED and the customer's money stays out for a person. That is the designed fail-closed answer and the alert fired for every one, but it is only acceptable if a real wallet cannot pay an intention before it has seen the plans: please confirm with Prisma. It is not in the default mix (LATE_AFTER_CANCEL in the mix is the realistic variant and passes).

F6 (Info): data growth

Per payment, in these runs: wire_log 4.3 rows, inbound_notification 2.0, complete_snapshot_method 2.0, bins_snapshot 1.0, wallet_charge 0.9, compensation_task 0.15. Nothing in the repo purges them. See F1.

  1. No money was lost and none was left charged without a sale in any scenario, including 11 hard kills, a 60 s database outage and Prisma faults, and the checker is proven able to fail. The structure that makes this true (state in the database, the intent persisted before the denial, the late approval annulled, one reversal per charge, never resend without asking) works.
  2. High, fix before volume: F1. A typed bind or CAST in one statement, a test on the plan, and retention. The A/B shows the gain (21.7 vs 9.4 payments/s, BINs p95 54 vs 2,115 ms at 50 checkouts). At the design load AC-07 holds today (3.6/s: p95 under 120 ms; 5.8/s: 206 ms, borderline); it does not hold under a burst, and ACCEPTANCE.md marks AC-07 as met without that qualifier.
  3. Medium: F2. Decide the 404 semantics with Prisma, shorten the time to the alert, add an operator "resend" command; at least put the 30 minutes in the RUNBOOK next to the COMPENSATION_FAILED alert.
  4. Low / ask: F5 (confirm the wallet cannot pay before it has the plans), F6 (retention policy).
  5. Operational: wire INV02_UNPERSISTED_DENIAL, COMPENSATION_FAILED, INBOX_WORK_FAILED and COMPENSATION_JOB_FAILED to alerts (all appeared exactly where expected); consider read-committed snapshot isolation for the database (readers waited on writers in every blocked-request sample).

7. What this campaign did not cover

  • A second live gateway node (AC-25 as written: "another node finishes the payment"): out of scope by user decision, because production runs a single node. What was proven is that a killed and restarted node finishes everything from the database (section 4.1); lease takeover by a live second node was not exercised.
  • Real Prisma, real network failures (loopback never drops packets), TLS, real card traffic, the Almacor POS, RouterQR.
  • A production-size database (the largest table here has about 25k rows, the real ones will have millions; F1 says that matters).
  • Disk-full, clock jumps, and memory pressure on the host.