GHSA-jh4v-gfqj-7rhx RabbitMQ Java client has frame-level OOM: Math.min(maxInboundMessageBodySize, 0) defeats frame size enforcement

• Maven com.rabbitmq:amqp-client: < 5.34.0 → fixed in 5.34.0

Vulnerability In `AMQConnection.java` (line 435-436), after `Connection.Tune` negotiation, the frame-max limit is set via: ```java _frameHandler.setFrameMax( Math.min(this.maxInboundMessageBodySize, frameMax)); ``` When `frameMax = 0` (meaning "unlimited" per AMQP spec), `Math.min(67108864, 0) = 0`. This value is then passed to `Utils.framePayloadLimit(0)` which returns `Integer.MAX_VALUE` (line 77-79 of Utils.java): ```java static int framePayloadLimit(int frameMax) { if (frameMax <= 0) { return Integer.MAX_VALUE; } // ... } ``` This completely defeats the `maxInboundMessageBodySize` protection (default 64MB) at the frame level. Attack Scenario A malicious AMQP server (or MITM) sends `Connection.Tune` with `frameMax=0`: 1. Client defaults: `requestedFrameMax = 0` (`ConnectionFactory.DEFAULT_FRAME_MAX`, line 82) 2. `negotiatedMaxValue(0, 0)` = `Math.max(0, 0)` = 0 (line 673-676) 3. `Math.min(maxInboundMessageBodySize, 0)` = 0 — 64MB cap defeated 4. `framePayloadLimit(0)` = `Integer.MAX_VALUE` — no frame size enforcement 5. Attacker sends a single frame with `frameSize = 0x1FFFFFFF` (~500MB) 6. `Frame.readFrom()` (line 135) executes `new byte[frameSize]` — OOM crash The frame does not need to be a body frame — method frames, header frames, or heartbeat frames with a crafted size field all trigger the allocation before any content-level check fires. Root Cause The AMQP spec uses `frameMax=0` to mean "unlimited", but `Math.min` treats it as the integer value zero. The intent of line 435-436 was to take the smaller of the two limits, but when one limit uses 0-means-unlimited semantics, `Math.min` always selects the zero, disabling the other limit. Impact • Default configuration is vulnerable: Both `requestedFrameMax` (client) and legitimate servers' `frameMax` in Tune may be 0 • Single-frame OOM: One malicious frame triggers up to ~2GB allocation (`Integer.MAX_VALUE` bytes) • Bypasses existing protection: `maxInboundMessageBodySize` (introduced to cap allocations at 64MB) is entirely defeated at the frame level • Different from ValueReader OOM: This is a frame-layer allocation in `Frame.readFrom()`, not a value-layer allocation in `ValueReader.readBytes()` Affected Code • `AMQConnection.java:435-436` — `Math.min` with 0-means-unlimited • `Utils.java:77-79` — `framePayloadLimit(0)` returns `Integer.MAX_VALUE` • `Frame.java:135` — `new byte[frameSize]` allocation site • `ConnectionFactory.java:82` — `DEFAULT_FRAME_MAX = 0` Suggested Fix ```java int effectiveFrameMax = (frameMax == 0) ? this.maxInboundMessageBodySize : Math.min(this.maxInboundMessageBodySize, frameMax); _frameHandler.setFrameMax(effectiveFrameMax); ``` This treats `frameMax=0` as "use maxInboundMessageBodySize as the cap" instead of "zero".