GHSA-7mwm-ghpf-9vrm Pillow: Double free in ImagingScaleAffine on calloc failure via Image.transform

• PyPI Pillow: < 13.0.0 → fixed in 13.0.0

``` ============================================================================ Double free in ImagingScaleAffine on calloc failure via Image.transform Location: src/libImaging/Geometry.c:866 in ImagingScaleAffine() Commit: a707c6b21c313880d0771e54965d142dd42358b6 ============================================================================ ``` There is a double free in Pillow's `Image.transform()` that crashes the process when an oversized transform runs out of memory. Instead of only raising `MemoryError`, Pillow frees the output image's memory once while handling the error and then again when the Python object is cleaned up. Anyone who can choose the output size of `Image.transform()` can trigger it, but only on a worker with a memory limit (a common hardening for image workers) or a 32-bit process. The result is a crash of the worker, plus corruption of its heap that has not been shown to be controllable. The bug is in `ImagingScaleAffine()` in `src/libImaging/Geometry.c` (line 866). When its `calloc()` for the per-column lookup table of `4 × width` bytes fails, it calls `ImagingDelete(imOut)`. When the call comes from `Image.transform()`, `imOut` is the `Imaging` struct owned by the live Python `ImagingCore` output object. `_dealloc()` in `src/_imaging.c` later runs `ImagingDelete()` on the freed struct again, writing its mutex and refcount in freed memory and freeing the pixel blocks a second time. The path is reached with `Transform.EXTENT`, or `Transform.AFFINE` with a pure scale/translate matrix, under the default NEAREST resampling. A very wide, 1-row output size makes the lookup table the allocation that fails while the image itself still fits. On 64-bit Linux with default overcommit, that requires an address-space limit such as `ulimit -v`, `uwsgi --limit-as` or `systemd LimitAS=`. `resize()`, `thumbnail()` and `rotate()` are not affected. The attached reproducer runs on a stock build with a 1 GB limit: `transform((300000000, 1), EXTENT)` raises and catches `MemoryError`, then the process dies with SIGSEGV in `free()` called from `_dealloc()`. A memory-checking build reports the same path as a heap use-after-free. The faulty line was inherited from PIL 1.1.7 in the original fork commit, so every Pillow release since 1.0 is affected. A patch is attached. Details `ImagingScaleAffine()` frees an image it does not own. On the one error path where its `calloc()` for the per-column lookup table fails, it calls `ImagingDelete(imOut)`. When reached through `Image.transform()`, `imOut` is the `Imaging` struct held by a live Python `ImagingCore` object; that object's `_dealloc()` later calls `ImagingDelete()` on the already-freed struct and frees its pixel blocks a second time. Shortest demonstration on the stock build (the address-space limit stands in for `ulimit -v` / `uwsgi --limit-as` / `systemd LimitAS=` on a worker): ```python import resource, gc from PIL import Image resource.setrlimit(resource.RLIMIT_AS, (1_000_000_000,) * 2) src = Image.new("L", (8, 8)) try: src.transform((300_000_000, 1), Image.Transform.EXTENT, (0, 0, 8, 8)) except MemoryError: pass # expected, and the process should carry on gc.collect() # instead: SIGSEGV in _dealloc -> ImagingDelete -> free() ``` The 300 MB output image allocates; `calloc(300000000, 4)` (1.2 GB) does not. `MemoryError` is raised as intended, but the interpreter then dies with SIGSEGV, gdb showing `free()` ← `memory_return_block`/`ImagingDestroyArray` (`src/libImaging/Storage.c:396,408`) ← `ImagingDelete` (`Storage.c:271`) ← `_dealloc` (`src/_imaging.c:188`). An AddressSanitizer build of the same sources (with `max_allocation_size_mb` used to make the `calloc` fail instead of an rlimit) reports `heap-use-after-free` in `ImagingDelete src/libImaging/Storage.c:257` called from `_dealloc src/_imaging.c:188`, on a 128-byte region "freed by" `ImagingDelete` ← `ImagingScaleAffine src/libImaging/Geometry.c:866` ← `ImagingTransformAffine Geometry.c:1043` ← `ImagingTransform Geometry.c:1135` ← `_transform src/_imaging.c:2153`, "previously allocated by" `ImagingNewPrologueSubtype Storage.c:53` ← … ← `_new src/_imaging.c:756`. Code path `Image.transform()` creates the output image as an ordinary Python-owned image and then transforms *into* it: ```python src/PIL/Image.py:3008…3021 im = new(self.mode, size, fillcolor) … im.__transformer( (0, 0, *size), self, method, data, resample, fillcolor is None ) ``` `__transformer()` rewrites `Transform.EXTENT` into an axis-aligned affine matrix (`src/PIL/Image.py:3040-3046`: `data = (xs, 0, x0, 0, ys, y0)`, `method = Transform.AFFINE`) and calls `self.im.transform(box, image.im, method, data, resample, fill)` (`Image.py:3117`). `resample` defaults to `Resampling.NEAREST` (`Image.py:2936`). In C, `_transform()` passes its *own* image as the destination: ```c /* src/_imaging.c:2153 */ imOut = ImagingTransform( self->image, imagep->image, method, x0, y0, x1, y1, a, filter, fill ); ``` `ImagingTransformAffine()` takes the scaling fast path because the filter is NEAREST and the matrix has no shear: ```c /* src/libImaging/Geometry.c:1035-1043 */ if (filterid || imIn->type == IMAGING_TYPE_I16) { return ImagingGenericTransform(…); } if (a[1] == 0 && a[3] == 0) { /* Scaling */ return ImagingScaleAffine(imOut, imIn, x0, y0, x1, y1, a, fill); } ``` and `ImagingScaleAffine()` allocates a table of `imOut->xsize` ints: ```c /* src/libImaging/Geometry.c:863-868 */ /* malloc check ok, uses calloc for overflow */ xintab = (int *)calloc(imOut->xsize, sizeof(int)); if (!xintab) { ImagingDelete(imOut); /* destroy`, and `free(im)`. `self->image` in the `ImagingCore` object still points at the freed struct. `_transform()` returns `NULL`, `Image.transform()` propagates `MemoryError`, the half-built output `Image` loses its last reference, and `_dealloc()` (`src/_imaging.c:188`) calls `ImagingDelete(imagep->image)` again: it locks `im->mutex` and decrements `im->refcount` inside freed memory, then walks `im->blocks` (already freed, typically reused by then) and frees every entry a second time. In the observed stock-build runs the freed `blocks` array had been overwritten, so `free()` receives a mangled pointer and the process segfaults; with other heap states it is a plain double free of a 300 MB block. Every other error return in `Geometry.c` (`ImagingError_ModeError`, `ImagingError_Mismatch`, `"bad filter number"`, `"bad transform method"`) leaves `imOut` alone; line 866 is the only one that deletes it. The delete is only correct for the other caller, `_resize()` (`src/_imaging.c:2026-2030`), where `imOut` is a fresh `ImagingNewDirty()` temporary that nothing else references. The delete predates Pillow's git history: it came in with the PIL 1.1.7 import, `9a640e315715` ("Forking PIL", 2010-07-30), so every release since Pillow 1.0 is affected. The output width is what decides whether the path is hit: the output image costs `width × height × pixelsize` bytes, the table an additional `4 × width`. A very wide, 1-row output (300 000 000 × 1 "L": 300 MB image, 1.2 GB table) makes the table the allocation that fails. `ImagingNewPrologueSubtype()` caps `xsize` at `INT_MAX/4 - 1`, so the table is at most ~2 GB. Impact Applications that let a user choose the output size of `Image.transform()` — `Transform.EXTENT` (crop-and-scale), `Transform.AFFINE` with a pure scale/translate matrix, or the `ImageTransform.ExtentTransform`/`AffineTransform` helpers, all with the default NEAREST resampling — can have an allocation failure turned into a use-af…