DEV Community

celticgemini001
celticgemini001

Posted on Edited on

White balance survives an upscale, your highlights do not

The most common worry about upscaling a photo is colour. People look at the result and say it feels colder, or yellower, or flatter, and they blame the resampler for shifting the white balance. I had that intuition too, so I measured it on eight distinct real photographs, three kernels and three factors. The resampler turned out to be exactly neutral on colour, and the thing that is not neutral is the last line of your pipeline: the float-to-8-bit cast.

The short version

Question Measured answer
Does the resampler move the per-channel mean? 0.0% for bilinear, 0.00004% for bicubic and Lanczos
Does int() truncation to 8 bits move it? 0.52%–0.82% — about thirteen thousand times larger
Does the resize keep the area of already-clipped highlights? bilinear keeps 65% of the area-scaled value at the median
Does bicubic or Lanczos create clipping that was not there? yes — 468 and 1880 pixels across 8 photos
What does that clipping cost inside those pixels? 93–95% of the local gradient is gone

So the honest headline is: the filter is innocent, the rounding mode is not, and the real damage is in highlight structure, which people then misread as a colour shift.

How the colour was measured

Two numbers, not a PSNR. The first is the per-channel mean of the frame before and after the resize, as a percentage of the before value. If a resampler were introducing a cast, this is where it would appear first. The second is a gray-world gain vector, g = mean(G)/mean(channel), which is what a one-click white balance preset would apply. If the resize changed what the auto-white-balance would do, the second number catches it even where the first is quiet.

Eight distinct photographs, 1080×1620 each, centre-cropped to 512×512. Factors 2, 3 and 4. Kernels bilinear, bicubic and Lanczos. All arithmetic in float64, with the quantisation step varied deliberately — that variation is the finding.

Result 1 — the filter is exactly neutral

Three arms, same input, same kernel. The only difference is what happens at the end of the resize.

Kernel @ factor float (no quantise) 8-bit, round-to-nearest 8-bit, int() cast
BILINEAR @ 2× 0.000000% 0.0005% 0.7145%
BILINEAR @ 3× 0.000000% 0.0006% 0.5229%
BILINEAR @ 4× 0.000000% 0.0003% 0.7453%
BICUBIC @ 2× 0.000038% 0.0109% 0.8179%
LANCZOS @ 4× 0.000034% 0.0177% 0.8179%

Bilinear in float is not "small" — it is zero, to every digit I printed. That is not luck: every output pixel is a convex combination of source pixels whose weights sum to one, so the frame mean is conserved by construction. Bicubic and Lanczos land at 3.4×10⁻⁵ percent, which is floating-point residue from a kernel with negative lobes, not a cast.

Now look at the third column. An int() cast on a value like 147.9 gives 147, not 148. Every pixel loses half a code on average, and because the loss is systematic rather than random it shows up in the mean: 0.52% to 0.82% depending on the image. On a channel mean of 110 that is nearly a full code.

This is worth being precise about, because it is the source of a claim I see often. If you have ever compared an upscaled image to the original and felt it was slightly darker, you were probably right — and the resampler was not the cause. Your image library was. Pillow's convert('RGB') and most numpy casts truncate; np.rint does not.

Result 2 — highlights are where the resize actually bites

Count every output pixel whose brightest channel is at or above 254. If the resize were geometrically faithful, that count should scale by the factor squared. Divide the real count by the factor-squared baseline and you get a number that should sit near 1.0. On a synthetic solid disc it does: 0.985, 0.986 and 0.971 for bilinear, bicubic and Lanczos. On real photographs it does not.

Kernel photos with clipping min median max
BILINEAR 5 / 8 0.1221 0.6508 0.6878
BICUBIC 5 / 8 0.6325 0.7417 0.8039
LANCZOS 5 / 8 0.7493 0.8004 0.8262

Bilinear keeps 65% of the expected highlight area at the median. That is the convex combination doing exactly what it is supposed to: a flat 255 region shrinks by roughly half a source pixel on every side, because the output pixel straddling the boundary is an average of 255 and its darker neighbour. The synthetic disc misses this because it is a single large blob — with a 341-pixel disc the boundary is a tiny fraction of the area, and the ratio sits at 0.985. Real photos have highlights the size of a coin, where the boundary is most of the total.

I checked the size dependence directly with squares of increasing side:

Clipped square, side area ratio at 4×
16 px 0.8789
32 px 0.9385
64 px 0.9690
128 px 0.9844

The smaller the highlight, the more of it you lose. A 16-pixel specular dot keeps 88% of its area; the loss is entirely at its rim, and its rim is all it has.

Bicubic and Lanczos go the other way — they overshoot. On a 100/255 edge, bilinear peaks at exactly 255.0 while bicubic reaches 266.4 and Lanczos 273.2, both outside the legal range. Those overshoots are then clamped. So the same step both shrinks the clipped region at its boundary and grows it outward into pixels that were never clipped. Counting the net gives a number near 0.8 and hides both effects.

Result 3 — how much clipping is manufactured, not inherited

Separate the two by looking at the source. Take a source pixel and its 3×3 neighbourhood. If that neighbourhood already contains a value at the clip threshold, an output pixel there is inheriting clipping. If it does not, and the output is still clipped, the filter created it: there was no clipped sample anywhere near, and yet the result is pinned at 255.

Kernel created total median/photo max/photo per megapixel photos affected
BILINEAR 0 0 0 0.0 0 / 8
BICUBIC 468 28 248 6.7 6 / 8
LANCZOS 1880 214 474 51.1 8 / 8

Bilinear creates exactly zero, and that is provable rather than lucky: a convex combination cannot exceed the largest of its inputs, so it can never push a pixel past a threshold no neighbour reached. Bicubic and Lanczos have negative lobes, so they can, and they do on almost every photograph in the set. The per-megapixel figure is the one to carry away — 51 manufactured clipped pixels for every megapixel of a Lanczos 4× output, spread thinly along every bright edge.

Result 4 — what those manufactured pixels cost

This is the number I did not expect to be large. Take the unclipped float resize as the truth, clamp it the way an 8-bit pipeline would, and look only at the pixels that were clipped without a clipped neighbour. Compare the spread of the brightest channel inside that zone, truth versus delivered.

Kernel photos with such a zone gradient, truth gradient, clipped retained mean error new-clip px (median)
BICUBIC 1 / 8 4.627 0.216 4.7% 2.794 codes 248
LANCZOS 4 / 8 4.172 0.280 6.7% 1.766 codes 322

Ninety-three to ninety-five percent of the variation across those pixels is gone, because everything that overshoots 255 is pinned to 255 and becomes uniform. The code values are not wildly wrong on average — 1.8 to 2.8 codes — but the shape has been flattened, and shape is what the eye reads as detail in a highlight. A specular glint, a strip of sky through a window, the wet edge of a leaf: all of it becomes a flat patch with a hard border where the gradient used to be.

The fix people try, and what it actually does

The natural response is to pull the exposure back afterwards. Halve everything and the clipped pixels are no longer at 255, so surely the detail comes back.

It does not. I applied that halving to both the clipped result and the truth, then measured the same zone. The error halves exactly — 2.794 becomes 1.397, 1.766 becomes 0.883 — because a uniform scale multiplies both sides of a difference. The gradient is not restored; the flat patch is simply moved to a darker flat patch. Rescaling cannot recover information that the clamp already destroyed, which is the whole point of a clamp.

What does work is not clipping in the first place: doing the resize in float and applying the tone curve afterwards, so the overshoot is still available when the curve is applied. That is the same argument as the bit-depth study on HDR photographs, reached from a different direction — there the precision was missing, here the range is.

Where the white balance order does matter

Colour balance is neutral overall, but the order of the gain and the resize is not exactly free. Gain-then-resize and resize-then-gain differ by 0.0012–0.0017 codes on average outside the highlights. Inside a highlight neighbourhood the same difference is 0.167–0.224 codes, a factor of 111 to 148. The two operations are both linear, so with no clamp they commute to within 5.4×10⁻⁵ codes; the entire discrepancy is the clamp, and it is concentrated where pixel values are near 255.

The maximum difference on a real photograph is 31–32 codes, depending on kernel. With a synthetic 1.6× push on the red channel it reaches 73 codes across 3.8% of the frame. None of that is visible in the shadows, and all of it is in the highlights.

So the practical rule: if you are applying any per-channel gain — white balance, a channel mixer, a saturation matrix — do it before the resize and in float, because doing it afterwards means multiplying values that have already been clamped. For a broader look at per-channel errors that look identical but are not, the matrix and range walkthrough covers the case where the bug is in the decode rather than the resize.

How to check your own pipeline in ten minutes

You do not need this experiment. You need three numbers from your own image:

  1. Record the per-channel mean of a crop before and after the resize, in float. If it moves at all beyond float noise, the filter is not the culprit — look at where the rounding happens.
  2. Count pixels at 255 before and after, then divide the after count by the factor squared. Far below 1 means your kernel is eating highlights; far above 1 means it is manufacturing them.
  3. Compare gain-then-resize against resize-then-gain on a copy. If they differ by more than a couple of codes, you have clamping in the middle of the chain.

If the first one fails, change int() to round(). If the third one fails, move the gain earlier. That is the whole fix.

Controls, so you know which numbers to trust

Check Result
resize at 1× is a no-op 0.0
gain = 1, clamped both arms 0.0
unclipped float path, gain ∘ resize 5.4×10⁻⁵ codes
grayscale input, gray-world gains exactly 1.0 / 1.0 / 1.0
flat 128 field after Lanczos 4× mean 128.0, zero clipped pixels
synthetic solid disc, area ratio 0.985 / 0.986 / 0.971
bilinear on a 100/255 edge max = 255.0 (no overshoot, as predicted)

Two honest caveats. The bicubic row in Result 4 rests on a single photograph — one crop produced a manufactured zone large enough to measure — so read it as "same direction, small sample", not as a bicubic-specific constant; the Lanczos row has four. And the bilinear zero in Result 3 is structural, not statistical: it holds on all eight because it cannot not hold.

The duplicate-photo trap is worth mentioning because it nearly cost me this article. My first pass used twelve files and reported a 0.31% colour drift. Two things were wrong: PIL.Image.resize returns 8-bit and my arithmetic truncated, and five of the twelve files were byte-identical copies of five others under different names. Hashing the crops caught the duplicates; switching to np.rint caught the drift. Both errors pointed the same way — toward a more interesting story than the truth, which is exactly the kind that survives a careless review.

If you want to watch your own crops go through the same steps rather than trust a table, run a photo through the upscaler in a browser tab and compare the highlight count before and after. The numbers here came from full-resolution files, not from that tool, so treat it as a quick sanity check rather than a reproduction.

Top comments (0)