One enduring myth in regards to PSU caps is that "they don't matter because they are not part of the audio circuit", until you realize that PSU caps are an AC path to ground. ...
Flip this around, look at it, and what do you see? The plate resistor and PSU cap are in fact in parallel with the coupling cap and following volume pot ...
I see the filter cap as being
in series when you look at the path of current for a tube stage.
Electron current is emitted from the hot cathode, flows out the plate and through the plate resistor, through the filter cap to ground, up through the cathode bypass cap (if present; through cathode resistor if not), and into the cathode completing the loop. This path is relevant to
alternating current.
Direct current cannot flow through the filter or bypass caps, so it goes through the cathode resistor (establishing bias voltage) and through the decoupling resistor in the power supply (providing a convenient way to set the supply node voltage).
... Flip this around, look at it, and what do you see? The plate resistor and PSU cap are in fact in parallel with the coupling cap and following volume pot (or tone stack), at least when it comes to AC (aka audio). Therefore, the reactance and ESR of the cap will affect the gain structure. ...
Show me an example of this as a measurement of a physical circuit.
Retreating to the theoretical for a moment, a "perfect" filter cap would look substantially like 0Ω for all a.c. and like ∞Ω for d.c. If the parallel view held, we have 0Ω in parallel with the plate load, coupling cap, grid reference resistor/volume pot, which gives us 0Ω total resistance. So that seems to not hold for impacting "gain structure".
If the ∞Ω case for d.c. was instead the issue, now the imperfection of leakage (and maybe ESR, indirectly) seems to figure in as it could reduce supply voltage if we had an extreme case. Of course, the "extreme case" we're likely to see which would have a noticeable impact here is the cap failing as a short-circuit.
The reactance of the cap (~90Ω for a 22µF cap at 80Hz) matters most in determining how effectively circulating a.c. will be kept at that audio stage & filter cap, and isolated from disrupting other stages further upstream. And since that cap reactance is working against 1k-20kΩ of decoupling resistor, ESR, etc would have to be HUGE to have a discernible impact.
... Getting the "sound" of film PSU caps can be approximated by bypassing the big electro caps with a smaller film cap (like put a .1uF cap across the 20uF+ normal cap). This will take care of ESR/ESL and render the power supply effectively transparent. ...
Electrolytics
can have a rising impedance due to imperfections, which the small bypass caps seek to neutralize. However, this extra impedance isn't typically significant until you're up to the RF range, which means at worst you're looking at fighting oscillation gremlins or distortion products downward-modulated into the audio range.
Hi-fi folks usually point to noise or "harshness" as the main problems created by power supply electrolytics. I used Solens in the first scratch-built amp I made, partly due to some vague suggestions it "would sound better."
My only real reason for using them since has been "they never need replacing" and is the only benefit I'm sure I've gotten from them. It's also the only reason I'd offer others for why they might consider using film caps in the power supply.