Probably the biggest factors are amplification factor (mu) and to a lesser degree, interelectrode capacitance. In guitar amps, we are not necessarily liking perfection as we might in a laboratory instrument. One could make great big plates and increase the interelec capacitance, but at audio frequencies that's probably not much of a factor. Realize that you or I might well prefer the sound of a tube that is markedly inferior in terms of "pure" performance as measured on a bench full of test gear. We might like slightly more noise if it gives us greater dynamic range; or, we might like a compression effect that others would reject. At the same time, if I wanted a huge plate structure for more current-carrying capability, then it's awfully likely that my cathode would have to be bigger and then I would need a bigger filament to heat it up and then the filament current specs wouldn't be the same as 12AX7.
Manufacturers typically designed tubes to be able to run the highest frequencies or the highest currents possible or to withstand the highest plate voltages possible; within the nominal type application. No manufacturer tried to make a 12AX7 run microwave frequencies or handle 200 mils of current or to take 1 KV on the plate. The challenge for Sylvania, for example, when making a type that was brought out by RCA, for example, was to match some levels of minimum performance. Once RCA came out with the type, Sylvania probably wanted to be able to sell tubes into applications designed for that RCA type.
From the sheer number of tubes available, one might surmise that tubes were designed for particular and pretty specific applications. Color TV, for example, suggested (if not demanded) tube types that the manufacturers simply didn't think of building or think they could sell enough of to make it worth it before 1954. If you go back to the 40's, there were many fewer tube types. Over time, introduced were were damper diode types, very tough horizontal oscillator types, teeny little nuvistors to get the electrodes really, really close to improve UHF performance and very small to reduce interelec capacitance. I have 6016 rectifier tubes pretty similar to 5Y3's but they are rated for (I think) 20,000 feet and their filaments take a solid 2.5 minutes to warm up. You can put them into a tube tester and they will show as absolutely dead for about 80 seconds. You're thinking "what the hell can go wrong with a rectifier that's lighting up?"
I have little doubt that the manufacturers worked long and hard experimenting trying to make new types that they thought they could sell. Strategically, they also probably built tubes of special types to crowd other manufacturers out in specialized applications like aircraft, or, adapted existing types for specific design objectives. The improvements in terms of building the tubes more rugged, or capable of running at higher freqs, or lower voltages, came with time, experience, market perception, customer demand, and better machines to build the things. Ultimately, it was all about whether they could sell the things.