You want to set a Power goal. This is tough because people listen at various levels and headphones have a VERY wide range of efficiencies. My survey above shows to get 120dB SPL, different 32 ohm phones may need 35ma to 200mA. 120dB is insane, 110dB would be 10mA to 70mA. Or 14mA to 100mA peak.
A couple sections of 12AT7 is good for 10mA-30mA at most. If you pick sensitive phones you may be happy.
> output impedence of a cathode follower is aproximately 1/gm.
That is small-signal impedance. It is artificially lowered by negative feedback. The large-signal, Power impedance is always Rp. In this case, around 10k. So the efficiency is no better than 32/10,000 or 0.32%. (Compare with 20%-40% for a happy transformer-coupled design.) At headphone powers, this may be acceptable. But it also suggests a strained amplifier with high THD.
No, there is no magic "impedance matching" other than a transformer. Use the fattest tube in the house, run at high current.
And before all else: PLAGIARIZE!! Study all the designs you can find. The interweb makes this very much easier than it used to be. What did other do? Why did designers do this or that? How did it work out? (It may be tough to find honest criticism.) Google "12at7 headphone amp" turns up lots of examples. "12AT7+6080 tube headphone amplifier" may be not-so-sad.