Scientifically comparing personal audio equipment

Q; Most comparisons of headphones or amps or DACs are so subjective. Why aren't there many scientific studies comparing personal audio equipment? How can we become better at comparing different headphones or different amps in a more scientific way?

A: Comparative analysis isn't a common or popular modality of testing audio equipment. Even the most quoted "objective" blinded A/B tests (like those for headphones) are methodologically weak, with lots of observer-dependent limitations that significantly detract from any results they manage to come up with.

Rather than trying to randomize the source like a drug trial or a physics experiment, another way to scientifically approach this might be to study the observations of different judges by looking at their inter-rater reliability as a measure of scientific "soundness".

For example, we could look at your data using Fleiss' kappa or a rank correlation coefficient (such as Spearmans, since your data will most likely be ordinal) as a measure of validity, since if our subjects were really under the influence of a placebo effect, there would be a very specific correlation (or not) between their scores, and if there was a true audio difference there would be a different specific correlation (or not) between their scores. So for example, if everything came up with poor correlation, we could make the argument that any sonic difference was attributable to the placebo effect.

Of course, the problem is that these projects are not viable because they are not really attractive to funding by the usual suspects, i.e. industry - after all, which DAC manufacturer would want to conduct an experiment which proved that their $1000 DAC was no better than a $100 DAC, or even the opposite, why would the manufacturer of a $100 DAC want to take a chance that the study would prove that the $1000 DAC was much better?