Blind listening tests are not the end all be all

[Tullman]

I have read people on here saying that if sound differences cannot be heard in a blind listening test that in fact there is no difference. One example is tweeks. Another might be mastering or cd-r. I think that the best way to determine sound is over a long period of time and familiarity with the playback equipment and room.

[Joseph]

Originally posted by Tullman
I have read people on here saying that if sound differences cannot be heard in a blind listening test that in fact there is no difference. One example is tweeks. Another might be mastering or cd-r. I think that the best way to determine sound is over a long period of time and familiarity with the playback equipment and room.

I agree totally with you Tullman.

[Dave]

Originally posted by Tullman
I have read people on here saying that if sound differences cannot be heard in a blind listening test that in fact there is no difference. One example is tweeks. Another might be mastering or cd-r. I think that the best way to determine sound is over a long period of time and familiarity with the playback equipment and room.
Good theory Tullman and I agree totally!

[Richard Feirstein]

Indivisual reports are very interesting. But even a lot of people reporting the same thing does not make it true. When lots of people report an effect, it is time to see if some fudge factor is at play since it is well established that the human brain is very easy to trick. False memory and self delution are very very powerful. And the brain's ability to remember sound is very very very very poor, thus making very quick A/B comparisons central to any comparitive expeience. Taking several min to change an interconnect does not cut it.

One way to confirm a reported audible effect is to see if you can still report that the effect is there when you do not see or know if that interconnect, marker pen, speaker spike, power conditioner, etc., is being used or not. If you only can "hear" the effect when you see what interconnect, spike, marker, or power condition is being used it belongs to the relm of self delution not reality. Now one can argue about how you go about doing a proper double blind study, but it is possible to design one and execute it over and over again. If a large sampe of listeners can hear the difference with statistical significane it add support to the truth of the reported effect. Since interconnects repeatedly fail to demonstrate that a large group of listeners can hear any difference with statistical significance it is just possible that what many people report is not real. Since much of what the CEO of Monster Wire and other venders of high end interconnects discuss as science is often total bull, it adds support that they are selling the Brooklyn Bridge and not something of real utility.

But still, some people I respect keep reporting these effects as something they truly hear; perhaps they are right, perhaps they are just tweaking themselves. Heck, sometimes I can't tell the difference between Coke and Pepsi. (Consumer reports and even Howard Stern have found that most people that think they know their Beer can't really identify one from the other when they don't see the can). This effect we are really talking about is what makes advertising so very very very effective. Brand identification and loyalty are all linked to that good old brain trick of self delution. Our economy runs on it; Monster Cable would be nowhere without it.

I suggest that you not throw out the double blind study as a useful tool that can help sort out truth from something else. I don't know of any good alternative. No well designed double blind study has been able to confirm the existance of ESP either, and there is fame and money to be found to the person who can prove it's existance. (Audio differences should be a lot easier to demonstrate, and I know of no such successful demonstraton). At the wire company where I worked one summer the tech guys found no merit to the claims some venders were making for interconnects made from the product produced by these tech guys on their behalf. It was deemed an inside joke, a fool and his money, etc.

[Dob]

If you want to take all the science out of it and make it totally subjective, I don't necessarily have an issue with that.

Consider drug effectiveness trials, which use a control group that takes placebos. Sometimes, those subjects taking placebos exhibit improvement. If one of those subjects INSISTS that the sugar pill they took made them better, how can I argue with that? For them, the sugar pill worked, even though science states that they have no medicinal benefit.

By the same token, if you insist that a tweak makes a sound difference, I can't refute that because I can't get inside your head and hear what you hear. And vice versa. Unfortunately, that doesn't leave much room for intelligent discussion, because there is no scientific rigor. Anything goes. If I claim that I don't listen to vinyl because I can hear the degradation caused by stylus wear after only one play, and I don't have to prove it with a blind test, and I don't have to provide any science to back it up, you are forced to accept that I can hear it.

Sort of makes the whole discussion pointless. Or, at least, lowers it to same level as arguing which Van Halen album is best or who's the best rock drummer. Everyone states his opinion, and in the end, everyone agrees to disagree.

[Scott Wheeler]

The problem with double blind tests is that they really find what is audible in double blind tests. Once you set up a double blind tests you have introduced at least three new variables. The envirement inwhich audiophiles normaly listen is gone. The way audiophiles listen to music has been changed. I don't rapidly switch back and forth between sources or amps while I listen to music, do any of you? And the state of mind has been affected. The fact that you are testing people changes their state of mind and adds stress. Now there are always the other added variables if you use them including switching devices, repetition of listening tests. That combined with the stress can cause listener fatigue.

Double blind tests seem to reveal tonal variations quite easily and harmonic distorion when there is a ton of it. But how do you know all the other variables aren't masking a plethora of other sonic characteristics? Until you can remove all those other variables all you can really say about double blind testing is that it reveals what is audible in double blind testing.

Now if you add to that equation a group of testers who already believe their tests will give a negative result. You are bound to get just that, a negative result.

Yes people easily fool themselves but I don't think double blind testing in this case really helps us figure out what is placebo and what really works if it is not changing the frequency response or adding a gross amount of harmonic distortion.

[Dob]

Originally posted by Richard Feirstein
If you only can "hear" the effect when you see what interconnect, spike, marker, or power condition is being used it belongs to the relm of self delution not reality.

To play the devil's advocate for a moment, the point is that the knowledge that the effect is being used IS PART OF THE EFFECT.

Example - suppose you were a big Milli Vanilli fan, until you found out that the two guys weren't the actual singers. Now you can't stand to listen to the albums. Why? The albums are the same as before, but your new knowledge has changed your perspective.

Another example - suppose you go to the ballet and are absolutely astounded at the leaping ability of the dancers. It is the best ballet you have ever seen! The next day, you read in the paper that the dancers were really suspended by wires. Does that change your opinion of the performance? Does this knowledge ruin it for you?

It works the other way as well. If I am convinced that my latest tweak will make my system sound better, and it does, why should I care about whether or not my knowledge of the tweak has any efffect on my perception? All I care about is that it sounds better, and it does.

[Dob]

Originally posted by Scott Wheeler
I don't rapidly switch back and forth between sources or amps while I listen to music, do any of you?

That's how I do all my mastering. I'm always comparing what is with what I want to try. It's also how I do all my sound comparisions. If your point is that this type of listening is somehow inferior to sitting back and becoming "emotionally involved", I forcefully disagree.

When I cook, I pay attention to details and work at making the meal the best it can be. When I eat, I relax and enjoy it. Don't confuse cooking with eating.

[Scott Wheeler]

Dob
Actually my point was only that it is not how most of us listen to music. If you listen that way on a regular basis than it won't be a variable for you. When you take the listener out of his or her envirement you introduce a variable or many variables. Our listening habbits are a part of our listening envirement so to speak.

[Dave]

Originally posted by Richard Feirstein
And the brain's ability to remember sound is very very very very poor, thus making very quick A/B comparisons central to any comparitive expeience. Taking several min to change an interconnect does not cut it.
Speak for themselves! Besides having at least 2 audiophiles listening at the same time it helps. ;)

[Richard Feirstein]

Dob: "Consider drug effectiveness trials, which use a control group that takes placebos. Sometimes, those subjects taking placebos exhibit improvement. If one of those subjects INSISTS that the sugar pill they took made them better, how can I argue with that? For them, the sugar pill worked, even though science states that they have no medicinal benefit. "

I believe this expresses a basic misunderstanding of the placebo effect. The placebo effect is real. Double blind studies have shown repeatedly that it is statistically significant. What "science" is after for a drug or procedure that is under study is whether or not a particular drug or procedure is more effective than the placebo effect. That is what you should want here and no less!

Thus a new interconnect may enhance the listening experience of the listener who knows he or she has a new interconnect. What one should also want to know is if the effect is merely a brain trick or is the audio actually different. Only double blind studies properly controlled can establish that. The other comments posted here concerning potential problems with the test environment are all variables that can be tested and controlled, not merely speculated about. Since it is understood that the brain's ability to remember subtle changes in sound is very poor the procedures utilized by those reporting here of their experince, while interesting, does not at all consider the fact that their reports may be a total self experience totally removed from objective reality. In the absence of cold objective facts at least offer a rational based upon good physics and not the phony science panned out by Monster Wire, etc., who use the right jargon poorly applied and abused with no actual basis in the laws of physics. Apparently the false science approach common up until the mid to late 1800's is still part of our human culture by those without any dedication to the scientific method.

These individual reports are merely the start and should not be the end of our search for truth and beauty.

[krabapple]

Originally posted by Tullman
I have read people on here saying that if sound differences cannot be heard in a blind listening test that in fact there is no difference. One example is tweeks. Another might be mastering or cd-r. I think that the best way to determine sound is over a long period of time and familiarity with the playback equipment and room.

Sorry, you can't just 'wish away' expectation bias. If we could, then medical trials (for example) would become much simpler and less expensive. No amount of familiarity with the sound or room is going to adequately control for that bias. We are *inclined* -- perhaps even evolutionarily so -- to perceive 'difference' even when it's not there. (Level matching is the other critical factor -- the human ear is pretty sensitive to small differences in loudness -- < 1 dB -- , within a normal range.).

Blind testing can be done over as long a period of time, with as much listening as you like and on whatever playback equipment suits you. The main drawback is that either blind or double blind testing require two people to perform, or else some sort of switching mechanism. A guy names Arny Kruger has developed software to allow you to do ABX comparison ( a form of double blind) at home, without the need for a 'helper'.

[krabapple]

Originally posted by Scott Wheeler
Dob
Actually my point was only that it is not how most of us listen to music. If you listen that way on a regular basis than it won't be a variable for you. When you take the listener out of his or her envirement you introduce a variable or many variables. Our listening habbits are a part of our listening envirement so to speak.

No one says you have to take the listener out of his or her normal environment, to do comparative listening.

(still, many of us *do* sometimes audition stuff in stores and salons...surely not our
usual listening environment. By your logic, perceptions during these sessions are not worth much. )

(I tend to agree, but for a different reason.)

[Richard Feirstein]

The best double blind test would be in a well designed environment (like your own listening room), where the listener gets to make the switch and gets to listen as long as he or she wants to the sound sample, over and over again, with the listner in control. Let the listner do this as long as he or she likes as often has he or she likes until a choice is made. When this has been done, over and over and over and over again, no statistical difference has ever been shown for an interconnect or speaker wire choice, ever. Not for speaker spikes, not for green markers, never. One must carefully control for level and avoid all visual and audio clues. Even DTS has failed to come forward and participate in a study against Dolby Digital, although Dolby Digital has claimed that it beat DTS in tests it ran (sure, we can trust its objectivity, right?) If any interconnect company had the data supporting their claims we would have seen the results published years ago. There is no supporting data; none. Just empty claims.

[Dob]

Originally posted by Richard Feirstein
I believe this expresses a basic misunderstanding of the placebo effect. The placebo effect is real. Double blind studies have shown repeatedly that it is statistically significant. What "science" is after for a drug or procedure that is under study is whether or not a particular drug or procedure is more effective than the placebo effect.

Hmmm, did my post contradict your statement? I don't see how.

What one should also want to know is if the effect is merely a brain trick or is the audio actually different.

Why? Each one is equally real to the listener. He may care, if he has an inquisitive mind, whether or not the audio is "actually different", but in the end result it only matters if the listener WANTS it to matter.

What bugs me is when those who wholeheartedly embrace the "power of suggestion" try to have it both ways - i.e., what they hear COULD be scientifically documented if science was as advanced as their golden ears.

[Scott Wheeler]

" No one says you have to take the listener out of his or her normal envirement to do comparitive listening"

I agree completely. The problem is that is exactly what most double blind listening tests have done. Yes it can be done right with just one variable in the equation. But none of the tests I have ever read about which found no differences between amplifiers and cables and the like have ever been done without taking the listeners out of their envirement. Not to mention numerous other variables that were introduced.

I also agree that auditions in audio salons aren't worth that much unless you are, say, getting a taste of what a component sounds like. I will not consider any purchases without living with the component for a while. Not always practical but.....

[krabapple]

Originally posted by Dob


Why? Each one is equally real to the listener. He may care, if he has an inquisitive mind, whether or not the audio is "actually different", but in the end result it only matters if the listener WANTS it to matter.

I agree with this. Whether the difference is real only matters when one is making *claims* to that effect ;> I'm used to writing stuff like, "they sounded different to me but I haven't verified that in a controlled comparison", rather than make a solid claim of audible 'difference' when there's reason to believe there might not be any,
and I haven' controlled for bias.

What bugs me is when those who wholeheartedly embrace the "power of suggestion" try to have it both ways - i.e., what they hear COULD be scientifically documented if science was as advanced as their golden ears.

What is curious to me is that, if in fact cables, for example, don't tend to sound different under controlled conditions, this shoudl be a *liberating* fact -- it means you don't *have* to spend $100/ft to get the 'best sounding' interconnect.

Yet some proponents of high-end -- and usually, quite high-priced -- electronics
seem to prefer battling that idea than actually testing it. The lengths to which certain audiophiles will go to resist the idea of controlled testing are amazing, as any time spent on audiophile forums will show. Typical reactions are: 1) I don't care what a blind test says, I hear a difference 2) listening under controlled conditions it too *different* from 'normal' listening 3) science can be wrong so how do we know it's not wrong in this case? 4) the equipment wasn't good enough to reveal the difference; 5) the people who took the blind test didn't have good enough ears. As each objection is addressed, the arguments are cycled, so that you eventually go through all of them again and again. The nearest thing I've seen to it are arguments between scientists and creationists.

[krabapple]

Originally posted by Scott Wheeler
" No one says you have to take the listener out of his or her normal envirement to do comparitive listening"

I agree completely. The problem is that is exactly what most double blind listening tests have done. Yes it can be done right with just one variable in the equation. But none of the tests I have ever read about which found no differences between amplifiers and cables and the like have ever been done without taking the listeners out of their envirement. Not to mention numerous other variables that were introduced.

I also agree that auditions in audio salons aren't worth that much unless you are, say, getting a taste of what a component sounds like. I will not consider any purchases without living with the component for a while. Not always practical but.....


There is a rather infamous blind test performed by Tom Nousaine , at the home of an audio salon owner (now deceased) who insisted he could hear differences between a Yamaha amp and a high-end one. Alas, he failed to do this under blind condition using conditions and equipment *he* specified, in *his* house. Also, Arny Kruger has been collected data for some time now on people who have used his ABX comparator to do blind tests in their own homes. You can look up either on a google groups; these are two examples of many. Several of the regulars on RAHE have reported blind test results done in their own homes.

But this is besides the point. It is a *fact* that sighted, nonlevelmatch comparison is prone to bias. THerefore, even if you don't believe the blind tests you've read about were rigorous enough, you *at least* have to acknowledge that many of the claimed differences being touted in audiophilia ARE *questionable* and unverified. Surely you cannot fault the blind testers for lack of rigor, and then 'blindly' accept the claims of The Absolute Sound or other reviewers who practice *no* rigor at all.

[Tullman]

Ok Richard, I will take a blind listening test. Replace my magnan Vi interconnect with zip cord. I know I will hear the difference. How? I know what to listen for. The magnan's will be more transparent have better timbre and detail. If I blind tested for speaker spikes on my Theil speakers I would listen to the bass. The tighter bass sound would be with the spikes in place.

When I audition equipment at home I am not under any sort of delusions just because I can see the equipment, or I know it is there. Either it sounds better (musical) or it doesn't. Heck, I would love it if I could get better sound from $10.00 interconnects. I could sell my magnans and spend the money on some lps or cds.

Blind listening tests are impractical when upgrading ones audio system at home. Extended listening is the way to go. Sometimes when I'm listening/auditioning something for the first time, I think wow this sounds better, but after living with the equipment for awhile I think something is missing. Yeah this component is more detailed but the mids just aren't as musical as the component I already have. In a blind listening test I might have said I like the detailed component better, but in the long run would have grown tired of the sound.

[Scott Wheeler]

The ABX comparitor in and of itself unfortunately intoruduces three unwanted variables. The ABX machine itself. the mode of listening with rapid back and forth comparisons, and the stress of a test conditions. You may say these are trivial and have no effect. I say no scientist worth his or her salt would ever make such assumptions. Variables are variables. More sublime variables have corrupted many a scientific investigation before.

I think to dismiss long term listening comparisons inwhich level matching can be statistically eliminated by listening at many levels with each component in the comparison by use of double blind test which do nothing to recreate the listening experiences claimed is a mistake. Thislong term multi level listening is sort of what I do and ultimately what others often do over extended listening sessions.

As for claims being questionable and unverified, yes speaking strickly on a scientific level I would not claim the differences I think I hear in different components is a scientific fact. But I still hold the position that the only thing that has been proven by double blind testing to be a scientific fact is that certain components that audiophiles claim to sound different don't reveal themselves as sounding different in some double blind tests. I say some because there have been double blind tests inwhich people have heard differences in these components. To ignore that data is to take a side. That would be very unscientific. Oh and yes I most certainly can fault anyone who conducts a nonrigorous test in the name of scientific investigation. Those people should be publicly flogged. Some of the tests conducted by members of the AES that I have heard about, if what I heard was true, were irresponsible and should be exposed for what they were. As for comparisons to this debate and creationism, the comparison is absurd. No one is claiming devine intervention here. Do you really think there are no good scientists who think they hear diferences between these components such as amps and cables?

[Dob]

Originally posted by Tullman
Sometimes when I'm listening/auditioning something for the first time, I think wow this sounds better, but after living with the equipment for awhile I think something is missing. Yeah this component is more detailed but the mids just aren't as musical as the component I already have. In a blind listening test I might have said I like the detailed component better, but in the long run would have grown tired of the sound.

That's a valid observation but it misses the point, which is whether or not there is a DIFFERENCE. Double blind testing of equipment has nothing to do with value judgements. Many times I have heard differences between A and B, but I can't decide which one is "better". And even if I do choose which one is "better", that is a totally subjective evaluation that someone else is perfectly free to disagree with. EQ is a good example - what sounds bright to me may sound dull to you, and vice versa.

If I need extended listening just to hear a difference, let alone try to decide if it's an improvement, that piece of equipment/tweak better be a) cheap, b) small, and c) convenient to use. Otherwise, I'd rather not bother with it.

[Richard Feirstein]

Good posts. Now for the reality test:

Best series of tests I observed where tried by AR years ago. Have live players behind a screen and then some AR3a speakers and switch between them and see if the audience can tell live from speaker reproduction. Closer than you would think.

Then I had a chance in the 60's at a NY Audio Society meeting, (good excuse to go out for Chinese food), they had a 15 ips master stereo tape and a tone and level compander. They ran the output into a very well respeced tube amp and also into a very poorly regarded cheap sold state receiver (Radio Shack's old brand name item). When the eq was set so that both were flat and the levels were matched and both were carefully driven below clipping levels, none of us could hear the difference. We were true snobs and there were were faced with this! Perhaps today with better source material and better amps and speakers we would all hear a big big difference, perhaps. Care to give that one a try?

Recently some people doing SACD and DVD-A mastering have had the opportunity to compare right off the mixing board the live or taped input with the SACD and high rez PCM output. None were double blind mind you but lots of people have reported that they hear differences between the input and the output. Most seem to report that SACD (DSD) output is most like the input but the jury is still out on this.

[Tullman]

Originally posted by Dob


That's a valid observation but it misses the point, which is whether or not there is a DIFFERENCE. Double blind testing of equipment has nothing to do with value judgements. Many times I have heard differences between A and B, but I can't decide which one is "better". And even if I do choose which one is "better", that is a totally subjective evaluation that someone else is perfectly free to disagree with. EQ is a good example - what sounds bright to me may sound dull to you, and vice versa.

If I need extended listening just to hear a difference, let alone try to decide if it's an improvement, that piece of equipment/tweak better be a) cheap, b) small, and c) convenient to use. Otherwise, I'd rather not bother with it.

If you can't decide which one is better then you need to spend more time listening. Sometimes I need 2 seconds to know whether a component is an improvement or not.

Being an audiophile is sometimes a hard job. There were times when I dragged home 100+ pound mono blocks. I have also home auditioned preamps, cd players, tuners, cable and other acessories.

[Tullman]

Originally posted by Richard Feirstein
Good posts. Now for the reality test:

Best series of tests I observed where tried by AR years ago. Have live players behind a screen and then some AR3a speakers and switch between them and see if the audience can tell live from speaker reproduction. Closer than you would think.

Then I had a chance in the 60's at a NY Audio Society meeting, (good excuse to go out for Chinese food), they had a 15 ips master stereo tape and a tone and level compander. They ran the output into a very well respeced tube amp and also into a very poorly regarded cheap sold state receiver (Radio Shack's old brand name item). When the eq was set so that both were flat and the levels were matched and both were carefully driven below clipping levels, none of us could hear the difference. We were true snobs and there were were faced with this! Perhaps today with better source material and better amps and speakers we would all hear a big big difference, perhaps. Care to give that one a try?

Recently some people doing SACD and DVD-A mastering have had the opportunity to compare right off the mixing board the live or taped input with the SACD and high rez PCM output. None were double blind mind you but lots of people have reported that they hear differences between the input and the output. Most seem to report that SACD (DSD) output is most like the input but the jury is still out on this.

Richard they should have taken the damn EQ out. It would have been easy to hear the difference.

[Richard Feirstein]

Exactly right. What they were demonstrating was the fact that non-linearity in frequency response from a cheap Aliance receiver was easy to spot. Once you took a speaker or amp that was poorly designed from a frequency response perspective, and carefully and accurately applied EQ, it was a lot harder if not at all possible to tell one amp or one speaker from another. This was not a scientific test at all, just fun to raise some questions about our notions and prejudices about what we hear.