Ars Technica And The Million Dollar Cock-up

If only the folks at Ars Technica and the James Randi Educational Foundation knew about Wikipedia (my emphasis):
An ABX test is a method of comparing two choices of sensory stimuli to identify detectable differences between them. A subject is presented with two known samples (sample A, the first reference, and sample B, the second reference) followed by one unknown sample X that is randomly selected from either A or B. The subject is then required to identify X as either A or B.
What do you call an ABX test that skips that last bit in bold and let's a subject say "I dunno" when presented with X? Useless.

Here's Ars' explanation of the relevant part of the test procedure:

"Then, behind the curtain, one of the JREF persons would draw a ping-pong ball out of a bag to select the "X" cable. A colored ball indicated the "X" test would be done with the AudioQuest cable, while a white ball meant the Amazon cable. We would then plug in the appropriate cable and play the listener’s audio sample for a third time, and at the end, the listener was asked to identify which cable was used. A "null" response of "I don’t know" or "no discernible difference" was also allowed and was counted as a miss"

Oh well. Ars Technica and The James Randi Education Foundation failed their own ABX 101 test. Oops!

But that didn't stop Ars Technica's man on the scene Lee Hutchinson from titling his story, "The audiophile’s dilemma: strangers can’t identify $340 cables either" and from waxing poetic in his analysis of their botched Million Dollar Challenge Ethernet test. And there's this bit of noodling:

"The statistical uncertainty makes it impossible to say exactly what the applicability of the results are beyond the scope of the test [they tested 7 people in total]. In a version of the test with forced A or B responses, we might be able to draw some generally applicable conclusions—however, here, we cannot. We can simply say that under these test conditions, almost all respondents indicated there was no discernible difference."
Lee Hutchinson also repeatedly quotes me as saying the differences I heard between various Ethernet cables were "not subtle or slight" and "as plain as day." This, according to Lee, was "The claim being examined..." in Vegas. I'm honored. The problem is, I was clearly saying these "not subtle or slight" and "as plain as day" differences were heard in my system. If I listened through the Ars test setup, Grado RS2e headphones connected to a Dell M2800 laptop, my guess is I'd be hard-pressed to hear the difference between an MP3 and an M16. Details, details. Let the show go on.

I had written a lot more about this fiasco but there's no point in wasting words on a botched test. While I suppose I should be grateful for all of the attention Ars has heaped on me, again, if people from a tech site coupled with a group who are supposedly expert in these kinds of tests can't even get the most basic aspect of an ABX test right, what does that say for their competence when it comes to, you know, anything else.

COMMENTS
DrPizza's picture

The lack of audible distinction in the non-blinded tests is even more damning!

If I listened through the Ars test setup, Grado RS2e headphones connected to a Dell M2800 laptop, my guess is I'd be hard-pressed to hear the difference between an MP3 and an M16.

Then I'd suggest a visit to a doctor; it appears that you have suffered considerable hearing loss.

Michael Lavorgna's picture
...and state any professional affiliation you have with this matter, chief. If not, I'll do it for you.
Michael Lavorgna's picture
"DrPizza is probably PETER Bright, yep." That's your colleague, Lee Hutchinson.

Just so people know, this is you: arstechnica.com/author/peter-bright.

As a friendly reminder, here's my favorite quote of yours:

"...you appear to be complicit in an industry whose sole purpose is separating the credulous from their money."
All of this over the fact that I heard a difference between Ethernet cables.

I've taken the liberty of blocking your account since you a) are unprofessional, and b) have nothing of value to add to this discussion. If you'd like to continue a discussion with me, just shoot me an email.

Cheers.

cundare's picture

My impression of the Ars guys is that they suffer from the "brightest person in the room" syndrome. When they were in grade school, they got so used to being considered bright that they later became arrogant and lazy. Have you ever read any of the gibberish published at that site about patent law? As a patent attorney, I find it almost funny to read the rabid pieces every few months that are so obviously written by authors who have no friggin' idea what they're talking about. I say "almost," because it's also frustrating to see so many other Arsholes agreeing with the gibberish these guys publish -- and to see the most idiotic pieces picked up by other "smartest guy" geek sites, like Slashdot, without any attempt whatsoever to verify or confirm results.

Really, for Ars to have any credibility, it would have to require an author to have even the most rudimentary understanding of law before they let him or her write about the law -- and, of course, require an author to have some credentials in the physical or electrical science before attempting to implement such a sophisticated implementation of the scientific method as an ABX audio test.

Michael Lavorgna's picture
...(pun accepted ;-) has taken to referring to me as "Mr. Lasagne" in the comments section on the Ars site. Seeing as the last time someone said that and thought it was a) worth saying, and b) somehow clever was way back in grade school. This tells me a lot about Peter Bright's level of maturity.

We've been through this before but it's worth repeating - the first Ethernet article Ars Technica posted was written by Lee Hutchinson. This article was essentially a re-telling of a post on Reddit. Lee didn't even bother to fact check his piece which used the same incorrect price as stated in the Reddit article. Lee also misrepresented my position to the point of simply making up a story. And on and on....

I agree with you. One assumes that Ars Technica would hire competent and knowledgeable writers who are interested in reporting the facts. That clearly is not always the case.

Venere 2's picture

Ars technica's new study reveals climate change is a hoax. It is not man made, it is not even happening… Yay let's pollute!

myrantz's picture

From the article:

We got through seven subjects (or eight, counting the first demonstration test where the "X" selection was done in full view of everyone). Only two subjects even attempted to identify the "X" cable; one was correct and one was not. The remaining six subjects said that they could hear no discernible difference. According to the test criteria, this many misses counted as a meaningful result, and the test was terminated.

Am I correct in reading that a single person has correctly identified the X. Assuming a he, did he do the full 10 times and correctly identified X every single time? If the cable swap is randomised as well this could well be significant.

Are you able to track down this mystery person and ask him what cues he used to identify X, or did he just plain guessed :D?

Michael Lavorgna's picture
One subject correctly identified X although we don't know how many times.

I consider this entire test a waste of time. If you want to read about results based on listening over months through a hi-fi in a treated room, as opposed to minutes through a laptop, you can read my reviews ;-)

Ex2bot's picture

It wouldn't be difficult to resolve this argument. You can agree to a blind test, where a third party tests your ability to hear the difference in Ethernet cables. Have a trusted friend keep an eye on the third party to make sure they accurately record your responses.

Problem solved. No need for a dog and pony show. Better yet, have a networking professional test the two cables for differences in data transmission. You'll be vindicated.

Michael Lavorgna's picture
...for the contents of your comments-you've never read what I've written about Ethernet cables. Instead, you are basing your view of my view on this subject on what Ars Technica has written. Since they have misrepresented, i.e. made up, most everything they've said about me in regards to this matter, for example I've never recommended nor been a proponent of buying a $10,000 Ethernet cable, it's no wonder what you are saying has nothing to do with anything I've written.
"Better yet, have a networking professional test the two cables for differences in data transmission."
That would be a great test if you believe the only issue here is data transmission. I don't.

As far as being vindicated goes, based on the fact that I've never said what Ars Technica has "reported" coupled with the fact that Ars and JREF do not know how to run an ABX test, and the mistakes made were of the utmost amateurish sort, yet they reported on the meaningless results anyway, the only person that needs vindication are the people from Ars and JREF.

Ex2bot's picture

I have to apologize for that comment, matt . It was pretty late at night in my neck of the woods and I'd had been sampling the vino with some friends. I've always enjoyed your singing, so I can't imagine what would possess me to say all that. Your bands alright too, but your drummer sounds like he's using sh**ty samples or something. Maybe I'm just old fashioned.

So no hard feelings, I hope. Catch ua next time your in town!

Bill

cundare's picture

Mike, I've got an even better idea. What if _you_ designed the test parameters and methodology -- right down to specifying components and source material -- and let Ars conduct the test. Each party could select half the testers and both would assigned representatives (unidentified to the testers) to monitor for compliance. I think _that_ would go a way toward resolving the issue.

Of course, convincing Ars or Randi to go this route would be hard. But even their denial would help discredit them. After all, it would be easy to create a test that resulted in negative results -- nobody being able to hear the difference -- but if a double-blind test produced verifable, repeatable positive results, it would be hard to argue that there was something wrong with the methodology.

Yeah, I know this will never happen. But wouldn't it be cool?

Michael Lavorgna's picture
...for me three years ago when I wrote the two related reviews. Ars/Randi attempted to refute what I wrote, that I heard a clear difference between different Ethernet cables, and they failed miserably at it.

It has been my experience that Ars Technica has zero interest in journalism or even in simply reporting facts. My opinion of JREF is they are clearly incapable of getting even the most basic aspects of an ABX right. They are also self-proclaimed "debunkers", not an impartial group interested in conducting a valid ABX test.

What would be cool, for me, is for Ars to report the truth. I'm not going to hold my breath ;-)

stephsrecords's picture

and sell a lot of magazines -- Invite the Ars Technica and The James Randi Education Foundation people over for a test or set up a test for the general public at the next big audio show where the system would show off the difference a good quality Ethernet cable will make. I would put money down that a proper system fed a variety of recordings from mp3 to DSD would convince and enlighten them.

Bromo33333's picture

Not only did they not run an actual ABX test, the number of subjects (6) wouldn't give statistically significant results.

But these guys know all of that - they know how to run a proper ABX test. They also know how much data they need for something that's statistically significant. (And I am sure from their comments if they thought they needed help they'd have an army of people there to help them).

But ... I suspect they came to similar conclusions to the author of this blog post, and ended up "throwing the test" -- funny how they put in a crummy soundcard instead of the better one in the Macbook pro (though, frankly both of those wouldn't be as revealing as an actual dedicated device), and they played on the subjects sense of insecurity by allowing a "I don't know" result (or allowed a hand selected few a way out).

These guys are publicity hounds, with all the hype if they came to conclusions that their readership wouldn't agree with. Well I have seen how they treat interlopers who don't tow the line with them. Imagine an evidence based apostasy would do the Ars and the Randi crew?

They don't have the luxury of validating anything, without losing their core group. They will not and cannot be objective, and clearly can throw a test.

VK's picture

That's how i describe this test, Ars Technica and The James Randi Education Foundation.

All the best to you Michael!

monetschemist's picture

by any reasonable definition of ABX test.

The point of an ABX test is to prove that X is reliably selected.

If it isn't that doesn't prove anything (not the hypothesis being tested).

Not to mention that testing it with 7 of your buddies (or 6 or whatever small N was used) doesn't permit any kind of statistically viable conclusion.

What a bunch of wankers.

jolon's picture

I can't believe they went to so much effort to run the test (buy an expensive cable, travel to Las Vegas), and yet got it so wrong. Why didn't they ask an audiophile for setup advice?

It's the equivalent of a couple of journalists evaluating medical equipment.

cliffjumper68's picture

Why couldn't they use even a AQ dragonfly DAC so that you have a better signal? They used a cheap switch and the same cheap cable for the NAS to switch connection. They only changed the last ethernet connection cable. Lastly, they used a open back headphone in a crowded room of people talking. Loaded crowd aside it was a loaded test.

Bromo33333's picture

They didn't make any big claims.

But another question, if this was a "Million Dollar Challenge" who was the guy they would have had to write the check to had the cable been shown to improve?

THis whole thing is fishy.

Michael Lavorgna's picture
Here's Lee from his Ars article:
It’s important to note here that while the on-stage demonstration adhered to a defined test protocol, the million dollar prize wasn't staked on its outcome—not yet. In keeping with all past public JREF "Million Dollar Challenge" tests, the public test on stage was a preliminary one; if the cables passed, the next step would then be to reach out to AudioQuest and construct a mutually agreed-upon formalized test, upon which would ride the cash.

But as it turned out, that wouldn’t be necessary.

Here's my read on that statement - JREF will not risk the $1M unless they know the outcome of the test. James Randi is, after all, a magician and he's also no fool.
drblank's picture

an admitted con artist. LOL

rbrooks's picture

Hopefully former Stereophile Assistant Editor Stephen Mejias will soon weigh in on this outrage from his current desk at Audioquest.

Stephen Mejias's picture

Hi there, rbrooks. Thanks for the prompt. We've been keeping up with the brouhaha (ha ha). One thing that seems evident to me is that the staff at Ars Technica is having a lot of fun. I can't help but admire the level of enthusiasm and pure joy with which they go about their work. Several of us here at AQ have been following along, happily. The image gallery depicting our Vodka being torn apart turned out to be a rather nice compliment to our work. (Although I do think it was a shame to destroy such a great cable. It could have been donated to someone who cares more about music. Hmm...)

While we have enjoyed the entire experience, and I hope that it's been successful for Ars' web traffic and as much fun for their readers as it seems to have been for them, we really see no reason to lecture the already informed or to spend time and energy arguing with those who honestly believe we're charlatans. (Ever hopeful, however, I do wish that we'd have the chance to sit in a room with the Ars staff, maybe share a drink, discuss our backgrounds in music and audio, listen to some songs, and get to know one another better. I think we'd all find that we're pretty likable people, working as hard as we can to make a good product.)

We also hugely appreciate Michael Lavorgna's efforts to provide his own informed, educated, honest perspective. Journalists like ML are, in my opinion, truly changing the digital-audio conversation. ML and others like him are digging deeper, listening harder, making it okay for audio enthusiasts to ponder fascinating topics that might otherwise be left unturned or neglected. This is great for audio, great for music, great for our hobby. The importance of the discussion and the deeply curious, honest investigation of the effects of high-frequency noise and jitter on the audio signal, in my opinion, cannot be overstated.

Plus, it's fun. Shouldn't anyone interested in music and technology want to learn more, experience more, dig deeper?

Along those lines, we're working on some very revealing difference files and compiling other clear evidence of differences, including directional differences. We want to do this for the sake of all the reasonable skeptics who haven’t had the opportunity to hear for themselves. If all goes well, it'll inspire more people to actually listen for themselves, to decide for themselves, and to have their own good, honest fun -- unaided by so-called "amazing" debunkers (ha ha).

Stephen Mejias's picture

Sorry! I forgot to add my signature to my post.
Love,
Stephen Mejias
AudioQuest

Clever Dean's picture

to show up at AQ with a computer server/client testing rig. Why won't you take them up on the offer?

Ex2bot's picture

I'm sure your cable is well-made, but no amount of effort put into an Ethernet cable will change the sound. The networking protocol (TCP/IP over Ethernet) handles issues of data integrity.The data will either be transmitted identically or it will be re-transmitted. In the event the cable is defective, the transmission will simply fail. Sending audio data digitally through a network cable is very different than analog audio through speaker wire. The same is true for audio data transmitted over HDMI.

Buy high-quality cables so they're less likely to be or become defective, but don't think there's some kind of magic that can be done to digital cables to make your music sound better. On the other hand, high-quality DACs will make a significant difference, as others have mentioned.

Hugo450r's picture

..was both hilarious and fitting.

Bravo, Mr Lavorgna!
Bravo!

Michael Lavorgna's picture
I sure did.

Cheers.

Clever Dean's picture

about both the AQ cable and the actual testing were validated.

The AQ cable is marginal 6a which is a problem especially when a $13 cable is blowing by it on a $10,000 Fluke tester.

And the entire testing rig brought by Ars is abysmal.

Randi and Ars should have brought in a person that has feet firmly planted in the world of Enterprise networking and is also an entrenched audio enthusiasts.

Not to avail themselves of a few of those people was a mis-step.

I'm not even sure what to make of the result that a marginal $340 cable and a completely failing $2.50 cable was null (If I take the testing at any value).

This has actually muddied the waters even further.

Michael Lavorgna's picture
...that people have no qualms about questioning everything I say but when anyone says "measurements" all of that hard-nosed skepticism melts away and you lap up the results like a puppy dog.

I don't trust anything Ars Technica does or says. I would not put it past them to have tampered with their sample.

Yes their test was abysmal. On purpose, imo of course.

Clever Dean's picture

I'm not lapping up anything. When I voiced my opinion about the DAR blind test interview with Barry you rightfully called me on it.

So much so I agreed with your POV and requested that you delete my posts on the matter. It was a sign of respecting your position and point.

But somehow Lee or someone @ Ars tampered with something. I think Lee and Ars should be extended the same credibility that you asked me to extend your friend and fellow journalist.

Am I somehow being unfair?

Michael Lavorgna's picture
Ars and Lee lost my respect and any semblance of credibility when they published their first factually inaccurate sensationalized bullshit article about $10,000 Ethernet cables. They continued to misrepresent, that's being kind, my position in order to jazz up their story. It should be clear to any thinking person that they will say anything irregardless of whether or not it's true to attract clicks, support their position, and feed their readers the pablum they desire.
Clever Dean's picture

where I'm lapping up anything they did? All I have are questions about everything they did. I even called their testing rig abysmal. Which it is. Please feel free to point out any of my kool aid drinking. But I don't think you can support your accusation.

I also called into issue what I saw with the tear down. So much so that I find that my concerns where justified. All this is opportunity for AQ to make a slight revision to their product assembly procedure.

IMO I don't think their cables are 6A but solid 6. The NEXT is too much into the quick. Stop and think for second. I deduced this on a photo. I may
know a thing or two and capable of objective thought.

For Ars it's to rethink in the entirety the testing procedure.

If you lost all respect for Ars then why have on-going back channel communications with them via email?

I'm sure if you asked Lee to make a retraction that Ars would. You have his email.

Michael Lavorgna's picture
...without question. And the tear down. How do you know Lee didn't mess with the cable before taking those pictures?

The email exchange with Lee was initiated by me. I pointed out all of the errors in his first article about the $10,000 Ethernet cable. You can see for yourself that this meant absolutely nothing since he/Lee continues to misrepresent what I've written and what I recommend. But why care about silly stuff like fabrication when they have "tests".

The fact of the matter is this entire stupid story was cooked up by some twit on Reddit, swallowed hook, line, and sinker by Ars and HotHardware, neither of which even bothering to fact check something as basic as pricing information when AQ has a price book on their website, and no one bothered to read and report on what I wrote. Nothing like a juicy pull quote to fabricate your point.

The reviews of mine they linked to and used as fodder for their bullshit were years old yet none of these "reporters" thought it might make sense to ask me a question.

Again, any thinking person can only conclude that Ars is interested in making up a story, not reporting the truth.

The fact that Ars, a tech site, and the JREF folks put on an embarrassingly amateurish meaningless charade parading as a scientific test is just icing on the bullshit cake. The further fact that this dipshit show appears on the ARS homepage as a featured article only serves to shout out the fact that they are shoveling horsehit to their readers. If you read the comments, some readers like the taste.

Clever Dean's picture

Just want to get that out of the way. I trust his operation, his honesty, but fully agree that there is the angle of bias. No way around that and I'm not looking for a way around that. You're correct 100%

But the cable is a solid CAT6 and good for Gbit networking. So I've no problems in that regard.

If you had emails back and forth and the issue of $10,000 being attributed to you. Ars should have resolved that. That's on them and it does take shine away in addition to not thinking the testing through. I'm STILL effing baffled by that. Just a head scratcher.

I've lost count of how many cables I've terminated in the field the past almost 18 years. It's from that perspective that I didn't see anything amiss about the tear down. Just my opinion.

So in the end I'm not satisfied with their testing. I'm not happy that they didn't correct the $10,000 article. I'm not happy they didn't explicitly state where you make a more than reasonable recommendation on a well sub $100 Ethernet cable.

I have the Cinnamon cable in and it's built really well. I'm trying to find out what terminations they use and don't know if I'm going to cut on the cable or not. How much will my curiosity cost me :-)

From the what I can see of the termination though I *think* it will scope out better than the Vodka. That's a conversation for another day.

Michael Lavorgna's picture
They strike me as being all about no nonsense, good value, and solid products.

Before you cut, ask AudioQuest whatever question you have.

Bromo33333's picture

I think BJC is an outstanding operation. They certify that their cables meet whatever standards are available. I think you could do a lot worse than those guys.

From what I gather, the AQ's claim is that they shield the living sh*t out of their cables. I'm contemplating trying them out. But also Oyaide ...

It's a good time to be an audio hobbyist

The only time you fail is when you get caught up in the noise. Cable debates have been going on for decades, and will continue to go on for decades. If you are posting and not listening to music ... that's your problem!! ;-)

Bromo33333's picture

The measurement was incomplete.

They confirmed that the cable conformed to the Cat-6a standard (yay! And nobody should be in doubt). And despite their badying about the term "marginal" they don't really talk about how the "control" cable they used in their abortive not-ABX test got blown out of the water.

The BlueJeans cable (hello! where did this come from?) was brought in and didn't perform as well as the AQ Vodka, actually if you look at the data. BlueJeans was honest, but Ars kind of did a sleight of hand and changed their comparison to the Bluejeans cable, and not the cable used in the test. Abracadabra!

But, and this is important, they didn't test for RF immunity and emissions. That would directly test the Audioquest claims about their shielding, especially the dissipative shielding. I will also mention that the Randi cable and the BlueJeans ones are unshielded. There are basic controlled standards that could be adapted to evaluate the effectiveness of the shielding, and given peopel an idea of the impact of routing the cable near sensitive electronics. But ... they didn't bother. (Not for nothing it would cost about $2-5k to do the test. But if you are saying you want to debunk something, you actually are then under the burden of actually verifying or disproving actual claims).

So, yes, they changed the test, introduced a new cable, and did a little shell game so they wouldn't be forced to admit tha the AQ cable blew the pants off of the "cheapo no-name Ethernet cable" that they were claiming was just fine.

In case you weren't watching or missed the bit of sleight of hand.

Clever Dean's picture

to the switching over to the BJC and the AQ. Ars readily admits that the Amazon cable was junk.

All I saw was a BJC used as an example of a cable that passed the spec way past margins.

Reed's picture

A while back I wanted to settle this on my own. I purchased a Manley Skipjack that allows you to switch between cables with the push of a button. What I found is that I couldn't rely on any opinion I thought I had, when the switches happen in somewhat rapped fire succession. It was too mind numbing.

I then tried various cables for a week then changed back to a different cable after a week. For instance I would try a Nordost LS series cable, then switch to a Blue Jeans Cable LC-1 after a week. I would swear that when I switched to the Nordost, the sound was detailed and bright, but lacked base. After a couple days the cable seemed well balanced. When I switched back to the Blue Jeans Cable, it seemed much better rounded but lacked detail. After a couple days, it in no way seemed to lack detail.

What's interesting is that I don't experience that when I switch components. I always seem to consistently prefer a certain component over another. I came to the conclusion that either my initial preference was preconceived, my hearing adjusts or there was simply no difference. I now use studio type cables. I have Mogami Gold Studio balanced cables and Blue Jeans Cable LC-1.

Maybe others have different findings, burn my somewhat exhaustive effort has settled my mind on the matter.

Reed's picture

Not burn....auto correction seems to make wild guesses every now and again.

robpriore's picture

I've been an audio fanatic for 25 years. Cables matter. I know because I've tested with my ears. And it's only my ears that matter, since they are uniquely mine. Your ears may hear something else, there's no way for me to know.

To be perfectly frank I've tested USB cables and cannot tell a difference, but there is a big qualifier on that, just like Michael said, it's being tested in "my system." In my system USB cables should not make a difference because the power is physically isolated, that's my audiophilleo and pure power rig, and of course they don't make a difference to MY ears, the only ones that matter.

Same goes for speaker wires an interconnects. I developed my own testing protocol around these categories and my ears hear the difference.

I'm very happy with my choices and nothing anyone can "say" will change that. I'm open to "hearing" new ideas, and old ones too, and testing out what makes sense.

Does a $300 Ethernet cable make sense for me? No, because I don't have an Ethernet cable in my system. If I did I would test the Staples version and a few others from various manufacturers including AQ. Just that statement alone would incite some folks who refer to us as audiophools.

But if you demand the best, like a lot of us, then you will match your components to the best parts you can find. Quality matters to people like me, and most of my gear is overbuilt, but that's because I want it to be. I was never trying to find the absolute cheapest possible method to the best sound. That doesn't mean I throw money away either. I just want good sound and that costs money because it takes manual effort to build good stuff, and a lot of my gear is hand made. Those hands cost money, and I'm especially paying for the golden ears the hands belong to. If the person or company building the product doesn't know good sound with their own ears then how can the product possibly sound good? I would say that one statement covers the overwhelming majority of computer anything, cables to hard drives. Is there really anyone that can't hear the difference between a laptop sound card and external USB DAC? If you can't I truly envy and pity you.

The last I have on this is that some call it gilding the lily. I just happen to prefer gold.

egsp's picture

see http://nwavguy.blogspot.com/2011/05/subjective-vs-objective-debate.html

It's not like in test after test, clear differences are shown and we are scrambling to explain them--quite the opposite.

All I ask is that reviewers explain how obvious differences disappear when you close your eyes!

If the differences are real you will be able to hear them even if you don't know which component is which. There is no way around this.

For those new to the debate there are very good explanations about how we hear and perceive that explain why we tend to hear differences. I've experienced it myself. It was humbling.

Without some attempt at objective testing we are just debating endlessly about nothing. It seems prudent to at least be aware of the objective data before plunking down cash. Buy nice stuff, but know what you are getting.

Otherwise we are fair targets for those calling us audiofools.

Michael Lavorgna's picture
...as it's typically presented is so grossly generalized and inaccurate that it's laughable. The idea that performing a valid blind test is a walk in the park is nonsense. The results of every single audio test that I'm aware of, including AES peer-reviewed papers, are contested = there has never been a blind test that proved anything to everyone.

If you've kept up with the Ethernet cable circus, you'll have seen that two professional organizations who tout their technical and testing abilities (Ars Technica and the James Randi Educational Foundation) performed an ABX test to disprove something I wrote about and they failed their own test because the testing protocols and procedures were so flawed that any results are meaningless.

Here are my thoughts on audio testing: "Blind Testing, Golden Ears, and Envy. Oh My!".

Audiophools. I've found that people who use this phrase are really talking about money. They are saying that audiophiles spend more money than they need to. I ignore such people since last I checked how much we spend on our enjoyment of a hobby is not governed by ignorance and envy.

egsp's picture

Thanks for your time in sending the comment and the link.

On a related tangent, one question in my mind is why differences in audio reproduction mean anything to me, as opposed to others. A blessing and sometimes a curse.

My roommate in graduate school was a genius, probably an IQ of 160 or more. I went through several iterations of stereo equipment and he listened to all them. He had no trouble identifying which set-ups sounded better to him, and seemed to enjoy them. In the end he went with the stereo a set up which was not much better than a boombox— Because it was small, cheap, and convenient.

I, on the other hand usually notice the music (and the sound quality), even in elevators and restaurants. It gives me special pleasure to hear well reproduced music, And that, I believe, the key to my condition. Music and sound, It’s why I’m here in front of my computer, on an audio forum, enjoying the music as well as the sound of the Heil AMT tweeters on my desktop monitors.

regards

Golden Ears's picture

BUT.... have them drink the two very similar Merlots they are comparing out of cups partially filled with mud, salt, pepper, and tabasco sauce and all stirred up.

And best of all, make sure the people doing the test are Miller beer drinkers- who have never touched wine in their life.

or have them evaluate two different types of glass for telescopes but let their own prescription be wrong and their glasses dirty.

In every case of these amateurs doing an ABX test and them never finding a difference, they are almost always people from outside the audio field, or ...well frankly audio enthusatists that may be listening for the wrong thing.

For instance, I had a discussion on a long plane ride in the 1980s with one of the Lyons Brothers that owned most of the good nightclubs in Boston about sound quality. His attitude- "Voltage is Voltage" nothing else matters... but you see he spent nearly every night in his deafeningly loud nightclubs, and likely can't hear much by now anymore.

It's also like asking some guy who served in Vietnam and Korea and both Gulf wars.. firing off millions of rounds of heavy artillery... which cable has better air in frequencies over 16K.

When you go to the hospital , and a doctor tells you you have a heart murmur, and lets you listen... to the person without a medical degree and speculation in cardiology , your heartbeat with a heart murmur sounds exactly like everybody else's heartbeat without the heart murmur.

Sometimes you have to learn how to hear the difference, a person without training cannot hear or quantify or understand those differences, or cannot discern between different aural differences in the presented music.

Just as I would expect most people NOT to be able to discern the difference between two years of similar Steinway grand pianos with a minor design change - or ones tuned by different tuners, I surely wouldn't be able to expect them to tell the difference between two different digital filters , two different USB cables.

When I think back to how unskilled my ears were when choosing my first pair of speakers in highschool... I was able to choose a pair of speakers that had a single better drum hit in a single recording , but I was unable to tell midrange differences...because frankly.... both speakers were so far off the mark...it was hard to tell what direction I should be headed.

For most people Budweiser works fine for them. Anything else is a waste, or there is a slight gain in taste , but they see it as not being worth it for what they are after...which MAY NOT BE TASTE ...it might be ease of drinkability, cost, just the ability to get a buzz....

Most audio purchasers are concerned with clarity, bass impact, and then volume. And if it doesn't sound good enough at lower volumes because the small speaker can't energize the room enough until the walls start storing and re-radiating bass back ...their solution is to turn it up until it perceptually sounds "better".

I got my nickname because I could put speakers in the right place and match cables well to gear... but I do not have the training a conductor would have, nor would I bet I could tell the difference between different years of recorded Steinways ... but I have an idea when the whole of the music presentation becomes more realistic over all...which is very different from the way a Steinway is voiced, or the way a conductor matches his orchestra to a venue, or the way a recording engineer looks a spectral balance and lift in certain areas in frequency when placing his MICs----> knowing in advance he has to compensate for generational recording losses in the recording chain (and in the end users gear) ... before he gets to the end product for sale.

To expect some guy who reviews cameras, Hard drives, and rechargeable battery packs, mobile phone operating systems, to get the different between speaker cables is foolhardy. Its also as silly as asking him to pick the better glass or gamma for a mobile phone display...

Look... I can drive, but I don't have the stupidity to think I can get behind the wheel of 2 nearly identical Formula 1 car rear wings and let you know which one is performing better at the point of release of traction in regards to buffeting and downforce in a 165mph turning pulling 1.3 Gs.

But these guys at what amount to "Tech gadget toyshops", think they can tell differences like a truly specialized product designer...

....silly.

gorkuz's picture

While we've disagreed in the past, I'm with you on this one, Michael. Most practitioners of "blind test" voodoo are, indeed, blind-ly confident of their faulty methodology. And most times their audio setups are woefully inadequate in resolution to be definitive to begin with, as well. Being unused to a particular system is another serious issue for the testers. One can always pick out far more in one's own from simple familiarity.

I recently participated in a test of a Behringer crossover. The test was to determine if the ADC-DAC conversion within was audible...in that system. The system, I might add, was quite good and I would not fault it for resolution. The conclusion? We could not get reliable results, so the tester/owner of the gear concluded quite confidently that the Behringer unit (complete, as opposed to the conversion section) was not audible (meaning "perfect"). Pointing out that all of the testing involved running the signal through the input and output buffering opamps in all cases was swamping the differences made by the apparently decent converters was ignored. Those opamps are ones I do not care for and automatically replace with much better ones when I have modded Behringer equipment, with very significant improvements. This is typical. The real question is of where the blinding is actually being applied.

This is how most ABX tests go, when examined critically. I fully believe you heard differences. I deal with many cables as well as design them, and have never heard two sound identically alike if checked closely enough with a system capable of separating them. Seems like quite a lot of people will always immediately insist that all "bits are bits" and need to learn the lesson that no two wires are quite alike...unless they are, actually, the same wire... This is the same sort of resistance that has occurred every time that upgrade cables of any type first appear. Wasn't it only recently that we went through the same thing with USB cables? Analog IC's? Speaker cables? Power cords? How many times do we have to learn the same thing over and over again? (Some ignorami haven't even gotten past even one of them.) Fortunately most audiophiles can hear and decide for themselves, so a market gets created anyway to the fortune of those that can appreciate it.

NeutronStar's picture

The article by Michael Lavorgna and comments (I must confess I did not read them all) lack a description of the setup tested. I guess there was a NAS connected by the test ethernet cable to a Dell Laptop and Grado headphones to listen, no idea of what software was running, kind of NAS, etc...
A much more revealing test would be to use a computer with already music in the HD a USB cable (AQ or Wal*MART and top notch USB AD converter and headphone amplifier with top notch headphones (STAX?) A panel of listeners, audiophiles and not, would hear (or not) which USB cable is used.
The electronics and headphones can and should be High End because the more revealing the more likely differences will be heard.
The test performed in Las Vegas seems lo me flawed and doomed from the very beginning

egsp's picture

Reading over my first reply, I realize it had a snarky attitude, I apologize and appreciate your calm response.

What I'd like to convey is that I don't think we can ignore objective testing. With all its limitations, It is the de facto/universal language by which disciplines communicate within and between each other. Instead of complaining about the difficulties in testing setups conducted by non-audiophiles,, it would seem much more productive to design and execute tests ourselves,ones that would help reveal the subtle differences. I believe subjective testing helps identify the potential differences then objective testing to separate the wheat from the chaff, if you will. Stereophile has done a great job of trying to correlate the the objective with the subjective experiences of the reviewers. This is no simple task, having just dabbled in some (humbling) objective testing myself. I respect reviewers, and hope they apply a part of their listening skills towards advancing the science of our hobby. Maybe a "Manhattan Project" all out assault on this issue?

egsp's picture

(laugh) reading through your blog I and other writings I see you have already addressed this topic many times.

X