Proof Poof

Let's listen to some music!

Since the goal of listening to music on the Hi-Fi is enjoyment, the only valid proof of anything to do with that experience lies in listening. Here at AudioStream, we listen to the products we review and report on what we hear. The value in this equation is simple; you are getting an honest and careful appraisal of how a given component, cable, app, or tweak affects our enjoyment of music. In my opinion, the more experience we have with this process and you have reading what we write, the more value you'll find in our reviews. Simple stuff really and not at all unique in the world of audio reviewing. So I have to wonder why some people expect/demand something more?

"Readers Look To You Guys for Proof!" was part of a recent comment here on AS. Really? That's like me expecting to see your High School GPA in your comments. Not only is this wrong-headed, it is obviously looking for proof in all the wrong places. It's also very common for these same proof seekers to demand measurements which I've repeatedly said is something we'd like to offer but cannot at present provide due to budgetary contraints. Besides, measurements obviously have no direct correlation to enjoyment. Simple stuff really but some people just can't take "not at present" for an answer.

If you want proof you'll have to look elsewhere. If you want to learn about the enjoyment of listening to music on the Hi-Fi and ideally how to increase that enjoyment, you've come to the right place.

COMMENTS
jim tavegia's picture

Members of the Stereophile community are easily aware of things that measure poorly, or just OK, that many like the sound of said gear in question. It is a good thing to measure especially as the price goes up, but each of us should be able to tell if we like something or not.

The real test should be to try a number of things and compare them, i.e. headphones ( surely more than 3 or 4); speakers, and especially formats and determine what you like, what you will live with (or hear the difference), and then can you afford it? It may be if one listens to some good CDs or some high rez files that some "fashion statement" headphones may not fare so well. But at least one would know if there is something better. So much of the beauty of a musical performance is thrown away with the use of the MP3 format.

Very good audio does not have to cost much these days, and yes, excellent does come at higher cost, but some great sounding bargains are out there if one does some reading here, also at innerfidelity.com, stereophile.com, and subscribes. Some research can save some money and lead one to some great sounding gear and some great music recommendations.

I don't buy things solely on reviews, but often do if something is a real bargain from a reputable manufacturer. It is easy to prove that there is more music embedded in higher resolution files, but most MP3 lovers don't care about THAT proof.

Learning to play a musical instrument properly is not easy, yet we discount those who have learned to do so by being lazy, taking the easy road and listening to their music on our phones, and at its lowest common denominator, the dreaded MP3. Kind of a sad tale to me.

VK's picture

... i think that the reason for so much proof claims is that some of the reviewed products are way too expensive, without any real 1+1=2 from the manufacturer (!!!) and (if you really love music) there are many other alternatives to enjoy your music without spending hundreds or thousands of dollars in some fringe products (98% of cables and tweaking gear for example).

I understand why you don't make measurements, but i think that a product from a manufacturer that doesn't explain how it REALLY works, don't deserve a review.

Best regards!

Michael Lavorgna's picture
"...i think that a product from a manufacturer that doesn't explain how it REALLY works, don't deserve a review."
Of course we disagree.
VK's picture

... so we must encourage the "snake oil market" by putting them in the spotlight?

As 24bitbob said we need some filter for snake oil products and consider the engineering behind the claims of the manufacturer. Also, as 24bitbob said, "it is largely through engineering endeavour that your ability, and mine, to enjoy music as we do has been allowed to happen."

Of course, you can disagree :-D

Best regards!

Michael Lavorgna's picture
I consider listening and improved enjoyment a form of verification.
"...it is largely through engineering endeavour that your ability, and mine, to enjoy music as we do has been allowed to happen."
Of course. No one said otherwise. But let's not confuse the role of the designer with the role of the listener.
stevebythebay's picture

Trying to glean whether anything would be of true interest for a potential purchase has a lot to do with how you go about doing your analysis. Seems that whether it's a review of a movie, a car, or pretty much anything you're left assessing whether you have many or but a very few sources of input. With films it's often a case of finding a reviewer whose sensibility is similar to your own. You build trust in their reviews over time. There are reviewers in all fields who clearly have biases in one direction or another. In audio you can find those who seem to focus on what excites them musically, while on the other end of the spectrum it's about what is is least colored in the end to end delivery of the reproduction. So, pick your poison. A problem we're all faced with is that the reviewers environment, unlike going to the cinema or the car lot, is likely to be very different than our own. Our home and components are most likely to vary by some or a great degree. That makes it all the more challenging to discern whether a reviewed item will have the same or any effect in our own environment. And like terminology in other areas (I'm thinking wine here) each reviewer has his/her own definition and experience that they develop over time. This and the fact that each of us may come to listening with very different human equipment and few or many years listening to live and recorded music.

Best to work with a dealer/manufacturer who is willing and able to let you audition their products so you can do your own analysis. That and developing an appreciation for a reviewers tastes, system, sources and what they use for music to do their evaluation are particularly helpful.

Michael Lavorgna's picture
...reviews are at best guides to help the reader narrow down the field of interest. I do my best to not only make my biases known but also to respect what the manufacturer has accomplished even if their product may not conform to my biases.
Best to work with a dealer/manufacturer who is willing and able to let you audition their products so you can do your own analysis. That and developing an appreciation for a reviewers tastes, system, sources and what they use for music to do their evaluation are particularly helpful.
I agree.
24bitbob's picture

Michael,

There's a host of reasons why consideration of objective criteria for the selection or review of equipment matters, some technical and some to iron out the fallibility of human hearing / perception. Lots of reviewers understand that, and lots of reviewers get that right.

I respect your position, and that of Audiostream, to declare that what you hear and the enjoyment that brings is what determines your approach. A consequence of that, for me is, I take some of what I read here as being worthwhile and informative, and other stuff I take as opinion pieces offering little credence. That's my view.

The harm comes when people then choose which equipment to buy or not buy. Money, for most of us is pretty real, and to lay down $00's or $,000's of dollars on a piece of equipment, for most people, requires judicial thinking. Unfortunately, too much of the HiFi, or Audiophile business fails many people in this regard. Websites like yours have an important role to play in helping people choose the wheat from the chaff. (And no, lots of people can't audition stuff before they buy). Unfortunately, there's a lot of chaff out there commanding very high prices. In my view, anything that offers declared sonic benefits that costs more than a small sum of money, needs to be scrutinised for an objective basis of such benefit. Audiostream too often fails to do that.

That said, I still read you website most days, and I appreciate your work, but I take some of it with a pinch of salt.

Thanks and regards,

Bob

Michael Lavorgna's picture
Our reviews are based on listening so they are in fact opinions. From my point of view, the fact that I do this for a living and spend my time immersed in this world allows me to take a broader view than the casual hobbyist. So my opinion is therefore more informed but still an opinion nonetheless.
"In my view, anything that offers declared sonic benefits that costs more than a small sum of money, needs to be scrutinised for an objective basis of such benefit. Audiostream too often fails to do that."
What do you propose in terms of scrutinizing what we review on an objective basis?
24bitbob's picture

Hi Michael,

Thank you for the reply:

"What do you propose in terms of scrutinizing what we review on an objective basis?"

To risk the scorn of many I would suggest that scrutiny of components should include some consideration of the engineering behind the claims. And to those who 'poof' at such a suggestion, I would add that it is largely through engineering endeavour that your ability, and mine, to enjoy music as we do has been allowed to happen.

I am not advocating an approach based on purely technical grounds (I'm not completely soulless), but such consideration would allow application of some sort of filter to remove 'snake oil'.

Have a great day (or is that evening where you are?)

Bob

VK's picture

Best regards!

stevebythebay's picture

A definition: "(of a person or their judgment) not influenced by personal feelings or opinions in considering and representing facts".

Good luck with that. We all must "measure" with our own ears, in the context of our own room/system. And as is an ancient precept: caveat emptor (that certainly applies to anything we read about). Trust is increasingly hard these days. And it's always wise to try before you buy. If you cannot at least get a money back guarantee or pass on whatever would create a financial hardship. I know whenever I put out money for a Kickstarter or Indiegogo crowd sourced offering I may never see anything come of it. Like going to Vegas. Win some, lose some. In home audio there's even more to lose over time as we ratchet up our purchases and eventually see some technologies disappear altogether, or at least no longer command our interest in the face of newer ones. At least it's not as bad as video which seems to change every year (B&W to color - VHS to LaserDisc to DVD to Blu-Ray to ...).

drblank's picture

is obviously the bottom line for any product. What i have a problem with is that some products used to get great reviews by the audio mags, yet the product was a pile of junk. Anyone remember the Bose 901's??? I don't know how many were around when they were first announced, but those things were getting rave reviews and awards and yet these days, the "audiophile" community doesn't even look at Bose for making anything serious. Plenty of products have fallen into the category of getting great reviews, but didn't really live up to the hype.

The other problem I have is when a new product is introduced and it's a product that's supposed to do XYZ and it's based on some form of new technology that's not been proven and when I read reviews, they rave about them, but then I read comments by consumers that listened to them said they did absolutely nothing. And when you talk to an engineer they have no idea what the product is doing from an engineering standpoint. So, there are a lot of tweak products that fit into this category.

Now, here's what I'm hoping for.

!. Reviewers that do listening tests should have some proof they actually have a well treated room. I've seen photos of some reviewers (not necessarily on this site) that have simply horrible rooms with lots of glass windows, horrible or lack of treatment, and I know their room sucks and that any review is going to be a waste of time reading.
2. Reviewers that do have some credibility with having trained ears. Harmon has a free s/w app that helps train and test one's listening abilities, yet I have yet to find any magazine reviewer admitted to taking the test and actually passing it. If a reviewer can't even get past at least Level 5 on each of the tests, then my take is they really don't know how to listen to even the most obvious things. BTW, I've reached level 6, but I still have a long way to go to pass all tests at the highest level so I wouldn't consider myself qualified to be a "professional" reviewer writing articles for magazines, yet I have gotten past the 50% level for these tests. Why don't more people get this app and go through the listening tests to find out how bad they listen or to help them get better trained?

I just want honest reviews and when there is a product that even the mfg can't explain and maybe show before/after measurements to actually prove the product actually does anything measurable, there needs to be a little more scrutiny than there is, some products slide right through and it damages the reputation of the mag/audiophile community when scam products get rave reviews when many of us know it's simply not going to actually do enough to warrant spending the money and that it is a scam product. There are a lot of tried and true products that do work that aren't getting attention when there are products that get lots of attention that don't really work and it's unfortunate that people get suckered into buying simply a product that doesn't work, when they could have spent the same amount of money on something else that does.

whell's picture

The onus is on the buyer, not the reviewer, the audio mags, or the retailer, to do appropriate due diligence before buying any product, including audio.

That said, it is getting increasingly difficult to do so for purchasers or audio products. I can go listen to and purchase "mainstream" audio gear from my local Best Buy (my local Best Buy had a Magnolia audio store within so there are a few pieces of "higher line" gear to choose from. But if my ear demands better and my budget allows, its not that easy to do appropriate due diligence on some of the better audio gear available. Where does one go, for example, to audition the new Schitt Yggdrasil DAC, for example. Sure, Schitt and other manufacturers have, for example, 15 day return policies but what if that 15 days falls during a period when I can't do much listening?

Reviewers play a key role in the due diligence process. But its still up to the buyer to gather the info they need before a purchase. If a particular review doesn't provide the info that you think you need, due to budget constraints,editorial format, whatever, then move on or at least use that reviewers feedback as a single data point in many more that you accumulate before deciding to purchase.

Asking a zebra to change its stripes, particularly when those stripes are fine where they are, seems a futile exercise that is limited in value to - maybe - making rhetorical points in an internet thread.

whell's picture

Sorry for the multi post. Hey Michael, if you're not gonna change your format than at least give me a delete button for my own posts! :-)

drblank's picture

some of these boxes you plug in the wall that's supposed to make your whole system sound better. Now, I don't know if this is what they are, but I wouldn't be surprised if they were. Why? Because I think some of these companies don't want to tell us what they are because they can be bought for only a couple of hundred dollars in a plastic case, yet some of these companies charge well over $1,000 for one is a fancy case.

7.83Hz Schumann Generator. Here's a site that sells some and they have a video that explains what some of things it does as a result of the signal. http://www.lessemf.com/schumann.html

If anyone knows more about this than i, please, by all means share in the knowledge. I just think that some products are repacked other products and they just don't want to tell us what they are for fear that we'll find out and buy the exact same product that's a much cheaper unit that does essentially the same thing.

If this does explain what some of these boxes are, I just think the magazines should find this out FIRST before promoting the product.

Steven Plaskin's picture

No doubt, the most controversial products I have reviewed are those made by Synergistic Research. The company will not divulge exactly what many of the products are and technically how they work. I reviewed these products because the manufacturer and their dealers offer a 30 day money-back trial period.

If on the other hand, this is still unacceptable to potential buyers, they should turn elsewhere for their audio products.

I would love to have a set of measurements to tell me what the best of any product is. We wouldn't need audio reviews and we would know that we made the wisest purchase possible. In the real world, this doesn't exist. Most of the measurements I see really don't tell you how something sounds in your setup. I have seen countless reviews that received big-time accolades from the reviewer, but didn't measure particularly well. I have seen blogs from people trying to quantify what they are hearing, but reaching erroneous conclusions do to an inadequacy of their testing tools and conceptual logic of their testing methods.

I agree with stevebythebay's approach; one I also use. Find a reviewer that seems to match what you are experiencing with the products under review. Also, some of you folks need to lighten up and realize this is all about enjoying music. Some of the extreme anger I have seen in the past (not recently) is beyond the pale.

As for myself, I have always been a magnet for controversial products. Sit back and enjoy the ride. Disagree or agree; but try to have some fun in the process.

CG's picture

The audio business is probably similar to most businesses.

* Some products are great values. Others maybe not so much. Same is true for food at restaurants, houses, cars, clothes, and so on.

* Some vendors are not as reputable as we'd like. Same is true for (see above.)

* The aforementioned slipperiness applies not only to customers, but often to their suppliers and technology partners. Any idea how many designs are blatant rip-offs? (How many people jump at the chance to buy a Thumpmaster 2000 amplifier for 1/3 the price?) Same is true for (see above.)

* Based on the slipperiness of some vendors, would you really be inclined to publish all the details of your product design? (Think Mercedes hands out pamphlets at every F1 race to their competitors explaining their aero tweaks for the track?)

* Similarly, if Amalgamated Amplifiers published every detail, just how much would that affect their sales? Is it realistic to think that skeptics who won't even listen to their products would suddenly change their minds? Or, would it just be more napalm to throw on the internet flames?

* I recognize that since brick and mortar stores are on the endangered list, it's hard to listen to everything. No question it's a problem for consumers. But, do you really need to try each and every piece of gear to find something you might like? Do you need to upgrade every few months? (If so, maybe you should build your own!) BTW... How many B&M businesses disappeared because people would use them as a demo spot and then go buy for a few bucks cheaper off the internet?

* Realistically, if Amalgamated Amplifier published more details, how many people in the audience would have the background and necessary information to really assess what those details? How might one correlate the details with what you hear?

* 3.87

My opinion is that reviews are opinions. Hopefully useful opinions. But, they are no substitute for choosing something based on your own values, tastes, and other personal considerations.

jim tavegia's picture

we don't get to audition as much any more, but that is as much our fault as anything else as we too often buy based on "price" alone. I remember auditioning my very first system, all of it and deciding which pieces I would include over others...all at a dealer who was a huge help. It was not all about the price and I decided it was what I wanted to pay and so I bought it. I can't do that any more as their are no dealers within 50 miles of me any more.

I have little choice but to trust reviews and maybe find a dealer who allows returns. That is not really fair, but most accommodating for them...at a cost. If I buy one of their "open box specials" I get a deal and hopefully the person who returned it bought something else from them, but 10% or more was lost, plus the shipping expense and possibly more counting the time.

It is kind of sad we have turned all of this, including our music, into nothing more than a commodity constantly looking for the best deal, when the reality is the best deal is really what we enjoy the most. Often we pay more for that. That is fine for me. I would rather pay more for a high rez download or a full format CD than $9.99 for an MP3 download. If I find a great used CD more does often cost less that a poor MP3 download.

Now Apple wants labels to give away their music for free for 3 months to help them launch their music streaming service. What a joke. When did Apple become poor and can't fund their own start-up? Business strategy and customer relations have become strange bedfellows. And to think that the internet is the root of all of this.

CG's picture

Give that man a, ahh, CD! I agree 98%!

My only point of minor dispute is that I believe this began prior to the Internet.

A friend used to be in the radio/electronics business. He ran a store started by his grandfather and father. He swore that things went down the toilet with the advent of 1-800 toll free numbers. Guys used to call him on the phone, on his 800 number, and bend his ear for more than an hour asking a million questions. OK, part of doing business. They'd later call to say that they had bought the $1500 radio elsewhere because the second place was willing to knock $5 off my friend's already discounted price. On his 800 number.

PDQ.Bach's picture

Four score and one years ago, Karl Popper brought forth the proposition that scientific research should seek not proof, but rather falsifiability, as its lodestone, because any number of experiments cannot ever prove a theory about the material world in the same way that a mathematical theorem can be proved by pure logic; but a single confirmed experiment can disprove a theory.

The Logic of Scientific Discovery (originally: Logik der Forschung, which translates more aptly as Logic of Scientific Research) has been a classic, and a theoretical mainstay, ever since.

Perhaps proof-seekers should peruse it occasionally. (It is freely available on the internet.)

monetschemist's picture

Scientific method takes such a beating sometimes!

A blind or double-blind test conducted under a set of conditions X that does not detect a statistically significant difference between A and B has nothing whatsoever to say about any other kind of test conducted under a different set of conditions Y.

And in any case a blind or double-blind listening test is not guaranteed to eliminate observer bias. Such tests maximize observer uncertainty; how does that affect the ability of the observer to discriminate between A and B?

How does a random selection of participants assure us that we are getting a test sample that doesn't incorporate a built-in bias or outright inability to discriminate ("all I listen to is 256K AACs on my iPhone earbuds")? Or even incorporating a significant number of participants that DON'T WANT to detect a difference?

And as for measurements, if we apply a given set of tests to two different pieces of equipment and both measure somewhat similar in those tests, are we really justified in thinking "oh well they are going to sound the same"? Maybe some of us will notice no difference; others will notice a difference but not have a general preference one way or the other; and still others will notice the difference and strongly prefer one over the other. Or maybe all will agree that A sounds "better" than B and that the measurements taken don't actually tell us very much.

So, given that Michael and Steve take the time to carefully explain what they are hearing that pleases and displeases them as they review equipment, that is a great service to us all. And given the format of this site, those of us with different conclusions in relation to the same equipment are free to add their comments. On the other hand, people who respond to reviews with comments like "not enough measurements" or "the theory doesn't support your conclusions"... well, thanks. I'll bear that in mind.

monetschemist's picture

Scientific method takes such a beating sometimes!

A blind or double-blind test conducted under a set of conditions X that does not detect a statistically significant difference between A and B has nothing whatsoever to say about any other kind of test conducted under a different set of conditions Y.

And in any case a blind or double-blind listening test is not guaranteed to eliminate observer bias. Such tests maximize observer uncertainty; how does that affect the ability of the observer to discriminate between A and B?

How does a random selection of participants assure us that we are getting a test sample that doesn't incorporate a built-in bias or outright inability to discriminate ("all I listen to is 256K AACs on my iPhone earbuds")? Or even incorporating a significant number of participants that DON'T WANT to detect a difference?

And as for measurements, if we apply a given set of tests to two different pieces of equipment and both measure somewhat similar in those tests, are we really justified in thinking "oh well they are going to sound the same"? Maybe some of us will notice no difference; others will notice a difference but not have a general preference one way or the other; and still others will notice the difference and strongly prefer one over the other. Or maybe all will agree that A sounds "better" than B and that the measurements taken don't actually tell us very much.

So, given that Michael and Steve take the time to carefully explain what they are hearing that pleases and displeases them as they review equipment, that is a great service to us all. And given the format of this site, those of us with different conclusions in relation to the same equipment are free to add their comments. On the other hand, people who respond to reviews with comments like "not enough measurements" or "the theory doesn't support your conclusions"... well, thanks. I'll bear that in mind.

Rick Tomaszewicz's picture

...reading Malcolm Gladwell's "blink". In it he recounts the cola wars and other marketing strategies involving sensation transference. (For instance, if you change the colour of the soda can, people swear you've changed the formula of the soft drink.)

Guess what popped into mind? Our hobby. It's too awkward to admit how easily we're manipulated. We're nowhere near as objective or as smart as we think we are. (To say nothing of how adaptable our hearing is.) And, after spending a lot of money on a new piece of gear, or publicly and loudly extolling its virtues, it's tough to admit we were wrong.

However, to be fair, Gladwell does explain the weakness of the "sip test" during the cola wars. Real evaluation takes time. Hence the merits of extended auditioning of gear, rather than quick, blind ABX comparisons.

I suppose the fairest gear audition would be blind ABX comparisons extended over time and many different types of music. (That's the way a scientist testing a new drug or theory does it.) But, who's got time for that? Even professional reviewers would struggle setting up the conditions for blind ABX comparisons during extended gear auditions. But, if they did.....?

audiogeorge's picture

FYI the entire audio-music industry has gotten the polarity of most digital music wrong (approximately 92% of the time) to the detriment of composers, musicians, and music lovers' enjoyment and emotional involvement with the music they love.  Please see the information at the links:

http://www.AbsolutePolarity.com

and

http://www.PolarityGeorge.com

Respectfully submitted,

George S. Louis, Esq., CEO
Digital Systems & Solutions
President San Diego Audio Society
Website: www.AudioGeorge.com
Phone:  619-401-9876