The Trouble with Listening

Reader Rich Davis asked in reference to the Meridian Explorer:
Is it possible to post measurement test results? I'm interested in knowing the dynamic range, jitter, etc. etc. for each bit rate. Just so I can compare [with] other DACs?

16 bit sounds great, but I have [downloaded] some 24/96 and 24/192 of the same recordings (only they were mastered at later dates and most likely with different equipment) and the 24 bit recordings sound incredible. I HIGHLY suggest going to places like [HDtracks] and obtaining later remastered versions of any recordings you already have.

Some of the recordings are just wonderful sounding on this DAC at 24 bit.

We are currently not equipped to provide measurements at AudioStream but this is something we may consider in the future. Part of the reason for this decision is budgetary, part logistic, and part pragmatic. It seemed to make sense to start with subjective reviews since its through listening that we determine the real value of a piece of hi-hi gear. But let me try to flesh out this position in more detail since this question of measurements and subjective reviews has come up before.

I would start by asking a question—Is listening sufficient to determine the worth of an audio component? Or to put it another way—Is listening sufficient in determining what we enjoy listening to? Well if we put it that way, of course it is. But is listening for enjoyment good enough?

People have ideas and you really can't blame them. Ideas, in general, are good things to have. I say in general because we can all point to some ideas that are in fact not good like the notion of racial supremacy for one easy example. The ultimate value of an idea is determined when that idea is put into practice, to state the plainly obvious. The question for us is, and by us I mean people interested in listening to music on the hi-fi, are the ideas about what comprises listening more important than listening itself? Are objective goals more important than experience?

Let's get specific. Can we enjoy a hi-fi component even if it does not measure well? The answer is clearly yes based on the evidence at hand, which exists mainly because of the work John Atkinson and the rest of the team at Stereophile do each month. In their formal reviews, as opposed to the columns, Stereophile performs measurements of each component under review in addition to the subjective listening impressions offered by the reviewer and there have been cases, most recently with the Croft Acoustics Phono Integrated amplifier (see review), where the listening impressions and measurements diverge. Instances where a given component measures poorly but sounds great.

What we're really positing is the idea of accuracy versus enjoyment. For example, let's say I own that Croft integrated amplifier and I have been listening to and enjoying music through it for months. Along come the measurements in Stereophile which includes these comments from John Atkinson, "However, I must admit to being puzzled why Art Dudley liked the sound of the Croft Phono Integrated as much as he did. To me, it seems, at best, inadequately engineered, and at its worst—that nonflat RIAA response, the high levels of harmonic and intermodulation distortion—just plain inadequate."

Does this fact erase all of the enjoyment I've already partaken in courtesy of my Croft amp? Of course not. You can't undo past enjoyment(s). Will this information hinder future enjoyment? Perhaps. It depends on how much value you place in measurements. Of course what we're really talking about are the goals of hi-fi and there are certainly more than one but the goal of enjoying listening to music is, I would suggest, paramount to all others.

In terms of ideas that would like to overwhelm experience, there are people who suggest that 320kbps lossy files are good enough and indistinguishable from higher resolutions. Others offer CD-quality as the best we need. Others that 24/96 is the absolute maximum fidelity that makes sense, while others still point to higher sample rates or DSD as being the quality we should strive for. Then we have people who suggest that all bit-perfect media player software sounds the same. With hardware, we have people who suggest all properly designed DACs will sound the same, all cables sound the same, and on and on. And on. Endlessly. I'm sure we can find someone somewhere who feels that everything everywhere sounds the same.

I will also point out that the goals of the equipment manufacturer, where measurements and ideas, among other things, are critical to the design and manufacturing process, differ from those of the listener. As listeners we are obviously not involved in this process, we are concerned with their results. That said, some people enjoy thinking like a designer or manufacturer which is certainly another aspect of the hobby but we should not let this curiosity cloud our main goal, our one true job as listeners, which is simply finding hi-fi components that we enjoy listening to music through (the wrinkle in this simple equation is this enjoyment is not restricted to qualities relating to sound alone but that's another story).

The trouble with listening is its imperfect. Listening, especially listening for enjoyment, is not governed by a given set of rules. What's worse, in terms of hi-fi listening, is the fact that enjoyment does not always directly correlate to those things we measure. If listening for pleasure were governed by measurements alone, we could simply buy products that measure well and we could make our selections without ever listening, which is something I would not recommend doing since that's exactly what you're going to do with this thing once you get it home. You're going to listen to it. You're not going to measure it.

Where does that leave us? I'd say it leaves us with the best tools for the job at hand being always at hand. Which is to say that the best way to determine what we prefer listening to is to listen. Everything else, including measurements and value, follow from there.

bigrasshopper's picture

I'm sure there are many reasons why someone reads product reviews of one sort or another.  While some are on the edge their seats, expectantly seeking that one important breakthrough bit of news that may make them part with their cash, others, having been prompted to put out in the past are now somewhat sated.  Yet once we start something we have a tendency to continue when we have been rewarded.

  As a rewarded reader of audio reviews, I would naturally like more to read, more tantalizingly ambiguous facts to catch my eye as I stroll more leisurely now through the curiosities of the reviewers landscape.  Surely I must have missed something the first time around, something foundational, I'm sure of it.  In fact my naïveté is pointed out to me every time I read a review by a seasoned professional, and I've read a lot of em.  Some have probably reviewed more gear than articles I've read.  That's got to be "Like singing the same old song and twisting the words in a different way"- I like the way Neil wrings the last bit of meaning out that trailing wire of a syllable.   In a pinch I'll I try to reread an old article with a different perspective to see if it offers up some new sense of insight.  "The professionals" do this by pushing music tracks through the gear their sent.  That is surely  a more powerful drug than my imagination.  But their is a similarity.  Understanding this, I can forgive the writers for the damage they inflict to their audience in their attempt to give us the benefit of their unending experience.  Especially when we are urgently warned to listen, at all costs... but to read ? How important is that?  It's gotta be right up there... no ?  So by all means give us more bits to brood over,  maybe then after a good bout of brooding over two dimensional pictures of something, I'll pluck up the courage to ask a dozen retailers to send me their products and redeem myself of buying by measurement, with the consent of John Atkins' apparition of course.  Well, one could do worse.

You see how fickle readers can be.  It wasn't more than a dozen weeks ago that I was berating your work for lack of measurements.  Seriously though, thanks for the thoughts, they have me reassessing my own.  For the present there are no lack of standards here, just measurements.  When you can confirm a billionth of a second

jitter, by all means let me know.

Martin Osborne's picture

of a piece of equipment based on reading technical spec's or measurements and what it will do to the music I love. 

Naturally I enjoy the way Audiostream reviews, and I hope it doesn't change.










Michael Lavorgna's picture

I have no plans to change the way we review.

Tyll Hertsens's picture

Hi Michael,

Noticed the FaceBook chatter about this post and it inspired me to chime in on the subject. 

The listening experience is in a completely different domain than objective measurements and, while they're related, the map is not the territory. If you can only have one, a report about your actual listening experience is far more relevant to my potential experience with the product than objective measurements.

My full response here.

Michael Lavorgna's picture

I just read your full response and thank you for that. Very well put and thought provoking especially this part:

Subjective reviews are like scouting reports. They tell you about the territory of the actual listening experience with a particular product. Objective measurements, while informative and related, are relatively mute when it comes to describing that.

I'd add that the more terrain you cover, ideally the more you enhance your skills at scouting.

firedog55's picture

Quite a setup Frank has in the picture, especially for the time (looks like the late 60's early 70's to me from the TV and the two audio components toward the right). Can anyone identify the equipment? Are those Klipsch speakers?

Audreal's picture

The Chairman appears to be sporting a 3 channel system - Presto 825 / 3 track, McIntosh C22 preamp, Fisher R200 tuner, 15" coax speakers (possibly Altec Duplex 604?).

Azteca X's picture

...but your job involves a large number of DACs and other devices going in and out of your home.  For the average reader, we will not easily part with $500, 1000, 5000, etc.  So yes, listening is great.  But I haven't met you.  I've read tons of your work, sure, but I've never sat in your listening room and heard what you think is good sound.  While I don't anticipate we would diverge greatly on what sounds good, and you do offer helpfully detailed subjective impressions, all of that takes trust.  I do trust you for the most part, Michael, but measurements let us establish some common criteria.  If I know I'm someone who generally enjoys the sound of solid state equipment with extremely low THD and high SNR, or I enjoy gear with a slightly tubey/euphonic sound, etc. then I can feel more confident in the lead-up to ordering that piece of gear.  Many of us might only buy one thing a year, or every two or three.  So advising us to listen is good but when a tool exists (measurements, as done by John Atkinson and Tyll, to refer to your Interlink brethren) that can help us make that decision with confidence, why would we not want it?  I don't jump to measurements when I read Stereophile or Inner Fidelity reviews, but the measurements can tip me off as to whether it will probably tickle my fancy or have a potentially troubling aspect.

 As listeners we are obviously not involved in this process, we are concerned with their results

And what's the one universal way to evaluate those results?  Measurements.  There are hundreds of devices out there vying for our attention, so just listening isn't going to help us sort through that pile.

Michael Lavorgna's picture

I don't discount the usefulness of measurements but as I said in my response to Archimago below, I question how many people will actually find them useful.

It would help for me to understand what measurements you'd like to see for the gear we review, let's take DACs as an easy example. What measurements do you find most useful and how do they relate, in your experience, to what you hear?

Azteca X's picture

There are the fairly common ones:  

  • Frequency response (definitely 20-20k, preferably higher since a lot of them tout 192 and DSD support)
  • Total Harmonic Distortion
  • Jitter on each input (within reason - but USB, coax, optical would be great)
  • Noise floor
  • Signal to Noise Ratio

As it turns out, Archimago also does use one great measurement technique that would be useful involving the "least significant bit."  It's important to know if a 24-bit DAC can actually resolve 16 bits or more.
I'll link you first to Stereophile, where he got the idea:

And his detailed, example-laden post:

Do let me know if you have any questions, but all that seems like pretty straightforward stuff to let us know if the MegaResolutionSonataTransformer 5000 can shoot us the signal to a reasonable specification.  Whether it can give us the music - well, that's where your expertise comes into play!

Thanks for listening.  Your responsiveness on this site is one thing that has made me a fan.

Michael Lavorgna's picture

Thanks for taking the time to provide this information.


Azteca X's picture

Of course I just remembered a few this morning that would also be useful (though I'd say the ones I previously mentioned are the "must-haves").

Intermodulation distortion.  "Another test in which there may be a reasonably strong correlation with perceived sound quality" - ". Processors or players with poor analog output stages will also do badly on this test."
"There appears to be some correlation between a lack of intermodulation products and a lack of glare and hard textures."


Impulse response.


Square wave performance has less direct impact on perceptible sound but does show a bit about the time domain.

Channel separation gives us some clues about the engineering though JA does note that "Although there is no correlation between low crosstalk and soundstage width (an intuitive link), high channel separation indicates good overall engineering (footnote 4)."

Linearity takes us back to figuring out how many usable bits we actually have and determines the accuracy of what you're hearing volume-wise.  Noise floor could -120dB, for example, but if once you get to -90 it starts adding or subtracting a few decibels you could end up with an audible difference.  Low-level linearity means it is just as solid with the quiet, delicate stuff as with the in-your-face.  I imagine budget components measure worse on this one.

Have I exhausted you yet? wink The thing is, the more of the tests you include, the clearer our understanding of the device grows.  So what you do or don't include will be up to you, but I'd think a full barrage of tests would be great and necessary for everything in the $1000+ category.  At that price point, you should know whether or not your purchase is well-engineered.  Cheaper stuff (Dragonfly, for example) will not perform on the same level but looking at Sterophile's measurements from their review shows that on a technical level it really is doing well.  In those categories where competition is so fierce, measurements helps weed out the pretenders.

I don't know precisely how the Source Interlink family works, but I'm sure between the folks at Stereophile, Inner Fidelity and Sound & Vision folks you could get some info or files that could save you lots of time.

deckeda's picture

Measurements are themselves a "tool," not the answer. If a reviewer hears something weird, measurements may or may not hint as to why. If they don't reveal the "why," I don't get bothered by that -- the listener's ears are still what matters most. I believe JA has espoused this same intent of what he provides for Stereophile more than once, so those of you pining for his kind of measurements on this site, take note!

It's easy to understand the failings of human hearing and agree when "we can't hear differences" because that's (IMO) largely about perception and caring, not so much biological acuity, assuming a normally healthy person without recognized (i.e. measureable, ha!) hearing loss.

But as audiophiles, i.e. people who give a shit, we should all understand the corrolary: that what can be heard can't always be measured. That's why competing technologies and design philosophies exist. Designers try new things not to rip you off necessarily, but because they listen and need to compete in the marketplace. And when YOU listen, you'll be able to tell the difference in a thing's worth to you. Simple, really, if you want it to be.

Archimago's picture

Good topic, Michael.

As one in the "more objective" camp, I'm of the opinion that technical accuracy is the foundation of which "good sound" derives at this stage of scientific maturity. As Azteca observed above, the objective measurements act as a "common criteria" for all of us listening around the world. What is technically accurate is generally well defined based on the engineering of these devices - human hands designed these things based on the foundation of math and physics, the devices are not like some new discovery waiting to be characterized with no clear history of how the universe decided to put it together.

Your example of the John Atkinson quote is appropriate and I draw a different conclusion. JA has every right to be critical of "inadequate engineering". The fact, that Art Dudley still thinks it's good doesn't seem all that surprising since the ear/brain has a wide tolerance for what is good enough... I'm sure many of us played with equalizers that bump the bass or accentuate the treble in our rooms. So what if he liked the non-flat RIAA curve? That however is meaningless for those of us who perhaps might prefer another configuation of the EQ or desire a flat response because we run other forms of ear/room "correction". Knowing how the RIAA curve is reproduced therefore allows us to conceptualize the potential sound much more than a single listener's opinion. Who knows how his room, or speakers sound. I wouldn't know if I he may have been consciously or subconsciously affected by the brand, price, ergonomics, salesman, etc. Finally, who knows what his auditory acuity is like... My dad has great hearing too, having listened to much gear in his life and built a few himself - but clearly his hearing is limited to below 11kHz these days and I would not totally agree with his opinon knowing this essential fact no matter how much I might respect him or his experience with audio gear.

Show me an example of the opposite these days... An example where a piece of equipment was picked to pieces by the subjective reviewer and sounds bad by a number of others but detailed modern objective analysis (like the AP devices used by Stereophile - we're not talking about measurement suites done in the 60's and 70's!) was "fooled" and demonstrates fantastic dynamic range, frequency response, low harmonic distortion, low jitter, etc... I would certainly re-evaluate my stance if this were to happen. Until then, I maintain that objective measurements are the pre-requisite for which the subjective evaluation is used for confirmation & light reading entertainment.

By the way, since this site reviews mostly line-level gear like DACs, it's even easier to get measurements done than the Stereophile speaker reviews or the dummy-head measurements from InnerFidelity. Would love to see some test results from the likes of Synergistic Research products.

Michael Lavorgna's picture

We have not taken the intended audience into account. The usefulness of subjective reviews will vary based on the reader as will the usefulness of measurements. At present, AudioStream is obviously geared toward people who enjoy, for whatever reason, reading subjective reviews. While this doesn't discount the usefulness of measurements, I'll stick to my main point which is nothing beats listening when deciding what we prefer listening to.

On your take on the Stereophile example, its my experience that people read Art Dudley to read Art Dudley. He is not simply a set of ears attached to a brain that augments a set of measurements. That's one reason, I'd imagine, why Art also has his "Listening" column in Stereophile where the gear he talks about is not measured.

Archimago's picture

Seems like I wrote a rebuttal on this almost a month ago:

You are of course free to write to whichever segment of audiophiles you want. If that's the "main issue" for you, fine. So long as we can all hopefully appreciate how fraught with uncertainty opinions based just on one man/woman's experience is.

I do believe that some of us do like to engage in discussion on "facts" where we can find them. Measurements provides this substance for which to test our perceptions and understand what we like and *why*. A good example I think is the post you just put up on SOTM Sample Rate Conversion. I asked in the comments what type of digital filter was used to perform the upsampling. Is SRC1 different from SRC2/3/4? If so, how? Are there level changes? Does one setting use minimal phase filtering versus just a straightforward linear upsampling? Do they impose frequency response restrictions like in some of the "apodizing" filters out there? Measurements would allow you to easily answer the questions and give to your reads a level of insight not available otherwise (unless this is all in the product's manual). This would help them understand what makes a difference and what doesn't.

I know this may be too much for some folks, but it's answering questions like these that *advance* the hobby by allowing critical evaluation of gear and providing the opportunity to educate the audiophile - maybe even in turn the manufacturers to know what the public demands and what technical expertise to focus on.

Finally, one general comment about the "main issue". Isn't the main goal to get more people interested in the hobby? I believe it's not just about "what" to buy, but also why we buy. Over the years, so many have opined that the audiophile hobby is "dying". I believe a large part of this is the fact that "we" accept so much subjectivity almost without question that "we" become dissociated from reality. "We" do this to ourselves by accepting pseudoscientific spewage from questionable companies and end up on the fringe... Yeah, those guys over there with the cable risers, foil tape, tuning "beaks", green CDs, $5000 power cables - eyes rolling.

Michael Lavorgna's picture

I mean, its no secret what our approach is.

But just as you say I am free to are you (and even link to what you write ;-). One difference between us however is I don't visit your site and I don't comment about my feelings concerning what it is you do. There would be no point in doing so as we clearly see things differently.

Which is all fine by me. What I don't get is this whole missionary zeal to protect people, according to you unknowing people that you need to educate, from buying stuff that they think will add to their enjoyment. And that's exactly what we're talking about. A hobby concerned with the enjoyment of listening to music. 

If there's anything to roll one's eyes over its people who take themselves way too seriously.

Archimago's picture

I'm here because I choose to be curious as to what people think about the hobby and the internet is a great place to share ideas and participate in debates. If you have an argument with my points then let's discourse. I'm merely responding to your point of view since the forum is open. Also note that I did not put in any link initially but responded with I believe some reasonable thoughts in a respectable fashion.

? Missionary zeal - as if I achieve rewards in heaven or money in my pocket. Just thought it'd be nice for readers to consider other points. Subjectivists have responded to my blog posts and we've chatted... No problem...




Michael Lavorgna's picture

...that you are here to espouse your beliefs which I'd summarize with this quote of yours from your initial comment:

...I'm of the opinion that technical accuracy is the foundation of which "good sound" derives at this stage of scientific maturity.

And here's what I said in the piece itself:

Of course what we're really talking about are the goals of hi-fi and there are certainly more than one but the goal of enjoying listening to music is, I would suggest, paramount to all others.

The question is does your "technical accuracy" equate to my "enjoyment"? In my experience the answer is no. That's why we always have subjective reviews accompanying measurements and why there's never been a publication, in print or on the web, that consists of measurements alone.

But as I also said in the first paragraph above:

We are currently not equipped to provide measurements at AudioStream but this is something we may consider in the future. Part of the reason for this decision is budgetary, part logistic, and part pragmatic. It seemed to make sense to start with subjective reviews since its through listening that we determine the real value of a piece of hi-hi gear.

The fact that we do not provide measurements does not mean we do not value them rather it appears that our differences are rooted in how much value we place on them versus listening.

Steven Plaskin's picture

I obviously share Michael's views. I don't know of any set of measurements that totally explains how one DAC sounds compared to another. I would love to perform measurements of the equipment I review. The combination of measurements and careful listening evaluation as done in Stereophile should be a goal we strive for here at AudioStream. But if I had to select which evaluation technique was more valuable, I would select listening.

Bill B's picture

I come to Audiostream to read good informative reviews (and other topics).  Not because I am looking for a solely "subjective" review site.  I appreciate objective and subjective approaches, and they will always mix, anyway, even if we try to separate them.  Just like measurements have limitations (can't measure every possible aspect of things), so do people have limitations (hearing differences, placebo effects, etc.).  The two approaches complement each other.  E.g., I'm GLAD to know the Croft amp sounds good, AND that it has measured deficiencies.  That all tells me something useful 

Listening pleasure is what I'm after; but in reading reviews I need helpful info - which includes both subj & obj.

Michael Lavorgna's picture

I also feel that subjective reviews and measurements are complimentary and offering both is ideally the way to go.

bigrasshopper's picture

If or when this site begins to add  a measure of fact to their impressions, the format and presentation would certainly have to change, dramatically I would think.

When readers can comment on listening impressions and test results in one breath, that does change the nature of the process.   Unless the reviewer is also the tester who is going to respond to purely technical inquiries ?  If they are not essentially separate reviews how can they be integrated in a meaningful way on an interactive forum ?  Does the tester also have to be a participant ? It certainly seems to present more challenges than a fixed publication.

If Atkinson already has a full schedule who's going to do these ?  To what degree will the results be commented on by the tester or by the reviewer.   What sort of exposure to results will the reviewer have and when, will Micheal and Steve feel an impulse to sit up straighter in their chairs ?  Or be more guarded in their conclusions.  Or have to change a fundamental attitude about reviewing.  There aren't a lot of examples of people that have the breadth of experience to add meaningful commentary to a host of technical results.  Of course I'm not suggesting that it shouldn't  be tried.  If for instance Steve is curious about measuring, perhaps their are test he could do.  If he can explain the relevance and make it worthwhile, why not, it's a beginning.  My feeling is the more information the better.  I'm a reader of gear reviews, of course I want more.

As a casual reader, I am quite happy that this site exists at all.  I enjoy trying to figure reviewers out by what they choose to say and how they say it.  It's really as much about people as gear if not more.  I like to be up to date on what's current and what's relevant to critical listeners and enthusiasts.  When I'm ready to actually make a purchase decision I would like to believe that my choice can satisfy on multiple meaningful levels.  I'd like to believe that I can trust my ears, but without considerable more exposure, I'll remain  cautious, and look to try and sum all of the impressions I can gather, and that definitely includes tests.  

Currently I can't find any tests of this sites favorite two Dacs.  I will wait.

If you looking to collect a short list of what is important, isn't their a short list of all the usual suspects.  We want to test inherent resolution and the presence of things that interfere with it.  Right ?  Do you like it and why.

Give testing a test.  Let it evolve.  If my first comment seems cynical and more than a little ambiguous...well I sometimes weary of the compulsion to be a buyer or a seller.

Michael Lavorgna's picture

I don't see the inclusion of measurements changing the way we approach our reviews and envision incorporating them the same way that Stereophile does on their website, as a separate page attached to the review. But that's certainly the easy part.

As I said up front, the issues of logistics, i.e. where and by whom will gear be measured, is the more difficult detail to deal with. I like the way Stereophile handles measurements, as something separate from the review as well as the fact that the review is written without prior knowledge of the measurement results. This obviously means that there has to be someone else doing the measurements which also leads to budgetary concerns.

What would be helpful at this stage, is for those people interested in seeing measurements on AudioStream to provide some examples of exactly what measurements they find most useful and ideally how these measurements relate to what they hear.

junker's picture

I'd like to see more comparative review, and some shoot-outs with direct comparisons, rankings, and scores. If you want to do it purely based on listening, fine, and if you want it based on a mix of objective and subjective criteria I would be interested in that as well.

Louis Motek's picture

The single most influential factor for analog to digital conversion quality is the Jitter amount at the very point and time of conversion. All the development of quality in digital audio from the early 80's until today has to do with this very aspect. Apart from "nice extras" like wider bandwidth and bit depth, still, Jitter is the #1 thing that could really tell you that one solution is better than another.


Problem is, nobody can measure Jitter extremely accurately, nor extremely authoritatively. This is because the measuring equipment itself suffers under the exact same problem of Jitter. Obtaining extremely low Jitter measurement is possible, but requires schooling and very expensive equipment. Shamelessly, many publish "jitter numbers" (made up from thin air) but never show the methodology to arrive at such conclusions nor the the jitter spectrum which underlies the mere jitter deviation amount and is actually more important when desiring to ascertain some sort of insight into the possible coloration of sound.


Why don't you just hire a specialist? I recommend John Miles, who recently developed a (slightly less expensive ... only about $30k) phase noise test probe, currently marketed by Symmetricom.


You will not get more accurate measurements than that unless your lab money bag is very large indeed (which means you are a telecoms company on whose technology rests the backbone of all modern communications).


Louis Motek