Are Measurements Subjective?
If you frequent some of the hi-fi forums, you may have noticed a dust up over some measurements. No, not the endles Objective versus Subjective debate over the idea of measurements, rather there's some controversy surrounding some measurements generated by LH Labs, makers of the Geek Out DAC, comparing their DAC to a number of competitors. I've received numerous emails relating to these measurements since I reviewed two of the DACs represented in the LH Labs comparisons; the AudioQuest Dragonfly and the Meridian Explorer. "How can something that measures that poorly, sound so good?" being the standard question. "It doesn't." being my standard answer. Let me explain.
Thankfully, both the AudioQuest Dragonfly and Meridian Explorer were also reviewed in Stereophile where John Atkinson performed measurements of both DACs. I think its safe to say that JA's measurements are generally regarded as being industry standard both in the test equipment used and the associated results. The main issue with the measurements provided by LH Labs on their website is they do not jive with Stereophile's. Let's look at just one; Total Harmonic Distortion + Noise.
With the first sample of the DragonFly, a full-scale 24-bit signal actually clipped the bottom halves of the waveform with the computer's volume control set to its maximum, giving a THD+noise level of 3.8%. Backing off the control by one click (–0.17dB) reduced the THD to 2.14%, by a second click (–0.34dB) to 0.627%, and by a third click (–0.51dB) to 0.054%, below which the THD+N percentage plateaued. The second sample didn't clip with a 0dBFS signal at maximum volume, and the THD+N was 0.041% rather than 3.8%. According to Gordon Rankin, the volume control offers 64 steps of less than 1dB to –60dB and then mute (–100dB), but he used only 60 of those steps in the DragonFly, as the top four steps suffered from significant clipping into high impedances. "In retrospect," he wrote of the first sample, "I could have changed the maximum volume down a few more steps and then this would not have been an issue."So the older Dragonfly, not the newer V 1.2, in its most recent incarnation, 1.0c, had a THD+N of 0.041% according to Stereophile. That is a striking difference from the > 5% shown in the LH Labs graph which is even higher than the original Dragonfly, 1.0, that measured 3.8% according to Stereophile. I've also been told by AudioQuest that the improved version 1.0c was implemented as soon as this problem was discovered and they replaced any model 1.0 with a 1.0c free of charge on request. Why are these THD+N numbers a big deal? According to LH Labs:
Total Harmonic Distortion + Noise represents how well the device will play back "faithfully" to the original music. It generally will give you a good idea about how comfortable and natural you will feel when you listen to your music. We list measurements in both decibels and percentages here. With Dragonfly at maximum output volume, you can see 5% distortion and -25dB THD+N here. This will usually cause the music to sound unnatural.According to my ears, I'd go with Stereophile's findings (see review). It's worth noting that LH Labs measured their own Geek Out at 0.0027/0.0025 THD+N.
I contacted iFi regarding this same issue since their DAC, the iFi iDSD, was also included in the LH Labs post. The LH Labs measurements show a THD+N of 0.152% while iFi claims a THD+N measurement of 0.02%. iFi also took issue with other LH Labs measurements of their iDSD DAC including Signal to Noise Ratio and the FFT Spectrum offering their own results which do not match those on the LH Labs website and in each instance the LH Labs results are worse than those from iFi. Since Stereophile has not performed measurements on this iFi DAC, I offer their measurements as rebuttal.
iFi also took exception with the LH Labs Output Voltage measurements (1.481/1.482Vrms) of their iDSD DAC. From iFi:
The nano iDSD's normal operating voltage output is 1.65v. This is irrefutable.There are other disparities between the LH Labs measurements and the Stereophile measurements. For example, if we look at the LH Labs measurements for the Dragonfly's Maximum Output Voltage, we are shown 2.106 and 2.105Vrms, where the Stereophile measurements came in at 1.84Vrms. Here's what Gordon Rankin, who provided the circuit design for the Dragonfly, had to say about the LH Labs numbers:
Yet Competitor X's measured output voltage of 1.48v. The DAC operates directly from the Battery and so when the battery voltage drops near to the full discharge, the output voltage drops in-line with the decline of the battery voltage. So from this, one can work back that the on board battery was running near empty with the red-light warning of battery low on.
The iDSD shuts down at 3.7V and the battery normally is at 4.1V.
Battery drop - 3.7V/4.1V = 90.24%
Signal drop - 1.48V/1.65V = 89.7%
As can be seen the lower, measured signal level by Competitor X is very close to what one can expect based on the Battery Voltage drop from Full to shutdown.
When the battery is so low, its measurements of course are no longer meaningful.
We could only attain this level of distortion when we raised the signal to 110%.AudioQuest also took issue with the LH Labs 5kHz Square Wave measurements and sent along their own results.
I also contacted Gavin Fish of LH Labs and asked him about the apparent discrepancies between their measurements, Stereophile's, AudioQuest's, and iFi's. I received this reply from Larry Ho, LH Labs Lead Engineer:
Few simple facts here...What does all of this add up to? I'd say we need to fully understand the methods behind the measurements in order to fully understand their outcome. If this is not possible, and I'm sure I don't understand the methods behind all of these measurements, its also not possible to make an informed judgement based on them.
1. Dragonfly do have clipping in -0dBFS. John Atkinson also mentioned in his review. He needs to dial down a few clicks to get normal result.
IF there is a need to do a special favorable treatment to Dragonfly. I could do that, actually I did it in our first test. The THD+N is still around -70dB or so.
Or if I do that, Meridian and iFi will jump out say: Not Fair? ;-)
2. JTEST is only a simulation of the simplest way jitter by toggle LSB... The JTEST doesn't have the 32 bit version yet, at least to my knowledge so I decide to skip it. Also, there are many professions claim that test is not accurate.
3. As an US manufacturer, I won't have any special favor for an UK vendor. We did Explore, iDSD and Dragonfly at the same time. Meridian Explorer is normal there... Machine won't discriminate.
4. We have the most advanced 1G bandwidth scope on hand. With the distortion I saw on Dragonfly's output wave, we really don't need the big gun scope there.... Again, even if Audio Precision is NOT good enough, and don't need to see Geek out's test result. Why iDSD is performing much better than Dragonfly? Do I need to favor iDSD? :-)
One friend from Google is sending me 3 more good DACs, including Herus, micro streamer and D3.
I'm gonna publish that with the original test results.
Don't worry. Truth is the daughter of the time.