Thread: Meyer Moran result debunked - again

Posts: 111
Page: 1 2 3 4 5 6 ... 12 next

Post by canonical September 6, 2010 (1 of 111)
As discussed in the Meyer Moran thread:

/showthread/42987/42987/y#42987

... it was self-evident that the Meyer Moran paper was fatally flawed before it began, because, amongst many factors:

(a) The authors used CD resolution recordings to test the SACD format (which is almost absurdly incompetent)

(b) In a test to see if human beings can discern hi-res formats, the correct scientific methodology is to drop those who cannot hear difference from the study, and then repeatedly test the golden ears that can hear a difference, to check if the result is reproducible. By contrast, Meyer Moran average out the successes against the failures, effectively wiping out the successes.
--

There is now a new AES paper which finds that even lowly 88.2kHz is discernible from 44.2kHz:


TITLE: Sampling Rate Discrimination: 44.1 kHz vs. 88.2 kHz
AUTHORS: Pras and Guastavino, McGill University

http://www.aes.org/e-lib/browse.cfm?elib=15398


Abstract (May 2010)
------------------

"Sixteen expert listeners were asked to compare 3 versions (44.1kHz, 88.2kHz and the 88.2kHz version down-sampled to 44.1kHz) of 5 musical excerpts in a blind ABX task. Overall, participants were able to discriminate between files recorded at 88.2kHz and their 44.1kHz down-sampled version. Furthermore, for the orchestral excerpt, they were able to discriminate between files recorded at 88.2kHz and files recorded at 44.1kHz."

This result entirely contradicts the conclusion of the Meyer Moran paper.

Post by RWetmore September 6, 2010 (2 of 111)
That's most interesting. I would like to hear David Moran's comments on it.

Post by flyingdutchman September 6, 2010 (3 of 111)
16 "experts." I wonder who those "experts" are.

Post by Polly Nomial September 6, 2010 (4 of 111)
Woohoo!

Post by Fitzcaraldo215 September 6, 2010 (5 of 111)
canonical said:


There is now a new AES paper which finds that even lowly 88.2kHz is discernible from 44.2kHz:


TITLE: Sampling Rate Discrimination: 44.1 kHz vs. 88.2 kHz
AUTHORS: Pras and Guastavino, McGill University

http://www.aes.org/e-lib/browse.cfm?elib=15398


Abstract (May 2010)
------------------

"Sixteen expert listeners were asked to compare 3 versions (44.1kHz, 88.2kHz and the 88.2kHz version down-sampled to 44.1kHz) of 5 musical excerpts in a blind ABX task. Overall, participants were able to discriminate between files recorded at 88.2kHz and their 44.1kHz down-sampled version. Furthermore, for the orchestral excerpt, they were able to discriminate between files recorded at 88.2kHz and files recorded at 44.1kHz."

This result entirely contradicts the conclusion of the Meyer Moran paper.

Actually, I was beginning to wonder if ABX testing had ever under any circumstances revealed any statistically significant sonic differences between something and anything else. All I have ever seen have been the "no difference" findings trumpeted by the likes of Meyer-Moran and numerous others for a whole range of audio equipment and formats.

Post by audioholik September 7, 2010 (6 of 111)
canonical said:

This result entirely contradicts the conclusion of the Meyer Moran paper.

I'm not surprised.

Post by Fitzcaraldo215 September 8, 2010 (7 of 111)
Arnaldo said:

One intriguing point about the Pras/Guastavino AES paper is why they chose to compare 44.1kHz to just 88.2kHz. Higher sampling rates such as 96kHz, 176.4kHz, 352.8kHz (DXD), and pure DSD, have been more widely used by recording companies nowadays. Either way, with the participants in the tests being even "able to discriminate between files recorded at 88.2kHz and files recorded at 44.1kHz," a comparison with higher resolutions might have been somehow redundant.

There is an older AES paper about an extensive study in Germany of DSD vs. 192K PCM using very high caliber equipment. As is typical of ABX tests, no statistically significant difference was observed.

Post by lowie September 8, 2010 (8 of 111)
Great news! Finally, I can start enjoying my SACD collection again.

Post by eesau September 8, 2010 (9 of 111)
Hi,

this is an interesting paper but the authors need to carry our further research before the results can be accepted as a scientific fact.

There are some interesting results in the paper:

Using orchestral music material, 13 participants out of 16 were statistically able to detect

+ native 88.4kHz and native 44.1kHz from each other

However, they were not able to detect

- native 88.4kHz from the same signal down sampled to 44.1kHz
- nor native 44.1kHz from the 88.4kHz down sampled to 44.1kHz

There existed no statistically relevant detection using ”cymbals” or ”violin” material.

With ”guitar” and ”voice”, these participants were able to tell

+ native 88.4kHz from the same signal down sampled to 44.1kHz

but with this material, they were not able to tell

- native 88.4kHz and native 44.1kHz from each other
- nor native 44.1kHz from the 88.4kHz down sampled to 44.1kHz


3 participants out of 16 provided with reverse results that (may) indicate that they detected a difference but could not really tell which was which …

These participants doing detection in reverse were able to non-detect

+ native 44.1kHz from the 88.4kHz down sampled to 44.1kHz using ”guitar” material (but nothing else statistically relevant)

and further with the ”violin” material they were able to non-detect both

+ native 88.4kHz and native 44.1kHz from each other and
+ native 44.1kHz from the 88.4kHz down sampled to 44.1kHz

So, don't you think that the results are somewhat contradictory … and this, again, shows that AES is not very discriminative when accepting papers to their conventions.

24-bit 88.4kHz and 44.1kHz sample rates were used possibly because 24/88.4kHz is considered to be ”high resolution audio” and sample rate conversion is very straight forward. They could have used 48kHz and 96kHz as well but would you have been so interested in the results?

ABX is a scientific method to compare results and AES and McGill University are trying to be scientific. For some reason, ABX does not seem to work for audiophile use ….

All participants in the tests had musical training and a professional or scientific relation to digital audio with an average age of 30 years. They were describing high definition audio as having a better spatial reproduction, high frequency richness, precision or fullness …

Esa

PS. I'm not sure if Meyer Moran results have really been debunked ....

Post by Fitzcaraldo215 September 8, 2010 (10 of 111)
eesau said:

Hi,


So, don't you think that the results are somewhat contradictory … and this, again, shows that AES is not very discriminative when accepting papers to their conventions.


ABX is a scientific method to compare results and AES and McGill University are trying to be scientific. For some reason, ABX does not seem to work for audiophile use ….

All participants in the tests had musical training and a professional or scientific relation to digital audio with an average age of 30 years. They were describing high definition audio as having a better spatial reproduction, high frequency richness, precision or fullness …

Esa

PS. I'm not sure if Meyer Moran results have really been debunked ....

I think there are numerous problems with ABX and how some ABX tests are conducted. I have seen some reports of testing of a group together, for example. That should be a no-no, because some subjects may influence others in the same room non-verbally.

But, I think the really big problem is the small differences that generally tend to exist in audio today, assuming decent equipment. Not everyone is attuned to them, and even then they can often be fooled enough to not pass the "statistically significant" hurdle, which is quite demanding.

I almost invariably find, even with things as different sounding as speakers, that only certain specific passages clearly reveal the differences, if they exist. So, a lot of listening time is frequently wasted listening to stuff that sounds pretty much the same, until you get to the right parts. Much of the listening time needs to be spent spotting and selecting those tell-tale passages. Then, it's a matter perhaps of repeating them often enough to be sure.

Then, there is always the short acoustic memory problem, even when listening to 5-minute passages back to back. They may include 4 minutes of indistinguishable material and 1 minute of material that truly and clearly does exhibit the difference: a flute entrance here, a string crescendo there, etc. I think many test subjects just do not listen that way. So, they are more influenced by the long passages that sound pretty much the same. And, fatigue with the repetitions forces a premature "decision". That's just a problem of using repeated music as the source for any test.

I do not think all ABX tests are conducted where the inputs are perfectly time synchronized so that the subject could switch quickly and repeatedly on the fly back and forth between A, B and X, as well as skipping back and repeating shorter excerpts at will. It might yield greater accuracy and precision in the subject's selections if that could be done, AND if the subjects were clearly encouraged to do so. But, from what I have read about some of these tests, that is not what is done. Subjects may just be told how to operate the test mechanism, not how to use it to discriminate adequately. I am not sure that even so-called "audio pros" as subjects have a good listening strategy for this type of test.

Bottom line: ABX is a crazy, unnatural way to listen that is fraugt with problems. I have no doubt it can identify big differences. But, when we get to the smaller differences that are quite prevalent in audio, it's just not reliable. And, of course, an ABX result of no "statistically significant" difference does not mean there is no difference in reality.

Page: 1 2 3 4 5 6 ... 12 next

Closed