7 Keystones of Accurate HiFi Blind Test [Article]

"Audiophile myths debunked" is one of popular topics at a forums and an articles.
As rule, these "myths" is based on listening impressions, unexplained technically or small measurable difference is inaudible theoretically / probably.
To accurate check the "myths" recommended to use method, ensuring repeatability with same results in the similar conditions.

HiFi blind test is way of "objective" measurement of "subjective" perception.

Read about most important things, turning a HiFi test into a safe evidence.

Still we cannot direct access to foreign feelings. We cannot listen music as other person.

We can ask him/her only. But they can't exactly describe their feeling. Especially in the subtlest details, providing by modern musical equipment, software, HiFi test records.

It is main aim of the blind trial. We try eliminate subjective part of perception (price, bias, habit, etc., i.e. probable imaginary perception).

For even more "subjectivity" reducing, double blind test is used.

Blind test audio

ABX test may be used in ear comparison.

ABX test audio

If somebody claim that sound is different, it can't measure its audibility. Because we listen in the brains via ears.

Theoretically, measurement tools are more sensitive than human ears. But without trials we don't know where is audibility edge of difference.

Professional double blind test of music equipment is not home entertainment. It is hard long expensive work.

It doesn't necessarily audiophile blind test should be done at laboratory. At home nobody can prohibit us to do it.

But safe trial results are necessary as claim evidences.

These features are necessary to provide safe trial results:

1. Methodology
2. Protocol
3. Big number of measurements
4. Listening place
5. Testing equipment issues
6. Measurement precision issue
7. Careful condition control

Trial begins with methodology design. Methodology define trial's:

• aim,
• precision,
• implementation,
• other things, noted below.

Methodology should be designed first

Test protocol is paper, where recorded detailed data about conducted trial:

• equipment,
• measurement tools,
• participants (and its listening skill, occupation, education, other),
• trial conditions,
• other.

Hi-Fi test protocol

Testing and measurement equipment may be registered with unique identifier of item (serial number, as example).

To save time several participants may be accommodated in one room. In the room are placed several seat rows.

In the trial result interpretation, necessary take into account:

• each seat impact to sound,
• seat with and without listener impact differently,
• speaker sound is different for each seat.

Impact seat place to audio test results

Anechoic room allow to avoid bouncing. But speaker have different frequency response in the different directions. So different seat places also cause different frequency response in anechoic room even.

Generally, only 1 sitting place in the listening room is recommended.

Testing equipment (apparatus that will tested in the trial) should be checked to workability by way, that may be described in the methodology of the trial.

It is desirable, if testing equipment have unique identifier (serial number). It help to repeat experiment or found difference result reasons in case of doubt.
Because different equipment instances may have various figures and sound differently.

Updated: When we compare headphones, tested sample can't be isolated from participant technically. For solving this issue, synthesis of headphone features and comparison via single headphone unit were suggested [3].

In statistical calculations, measurement error values may be different. We don't know it exactly, but accept in first approach, that it have normal distribution [1]. So...

As example, we want to provide ear trial measurement precision 1%.

1/1% = 1/0.01 = 100 tryings.

So trying number 3 times higher 300 = 100 * 3 is recommended.

Defined measurement precision should be provided

The ear test have a many variables due human factor, above all.

Somewhat, it may be compensated by big number of tryings, participants, equipment items.

Big participant and trying number is recommended

In the "Measurement precision issue" part was considered example: providing 1% precision.

There are recommended 300 tryings.

However, it don't guarantee 100 % of true.

Participant skills can cause biasing of results, as example.

To compensate it with precision 1%, we should invite 300 = (1/1%)*3 = (1/0.01)*3 participants.

Thus total number of measurements is 90000 = 300 trying * 300 participants

We should account each feature, that can cause mistake probably. To a first approximation, we should invite participants with different skills and groups of participants with same skill should have same number of members.

In the trial conclusions we should group results by participant skills.

Conditions define experiment. Changing conditions can bias results significantly.

Careful control of conditions provide the test exactness

Example:

Somebody want to compare DACs. He asks peoples who have the devices, check it on some public HiFi test CD.

These tests, as rule, performed at home without exact condition control. We have experiment at different speakers, listening places, the ambient noise, etc. Even experiment, described in details, may have subtle, at first glance, issues, that impact upon result.

As rule, before experiment we can't know exactly, that details are important in the trial. Only result figures can show, that details are unimportant in our experiment: impact to result into allowable error margin.
As example, we cannot expect listening skill affected trial results. But for some tests, it might be important.

1. Blind test, double blind test, ABX test are intended to "objective" measurement of "subjective" (immeasurable) perception.
2. Properly performed and recorded hi-fi blind test may be a serious objective evidence of subjective perception.
3. The trial may be performed at home for personal purposes, but can't be used as technical evidence.
4. The trial should be designed, performed in a manner that maximizes independence from number of participants and tryings and/or other. As reference value may be allowable changes into error margin.
5. The independence is achieved by big number of participants and tryings, equipment items, etc.
6. Even correctly designed and performed HiFi test don't guarantee absolute truth. Because new knowledge may open new details.

Share the article in the social networks, forums

    Write comments, your thoughts about hifi blind tests
        at the article post on Audiophile Inventory's facebook page
        and elsewhere.

Additional: popular HiFi test kinds

• lossless vs mp3 test
• 16 bit vs 24 bit
• PCM vs DSD
• FLAC vs WAV
• acoustic cables

Yuri Korzunov,

Audiophile Inventory's developer

2017 July 22

1. Normal distribution (statistics)
2. Bob Snyder. Music and Memory: An Introduction
3. About testing headphones
4. One of careful test examples
5. How Well Can You Hear Audio Quality?

Video: Some New Evidence that High School and College Students May Prefer Accurate Sound Reproduction

Read more about HiFi tests

• Hearing Test vs. Measurements Audio. Where True? >
• Why We Can't Compare Different Audio Formats >
• Could a Music Be Used for Sound Quality Tests? [Article] >

How to Improve Sound Quality

Where is the Limit of Audio Quality?

How to Choose the Best CD Ripping Software (Mac OS, Windows)

How to Convert ISO to FLAC on Mac OSX, Windows [Step-by-step Guide]

How to Convert ISO to DSF on Mac OS, Windows [Step-by-step Guide]

How to Choose the Best Audio Converter Software [Article]