Do I really need POLQA?
The question is understandable, as the cost of POLQA can be a bit higher than the cost of PESQ.MultiDSLAcustomers recognize great value and sometimes want to pursue it to the limit! Well, predictably, the answer is "It depends..." but to be more helpful, Opale has published a video describing specific cases where POLQA is the best solution and also when the PESQ metric - about 10 years older than POLQA - might be perfectly good enough. The video is available here , in case you missed it.
From time to time, customer questions bring to mind the practical reality of the PESQ vs. POLQA debate, and that's exactly what happened recently, when we got this question:
"Why do I only get a PESQ score of 1.90, when POLQA gives 3.86?"
How do you answer a question like that?
"Why do I only get a PESQ score of 1.90, when POLQA gives 3.86?"
MultiDSLA Analysis
But getting back to that question, here's how the two sets of results - POLQA first, then PESQ - came about:
So how does the survey work?
Step 1: Listen
Listening to the degraded signal alone doesn't reveal much. In fact, it sounds reasonably good, not excessively distorted, and there are no "gaps" in the speech. But then listening to the reference signal a few times reveals something interesting - the speech in the degraded signal actually comes out faster during the first of the two sentences!
Step 2: Look
Next, there's a visual clue, clear enough in the screenshots above. The weakest of the three traces - the "error surface" in the POLQA analysis - shows small, but fairly consistent errors. The same trace in the PESQ analysis (highlighted) shows larger errors throughout the first sentence and right at the beginning of the second. What is going on?
Step 3: Going Further
It's time to go through some of the other POLQA and PESQ graphs provided by MultiDSLA to see what we can discover. When we get to the Sample (Frame) Time Offset view, this is what we see, again POLQA first:
Well, they both tell the same story, sort of...
Latency/Delay Changes
POLQA tells us that during the first of the two sentences, the one-way latency (delay) decreases by more than 200 ms over a period of about 2.1 s. POLQA indicates that this is an approximately linear fit, corresponding to a speech "speed-up" of about 9.5%. Interestingly, speech pitch appears normal while this is occurring. One would expect that "speeding up" speech by nearly 10% would raise the pitch, but that is not what we hear.
What does PESQ say? Well, PESQ shows a similar overall effect but interprets the latency reduction as a series of discrete steps rather than a continuous process. So what's going on?
Codec behavior discovered
We find the explanation in the way some codecs handle adverse network conditions through a process called "time warping." Codecs with this capability belong to a class called relaxed code excited linear prediction (RCELP) that can vary the read rate of a jitter buffer while approximating the original signal pitch. Reading a signal faster than normal allows the codec to regain the time it "lost" after the jitter suppression buffer was increased to accommodate a delayed packet arrival time.
So what? And how does this explain the low PESQ score and the high POLQA score?
Mobile Codecs - State of the Art
The example in this blog is from a Skype Internet call, and the effect is pronounced and easy to see. But the same technique works on a smaller scale in codecs used in mobile networks, and that's where the limitations of PESQ became clear over a decade ago. This prompted interested parties to publish technical reports suggesting that inappropriate use of PESQ measurements could result in unfair, or even invalid, characterization of codec performance. You can see an example of this here.
And now there is POLQA
The partial obsolescence of PESQ was also one of the driving forces behind the development of a new objective voice quality analysis technique, which eventually led to the first commercial version of POLQA in 2011.
So, our client's simple question about PESQ vs. POLQA took us a long way, but we got to the explanation and discovered - guess what? That you have to use the right tools for the job!
Opale Systems
Thursday, 23rd June 2022 
