Snake Oil

Look what the Ministry found. Remember the business with Harrow Borough Council and the secret Israeli lie detector? We noted that it was unlikely that the signal it claimed to detect would be transmitted through the telephone system; an expert pointed out that it might be manifested in other ways; eventually we obtained a copy of the patent including a reference implementation in MS Visual Basic, and discovered that even if this was so, that wasn’t what it did.

In fact, it did very little of any use, and the actual content of the code directly contradicted Nemesysco’s claims regarding the system. For example, far from measuring 129 different parameters, it measures two, and claims to derive information on no less than eight different scales from these. And the actual judgment of truth or falsehood is based on entirely arbitrary reference values.

Swinging from tragedy to comedy, someone who was almost certainly Nemesysco founder Amir Liberman was then sighted sock-puppeting in the comments, and further inquiries showed probably the same person spamming Wikipedia. Very funny. Anyway, the Ministry has found an article from the International Journal of Speech Science and Law i which a pair of Swedish academics scrutinise the claims of some supposed lie detectors, Nemesysco’s among them. You can read it here. Here are some highlights:

The author describes the program as ‘detecting emotional status of an individual based on the intonation information’. But whereas intonation in phonetics means variation in pitch encoded by fundamental frequency (albeit almost always accompanied by other prosodic factors) the author of the LVA mistakenly believes that what he calls ‘thorns’ and ‘plateaus’ represent intonation..

Don’t get scratched by them thorns.

When an analog signal is digitized the complex continuous variation found in the original signal is replaced by a simplified discrete representation. How closely this representation matches the original depends on the sampling parameters but the match will never be perfect. It is in the digitization process that the ‘thorns’ and ‘plateaus’ are created. There is obviously an indirect relationship between thorns and plateaus and the original waveform, but the number of thorns and plateaus, which is the very basis for all computations in the LVA, depends crucially on sampling rate, amplitude resolution and the threshold values defined in the program. It is therefore correct to say that these computations are basically no more than statistics based on digitization artefacts.

And that is all there is. There is nothing special with these computations, except that there is no theoretical basis for them or independent motivation for the proposed ranges… The program would analyze any sound the same way, be it a man speaking, an idling car engine, a dog barking or a tram passing by. Secondly, the number and distribution of thorns and plateaus depend crucially on a number of factors that have to
do with how the digitization is performed. Different sampling frequencies and amplitude resolutions would produce different results.

the code is rather messy and not particularly well structured and we decided it would not be worth the time and effort to clean up the code in order to convert it into a running program. The Damphouse et al. group report that the program crashed repeatedly during their experiments so it is obviously rather unstable too

Ouch. But it gets worse.

The performance of LVA on the VSA database … was similar to that observed with CVSA. That is, neither device showed significant sensitivity to the presence of stress or deception in the speech samples tested. The true positive and false positive rates were parallel to a great extent.

That is to say, the results were entirely down to chance. And finally….

The output of an analysis is structured much along the same lines as horoscopes…To sum up by saying that there is absolutely no scientific basis for the claims
made by the LVA proponents is an understatement. The ideas on which the products are based are simply complete nonsense

Just for good measure, it seems that Liberman promoted himself as a significant Israeli mathematician whilst trying to sell the program in Sweden; it turns out he is no mathematician of any kind. However, he did know just what to do; sue the International Journal, which took the article off line. So much for that. Now for the DWP.

  1. Now I feel guilty for not tipping you off about that, it was in the swedish papers.

  2. Has anyone sent the DWP or Harrow Council, an official written “Data Subject Notice” under the Data Protection Act 1998 section 12 Rights in relation to automated decision-taking ?

    “(1) An individual is entitled at any time, by notice in writing to any data controller, to require the data controller to ensure that no decision taken by or on behalf of the data controller which significantly affects that individual is based solely on the processing by automatic means of personal data in respect of which that individual is the data subject for the purpose of evaluating matters relating to him such as, for example, his performance at work, his creditworthiness, his reliability or his conduct. “

  3. yorksranter

    Dunno about that; the Ministry’s doing a FOIA request.

  1. 1 quack quack oops « Alternate Seat of TYR

    […] and misses; notably, they are apparently too chicken to point out that they are claiming to get 129 dimensions of data from only two actual measurements. This wouldn’t even involve the Lacerda/Eriksson paper; it would just involve reading their […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: