Interfering with Speech-to-Text

(report on a preliminary experiment)

David Prager Branner

Hack and Tell, New York
4 February, 2014

Goal

  • Goal: To measure the effectiveness of different ways to interfere with automated voice transcription.
  • Remarks: Normally we want to improve the quality of voice-recognition, for instance in the use of tools for the hearing-impaired, automated stenography, or giving verbal instructions to a robotic servant.
  • Today I want to consider the reverse question: What means do I have to confound automatic voice recognition and transcription?
  • I used Google's automated transcription of voicemail for this project, because of its convenience.

Measurement tool

  • Find the largest number of words in the original that occur in the correct order in the transcription. Ratcliff/Obershelp algorithm in Python difflib; input: lists of words.)
  • Google returns both a whole string of text and a word-by-word list, showing the confidence with which each word is transcribed:

            gc-word-high vs.
            gc-word-med2 vs.
            gc-word-med1

    Here I ignore these confidence values and only treat the whole string. This lowers accuracy somewhat.
  • I consider only whole words, normalized (with NLTK and other tools) to ignore small morphological variations (so “saves”, “saved”, “saving” → “save”). Numerals zero to ten (ordinal and cardinal) are also converted to words. All this raises accuracy somewhat.

Strictures

  • Limit myself to procedures that are laborious neither to execute nor to understand, and limit myself to standard English.
  • Must be usable on the fly, without preparation. No codes, ciphers, steganography, or heavy use of filler words (e.g., “wink wink”, “nudge nudge”).
  • Also no resorting foreign language or non-standard accents (in spite of their considerable utility).
  • Reported values are the average of five trials per type of interference. (These slides contain only one of the five trials, for each of nine types of interference, plus a plain recording.)

Summary of results

type of interferenceaverage
accuracy
ease to
produce
ease to
understand
Natural voice (no interference) 0.8highhigh
acoustic interference
Vigorous tapping 0.4highmedium
Flapping paper 0.5highhigh
News report (1010 WINS) 0.5highhigh
Music (John Coltrane, Blue Trane, 1957) 0.6highhigh
White Noise (from SimplyNoise) 0.8highhigh
funny voices
Whispering 0.5highmedium
Heavy Breathing 0.6lowlow
Falsetto 0.4lowlow
composite
Whispering while waving paper 0.1mediumlow

Measuring transcription accuracy

Google supplies a transcription composed of well-formed English words, so there is no question of typographical errors, for which edit distance would be a useful quantification tool.

Instead, given an original string s1 and a transcription string s2, I look for the longest common substring of s1 in s2 together with whatever other matching material is found on either side of it interspersed with non-matching material.

The Ratcliff/Obserhelp algorithm was developed to accomplish this task by John W. Ratcliff and John A. Obershelp in 1983 for application to problems in educational software (Ratcliff and Metzener, 1988). Ratcliff called it "Wild-card search that doesn't require wild cards; … the algorithm creates its own wildcards." It is not the most efficient LCS algorithm, though.

The algorithm is implemented in Python using a queue rather than recursion.

References

The Ratcliff/Obserhelp algorithm

  1. The Ratcliff/Obershelp algorithm (accessed 20140130).
  2. See also mention (without code) of two C implementations (accessed 20140130).

Python difflib

  1. Docs for Python difflib.
  2. Source-code for Python difflib.

Data (1: Newswire)

Original text (composite of recent news material):

The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. … Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.

(Nicole Hong and Prabha Natarajan, “Emerging Markets Selloff Spreads”; on line at the Wall Street Journal, dated 20140129 and accessed 20140129.)

Data (1: Newswire, Original)

sampleaccuracyease to
produce
ease to
understand
0.8highhigh

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.The Decline 4 Fuel by concerns that efforts by emerging markets central banks to tighten monetary policy won't be enough to protect their economies against an exit of invest your money. Although higher. Rates are supposed to entice investors to continue investing in emerging markets times. These, analysts at the total. The higher interest rates may take on the economic growth of those nations. Maybe too high.

Data (1: Newswire, Vigorous tapping)

sampleaccuracyease to
produce
ease to
understand
0.4highmedium

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.The client was little 12th past 5 or 10 North Central Park Type of monetary policy won't be enough to protect their economies again for the okay bye bye, also higher. Rates are doing. Tyson. Vestess to invest in Virginia. Geez, panelist said the total pilots great vacate the economic growth presentations, maybe 2. Bye.

Data (1: Newswire, Flapping paper)

sampleaccuracyease to
produce
ease to
understand
0.5highhigh

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.I just fine for few. My concern is that after 5:30 or 6. Thanks to tighten monetary policy 4' enough to protect their economies again, if you've got investor money for a little higher rates, as opposed to intense investors to continue investing emerging market currency, analysts said fits hold a higher interest rates may takes on the on the growth of those nations. Maybe 25.

Data (1: Newswire, News report)

sampleaccuracyease to
produce
ease to
understand
0.5highhigh

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.Who claims. We're sure. My concern is that efforts by emerging markets info. Thanks, tycoon monetary policy just be enough to protect their economies again for the exact of investor money, although higher rates, as opposed to entice investors to continue investing in emerging markets. Alright. Thanks and I was at the toll of a higher interest rates may take on the economic growth reservations baby 2. Hugs. Bye.

Data (1: Newswire, Music)

sampleaccuracyease to
produce
ease to
understand
0.6highhigh

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.I'm calling to see what concerns. The efforts by emerging market central banks to tighten monetary policy won't be enough to protect their economies. Again today cos investor money for a little higher ready so supposed to entice investors continue investing in emerging market currency, analysts say that the total HI. Sprayed may take part of the economic growth of suspicion. Maybe 25.

Data (1: Newswire, White Noise)

sampleaccuracyease to
produce
ease to
understand
0.8highhigh

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.Yes, calling to ask you by concerns. That efforts by emerging markets central bye to tighten monetary policy won't be enough to protect their economies. Again the exit, of investor money, Hello Hi. Rates are supposed to entice investors to continue investing in emerging market currency, analysts said, the toll of a higher interest rates may take on the economic growth of those nations they'd be too high.

Data (1: Newswire, Whispering)

sampleaccuracyease to
produce
ease to
understand
0.5highmedium

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.Time for your bike is that efforts by Mark at central banks to tighten monetary policy won't be enough to protect our economy. I can't accept a investor putting ohh Hi, Alright. I suppose to entice investors to continue investing in emerging market starts. I don't know what said, Put all the under Interest rates paid A on the economic growth info stations, maybe July.

Data (1: Newswire, Heavy Breathing)

sampleaccuracyease to
produce
ease to
understand
0.6lowlow

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.Hey clients were fuels by concerns that efforts by emerging markets central banks to tighten monetary policy won't be enough to protect their economies again stagnate of investor money. Also, I always are supposed to entice investors to continue investing in emerging markets parents East, and I was set Apple are interest rates me take on the economic growth associations, maybe 2. Bye.

Data (1: Newswire, Falsetto)

sampleaccuracyease to
produce
ease to
understand
0.4lowlow

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.The client was little concerned, the difference, but I'm going to work at central banks to talk to monica closing cost to protect our economy. As you can still exist of investor money. But I don't know what I'm supposed to rain forest investors to continue investing in emerging markets time please. Hey fast calling her cruise from. It's maintained by economic growth of those nations maybe tomorrow how high.

Data (1: Newswire, Whispering & Flapping)

sampleaccuracyease to
produce
ease to
understand
0.1mediumlow

Original textTranscription
The declines were fueled by concerns that efforts by emerging-market central banks to tighten monetary policy won't be enough to protect their economies against an exit of investor money. Although higher rates are supposed to entice investors to continue investing in emerging-market currencies, analysts said the toll the higher interest rates may take on the economic growth of those nations may be too high.Hello HI. Your talk to you sir okay bye bye, bye bye hey. I think that bye bye, alright bye bye, bye. Hello, thanks bye bye economic growth.