Pitch-synchronous speech processing

STATEMENT OF GOVERNMENT INTEREST

The invention described herein may be manufactured and used by or for the Government for governmental purposes without the payment of any royalty thereon.

BACKGROUND OF THE INVENTION

The present invention relates generally to synthetic speech systems and more specifically to a pitch synchronous method of transforming speech into vectors for speech processing.

Signal processing for speech, speaker, or language recognition, or for other speech applications, generally consists of a pre-processing step that reduces the speech to a series of vectors, on per time interval, where that interval is typically chosen to lie between five and twenty msec, and successive intervals may overlap. The most commonly used vector representation is the mel cepstrum, which is the Discrete Fourier Transform (DFT) of the logarithm of the non-uniformly low-pass filtered sampled magnitude of the spectrum of that speech segment. The non-uniform filtering and sampling provide roughly constant Q for each channel. A typical output vector might have twenty-eight scalar elements.

The task of processing speech into preprocessing vectors is alleviated, to some extent, by the systems disclosed in the following U.S. Patent, the disclosures of which are incorporated herein by reference: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0005">U.S. Pat. No. 5,008,941 issued to Sejnoha</li><li id="ul0002-0002" num="0006">U.S. Pat. No. 5,148,489 issued to Erell et al</li><li id="ul0002-0003" num="0007">U.S. Pat. No. 5,337,301 issued to Rosenberg et al</li><li id="ul0002-0004" num="0008">U.S. Pat. No. 5,469,529 issued to Bimbot et al</li><li id="ul0002-0005" num="0009">U.S. Pat. No. 5,598,505 issued to Austin et al</li><li id="ul0002-0006" num="0010">U.S. Pat. No. 5,727,124 issued to Lee et al</li><li id="ul0002-0007" num="0011">U.S. Pat. No. 5,745,872 issued to Sonmez et al</li><li id="ul0002-0008" num="0012">U.S. Pat. No. 5,768,474 issued to Neti</li><li id="ul0002-0009" num="0013">U.S. Pat. No. 5,924,065 issued to Eberman</li><li id="ul0002-0010" num="0014">U.S. Pat. No. 6,059,602 issued to Stadin</li></ul></li></ul>

The Stadin is interesting as it is for a powered roller skating system using speech recognition sensors and synthesized speech data processing.

The best reference is the Eberman patent which shows a computerized speech processing system with speech signals stored in a vector codebook and processed to produce corrected vectors.

Generally, speech processing includes the following steps. In a first step, digitized speech signals are partitioned into time-aligned portions (frames) where acoustic features can generally be represented by linear predictive coefficient (LPC) “feature” vectors. In a second step, the vectors can be cleaned up using environmental acoustic data. That is, processes are applied to the vectors representing dirty speech signals so that a substantial amount of the noise and distortion is removed. The cleaned-up vectors, using statistical comparison methods, more closely resemble similar speech produced in a clean environment. Then in a third step, the cleaned feature vectors can be presented to a speech processing engine which determines how the speech is going to be used. Typically, the processing relies on the use of statistical models or neural networks to analyze and identify speech signal patterns.

In an alternative approach, the feature vectors remain dirty. Instead, the pre-stored statistical models or networks which will be used to process the speech are modified to resemble the characteristics of the feature vectors of dirty speech. This way a mismatch between clean and dirty speech, or their representative feature vectors can be reduced.

By applying the compensation on the processes (or speech processing engines) themselves, instead on the data, i.e., the feature vectors, the speech analysis can be configured to solve a generalized maximum likelihood problem where the maximization is over both the speech signals and the environmental parameters.

The present invention is an alternate method and means for performing this first step of transforming speech into a standard series of vectors where each vector represents the sampled magnitude of the spectrum of one pitch period for voiced speech or one pseudo pitch period for unvoiced speech. The subsequent speech processing steps can then be performed with these new vectors as inputs.

SUMMARY OF THE INVENTION

The present invention is an alternate method and means for performing the first step of transforming speech into a standard series of vectors where each vector represents the sampled magnitude of the spectrum of one pitch period for voiced speech or one pseudo pitch period for unvoiced speech. The subsequent speech processing steps can then be performed with these new vectors as inputs, provided these subsequent steps are adapted to the new vectors with suitable training protocols and data.

The invention involves two main steps: <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0023">1. divide the speech into pitch periods, or into pseudo pitch periods for unvoiced speech, where the breaks occur, for example, at the first zero-crossing preceding each glottal pulse for voiced speech and at any arbitrary point for unvoiced speech, and</li><li id="ul0004-0002" num="0024">2. compute the log-magnitude of the Discrete Fourier Transform (DFT) of each pitch-period waveform, and interpolate each log-magnitude spectrum to a common regular grid which can accommodate the spectrum of a waveform having the longest pitch period anticipated.</li></ul></li></ul>

BRIEF DESCRIPTION OF THE DRAWINGS

<figref idref="DRAWINGS">FIG. 1</figref> is an illustration of a complete speech preprocessing system of the present invention;

<figref idref="DRAWINGS">FIG. 2</figref> is a diagram of the pitch estimation component; and

<figref idref="DRAWINGS">FIG. 3</figref> is a diagram of the output of the pitch period segmentor of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention is a speech processing system and process for transforming speech into a standard series of vectors where each vector represents the sampled magnitude of the spectrum of one pitch period for voiced speech or one pseudo pitch period for unvoiced speech. The subsequent speech processing steps can then be performed with these new vectors as inputs, provided these subsequent steps are adapted to the new vectors with suitable training protocols and data. A block diagram of the proposed process is illustrated in FIG. <b>1</b>.

The process of <figref idref="DRAWINGS">FIG. 1</figref> has two main steps: <ul id="ul0005" list-style="none"><li id="ul0005-0001" num="0000"><ul id="ul0006" list-style="none"><li id="ul0006-0001" num="0030">1. divide the speech into pitch periods, or into pseudo pitch periods for unvoiced speech, where the breaks occur, for example, at the first zero-crossing preceding each glottal pulse for voiced speech and at any arbitrary point for unvoiced speech, and</li><li id="ul0006-0002" num="0031">2. compute the log-magnitude of the Discrete Fourier Transform (DFT) of each pitch-period waveform, and interpolate each log-magnitude spectrum to a common regular grid which can accommodate the spectrum of a waveform having the longest pitch period anticipated.</li></ul></li></ul>

The process of <figref idref="DRAWINGS">FIG. 1</figref> begins as acoustic data is processed for silence detection <b>100</b> to determine which part of the data stream has speech or silence. The speech sequence is converted into a stream of windows of LW speech samples each. The length LW should be comparable to the duration of a syllable. A given window is said to contain speech if its average power exceeds a suitably chosen threshold POW_TH and is otherwise classified as silence, e.g. POW_TH may equal the noise variance per sample.

Once the portion of the data stream containing speech is flagged the pitch estimator <b>200</b> can process the flagged data stream.

The pitch estimation component is illustrated in FIG. <b>2</b>. The input data used to estimate the pitch is the stream of classified speech/silence windows, and the minimum and maximum anticipated pitch period, P_MIN and P_MAX respectively. A register of length K=┌2P_MAX/LW┐ LW¹is sequentially filled with samples from a contiguous sequence of windows containing speech until the capacity of the buffer is reached or a silence window is found on the input stream. Then the following operations are performed on the retrieved speech segment: <ul id="ul0007" list-style="none"><li id="ul0007-0001" num="0000"><ul id="ul0008" list-style="none"><li id="ul0008-0001" num="0035">1. The N-point DFT of the speech segment is computed with <maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mrow><mi>N</mi><mo>=</mo><msup><mn>2</mn><msup><mrow><mo>⌈</mo><mtable><mtr><mtd><mrow><mi>log</mi><mo>⁢</mo><mstyle><mtext> </mtext></mstyle><mo>⁢</mo><mi>K</mi></mrow></mtd></mtr><mtr><mtd><mn>2</mn></mtd></mtr></mtable><mo>⌉</mo></mrow><mrow><mo>+</mo><mn>1</mn></mrow></msup></msup></mrow></math></maths></li><li id="ul0008-0002" num="0036"> and the square-magnitude of each transform coefficient is computed to yield a power spectrum.</li><li id="ul0008-0003" num="0037">2. The frequencies at which the power spectrum has local maxima are determined.</li><li id="ul0008-0004" num="0038">3. A locally normalized spectral envelope is computed by dividing the value of the power spectrum at each peak by the geometric mean of the two adjacent peaks. For the first and last peaks the power spectrum is normalized by the value of the single adjacent peak.</li><li id="ul0008-0005" num="0039">4. If there are no frequencies at which the normalized spectral envelope is greater than ten, the speech segment is declared to be unvoiced; otherwise it is declared to be voiced.</li><li id="ul0008-0006" num="0040">5. For unvoiced speech segments the pitch is set to the default pitch P_DEF.</li><li id="ul0008-0007" num="0041">6. For voiced speech segments a primary pitch estimate is extracted from the normalized spectral envelope using the following heuristic. If there are fewer than five normalized spectral peaks which exceed a threshold of ten, then the lowest frequency in that set of spectral values yields the primary pitch estimate. Alternatively, if there are five or more normalized spectral peaks greater than ten, one first finds the maximum normalized spectral peak from the set of frequencies which are lower than the lowest frequency satisfying the threshold condition. If such a maximum exists and is greater than five and occurs at a frequency which is within twenty percent of half the lower frequency at which the normalized spectrum is greater than ten, then the lower of the two frequencies gives the primary pitch estimate, otherwise the higher of the two frequencies is used as the primary pitch estimate.</li><li id="ul0008-0008" num="0042">7. If the current and previous speech segments are not separated by silence and they were both declared as voiced, a secondary pitch estimate for the current segment is computed. First the means and standard deviation of the ensemble of pitch period lengths of the previous speech segment are computed. If the standard deviation is less than ten percent of the mean and the mean is less than P_MAX, then the mean pitch period length for the previous segment is used as the secondary pitch estimate for the current speech segment.</li><li id="ul0008-0009" num="0043">8. The final pitch estimate p_est for voiced speech segments is obtained as follows. If only the primary pitch estimate is available, it is used as the final estimate. When the secondary pitch estimate is also available the ratio of the primary estimate to the secondary estimate determines which of the two estimates is used as the final estimate. If the ratio is less than 1.3 and greater than 0.7, the primary estimate is used; otherwise the secondary estimate is used.</li></ul></li></ul>

The speech segments are segmented further into pitch periods as follows. <ul id="ul0009" list-style="none"><li id="ul0009-0001" num="0000"><ul id="ul0010" list-style="none"><li id="ul0010-0001" num="0045">1. If the current speech segment is starting and the current and previous speech segments are separated by silence, find the maximum peak of the speech waveform in the time interval of duration P_MAX starting at the beginning of the current speech segment. Otherwise, find the maximum peak within the time interval starting 0.7*p_est time units ahead of the last located peak and ending 1.3*p_est time units ahead of the last located peak. Let s_max and t_max be the value and the time index of the located maximum, respectively.</li><li id="ul0010-0002" num="0046">2. Find the minimum value of the speech waveform in the time interval of duration p_est/2 ending at t_max. Let s_min be the value of the located minimum.</li><li id="ul0010-0003" num="0047">3. Position the time cursor at t_max.</li><li id="ul0010-0004" num="0048">4. Move back along the time axis until a peak is found which lies above a line of slope 0.5*(s_max−s_min)/p_est passing through the current peak and is contained in the time interval of length 0.3*p_est ending at t_max.</li><li id="ul0010-0005" num="0049">5. Repeat step 4 until another peak satisfying the specified conditions is not found. Let t_p be the time index of the last located peak.</li><li id="ul0010-0006" num="0050">6. If the current speech segment is declared as unvoiced, the start of the current pseudo pitch period is the minimum of t_p and the start of the previous pitch period (pseudo pitch period) plus P_MAX if there is a preceding pitch period (pseudo pitch period), or the maximum of t_p and the start of the current speech segment if the current pseudo pitch period is the first one in the current speech segment if the current pseudo pitch period is the first one in the current speech segment and the current and previous speech segments are separated by silence.</li></ul></li></ul>

<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="1"><colspec colname="1" colwidth="217pt" align="center" /><thead><row><entry namest="1" nameend="1" rowsep="1">TABLE 1</entry></row></thead><tbody valign="top"><row><entry namest="1" nameend="1" align="center" rowsep="1" /></row><row><entry>parameter values used to generate the example discussed below.</entry></row><row><entry>The symbol [*] denotes rounding to the nearest integer. The</entry></row><row><entry>sampling rate was F_s = 48000 samples sec.</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="4"><colspec colname="offset" colwidth="42pt" align="left" /><colspec colname="1" colwidth="77pt" align="left" /><colspec colname="2" colwidth="56pt" align="right" /><colspec colname="3" colwidth="42pt" align="left" /><tbody valign="top"><row><entry /><entry>Parameter</entry><entry>Value</entry></row><row><entry /><entry namest="offset" nameend="3" align="center" rowsep="1" /></row><row><entry /><entry>POW_TH</entry><entry>1000</entry><entry /></row><row><entry /><entry>LW</entry><entry>[16 * F_s/1000]</entry></row><row><entry /><entry>P_MIN</entry><entry>[1.4 * F_s/1000]</entry></row><row><entry /><entry>P_MAX</entry><entry>[25 * F_s/1000]</entry></row><row><entry /><entry>P_DEF</entry><entry>[6 * F_s/1000]</entry></row><row><entry /><entry namest="offset" nameend="3" align="center" rowsep="1" /></row></tbody></tgroup></table></tables><ul id="ul0011" list-style="none"><li id="ul0011-0001" num="0000"><ul id="ul0012" list-style="none"><li id="ul0012-0001" num="0052">7. If the current speech segment is declared as voiced the following rules are used to determine the start of the current pitch period. <ul id="ul0013" list-style="none"><li id="ul0013-0001" num="0053">(a) If the current and previous speech segments are separated by silence and the current pitch period is the first one in the current speech segment, the start of the current pitch period is the maximum of the zero-crossing preceding t_p and the start of the current speech segment. If there is no zero-crossing, the start of the current pitch period is the start of the current speech segment.</li><li id="ul0013-0002" num="0054">(b) If the current and previous speech segments are adjacent in time and there is a zero-crossing between t_p and the start of the previous pitch period, the start of the current pitch period is the minimum of trhe zero-crossing immediately preceding t_p and the start of the previous pitch period plus P_MAX. If there is no zero-crossing between t_p and the start of the previous pitch period, the start of the current pitch period is the start of the previous pitch period plus p_est.</li></ul></li></ul></li></ul>

This procedure is repeated until the end of the current speech segment is reached. <figref idref="DRAWINGS">FIG. 3</figref> shows the segmentation into pitch periods and pseudo pitch periods of a speech segment 100 msec long, where the breaks are indicated by asterisks.

For each pitch period or pseudo pitch period the N-point DFT is computed with N equal to the length of the period in question and the log-magnitude of each transform coefficient is computed. Finally, each log-magnitude spectrum is linearly interpolated to a common regular grid with frequency resolution 1/P_MAX.

One example of the invention illustrated the pitch-synchronous spectral representation of the sentence “The little blankets lay around on the floor.” as delivered by a female speaker. The speech was sampled at a rate of F_s=48000 samples/sec with 16-bit resolution. The values of the parameters used to generate this example are listed in Table 1.

While the invention has been described in its presently preferred embodiment it is understood that the words which have been used are words of description rather than words of limitation and that changes within the purview of the appended claims may be made without departing from the scope and spirit of the invention in its broader aspects.