State transition model design method and voice recognition method and apparatus using same

0001The present invention relates to a design method for a state transition model used, for example, for a voice recognition model. The present invention also relates to a voice recognition method and apparatus using a state transition model designed to recognize voices at high speed.

0002In the following, a hidden Markov model (HMM) is used as a voice recognition model by way of example.

0003As the speed of a computer has drastically increased, studies on practical uses and production of voice recognition systems have been made extensively. These systems incorporate HMM which is a statistical model. A triphone HMM has been widely studied because this shows better performance than other HMMs. With this triphone HMM, differences in phone environments such as preceding and succeeding phones are classified finely. This triphone HMM has a number of models so that trainability of data degrades and models of high performance cannot be configured. Furthermore, the computation amount becomes large in proportion with the number of models, posing a critical issue on voice recognition which is required to process in real time.

0004Several methods for solving these problems have been studied basing upon a concept of "shared structure HMM". <ul id="ul0001" list-style="none" compact="compact"><li>(1) A generalized triphone HMM which shares HMMs themselves having similar acoustic characteristics of the whole phone section (K. F. Lee, H. W. Hon, Large-vocabulary speaker-independent continuous speech recognition using HMM, ICASSP88, pp. 123-126)</li><li>(2) A shared-state HMM which shares the states of HMMs having similar acoustic characteristics of the whole phone section (Mei-yuh Hwang, X. D. Huang, Subphonetic modelling with Markov States - SENON, ICASSP92, pp. 133-136, S. J. Young, P. Woodland, The use of state tying in continuous speech recognition, Eurospeech 93, pp. 2203-2206, 1993).</li><li>(3) A tied mixture HMM which shares the distributions of HMMs having similar acoustic characteristics of the whole phone section (J. Bellegarda, D. Nahamoo, Tied mixture continuous parameter models for large vocabular isolated speech recognition, ICASSP89, pp. 13-16, D. Paul, The Lncoln robust continuous speech recognition, ICASSP89, pp. 449-452).</li></ul>

0005Of these and others, a shared-state HMM using successive state splitting (SSS) proposed by Takami and realizing both the above (1) and (2) is known as a method of generating a shared-state triphone HMM of high precision because a shared state is determined in a top-down manner while considering phone environments (refer to Takami, Sagayama: "Automatic generation of hidden Markov network by SSS", Papers of the Institute of Electronics, Information and Communication Engineers, J76-DII, No. 10, pp. 2155-2164, 1993).

0006X. D. Huang, S. J. Young, et al. have proposed a method of generating a shared-state triphone HMM through bottom-up merge and obtained good results. Takahashi, et al. have proposed a method of generating an HMM which method synthesizes the above (1) to (3) (refer to Takahashi, Sagayama: "HMM for four hierarchical-level shared structure", Technical Reports of the Institute of Electronics, Information and Communication Engineers, SP94-73, pp. 25-32, 1994-12).

0007In this invention, all triphones are prepared and the states of these triphones are clustered. In this context, it is analogous to the methods by X. D. Huang and S. J. Young. However, different from clustering through merge considering only local likelihood, top-down clustering considering the whole acoustic space is performed and this clustering is efficient because of consideration of the whole acoustic space.

0008Although the same top-down scheme as SSS is used, SSS has an inefficient point that an ending state of one triphone is not shared with a starting state of another triphone because of successive state splitting (SSS). Since voices are generally continuously converted, it can be considered relatively natural that a connectable ending state of a triphone and the starting state of the next triphone are to be shared. The method by S. J. Young considers a share of only states within a phone class and cannot share states between phone classes. These disadvantages of SSS have been solved by Takami by incorporating merge into the processes of successive splitting (refer to Takami, "Efficiency improvement of hidden Marcov network by state splitting method", Papers of Lectures of Acoustical Society of Japan. 1-8-4. pp. 7-8, 1994-10). Takahashi and et al. have solved the above disadvantages by incorporating a tied-mixtured HMM. However, the present inventors consider more desirable that the above disadvantages are to be solved from the viewpoint of a state level.

0009Another disadvantage of SSS is that if an arbitrary speaker HMM is generated by successive state splitting, this splitting becomes dependent upon the arbitrary speaker. It is therefore necessary to use a specified speaker in obtaining a state shared structure. This poses other problems that a large amount of data is required for the specified speaker and that it is necessary to use the state shared structure of the specified speaker for other arbitrary speakers.

0010The invention has been made under the above circumstances. According to one aspect, the present invention aims to provide a state transition model design method and apparatus capable of recognizing voices at high speed, and a voice recognition method and apparatus sing the state transition model.

0011According to another aspect, the present invention aims to provide a state transition model design method and apparatus capable of sharing states between phone lasses or within a phone class, and a voice recognition ethod and apparatus using the state transition model.

0012According to another aspect, the present invention ims to provide a state transition model design method nd apparatus capable of obtaining a state shared tructure of phones of an arbitrary speaker and capable of efficiently designing a state transition model, and a voice recognition method and apparatus using the state transition model.

BRIEF DESCRIPTION OF THE DRAWINGS

0013Fig. 1 is a flow chart illustrating processes according to a first embodiment of the invention.

0014Fig. 2 shows states of an HMM and a state transition model diagram.

0015Fig. 3 is a flow chart illustrating top-down clustering processes.

0016Fig. 4 is a diagram illustrating a state shared type HMM.

0017Fig. 5 is a block diagram illustrating a voice recognition process used by a voice recognition apparatus of the embodiment.

0018Fig. 6 is a table showing the results of recognition of 100 sentences spoken by 10 arbitrary speakers, the recognition being made by using grammars constituted by 1000 words and the voice recognition apparatus of the embodiment.

0019Fig. 7 is a flow chart illustrating processes by a second embodiment.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

0020Exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

0021The embodiments (inclusive of flow charts) of the invention are reduced in practice under the control of a CPU in accordance with a control program stored in a ROM or RAM. This control program may be stored in a removable storage medium such as a CD-ROM mounted on a voice recognition apparatus.

0022A method of designing a state shared structure model for voice recognition according to a first embodiment of the invention will be described.

0023Fig. 1 is a flow chart illustrating the processes of the first embodiment.

0024Referring to Fig. 1, reference numeral 101 represents a means (process) for designing initial clusters, reference numeral 102 represents a means (process) for top-down clustering such as general LBG for generating clusters raised to a power of 2, i.e., a means (process) for finely classifying clusters starting from a small number of clusters and sequentially increasing the number of clusters, reference numeral 103 represents a means (process) for determining a common status structure (or state shared structure) of a triphone HMM (modeling considering both preceding and succeeding phones), and reference numeral 104 represents a means (process) for studying (learning) a triphone HMM of the state shared structure.

0025The details of these means (processes) will be described.

(1) Design of Initial Clusters (101)

(A) All triphone HMMs are learnt using data of an arbitrary speaker.

0026<ul id="ul0002" list-style="none" compact="compact"><li>(a) Phone HMMs of one distribution are learnt at the proper number of states.</li><li>(b) A right environment type (right-context) HMM is learnt by using the phone HMMs as initial models.</li><li>(c) A both-side environment type (triphone) HMM is learnt by using the right-context HMMs as initial models.</li></ul>

(B) All states of triphone HMMs are used as initial clusters.

0027Fig. 2 is a diagram illustrating HMM and showing a general state and a state transition model.

0028In Fig. 2, a state transition probability is indicated by a, an output probability at the corresponding state is indicated by b, a mean value of output probabilities is indicated by µ, and a corresponding dispersion is indicated by σ.

(2) Top-down Clustering by LBG scheme (102)

0029The top-down clustering is performed by an LBG scheme using a distance scale considering the output probability distribution. Clustering is defined only by the output probability b which is considered to be an important parameter for obtaining a likelihood to HMMs, by neglecting the state transition probability a.

0030This process is illustrated in the flow chart of Fig. 3.

0031At Step S1, 1 is set to m. At Step S2, one class Φm is generated which contains all initial clusters (φi). At Step S3 it is checked if the value m is equal to the total number M (e.g., 600) of clusters. If equal, the process is terminated, whereas if not, the process advances to Step S4.

0032At Step S4, a new cluster Φm is generated from all the initial clusters (φi) belonging to the old cluster Φm by using the following equations (1) and (2). Specifically, the new cluster Φm is generated by using the mean value µ of output probabilities and a corresponding dispersion σ squared. In the equations, m indicates the cluster number, and N indicates the total number of initial clusters belonging to the class Φm.<maths id="math0001" num=""><img file="EP0750293A2_D0001.tif" /></maths><maths id="math0002" num=""><img file="EP0750293A2_D0002.tif" /></maths>

0033Next, obtained at Step S5 are an initial cluster φp, among the initial clusters φi belonging to the new cluster Φm, remotest from the cluster Φm, and an initial cluster φq remotest from the initial cluster φp. As the distance scale d (φp, φq) between the two initial clusters, a Kullback information quantity, a Chernoff distance, a normalized Euclid distance, a Euclid distance, or the like may be used. In this embodiment, a Bhattacharyya distance is used which can be calculated by the following equation (3) in the case of a single Gaussian distribution.<maths id="math0003" num="(3)"><math display="block"><mrow><mtext mathvariant="italic">d</mtext><msub><mrow><mtext>(φ</mtext></mrow><mrow><mtext mathvariant="italic">p</mtext></mrow></msub><msub><mrow><mtext>,φ</mtext></mrow><mrow><mtext mathvariant="italic">q</mtext></mrow></msub><mtext>)=</mtext><mspace linebreak="newline" /><mfrac><mrow><mtext>1</mtext></mrow><mrow><mtext>8</mtext></mrow></mfrac><msub><mrow><mtext> (µ</mtext></mrow><mrow><mtext mathvariant="italic">p</mtext></mrow></msub><msub><mrow><mtext> - µ</mtext></mrow><mrow><mtext mathvariant="italic">q</mtext></mrow></msub><msup><mrow><mtext>)</mtext></mrow><mrow><mtext>t</mtext></mrow></msup><mtext>(</mtext><mfrac><mrow><msub><mrow><mtext>Σ</mtext></mrow><mrow><mtext mathvariant="italic">p</mtext></mrow></msub><msub><mrow><mtext>+Σ</mtext></mrow><mrow><mtext mathvariant="italic">q</mtext></mrow></msub></mrow><mrow><mtext>2</mtext></mrow></mfrac><msup><mrow><mtext>) </mtext></mrow><mrow><mtext>-1</mtext></mrow></msup><msub><mrow><mtext>(µ</mtext></mrow><mrow><mtext mathvariant="italic">p</mtext></mrow></msub><msub><mrow><mtext>-µ</mtext></mrow><mrow><mtext mathvariant="italic">q</mtext></mrow></msub><mtext>)</mtext><mspace linebreak="newline" /><mtext>+ </mtext><mfrac><mrow><mtext>1</mtext></mrow><mrow><mtext>2</mtext></mrow></mfrac><mtext>ln</mtext><mfrac><mrow><mfenced open="|" close="|"><mrow><msub><mrow><mtext>(Σ</mtext></mrow><mrow><mtext mathvariant="italic">p</mtext></mrow></msub><msub><mrow><mtext>+Σ </mtext></mrow><mrow><mtext mathvariant="italic">q</mtext></mrow></msub><mtext>)/2</mtext></mrow></mfenced></mrow><mrow><msup><mrow><mfenced open="|" close="|"><mrow><mtext>Σ</mtext><msub><mrow><mtext>​</mtext></mrow><mrow><mtext mathvariant="italic">p</mtext></mrow></msub></mrow></mfenced></mrow><mrow><mtext>1/2</mtext></mrow></msup><msup><mrow><mfenced open="|" close="|"><mrow><mtext>Σ</mtext><msub><mrow><mtext>​</mtext></mrow><mrow><mtext mathvariant="italic">q</mtext></mrow></msub></mrow></mfenced></mrow><mrow><mtext>1/2</mtext></mrow></msup></mrow></mfrac></mrow></math><img file="EP0750293A2_D0003.tif" /></maths> where µi and Σi indicate a mean value and a dispersion, respectively.

0034Next, at Step S6, the initial clusters φi belonging to the cluster Φm are divided into new clusters Φm and Φ(m+1) nearer to the initial clusters φp and φq obtained at Step S5.

0035The above process will be described with reference to Fig. 4. In an acoustic space 401, assuming that the cluster Φm is positioned generally at the center of the acoustic space 401 and the cluster φp is positioned near at the right end of the acoustic space 401, then the cluster φq is positioned near at the left end of the acoustic space 401. If the initial clusters φi are divided into the new two clusters nearer to the initial clusters φp and φq, the acoustic space 401 is divided at generally the center thereof into two spaces and the total number M of new clusters is two.

0036At Step S7, K-means clustering is performed for the new clusters Φi by using all the initial clusters. This K-means clustering is performed until a preset number of iterations is performed or the total distortion Dm becomes a threshold value or smaller, to search a cluster Φd having a maximum total distortion, and d is set to m to return to Step S3.

0037The total distortion of each cluster can be obtained by the following equation (4).<maths id="math0004" num=""><img file="EP0750293A2_D0004.tif" /></maths>

0038If the total number M of clusters exceeds the preset number (e.g., 600), the process is terminated. In this manner, the shared state of M clusters can be determined.

(3) Determination of a state shared structure of Triphone HMMs (103)

0039Each state of the triphone HMMs designed at Design of Initial Clusters (101) is assigned a nearest cluster among the clusters designed at Top-down Clustering (102) to determine the state shared structure of triphone HMMs by using the shared state numbers. For judgement of a distance, the Bhattacharyya distance was used and the states were assigned. In this manner, the acoustically nearer states are shared between triphone HMMs or in a single triphone HMM.

0040In Fig. 4, a symbol such as /a·Z·i/ indicates a single triphone. In the example shown in Fig. 4, a model having three states is shown. This triphone is a phone "Z" having a right phone "1" and a left phone "a". For example, in Fig. 4, the first states of /a·Z·i/, /a·Z·y/, and /a·Z·a/ are represented by the same state 402, the second states of /a·Z·i/ and /a·Z·y/ are represented by the same state 403, and only the second state of /a·Z·a/ is represented by another state 404. All the first to third states of /a·Z·i/ and /a·Z·y/ are shared by the same state, and so they cannot be discriminated. However, for example, the phone series and triphones of "azia" and "azya" are as follows. <tables id="tabl0001" num="0001"><table frame="all"><tgroup cols="5" colsep="1" rowsep="0"><colspec colnum="1" colname="col1" colwidth="31.50mm" /><colspec colnum="2" colname="col2" colwidth="31.50mm" /><colspec colnum="3" colname="col3" colwidth="31.50mm" /><colspec colnum="4" colname="col4" colwidth="31.50mm" /><colspec colnum="5" colname="col5" colwidth="31.50mm" /><tbody valign="top"><row><entry namest="col1" nameend="col1" align="center">azia (phones)</entry><entry namest="col2" nameend="col2" align="center">a</entry><entry namest="col3" nameend="col3" align="center">z</entry><entry namest="col4" nameend="col4" align="center">i</entry><entry namest="col5" nameend="col5" align="center">a</entry></row><row><entry namest="col1" nameend="col1" /><entry namest="col2" nameend="col2" align="center">qAz</entry><entry namest="col3" nameend="col3" align="center">aZi</entry><entry namest="col4" nameend="col4" align="center">zIa</entry><entry namest="col5" nameend="col5" align="center">iAq</entry></row><row><entry namest="col1" nameend="col1" align="center">azya (phones)</entry><entry namest="col2" nameend="col2" align="center">a</entry><entry namest="col3" nameend="col3" align="center">z</entry><entry namest="col4" nameend="col4" align="center">y</entry><entry namest="col5" nameend="col5" align="center">a</entry></row><row rowsep="1"><entry namest="col1" nameend="col1" /><entry namest="col2" nameend="col2" align="center">qAz</entry><entry namest="col3" nameend="col3" align="center">aZy</entry><entry namest="col4" nameend="col4" align="center">zYa</entry><entry namest="col5" nameend="col5" align="center">yAq</entry></row></tbody></tgroup></table></tables>

0041A silent portion without a phone is represented by q. Since qAz, aZi, and aZy have the same shared state, the two words "azia" and "azya" cannot be discriminated at this point. However, if zIa and zYa, or iAq and yAq, have not the same state shared structure, the two words can be discriminated at one of these points and there is no problem of practical recognition processes.

0042In some case (particularly if the total number of shared states is small), all states of triphones having the different middle phones may share the same state. In such a case, if division is necessary, all triphones can be modified to have different acoustic characteristics by assigning a shared state number obtained by adding 1 to the total shared state number, to the state (e.g., middle state) of each triphone to become discriminable.

(4) Learning state shared triphone HMMs (104)

0043In accordance with the state shared structure determined at (3), the states of triphones are tied to one for performing tied-state learning. This learning may use conventional methods such as EM-algorithm.

0044Fig. 5 is a block diagram illustrating a voice recognition process used by the voice recognition apparatus of the invention.

0045In this embodiment, HMMs 505 are generated by the above described procedure 510. A voice section is extracted by an extractor 501 from a voice signal input from a microphone or the like. The extracted voice signal is analyzed by an acoustic analyzer 502. A likelihood calculator 503 obtains a likelihood of each state of HMMs 505. By using the obtained likelihood, a grammar 506, and a voice recognition network 507, a language searcher 504 searches a language series having the largest likelihood and outputs it as the voice recognition results.

0046Fig. 6 shows the results of recognition of 100 sentences spoken by 10 arbitrary speakers, the recognition being made by using grammars constituted by 1000 words and the voice recognition apparatus of the embodiment. In Fig. 6, a sentence recognition rate (%) indicates a percentage of sentences whose input voices were all correctly recognized, and a word recognition rate (%) is a percentage of correctly recognized words in a spoken sentence.

0047As above, with the voice recognition performed by using the state shared structure with 600 shared states in total generated by the procedure of the first embodiment, sentence and word recognition rates much higher than conventional phone HMM, right-context HMM, and triphone HMM were obtained.

0048Next, the second embodiment of the invention will be described.

0049The above-described clustering algorithm uses a distance scale considering the dispersion σ. Therefore, if the number of initial clusters φi and the number of final clusters are very large, the calculation amount is immense. Therefore, if a distance calculation requiring a large calculation amount for calculating the distances between all clusters is used, a correspondingly longer time is required. In view of this, two calculation types, a simple distance calculation and an accurate distance calculation for calculating accurate distances, are used. The simple distance calculation is used for clusters of a first group starting from the first cluster to an intermediate cluster among the total number of clusters, whereas the accurate distance calculation is used for clusters including the cluster next to the intermediate cluster to the final cluster. In this manner, the time required for distance calculation is shortened and the process can be speeded up. In this second embodiment, the simple distance calculation uses the Euclid distance and the accurate distance calculation uses the Bhattacharyya distance.

0050Fig. 7 is a flow chart illustrating processes by the second embodiment.

0051First, at Step 701, a cluster Φm containing all initial clusters φi is generated. This corresponds to Step S2 in Fig. 3. At Step 701 it is checked whether the total number M of clusters have been obtained. If smaller than M, the procedure continues, and if M, the procedure is terminated. At Step 703, it is judged whether the next clustering uses the simple distance calculation or the accurate distance calculation. If the number (m) of clusters is smaller than total number M (e.g., 600) of clusters subtracted by x (e.g., 10), i.e., from the first cluster to the 590-th cluster, the flow advances to Step 704 to execute clustering with the simple distance calculation.

0052If the number (m) of clusters is (M - x) or larger, the flow advances to Step 705 to execute clustering with the accurate distance calculation to the final cluster M. The processes at Steps 704 and 705 are different in there calculation methods and correspond to Steps S4 to S7 of Fig. 3. Namely, Step 705 uses the Bhattacharyya distance and are the same processes at Steps S4 to S7 of Fig. 3, and Step 704 uses the Euclid distance and calculates the distances at Steps S4 to S7 by the Euclid distance. After Step 704 or 705, one cluster is added at Step 706 and the flow returns to Step 702.

0053The distance calculation in this embodiment may use other distances different from the Bhattacharyya distance and Euclid distance.

0054In the above embodiments, HMM is used as the voice recognition model. Instead of HMM, other models may be used if they are state transition models having distributions. Although the triphone is used as a model unit, the recognition unit may be music or other information.

0055In the above embodiments, although voice recognition is used, the above embodiment procedures are applicable to model design of pattern recognition by using models having similar distributions.

0056The invention is applicable to a system having a plurality of equipments and to a single equipment. The invention is applicable to a program embodying the invention and supplied to a system or equipment.

0057As described so far, the features of the embodiments reside in (1) that clusters are generated through top-down clustering considering the whole acoustic space, (2) that states can be shared between phone classes and in each phone class, and (3) a state shared structure of an arbitrary speaker can be generated directly. Therefore, a triphone HMM of an efficient state shared structure can be designed through top-down clustering. By using the voice recognition model designed by the procedures of the invention, high speed and high performance voice recognition can be realized.