Device and method for extracting musical phoneme, and record medium

Description: TECHNICAL FIELD [Detailed description of the invention]

【0001】

[Technical field to which the invention belongs]

The present invention relates to a musical tone element extraction device and method that can be used when creating a MIDI file or the like based on an input sound from a microphone or the like, and a storage medium, and particularly controls the extraction form of the musical tone element or performs extraction processing. The present invention relates to a musical tone element extraction device and method capable of improving the sound processing for confirmation, etc., and a storage medium, and relates to a technique applicable to, for example, musical tone generation based on an input sound signal, musical tone control, or transcription processing.

【0002】

[Conventional technology]

A musical tone element extraction technique for extracting musical tone elements such as pitch and length from an input sound signal has been conventionally known. For example, in a transcription reproduction device that transcribes music based on the input sound from a microphone, such musical tone elements. Extraction technology is used. A conventional transcription / playback device analyzes and processes an input waveform signal (voice, etc.) input from a microphone, and faithfully transcribes the pitch of the input waveform signal. The user auditions the transcription result by playing it through a sound source (musical tone generator), and evaluates the transcription result. As a result of the evaluation, the same transcription process was repeated again and the transcription result was edited so that the desired transcription result could be obtained.

【0003】

[Problems to be Solved by the Invention]

When a musical tone element is extracted from a sound signal input by a microphone and transcribed, the phrase of the transcribed song is often pronounced with an appropriate tone set on the playback sound source side. In such a case, if you do not perform transcription processing once and then perform pronunciation processing with an arbitrary tone on the playback sound source side based on the transcription result, what kind of song phrase will be the transcription result according to the playback tone? There was a problem that it was difficult to understand whether it could be heard by impression, and it was troublesome to repeat a series of voice input and transcription processing until the voice input of the desired phrase according to the timbre was performed. .. In addition, if the sound to be pronounced is difficult to hear, or if the input waveform signal (voice itself) is too high or too low, it will be difficult to audition and evaluate the transcription result. It was. Furthermore, when trying to perform transcription processing for pitches that cannot be expressed by voice, which is an input waveform signal (for example, when inputting a phrase for a bass part by voice, it is difficult to input voice in the range of the bass part). In this case, there is a problem that the desired pitching result cannot be obtained unless the MIDI file which is the transcription result is edited later and the pitch is changed as a whole.

【0004】

The present invention has been made in view of the above points. For example, even when transcribing a pitch (sound range) that cannot be expressed by voice or is difficult to express by voice, the pitch corresponding to a desired tone color without editing later. It is an object of the present invention to provide a musical tone element extraction device, a method, and a storage medium capable of performing appropriate pitch conversion when extracting musical tone elements so that the MIDI file of the above can be transcribed.

【0005】

Further, the present invention enhances the musical tone playing function by causing the musical tone signal based on the musical tone element extracted based on the microphone input sound to be produced in real time, and enables the musical tone based on the extracted musical tone element to be quickly confirmed. It is intended to provide a musical tone element extraction device and method and a storage medium. Further, the present invention makes it possible to reproduce a musical tone signal based on a musical tone element extracted based on a microphone input sound in real time with a specified timbre, or to perform a demo performance sound according to a specified timbre. Provides a musical tone element extraction device, method, and storage medium that are convenient for musical tone element extraction work such as transcription processing by enabling prompt confirmation of a specified tone color and a musical tone based on the extracted musical tone element. It is something to try.

【0006】

[Means for solving problems]

The music element extraction device according to the present invention is detected by a pitch input means for inputting a sound signal, a pitch detecting means for detecting the pitch of a sound signal input from the microphone input means, and the pitch detecting means. The extraction process that extracts the data determined based on the pitch as a musical sound element, and the data determined based on the pitch obtained by shifting the pitch detected by the pitch detecting means as a whole. It is provided with an extraction means for performing one of the extraction processes for extracting as a musical sound element. Since the pitch detection means faithfully detects the pitch of the input sound signal, when a phrase about a part that the user cannot utter because the sound range is too low, such as a bass sound, is input by the user. Create music sound data that detects the pitch of the sound signal uttered by. In such a case, the extraction means performs an extraction process of extracting as a musical tone element data determined based on the pitch obtained by shifting the pitch detected based on the input sound as a whole. It is possible to easily create musical tone element extraction data converted into a pitch (sound range) that cannot be uttered. Therefore, when this is applied to the transcription process, it is possible to easily perform transcription in a range suitable for a desired tone color, a performance part, or the like.

【0007】

A musical tone element extraction device according to another aspect of the present invention includes a microphone input means for inputting a sound signal, an extraction means for extracting a musical sound element from a sound signal input from the microphone input means, and the extraction means extracted by the extraction means. It is provided with a real-time performance instruction means for instructing the musical tone signal forming means to form a musical tone signal based on the musical tone element in real time. As a result, when the user inputs the sound signal of an arbitrary phrase with the microphone, the musical sound signal based on the musical sound element extracted based on the microphone input sound can be immediately produced in real time, and the musical sound playing function is abundant. Become. For example, by performing it in combination with a control that arbitrarily specifies the timbre of the generated sound or shifts the range arbitrarily, it is possible to pronounce an arbitrary phrase input by voice with a microphone in any timbre and range. This musical effect can be obtained, and the musical tone playing function can be enriched. In addition, since the musical tone signal based on the extracted musical tone element can be quickly confirmed without going through the automatic performance reproduction process, it is possible to immediately confirm whether the transcription state is appropriate when transcribing the microphone input sound. It can be done easily and is very easy to use. For example, in the pitch detection process in the transcription process, the most suitable note pitch is extracted by rounding the pitch detected from the input sound to the nearest note pitch, but this pitch rounding process does not work. In that case, what the user intended to input with one sound (1 note pitch) is extracted as 2 sounds (2 note pitch), or conversely, what he intends to input with 2 sounds (2 note pitch). May be extracted as one note (one note pitch). In such a case, in order to enable appropriate adjustment by changing the user's voice input method or changing the analysis parameters, the performance based on the musical tone element extraction result of the input sound as in the present invention. It is extremely advantageous to be able to pronounce the phrase immediately and listen to it for confirmation.

【0008】

A musical tone element extraction device according to still another aspect of the present invention includes a mic input means for inputting a sound signal, a means for designating a desired timbre, and an extraction means for extracting a musical tone element from the sound signal input from the mic input means. And, the formation of the musical tone signal based on the musical tone element extracted by the extraction means is performed in real time with the designated tone color, or the formation of the musical tone signal based on the performance data stored in advance is performed by the designated tone color. It is provided with a performance instruction means for instructing the musical tone signal forming means to form the musical tone signal according to the selection. As a result, when real-time performance is selected, the same merits as described above can be obtained. In addition, it is possible to confirm how the musical tone performance of the phrase based on the musical tone element extracted from the microphone input tone signal can be heard in relation to the designated timbre. Alternatively, when a musical tone based on pre-stored performance data is to be pronounced with a designated timbre, the designated timbre can be easily confirmed. It is meaningful for this kind of musical tone element extraction device and the transcription device using the music element extraction device to be able to confirm the designated timbre in this way. That is, what type of musical tone signal is based on the musical tone element extracted by the musical tone element extracting device or program according to the present invention, or the musical tone signal based on the data transcribed by the musical instrument or program including the musical tone element extracting device or program. Since it can be formed by using the same sound source, there are many different types of sound sources that may be used at the time of implementation, and the tones with the same name (for example, the names of musical instruments such as piano, bass, and flute). Even if it is a tone color), the sound quality may differ depending on the difference in the sound source, so it will be possible for the user to easily confirm such sound quality (confirmation of the sound quality for each tone color). Brings benefits.

【0009】

It should be noted that the musical tone element extraction device according to the present invention is not limited to the musical tone element extraction device or a single device or product named by a similar name, and in short, it has a musical tone element extraction function. Needless to say, any product form may be taken as a whole as long as it is present. For example, even if the product has an arbitrary product form such as a transcription / playback device or an electronic musical instrument, if the product part includes a device similar to the musical tone element extraction device according to the present invention, that part corresponds to the musical tone element extraction device. In addition, all the inventions disclosed in the specification and drawings according to the present application can be configured and implemented not only as device inventions but also as method inventions, and can also be configured and implemented as computer inventions or computers. It can be implemented in the form of a program of a processor such as a DSP, or can be implemented in the form of a recording medium that stores such a program.

【0010】

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments when the present invention is applied to a transcription / playback apparatus will be described in detail with reference to the accompanying drawings. FIG. 2 is a hardware block diagram of a personal computer operating as a transcription / playback device according to the present invention. The personal computer is controlled by CPU21. CPU21 has program memory (ROM) 22, working memory (RAM) 23, external storage device 24, mouse detection circuit 25, communication interface 27, external interface 2A, microphone interface 2D, keyboard (K) via data and address bus 2P. / B) Detection circuit 2F, display circuit 2H, sound source circuit 2J and effect circuit 2K are connected. The personal computer may have other hardware, but here, the case where the minimum necessary resources are used will be described.

【0011】

The CPU 21 performs processing based on various programs and various data in the program memory 22 and the working memory 23, and music information fetched from the external storage device 24. In this embodiment, as the external storage device 24, a floppy disk drive, a hard disk drive, a CD-ROM drive, a magneto-optical disk (MO) drive, a ZIP drive, a PD drive, a DVD, or the like is used. Further, music information and the like may be imported from another external device (for example, MIDI device) 2B via the external interface 2A and the sound source circuit 2J. The CPU 21 supplies the music information taken in from the external storage device 24 to the sound source circuit 2J, and pronounces it using the external sound system 2L.

【0012】

The program memory 22 stores the system-related programs of the CPU 21 and various parameters and data, and is composed of a read-only memory (ROM). The working memory 23 temporarily stores various data generated when the CPU 21 executes a program, and a predetermined address area of the random access memory (RAM) is assigned to each and is used as a register, a flag, or the like. To. Further, instead of storing the operation program, various data, and the like in the ROM 22, various data and any operation program may be stored in the external storage device 24 such as a CD-ROM drive. The operation program and various data stored in the external storage device 24 can be transferred and stored in the RAM 23 or the like. This makes it possible to easily perform a new installation or version upgrade of the operating program.

【0013】

In addition, it is possible to connect to various communication networks 28 such as LAN (local area network), the Internet, and telephone lines via the communication interface 27, and data (music with data) is connected to other server computers (not shown). Information, etc.) may be exchanged. This makes it possible to download operation programs and various data from the server computer. In this case, the personal computer serving as the client sends a command requesting the download of the operation program and various data to the server computer 29 via the communication interface 27 and the communication network 28. In response to this command, the server computer 29 transmits a predetermined operation program, data, or the like to another personal computer via the communication network 28. The personal computer receives these operation programs and data via the communication interface 27 and stores them in the RAM 23 or the like. This completes the download of the operation program and various data.

【0014】

The present invention may be implemented by a commercially available electronic musical instrument or the like in which an operation program or various data corresponding to the present invention is installed. In that case, the operation program and various data corresponding to the present invention may be provided to the user in a state of being stored in a storage medium such as a CD-ROM or a floppy disk that can be read by an electronic musical instrument.

【0015】

The input signal from the mouse 26 is converted into position information by the mouse detection circuit 25 and supplied to the data and the address bus 2P. The microphone 2C converts an audio signal or musical instrument sound into a voltage signal and outputs it to the microphone interface 2D. The microphone interface 2D converts the analog voltage signal from the microphone 2C into a digital signal and outputs it to the CPU 21 via the data and the address bus 2P. The keyboard (K / B) 2E is equipped with multiple keys for inputting character information and keys such as function keys, and has a key switch corresponding to each key. The keyboard detection circuit 2F includes a key switch circuit provided corresponding to each key of the keyboard 2C, and outputs a key event corresponding to the pressed key. In addition to these hardware switches, various switches may be displayed on the display 2G in the form of buttons so that they can be selected by software with the mouse 26. The display circuit 2H controls the display contents of the display 2G. The display 2G is composed of a liquid crystal display panel (LCD) and the like, and its display operation is controlled by the display circuit 2H.

【0016】

The sound source circuit 2J can simultaneously generate musical sound signals on multiple channels, input data and music information (MIDI file) given via the address bus 2P and external interface 2A, and based on this information, the musical sound signal. Occurs. In the sound source circuit 2J, as a configuration for simultaneously producing a music signal with multiple channels, one circuit is used in a time-divided manner to form multiple sounding channels, or one sounding channel is composed of one circuit. It may be in the form that is used. Further, any musical sound signal generation method may be used in the sound source circuit 2J. The musical tone signal output from the sound source circuit 2J is pronounced by the sound system 2L consisting of an amplifier and a speaker. An effect circuit 2K that gives various effects to the musical sound signal is provided between the sound source circuit 2J and the sound system 2L. The sound source circuit 2J itself may include an effect circuit. The timer 2N generates a tempo clock pulse for counting the time interval and setting the tempo at the time of playing the music information. The frequency of this tempo clock pulse is adjusted by a tempo switch (not shown). The tempo clock pulse from the timer 2N is given to the CPU 21 as an interrupt instruction, and the CPU 21 executes various processes at the time of automatic performance by the interrupt process.

【0017】

An embodiment in the case where the personal computer of FIG. 2 operates as a transcription / playback device will be described with reference to FIGS. 1, 3 to 10. FIG. 3 is a diagram showing a main flow when a personal computer operates as a transcription / playback device. CPU21 operates according to this main flow. Hereinafter, the operation of this main flow will be described in order.

【0018】

First, the initial setting process is performed in the first step. In the initial setting process, predetermined initial values are set for each register and flag in the working memory 23 of FIG. After the initial setting process is completed, the panel setting process, the performance input process, and the performance process are executed in order. In the panel setting process, processing corresponding to the operation status of various controls displayed on the display 2G is performed. The performance input process is a process in which the user inputs voice using the microphone 2C. In the performance processing, processing is performed according to whether the performance mode is the transcription mode or the playback mode.

【0019】

As shown in FIG. 4, the panel setting process includes a test mode selection process, a transcription mode selection process, a performance mode selection process, a device-driven selection process, and other selection processes. FIG. 1 is a diagram showing the first half of the details of the test mode selection process. In the test mode selection process, first, it is determined whether or not the test mode button is operated on the display 2G, that is, whether or not the test mode is selected. When a test mode is selected, processing related to various settings of each test mode is performed. If it is not selected, it returns immediately and the transcription mode selection process is executed.

【0020】

Processing related to various test mode settings includes level adjustment selection processing, tone color trial specification processing, real-time demo performance selection or instruction processing, octave shift change processing, and other specification processing. It is determined whether or not the setting is selected. These settings, selections, or instructions are made by the user operating the buttons or controls on the operation panel displayed on the display 2G. Alternatively, settings, selections, or instructions may be automatically made.

【0021】

If it is determined that the level adjustment selection process is performed, it is determined whether or not it is in the automatic setting mode. Here, the automatic setting mode is a transcription mode in which ambient noise and / or performance support sound is picked up by a microphone and detected as noise, and based on the input noise from the detected microphone or the amount of noise according to the performance support sound. This mode automatically sets the threshold value for each process. In the case of the automatic setting mode, it is further determined whether or not the mode is the auxiliary mode. When not in the automatic setting mode, it is determined whether or not the threshold value setting switch is in the high position. In the case of the high position, the threshold value aH for the high position is used as the threshold value, and in the case of the low position, the threshold value is aH. Set the low position threshold value aL as the level threshold value for pitch detection. (Note that the level threshold value for pitch detection is, for example, a level threshold value for detecting a section of an input sound signal in which a valid sound exists, and the valid sound detected in this way exists. Pitch detection processing is performed for the section to be used.) In the case of the automatic setting mode and the auxiliary mode, an appropriate support sound (metronome sound or back performance sound) selected by an arbitrary or user is generated. Is output from the speaker of the sound system 2L, and the input sound signal from the microphone 2C is recorded for about 5 seconds. When the auxiliary mode is not selected, the input sound signal from the microphone 2C is recorded for about 5 seconds without generating the above-mentioned support sound. Here, the auxiliary mode means that when the user inputs voice with the microphone 2C (speaks the desired melody to be transcribed) at the time of performance input described later, a metronome sound or a back performance is used as the performance support sound (auxiliary sound). This is a mode in which performance input is performed while making the sound sound. Therefore, when the auxiliary mode is not selected, the input sound signal from the microphone 2C is recorded without generating the performance support sound. In principle, in the automatic setting mode, the user does not actively generate voice toward the microphone 2C regardless of the auxiliary mode or the non-auxiliary mode, and the auxiliary mode is used. In the case, only the performance support sound (auxiliary sound) is generated while the input signal from the microphone 2C is recorded, and in the non-auxiliary mode, such a performance support sound (auxiliary sound) is not generated. The input signal from the microphone 2C shall be recorded. Therefore, in the automatic setting mode according to the auxiliary mode, the performance support sound is mainly picked up and recorded by the microphone 2C (although it includes ambient noise). Further, in the automatic setting mode according to the non-auxiliary mode, only the ambient noise is mainly picked up by the microphone 2C and recorded. The performance support sound (auxiliary sound) is not limited to the mode generated by the transcription / playback device shown in FIG. 2, but is generated by other appropriate modes (for example, another device or musical instrument, metronome, etc., or manually operated by the user. Of course, it may be generated by stepping motion).

【0022】

When the recording of the input signal from the microphone 2C in the automatic setting mode according to the auxiliary mode or the non-auxiliary mode is completed, the maximum value of the absolute value of the recorded sound signal is detected as the noise amount. That is, in this embodiment, not only normal noise (unwanted noise emitted from the surrounding environment) but also performance support sound is regarded as a noise signal, and the amount of noise is detected in advance. In this case, any method may be used for detecting the amount of noise. For example, this noise amount may be detected as follows. First, the level range of the audio signal from 0 to 32767 is divided into 100 units such as 0 to 99, 100 to 199, 200 to 299, and so on. Further, the recorded audio data is divided into unit time intervals of, for example, every second. Since this audio data is recorded for about 5 seconds, the maximum value in each unit time interval is detected, and it is the level range of 0 to 99, 100 to 199, 200 to 299, ... mentioned above. Detects whether it belongs to, and collects statistics over the entire unit time interval (for example, counting for each applicable level range). The level range with the highest count value (frequent) in the statistics is extracted, and either the maximum value, the center value, the minimum value, or the average value of the absolute value of the audio signal within that level range is noised. It may be detected as a quantity.

【0023】

For example, if the maximum level value in each unit time interval in the recorded 5-second audio data is 180,205,210,245,315, 100 to 199 is 1 point, 200 to 299 is 3 points, and 300 to 399 is 1 point. The number of counts is high in the range of 200 to 299 with 3 points. Therefore, any one of the maximum value 245, the center value 210, the minimum value 205, or the average value 220 of the three values may be detected as the noise amount. Alternatively, any one of the maximum value 299, the minimum value 200, and the center value 250 from 200 to 299 may be detected as the noise amount. In this way, an appropriate amount of noise can be detected by determining the amount of noise with reference to the concentrated and constant width level range of the maximum level. That is, for example, if only one maximum level in the recorded audio data for 5 seconds is detected as the amount of noise, there is sudden noise depending on the noise, so that the level is relatively high due to such sudden noise. Will be detected as the amount of noise, resulting in an inappropriate error. However, if the noise amount is detected by a method of statistically processing the noise level in a plurality of time intervals as in the above embodiment, an inappropriate noise amount due to such sudden noise can be detected. It can be avoided.

【0024】

Next, it is determined whether or not the value of the amount of noise detected in this way is within an appropriate range in the transcription environment. That is, when the amount of noise is too large and makes it difficult to detect the actual audio signal, it is determined that the amount of noise is not within the appropriate range. Therefore, if the amount of noise is not within the appropriate range (NO), for example, a warning such as "The amount of noise is large and the detection sensitivity drops" or / and "The amount of noise is large so that the room should be quiet". Display the sentence on the display 2G. If the amount of noise is within the appropriate range (YES), the threshold value a in each process in the transcription mode is determined according to the value. The threshold value a is a threshold value related to the input voice level used for pitch detection processing during voice analysis. Here, as the threshold value a, a value obtained by adding a predetermined constant value to the detected noise amount may be used, or a value obtained by multiplying the noise amount by a predetermined ratio may be used.

【0025】

As is clear from the above, it is desirable to prevent the user from emitting unnecessary sound into the microphone 2C during the 5-second microphone recording in the above-mentioned automatic setting mode for accurate noise detection. .. However, of course, it does not mean that the user should never emit sound during the microphone 2C input in this case, and the amount of noise (performance support sound and ambient noise) is larger when the user does not emit sound. The purpose is that it has a great advantage because it can be easily detected. For example, if some user voice is included in the input signal from the microphone 2C, if necessary, a simple human voice formant analysis or human voice band determination is performed, and the user is clearly identified. By taking measures such as cutting the part that is presumed to be voice or excluding the noise amount from being detected, even if the user voice is mixed during microphone recording in this automatic setting mode, it will be ignored. It is also possible.

【0026】

Returning to the explanation of the judgment step of "Are there level adjustment selection?" Before entering the automatic setting mode in Fig. 1, if it is judged that it is not the level adjustment selection process here, go to Fig. 5 and try the tone. Judge whether or not there is a designation. If the tone color test is specified, it is judged whether or not the real-time demo performance is specified. When the real-time demo performance is specified, the audio input of an arbitrary melody is performed in real time via the microphone 2C, for example, the input audio signal for the 5 seconds is transcribed, and this input audio is converted to the specified tone. Convert to the corresponding musical tone and perform a demo performance. In the demo performance processing in this case, each musical tone is generated according to the transcription result for about 5 seconds and the demo performance pronunciation processing is performed. In this case, the demo performance pronunciation processing performs the voice analysis processing of FIG. 9 (transcription) as described later according to the user's voice input, and outputs the musical sound of FIG. 10 according to the real-time analysis result (transcription result). It consists of executing processing in real time. That is, the pitch and pronunciation section are analyzed from the input voice, and the musical tone according to the analysis result (for example, the musical tone that is rounded to the required scale note pitch according to the analysis and the pronunciation on / off time is adjusted) is generated in real time. Be pronounced. Here, the microphone input voice is simply analyzed and transcribed, and the transcribed phrase is immediately pronounced with a specified tone, and the automatic performance progress processing is not performed. That is, the pronunciation timing is not controlled in the automatic performance, and the pronunciation timing is controlled in real time corresponding to the pronunciation on / off section extracted from the input voice. In addition, it is not necessary to perform secondary processing such as recording and saving the transcribed data and modifying (editing) the data. However, as described above, the timbre of the generated musical tone can be arbitrarily specified, and as will be described later, it is also possible to arbitrarily change the range by octave shift (or pitch shift) control to generate the musical tone. In the present specification, a performance in which a musical sound based on a musical sound element extraction result (transcription result) of an input sound is immediately generated in real time is referred to as a real-time demo performance.

【0027】

If real-time demo performance is not specified, phrase data for demo performance (automatic) prepared with different or appropriate phrases (performance data for demo performance) for each instrument type (piano, guitar, etc.) (Performance data) is used to select and play a phrase for demo performance that corresponds to the specified tone, and the demo performance process is performed by pronouncing this phrase. The performance processing (normal demo performance processing) based on the existing data in this case is also about 5 seconds as described above. In this normal demo performance process, the automatic performance progress process is performed, and the demo performance is performed while controlling the pronunciation timing according to the automatic performance phrase data. The existing performance data for the demo performance may be sample data dedicated to the demo performance, but the transcribed phrase data (stored in the storage means) previously voice-input by the user may be used. Or, if there is preset automatic performance phrase data that can be used by the user at any time, it may be used.

【0028】

The above-specified timbre can be arbitrarily changed (that is, can be specified) by the user. By performing real-time demo performance or normal demo performance processing with musical tones corresponding to the tones specified in this way, the tones specified by the user for the transcription processing can be obtained as desired before the full-scale transcription processing is started. It will be possible to easily confirm whether the tone is a tone and whether the transcription process is performed accurately.

【0029】

One advantage of such a designated timbre audition function is that there are various types of musical instruments that form musical tone signals corresponding to transcribed scored (for example, MIDIized) phrase data, and the present invention What type of sound source is used for the musical tone signal based on the musical tone element extracted by the musical tone element extraction device or the program, or the musical tone signal based on the data transcribed by the musical tone element extraction device or the transcription device including the program or the program. Since it can be formed even if it is possible, there are many different types of sound sources that may be used at the time of implementation, and the tones with the same name (for example, the tones of musical instrument names such as piano, bass, and flute) are used. Even if there is, the sound quality may differ depending on the difference in the sound source, so it is useful for confirming such sound quality (confirmation of the sound quality for each tone color).

【0030】

Another advantage of the designated timbre audition function is that it makes it easy to check what kind of impression the user-specified timbre has in relation to the actual performance melody. For example, in the case of a real-time demo performance, it is possible to immediately confirm whether or not the specified timbre matches in relation to the desired song phrase to be transcribed by voice input by the user. Further, in the case of a normal demo performance, since the designated timbre can be confirmed in relation to a suitable existing phrase, it is easy to immediately understand the characteristics of the timbre. In addition to the purpose of confirming the specified tone, this function is also useful. For example, this designated tone can be fixed (semi-fixed), and in the real-time demo performance mode, melody phrases of various modes can be input by user voice, and these melody phrases can be listened to and compared with each other. In that case, since the timbre is constant, there is an advantage that it is easy to compare each melody phrase input by the user's voice. In addition, by performing real-time demo performance and normal demo performance in order with a common specified tone, it becomes easier to compare existing phrases with user-input phrases because they have the same tone, and it is easier to evaluate user-input phrases. Become.

【0031】

The above real-time demo performance can be advantageously used not only for the purpose of confirming the specified timbre, but also for the purpose of immediately listening to and confirming the musical tone element extraction result or the musical tone phrase based on the transcription result according to the user's voice input. it can. For example, in the pitch detection process in the transcription process, the most suitable note pitch is extracted by rounding the pitch detected from the input sound to the nearest note pitch, but this pitch rounding process does not work. In that case, what the user intended to input with one sound (1 note pitch) is extracted as 2 sounds (2 note pitch), or conversely, what he intends to input with 2 sounds (2 note pitch). May be extracted as one note (one note pitch). In such a case, in order to enable appropriate adjustment by changing the user's voice input method or changing the analysis parameters, the above real-time demo performance can be used to immediately convert the musical tone element extraction result or transcription result. It is extremely advantageous to be able to pronounce and listen to the based musical phrase. In that case, the embodiment may be modified so that the real-time demo performance is performed with a predetermined tone without specifying an arbitrary tone color, and either the real-time demo performance or the normal demo performance is selected. The embodiment may be modified so that only the real-time demo performance is selected (that is, the normal demo performance is not performed). When implementing such a transformation, change the judgment step of "with tone test designation" in Fig. 5 to the judgment step of "real-time demo performance", and when "real-time demo performance" is NO, the normal demo You can delete the step of playing.

【0032】

As a modification of the above real-time demo performance, the sound signal input to the microphone in real time is stored in a buffer, the sound signal stored in the buffer is read out, and transcription processing is performed so that the musical sound transcribed in this way is pronounced with a specified tone. You may. In that case, there may be a slight discrepancy between the time when the sound signal is input to the microphone and the time when the transcribed musical tone is pronounced. It is also possible to store the data transcribed for the real-time demo performance and modify the embodiment so that the data is reproduced and played.

【0033】

Returning to the explanation of the determination step of timbre trial designation in FIG. 5, if it is determined that the tone color trial designation is not specified, it is determined whether or not the octave shift has been changed. When a desired tone color is specified during the transcription process, or when the octave shift amount is set by operating a switch, it is determined here that there is an octave shift change (YES). If the octave shift is changed, it is determined whether or not the tone color correspondence is specified. The tone color correspondence designation is to set a predetermined octave shift amount corresponding to the designated tone color. Therefore, if the determination of "with tone color correspondence designation" is YES, a predetermined shift amount is selected and set corresponding to the designated tone color. In that case, in order to specify the voice signal to be the target of pitch detection in the transcription process (distinguish from noise), a setting item for objectively estimating the voice quality of the user who inputs the voice is provided (this setting). Can be set manually or automatically determined and set), using information such as user gender (male or female) and voice quality (high, normal, low) set in the setting item. , Determine or calculate the octave shift amount corresponding to the tone color specified by the user (that is, the tone color of the musical tone output generated based on the input voice) with reference to a predetermined table or the like or by using a predetermined calculation algorithm or the like. Set. For example, when a person with gender: female / voice quality: high transcribes with a bass tone, it is converted to a range lower than the actually detected range by a predetermined pitch (for example, lowered by 2 octaves) so that the transcription process is performed. Even when octave shift control is performed, or when a person who says gender: male / voice quality: normal transcribes with the bass tone, it is converted to a range that is different from the actually detected range by a predetermined pitch ( For example, the octave shift is controlled so that the transcription process is performed (lowering by one octave).

【0034】

In addition, in order to identify the voice signal for pitch detection of transcription (distinguish from noise), the user who inputs it on a trial basis before the actual (production) transcription processing actually voices (voice according to the singing pitch). ) Is input, the frequency band of the input voice is detected, the range considered to be uttered (sung) by the user is determined based on the detected result, and the range of processing for pitch detection is limited (that is,). , Detects the input sound filter value b and limits the band based on this filter value b), and also performs processing to reduce the load of signal processing, but considers the compatibility of the detected result with another tone. It is advisable to compare the user-spoken sound range and the appropriate sound range considered for each tone color, and determine the shift amount with reference to a predetermined shift amount setting table. For example, if a user who has only a low voice (as the result of detection is) wants to make a musical tone with a piano tone, decide on a scale higher than the actually detected pitch (such as raising it by one octave). To do. The user's range may be detected in advance, and the octave shift amount may be set based on whether or not the user's range matches the specified tone range, or even if both ranges match. Needless to say, it may be shifted arbitrarily.

【0035】

In FIG. 5, when the judgment of "with tone color correspondence designation" is NO, the octave shift value setting process that does not depend on the specified tone color is performed. For example, the octave shift value is determined according to an arbitrary shift amount (a specific value such as one octave or two octaves) set by the user. Alternatively, the input voice of the user may be analyzed and the octave shift value may be determined according to the user's range. Alternatively, the user's input voice is analyzed to determine the first octave shift value according to the user's range, and the desired second octave shift value set and input by the user is taken into consideration, and the final combination of the two is used. The amount of octave shift may be determined. The octave shift amount determined or set as described above is indicated by c. The range of the pitch detected from the input audio signal is changed according to the octave shift amount determined or set in this way, and the final pitch determination process or scale determination process is performed for the pitch whose range has been changed in this way. .. In this octave shift process, the pitch shift amount is not limited to the octave unit, and the shift may be set or determined in a range finer than the octave.

【0036】

On the other hand, in FIG. 5, when the determination of "the octave shift has been changed" is NO, it is determined whether or not "there is another designation". In the case of "with other specifications", the required processing is executed according to the specifications.

【0037】

Of course, the above-mentioned octave shift function can be combined with the above-mentioned "tone audition function" or "real-time demo performance" function. For example, suppose that a desired timbre (for example, a bass timbre) is first specified prior to voice input, and YES is determined in the octave shift change determination in FIG. 5 accordingly. This determines the amount of octave shift according to this specified timbre (eg, a "bass" timbre). Next, when the "timbre audition" function is selected, the judgment of "timbre trial designation" in Fig. 5 is judged to be YES, and the "real-time demo performance" function is selected. The judgment is YES, and "real-time demo performance" is executed. That is, the range of the input voice is shifted by a determined shift amount and the transcription process is performed, and the musical tone of the transcribed phrase is pronounced in real time with the specified bass tone.

【0038】

FIG. 6 is a diagram showing details of the transcription mode selection process of FIG. In the transcription mode selection process, first, it is determined whether or not the transcription mode button on the display 2G has been operated, that is, whether or not the transcription mode has been selected. When the transcription mode is selected, the processing related to the setting of each transcription mode is performed, and when there is no selection, the process returns immediately and the selection processing of the next performance mode is executed.

【0039】

The processing related to the selection setting of the transcription mode is performed according to whether or not there is a switch operation related to a change in the use of the auxiliary mode, a tone color change, a change related to waveform recording, and other changes. First, when it is determined that there is a change in whether or not the auxiliary mode is used, it is determined whether or not the setting by the auxiliary mode is related to the setting of the metronome sound. That is, in the auxiliary mode, it is determined whether the sound produced at the same time as the voice input is the metronome sound or the back performance. If it is determined that the metronome sound is related to the setting, various settings such as the pronunciation tempo and volume of the metronome sound are made. If the volume of the metronome sound is changed, the level threshold value of the transcription input voice may be changed. If it is determined that it is not related to the setting of the metronome sound, it means that the back performance is performed as a performance support sound. Therefore, here, various settings related to the chord pattern are made in order to automatically play the back performance. In addition, since the volume at the time of playing the chord pattern can be set in the same way in the chord pattern setting process, the threshold value of the transcription input voice is changed according to the volume of the changed chord pattern in this case as well. There is also. Whether or not the auxiliary mode is used If there is no change, it is determined whether or not the tone color has been changed. If there is a change, change to the specified tone. That is, the user arbitrarily sets a desired timbre as the timbre of the transcribed result by switch input, data input, or the like. If there are other changes, the specified changes will be processed.

【0040】

If there is a change in the waveform recording, it is determined whether or not the change is in the transcription recording time. When changing the recording time, the memory area in RAM23 is secured for the specified time. By securing a memory area in response to the transcription recording time, it is possible to set whether advance how much time the user (how many seconds) perform the play. Further, since it is not necessary to secure the memory area by this, the load of the program can be reduced. This is because by making the memory area variable, it takes time to allocate the memory, and there is a drawback that it is not suitable for real-time processing. Therefore, the burden can be reduced by allocating the memory area in advance in this way. In addition, there is also an advantage that the remaining amount of the memory area secured at the time of performance input can be displayed at a glance by displaying the remaining amount of the memory area. If the transcription recording time is not changed, it is determined whether or not the waveform recording is changed. If there is waveform recording, change the waveform recording mode. There are two types of waveform recording modes: a mode in which the waveform is recorded while transcribing, and a mode in which the waveform is not recorded while transcribing, so these can be changed and set alternately.

【0041】

FIG. 7 is a diagram showing details of the performance mode selection process. In the performance mode selection process, first, it is determined whether or not the performance mode selection button on the display 2G has been operated, that is, whether or not there is an instruction to change the playback state, and if there is an instruction, the process related to the change of each performance mode. If there is no instruction, it returns and executes the next device drive selection process. As the processing related to the change of the performance mode, the processing is performed according to whether or not there is a switch operation related to the change of the playback volume, the change of the tone color, the change of the speed (tempo), and other changes. First, when it is determined that the playback volume has been changed, the volume is changed. If it is determined that the timbre has been changed, the timbre is changed. If it is determined that the speed (tempo) has changed, the tempo is changed. If it is determined that there is another change, the change process is performed according to the instruction content.

【0042】

FIG. 8 is a diagram showing details of the device drive selection process. In the device-driven selection process, it is determined whether or not the transcription start instruction button, playback start instruction button, stop instruction button, and other instruction buttons on the display 2G have been operated, and processing is performed according to the determination result. First, when the transcription start instruction button is operated, in preparation for the transcription start, the transcription is prepared so that the transcription starts from the time when the voice is input. If there is a set auxiliary mode, the support sound (metronome sound or back performance sound) corresponding to the set auxiliary mode is started to be generated. When the playback start instruction button is operated, the performance processing flag corresponding to the operation is set, or the specified data or the data after the transcription is completed is played back. When the stop instruction button is operated, the currently executing process (transcription process or playback process) is stopped. If there is an operation of another instruction button, the process is performed according to the content of the operated button. For example, a pause instruction, a data feed process, a return process, and the like are executed. When the device drive selection process is completed, the selection process corresponding to the other selection switches is performed, and the panel setting process is completed.

【0043】

When the series of processing related to the panel setting in FIG. 4 is completed in this way, the performance input processing is performed by returning to FIG. This performance input process is a voice input work by the user using the microphone 2C. This voice input work captures the user's voice signal input from the microphone 2C. Next, in accordance with the operation of the transcription-related or performance-related buttons (not shown), the performance process is performed according to the instruction. For example, when the performance start start button is operated, the corresponding performance processing flag is set, and when the transcription processing start button is operated, the corresponding transcription processing flag is set. Further, since the performance processing is also performed based on the conventionally known automatic performance technique, the description thereof will be omitted here. Needless to say, the transcription process is performed according to the scale rounding condition selected by the user as described above.

【0044】

The basic operation of this transcription / playback device is described in Japanese Patent Application No. 9-336328, which the inventor of the present application filed earlier. Therefore, only the outline of the voice analysis process and the musical tone output process is described here. explain. First, there are two types of performance processing in FIG. 3, a transcription mode and a playback mode. In the transcription mode, the input voice is made into a musical tone, and the musical note is displayed as a note on the staff notation and the musical tone is pronounced by the sound source. In the playback mode, the selected music data or the transcribed and stored data is read out and sounded by the sound source. There are other modes, but these two types will be explained here. The musicalization of the input voice in the transcription mode is performed by the voice analysis process shown in FIG. Since the details of this voice analysis process are described in the above-mentioned prior application, they will be briefly described here. In the voice analysis process, pitch detection process is performed as shown in FIG. This pitch detection process applies to the level threshold value a determined in the test mode selection process of FIG. 1, the level threshold value aH for the high position, the level threshold value aL for the low position, and the input sound filter value b. It is done based on. Here, the input sound filter value b is a filter characteristic corresponding to the detection result of detecting the frequency band of the input voice. In the scale rounding process, the scale rounding process is performed according to the designated scale rounding condition. In the past, the voice analysis processing was completed by performing pitch detection and scale rounding processing, but in this embodiment, the result of the scale rounding processing is further octave-shifted based on the octave shift value c to finalize the scale. I try to decide. That is, the pitch data is shifted by transcribing so that the range corresponds to the tone set as the transcription tone.

【0045】

A musical tone signal is output from a sound source by the musical tone output process of FIG. 10, and is pronounced via a sound system. The musical sound output processing is the same as the conventional automatic performance processing, and is a series of processing in which performance data is taken in, a musical sound signal is generated by a sound source, and sound processing is performed by a sound system. In the sounding process, a value determined based on the threshold values a, aH, or aL determined in the test mode selection process of FIG. 1 is used as the volume value. That is, when the threshold value a is determined by the automatic setting mode, the volume value d based on the threshold value a is used, and when the threshold value setting switch is in the high position, it is for the high position. When the volume value dH is in the low position, each sound processing is performed based on the volume value dL for the low position. If the playback volume is changed in the performance mode selection process shown in FIG. 7, that volume is given priority. In the above-described embodiment, the case where the volume value is determined based on the threshold values a, aH, and aL has been described, but the rate of change in the volume is determined in addition to the volume value, and the rate of change is used as the rate of change. The volume may be changed accordingly.

【0046】

It should be noted that the octave shift as described above is difficult to understand for a user whose voice pitch is too low or too high when auditioned based on the result of voice analysis processing (transcribed data). Because there was something. Therefore, by shifting the pitch up and down in octave units, it is possible to secure the pronunciation at a pitch that is easy to hear, so that the audition based on the result of the voice analysis processing can be made smooth. In this case, it is possible to automatically perform an octave shift according to the set tone, so this octave shift is used for users who cannot speak at the pitch of the bass, such as when creating a bass phrase. By using this, it becomes possible to easily create a pitched sound in the bass range, and to easily audition the sound. On the contrary, the same is true when arranging high-pitched sounds. By shifting the octave in this way, it is possible to create sounds that cannot be produced by human voice.

【0047】

In the above-described embodiment, only the case of transcription has been described, but for example, the user sang by automatically assigning velocity information to each note using the level value when the user sang. Alternatively, the nuance of the performance may be reflected in the transcription result as it is. For example, it finds a valid position in the note section and detects the maximum peak of that part. After inputting all notes, search for the maximum value among the maximum values of the peak. The maximum value of all sections is set to 127, and the velocity is determined based on that. The largest value in all sections may be used as a reference.

【0048】

Further, in the above embodiment, in the real-time demo performance, the designated tone can be confirmed by pronouncing the musical tone transcribed based on the sound signal input by the microphone with the designated tone. The microphone input signal may be played in real time for purposes other than checking the tone. For example, the sound signal input by the microphone is transcribed in real time, and the pitch of the transcribed musical sound is changed in real time to produce the musical sound, or the transcribed musical sound is pitch-changed in real time and the pitch is not changed at the same time. You may try to pronounce it.

【0049】

Further, in the above-described embodiment, the case where the present invention is applied to the transcription / playback apparatus has been described, but the present invention is not limited thereto. Even if it is applied to a sound signal conversion device or a sound signal conversion method such as a voice changer or a harmony addition device that simply converts an input sound signal into a signal of a specified tone color or pitch and outputs it without performing transcription processing. Good. Alternatively, the present invention is applied to a device or method for extracting a musical tone element of an input sound signal, controlling some other musical tone according to the extracted musical tone element, or controlling an image displayed on a display. You can also. In any of the above-described embodiments, the sound signal input to the microphone is not limited to the human voice sound signal, and may be another sound signal (for example, an existing performance sound signal).

【0050】

In the octave shift control shown in the above embodiment, as described above, the shift amount is not limited to the width in octave units, and may be set in any pitch shift width less than an octave. In addition, this shift amount is determined by the set tone type, or by the relationship between the tone type and the user's personal sound quality setting (gender, voice pitch, etc.), or the tone type and input. The determination may be made based on the relationship with the detection result of the frequency band of the voice, or may be arbitrarily determined by the user.

【0051】

In the above embodiment, the present invention is implemented by a computer-configured device, but the present invention is not limited to this, and equivalent functions may be configured by a dedicated LSI, and equivalent functions may be provided by logic, gate array, storage, and the like. It may be configured by connecting a discrete circuit such as a memory. Further, when the software configuration is used, it is not limited to a general-purpose computer such as a personal computer, but a microcomputer deployed in a required device such as an instrument or a music device may be used, and a DSP ( It may use a digital signal processor).

【0052】

[Effect of the invention]

According to the present invention, there is an effect that even when transcribing a pitch that cannot be expressed by voice, a MIDI file having a pitch corresponding to a desired tone can be transcribed without editing later. In addition, there is an effect that a demo performance can be performed by the tone of the transcription result before the actual transcription processing. In addition, it produces various effects as described above.

[Simple explanation of drawings]

[Figure 1]

It is a figure which shows the first half part of the detail of the selection process of the test mode in the panel setting process of FIG.

[Figure 2]

It is a hardware block diagram of the personal computer which operates as a transcription reproduction apparatus including the musical tone element extraction function which concerns on this invention. [Fig. 3]

It is a figure which shows the main flow of the transcription reproduction apparatus including the musical tone element extraction function which concerns on this invention.

[Fig. 4]

It is a figure which shows the detail of the panel setting process of FIG.

[Fig. 5]

It is a figure which shows the latter half of the detail of the setting process of the test mode in the panel setting process of FIG.

[Fig. 6]

It is a figure which shows the detail of the selection process of the transcription mode in the panel setting process of FIG.

[Fig. 7]

It is a figure which shows the detail of the performance mode selection process in the panel setting process of FIG.

[Fig. 8]

It is a figure which shows the detail of the equipment drive selection process in the panel setting process of FIG.

[Fig. 9]

It is a figure which shows the detail of the voice analysis processing in the performance place of FIG.

[Fig. 10]

It is a figure which shows the detail of the musical tone output processing in the performance processing of FIG.

[Explanation of symbols]

21 ... CPU, 22 ... ROM, 23 ... RAM, 24 ... external storage, 25 ... mouse detection circuit, 26 ... mouse, 27 ... communication interface, 28 .. .Communication network, 29 ... server computer, 2A ... external interface, 2B ... other external devices, 2C ... mouse, 2D ... microphone interface, 2E ... keyboard, 2 ... Keyboard detection circuit, 2G ... display, 2H ... display circuit, 2J ... sound source circuit, 2K ... effect circuit, 2L ... sound system, 2N ... timer, 2P ... data and Address bus