Information search method, device, and program, and computer-readable recording medium
12 claims: 11 independent, 1 dependent
- 1クライアントからの文字列を取得する文字列入力手段と、該文字列が含まれる情報をインデックスDBから取得する文字列情報検索手段と、該文字列と検索された情報との類似度を算出する類似度算出手段と、類似度の高い順に検索された情報を出力する出力手段と、を有する情報検索装置であって、 前記文字列情報検索手段は、 入力された前記文字列が複数の単語からなる場合に、各単語に基づいて、単語及び該単語の 文書中における文単位での 出現位置を格納したインデックスDBを検索し、 各単語を含む文書dと当該文書dでの各単語の出現位置を取得し、 文字列を分割して得た単語の集合をQとし、Pos d (q i )を文書d中の単語q i の出現位置とし、Pos dk (q j )を文書d中の単語q j の出現位置からk減算した値とし、kを所定のカウンタ値とし、αを所定の係数とし、Count(Pos,Pos)を、2つの位置データが与えられた時に、その一致度を返す関数とした場合に、 前記類似度算出手段は、 単語の出現位置間の一致度score(Q,d,k)を により算出し、当該一致度score(Q,d,k)に基づいて 類似度を算出することを特徴とする情報検索装置。
- 2nを0以上の定数としたときに、前記類似度算出手段は、 により、前記文字列と文書dとの類似度sim(Q,d)を算出する ことを特徴とする請求項1に記載の情報検索装置。
- 3前記インデックスDBは、 各単語の 出現位置をビット列で表現し、 前記文字列情報検索手段は、 前記インデックスDBを検索し、前記ビット列による単語の出現位置を取得する手段を含み、 前記類似度算出手段は、 前記インデックスDBの前記出現位置のビット列を用いてビットの論理演算により類似度を求める手段を含む請求項1 又は2に 記載の情報検索装置。
- 4前記入力された前記文字列が複数の単語からなる場合に、該単語に基づいて、近接を考慮しない単語が登録されている近接未考慮単語リスト記憶手段を参照して、該単語が該近接未考慮単語リスト記憶手段に存在する場合は近接を考慮しない単語と判断する文字列選択手段を更に有し、 前記文字列情報検索手段は、 前記文字列選択手段において、近接を考慮しない単語であると判断された単語については、全ての文に該単語が存在することとして前記インデックスDBを検索する手段を含む請求項1 又は2に 記載の情報検索装置。
- 5前記文字列情報検索手段は、 前記入力された文字列から得られた複数の単語に基づいて、 前記 インデックスDBを検索し、当該複数の単語が存在する文のIDを検出し、文書保存DBから当該IDに対応する文を検索し、当該文に該入力された文字列が存在している場合のみ検索結果を前記類似度算出手段に出力する連接確認手段を更に有する請求項1 又は2に 記載の情報検索装置。
- 6クライアントからの文字列を取得し、該文字列が含まれる情報をインデックスDBから取得し、該文字列と検索された情報との類似度を算出し、類似度の高い順に検索された情報を出力する装置における情報検索方法であって、 文字列情報検索手段が、入力された前記文字列が複数の単語からなる場合に、各単語に基づいて、単語及び該単語の 文書中における文単位での 出現位置を格納したインデックスDBを検索し、 各単語を含む文書dと当該文書dでの各単語の出現位置を取得する 文字列情報検索ステップと、 文字列を分割して得た単語の集合をQとし、Pos d (q i )を文書d中の単語q i の出現位置とし、Pos dk (q j )を文書d中の単語q j の出現位置からk減算した値とし、kを所定のカウンタ値とし、αを所定の係数とし、Count(Pos,Pos)を、2つの位置データが与えられた時に、その一致度を返す関数とした場合に、 類似度算出手段が、 単語の出現位置間の一致度score(Q,d,k)を により算出し、当該一致度score(Q,d,k)に基づいて 類似度を算出する 類似度算出 ステップと、 を有することを特徴とする情報検索方法。
- 7nを0以上の定数としたときに、前記類似度算出手段は、 により、前記文字列と文書dとの類似度sim(Q,d)を算出する ことを特徴とする請求項 6 に記載の情報検索方法。
- 8前記インデックスDB は 、 各単語の 出現位置 を ビット列で表現 し、 前記文字列情報検索ステップにおいて、 前記文字列情報検索手段は、前記インデックスDBを検索し、前記ビット列による単語の出現位置を取得し、 前記類似度算出ステップにおいて、 前記類似度算出手段は、前記インデックスDBの前記出現位置のビット列を用いてビットの論理演算により類似度を求める請求項 6又は7に 記載の情報検索方法。
- 9前記入力された前記文字列が複数の単語からなる場合に、該単語に基づいて、近接を考慮しない単語が登録されている近接未考慮単語リスト記憶手段を参照して、該単語が該近接未考慮単語リスト記憶手段に存在する場合は近接を考慮しない単語と判断する文字列選択ステップを更に有し、 前記文字列情報検索ステップにおいて、 前記文字列情報検索手段は、前記文字列選択ステップにおいて、近接を考慮しない単語であると判断された単語については、全ての文に該単語が存在することとして前記インデックスDBを検索する請求項 6又は7に 記載の情報検索方法。
- 10前記文字列情報検索ステップにおいて、 前記文字列情報検索手段は、前記入力された文字列から得られた複数の単語に基づいて、 前記 インデックスDBを検索し、当該複数の単語が存在する文のIDを検出し、文書保存DBから当該IDに対応する文を検索し、当該文に該入力された文字列が存在している場合のみ検索結果を前記類似度算出手段に出力する請求項 6又は7に 記載の情報検索方法。
- 11請求項1乃至 5 のいずれか1項に記載の情報検索装置を構成する各手段としてコンピュータを機能させる情報検索プログラム。
- 12請求項 11に 記載の情報検索プログラムを格納したコンピュータ読取可能な記録媒体。
Independent claims12
89 paragraphs, as filed
The present invention relates to an information retrieval method, an apparatus and a program, and a computer-readable recording medium, and the present invention relates to an information retrieval method, an apparatus and a program for constructing a full-text search index for searching a large amount of documents by a character string. And computer-readable recording media. More specifically, the present invention relates to an information retrieval method, an apparatus and a program, and a computer-readable recording medium when the character string of the search condition is composed of two words (morphemes) or more.
A system has been realized in which a system for searching and presenting a document containing a character string input as a search condition from a large number of documents is realized. The information retrieval device generally performs a process (called lookup) of identifying a document including a character string input as a search condition from the document information stored in the character string DB of the information retrieval device. A process (called ranking) for calculating the similarity (also called a score) between a character string and each document containing the character string is performed. Then, the information retrieval device displays documents (or document IDs) as search results in descending order of similarity. In this specification, "document" is a set of sentences of a certain unit, and "sentence" is a character string of units separated by punctuation marks. For example, one document file can be thought of as one document. Hereinafter, examples of conventional techniques will be described in more detail.
FIG. 1 shows a configuration example of the conventional information retrieval device 10. In FIG. 1, an information retrieval device 10, a client 20, and a network 30 constitute an information retrieval system. In the figure, the character string index DB3 stores the information of the document group to be searched (in this specification, "document" may be used to mean "document group") in a data structure that is easy to search. In this example, the inverted index is stored. The conventional inverted index has a document ID in which the word appears and a word-by-word appearance position of the word in the document for each word. In addition, the frequency of occurrence of the word in the document may be included in the inverted index.
In the information retrieval device 10 shown in the figure, the character string transferred from the client 20 is input from the client input receiving unit 1. The input character string is divided into words in the character string information search unit 2. Then, the character string information retrieval unit 2 acquires the document ID of the document including the word and the appearance position of the word in the document from the character string index DB3 for each of the divided words.
By the way, the words constituting the search condition may include a plurality of words (morphemes) obtained by dividing the compound word "Tokyo" such as "Tokyo" and "Tokyo". In such a compound word, two words are adjacent to each other and have one meaning. Therefore, in the lookup process, the process of checking whether the two words are adjacent to each other in the search target document is performed. It is common to do it. This is called concatenation processing. Specific examples of the concatenation processing in the information retrieval device 10 are as follows.
When the input character string is "Tokyo", the character string information retrieval unit 2 in the information retrieval device 10 divides "Tokyo" into "Tokyo" and "capital". Then, the character string information retrieval unit 2 acquires the document ID in which each word acquired by division exists and the appearance position thereof from the character string index DB3. Figure 2 shows an example of the acquired information. In FIG. 2, it is shown that "Tokyo" is included in the documents with document IDs 133, 144, 170. In the document with document ID 133, the frequency of appearance of "Tokyo" is 2, and it appears at the 5th and 22nd places in the document. In the document with document ID 144, the frequency of appearance is 3, and the frequency of appearance is 1 and 11 in the document. It is shown that it appears in the 18th and 18th places. Similarly, for "capital", the appearance document, appearance frequency, and appearance position of each word are shown.
Then, the character string information retrieval unit 2 checks for the presence or absence of a document in which "Tokyo" appears next to "Tokyo", and "Tokyo" appears next to "Tokyo" in the document with document ID 144. Make sure that. As a result, the character string information retrieval unit 2 can output the document ID 144 as a document containing the compound word "Tokyo".
Since it is initially unknown which of the divided words constitutes a compound word, the character string information retrieval unit 2 searches all documents having any of the words included in the search conditions. On the other hand, there is a problem that the concatenated processing is performed, which requires a large processing cost. Further, in the character string index having the position information in word units, there is a problem that the position information uses a lot of resources.
After the lookup process is completed as described above, the similarity calculation unit 4 uses the information obtained from the character string index DB3 to input the character string and each document in which the character string appears. The degree of similarity with is calculated, and the result is transferred to the client output unit 5 in the order of documents having the highest degree of similarity.
The similarity between a character string and a document is calculated using, for example, TFIDF (Term Frequency Inverse Document Frequency) (Non-Patent Document 1). FIG. 3 shows the similarity calculation unit 4 in this case in detail. As shown in FIG. 3, the similarity calculation unit 4 uses the word importance similarity calculation unit 41, which calculates the similarity using the word importance (idf), and the similarity calculation unit 41, which uses the word frequency (tf) to calculate the similarity. It is composed of the word frequency similarity calculation unit 42 for calculating, and the value obtained by multiplying these two values is the similarity between the document and the word. The degree of similarity between the character string, which is a search condition, and the document can be obtained by calculating the above-mentioned similarity for all the words constituting the character string and obtaining the sum of these similarities.
<maths num="1"><img file="JP5116775B2_D0001.tif" /></maths> The above formula shows the calculation method. sim (Q, d) is a function that expresses the similarity between the search condition Q and the document d, and w<sub>di</sub>Is the score of the word w that constitutes the search condition Q, and the score is calculated from the tf of the word w (the number of occurrences of the word w on the document d) and the idf (the number of documents in which w appears / the total number of documents). Here, the similarity is calculated without using the position information of the words.
In addition, when calculating the similarity by looking at the concatenation, for example, when a compound word consisting of two words, q1 "Tokyo" and q2 "Miyako", "Tokyo" is input, two similarity calculations are performed. There is a way. As it is, there is a method of calculating TFIDF by regarding q1 and q2 as w1 and w2, and a method of calculating the similarity by regarding the compound word in which q1 and q2 are connected as one word w. .. The former ignores the position information, and the latter only uses the position information with a score of 1 when they are connected and a score of 0 when they are not connected. Further, in the prior art, even when a plurality of words are input, the appearance position information of the words is not considered.
As a method for calculating the similarity between a character string and a document, there is also a method called BM25 (Non-Patent Document 2). However, in this method as well, when the character string contains a plurality of words, the similarity is calculated without considering the appearance position information in the sentence of each word, as in TFIDF.
In addition, there is also a method of calculating the similarity in consideration of the fact that a plurality of input words appear in the vicinity (hereinafter referred to as "proximity") when there are a plurality of input words for English (for example, non-patent documents). 3).<nplcit num="1"><text>Gerard Salton and Chris Buckley, Term Weighting Approaches in Automatic Text Retrieval, Information Processing and Management: an International Journal, Pages: 513-523 Vol.24, Issue 5, 1988.</text></nplcit><nplcit num="2"><text>Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Asrron Gull, and Marianna Lau. Okapi at TREC3. In text Retrieval Conference, pages 21-30, 1992.</text></nplcit><nplcit num="3"><text>Tao Tao and ChengXiang Zhai. An exploration of proximity measures in information retrieval. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp.295-302. New York, NY, USA, 2007. ACM Press.</text></nplcit>
<p> As described above, the character string index (inverted index) having the position information in word units has a problem that the amount of the position information is large and the concatenation processing requires a large processing cost.</p><p> Further, in the similarity calculation method described in Non-Patent Document 1 and Non-Patent Document 2, even if the input character string is composed of a plurality of words (or morphemes), the position where the input character string appears. Since the similarity is calculated regardless of the above, there is a problem that the accuracy of the information retrieval device is low.</p><p> Further, in the similarity calculation method described in Non-Patent Document 3, when there are a plurality of input words, it is proposed to calculate the similarity in consideration of the appearance of the plurality of words in the vicinity (proximity). However, it is not assumed that one word is composed of multiple morphemes.</p><p> The present invention has been made in view of the above points, and provides a character string index for solving the conventional problem, and calculates the similarity using close information for the entire document. It is an object of the present invention to provide an information retrieval method, an apparatus and a program, and a computer-readable recording medium capable of improving search accuracy and reducing calculation cost.</p>
<p> In order to solve the above problems, the present invention presents a character string input means for acquiring a character string from a client, a character string information retrieval means for acquiring information including the character string from an index DB, and the character string. The character string information search means is an information retrieval device having a similarity calculation means for calculating the similarity with the searched information and an output means for outputting the searched information in descending order of similarity. , When the input character string consists of a plurality of words, the word and the word are based on each word.<u style="single">On a sentence-by-sentence basis in a document</u>Search the index DB that stores the appearance position,<u style="single">Obtain the document d containing each word and the appearance position of each word in the document d,</u><u style="single">Let Q be the set of words obtained by dividing the character string, and Pos</u><sub><u style="single">d</u></sub><u style="single">(q</u><sub><u style="single">i</u></sub><u style="single">) The word q in the document d</u><sub><u style="single">i</u></sub><u style="single">As the appearance position of, Pos</u><sub><u style="single">dk</u></sub><u style="single">(q</u><sub><u style="single">j</u></sub><u style="single">) The word q in the document d</u><sub><u style="single">j</u></sub><u style="single">The value obtained by subtracting k from the appearance position of, k is a predetermined counter value, α is a predetermined coefficient, and Count (Pos, Pos) is a function that returns the degree of coincidence when two position data are given. If you do</u><u style="single"> The similarity calculation means calculates the score (Q, d, k) of the degree of agreement between the appearance positions of words.</u><maths num="5"><img file="JP5116775B2_D0002.tif" /></maths><u style="single"> Calculated by, and based on the match score (Q, d, k)</u>It is configured as an information retrieval device characterized by calculating the degree of similarity.</p><p> Further, the present invention acquires a character string input means for acquiring a character string from a client and information on a document including the character string from an index DB that stores word appearance position information for each sentence in the document for each word. An information search device having a character string information search means for calculating, a similarity calculation means for calculating the similarity between the character string and a document, and an output means for outputting information of documents searched in descending order of similarity. Therefore, when the input character string is composed of a plurality of words, the character string information search means searches the index DB based on each word to obtain sentence-based appearance position information for each word. Is included for each document, and the similarity calculation means calculates the similarity between each document and the character string based on the degree of co-occurrence of the plurality of words in each document in sentence units. It can also be configured as an information retrieval device characterized by including means.</p><p> In the information retrieval apparatus, the similarity calculation means may determine the number of sentences in which the plurality of words co-occur in a certain document as the degree of co-occurrence in the document.</p><p> The present invention can also be configured as an information retrieval method as a processing method executed by the information retrieval apparatus. Further, the present invention can also be configured as an information retrieval program that causes a computer to function as each means for configuring the information retrieval device, and as a computer-readable recording medium that stores the information retrieval program.</p>
<p> As described above, according to the present invention, when a search is requested using two or more words (or one word composed of two or more morphemes) as a search condition, the two words are used by using the appearance position information of the words. It is possible to calculate the similarity considering that appears in the vicinity or that the two words co-occur in the sentence. This may improve the search accuracy. Further, the similarity can be calculated at low cost by storing the appearance position in bits and realizing it only by bit operation.</p><p> Further, by calculating the similarity based on the degree of proximity (distance) of words or the degree of co-occurrence of words without performing the concatenation processing, the processing cost is reduced as compared with the conventional technique of performing the concatenation processing. In addition, the reproducibility is improved as compared with the conventional technique. Furthermore, by constructing the character string index using the position information of the sentence unit instead of the position information of the word unit as the delimited section, compared with the conventional character string index using the position information of the word unit. The amount of information can be reduced.</p>
<figref num="1">It is a block diagram of an information retrieval system.</figref><figref num="2">It is a figure which shows the example of the information acquired from the conventional character string index DB.</figref><figref num="3">It is a figure which shows the similarity calculation part 4.</figref><figref num="4">It is a figure which shows the outline structure example of the information retrieval apparatus which concerns on embodiment of this invention.</figref><figref num="5">It is a figure for demonstrating the outline of the information retrieval method which concerns on embodiment of this invention.</figref><figref num="6">It is a system block diagram which concerns on embodiment of this invention.</figref><figref num="7">It is a block diagram of the similarity calculation part 4'in the 1st Embodiment of this invention.</figref><figref num="8">It is a flowchart of a series of operations of the information processing apparatus 10 in the 1st Embodiment of this invention.</figref><figref num="9">It is a flowchart of the similarity calculation process using the position proximity information in the 1st Embodiment of this invention.</figref><figref num="10">It is a figure which shows the structural example of the data stored in the character string index DB3'in the 1st Embodiment of this invention.</figref><figref num="11">This is an example of storing position information by a bit string of the character string index DB3'in the first embodiment of the present invention.</figref><figref num="12">It is a figure for demonstrating proximity calculation in the case of using the position information using a bitmap.</figref><figref num="13">It is a figure for demonstrating the number of co-occurrence in a sentence.</figref><figref num="14">It is a block diagram of the information retrieval apparatus 10 in the 2nd Embodiment of this invention.</figref><figref num="15">It is a figure which shows the case where the word which does not consider the proximity in the 2nd Embodiment of this invention is input.</figref><figref num="16">It is a block diagram of the information retrieval apparatus 10 in the 3rd Embodiment of this invention.</figref><figref num="17">It is a figure for demonstrating the connection realization method in the case of having the position information in the sentence unit in the 3rd Embodiment of this invention.</figref>
Code description
1 Client input reception unit, character string input means 2 Character string information retrieval unit, character string information retrieval means 3,3'String index DB 4,4'Similarity calculation unit, similarity calculation means 5 Client output section, output means 6 String selection section 10 Information retrieval device 12 Proximity unconsidered word list memory 14 Document storage DB 20 clients 30 networks 41 Word Importance Similarity Calculation Department 42 Word frequency similarity calculation unit 43 Word position similarity calculation unit
Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, a schematic example of the embodiment of the present invention will be described with reference to FIGS. 4 and 5.
FIG. 4 is a diagram showing a schematic configuration example of the information retrieval device according to the present embodiment. As shown in FIG. 4, this information retrieval device includes a character string input means 1 for acquiring a character string from a client, a character string information retrieval means 2 for acquiring information including the character string from a DB, and the character. It is an information retrieval device having a similarity calculation means 4 for calculating the similarity between a column and the searched information and an output means 5 for outputting the searched information in descending order of similarity, and is a character string information retrieval. When the input character string consists of a plurality of words, the means 2 searches the index DB3'that stores the word and the appearance position information of the word based on each word, and the distance between the appearance positions of the word. The similarity calculation means 4 includes a means for calculating the similarity based on the distance between the appearance positions of the words.
Further, the index DB3'holds information that aggregates the word and the appearance position information in the sentence of the word for each sentence, expresses the appearance position information by a bit string, and searches the index DB3 by the character information search means 2. A means for acquiring the appearance position information of a word by a bit string may be included, and the similarity calculation means 4 may include a means for obtaining the similarity by a logical operation of bits using the bit string of the appearance position information of the index DB3'.
Further, in the information retrieval device, when the input character string is composed of a plurality of words, the proximity-unconsidered word list storage means in which words that do not consider proximity are registered based on the words is referred to. Further, when the word exists in the proximity-unconsidered word list storage means, it further has a character string selection means for determining that the word does not consider the proximity. For a word that is determined to be a word that is not considered, a means for searching the index DB3 may be included assuming that the word is present in all sentences.
Further, in the information retrieval device, when the character string information retrieval means 2 stores the sentence number and the appearance position information in the index DB3 for each sentence, it is based on the word obtained from the input character string. , The index DB3 is searched, the obtained sentence number is searched from the document storage DB in which the document is stored, and the search result is used as the similarity calculation means only when the input character string exists. It may be possible to further have a connection confirmation means for outputting.
FIG. 5 is a diagram for explaining an outline of the information retrieval method according to the embodiment of the present invention. The information retrieval method acquires a character string from a client, acquires information including the character string from a DB, calculates the similarity between the character string and the searched information, and searches in descending order of similarity. This is an information retrieval method in the device that outputs the input information (step 3), and when the character string information retrieval means consists of a plurality of words, the word and the word are based on each word. The character string information search step (step 1) for searching the index DB that stores the appearance position information of the word and finding the distance between the appearance positions of the word, and the similarity calculation means are based on the distance between the appearance positions of the words. It has a step (step 2) of calculating the similarity.
Further, in the information retrieval method, when the index DB holds information that aggregates a word and the appearance position information in the sentence of the word for each sentence, and the appearance position information is expressed by a bit string, character string information. In the search step, the index DB is searched, the appearance position information of the word by the bit string is acquired, and in the similarity calculation step, the similarity is obtained by the logical operation of the bit using the bit string of the appearance position information of the index DB. Good.
Further, in the information retrieval method, when the input character string consists of a plurality of words, the proximity-unconsidered word list storage means in which words that do not consider proximity are registered based on the words is referred to. If the word exists in the proximity-unconsidered word list storage means, a character string selection step of determining that the word does not consider proximity is further performed, and in the character string information retrieval step, a word that does not consider proximity in the character string selection step. For the word determined to be, the index DB may be searched assuming that the word exists in all the sentences.
Further, in the character string information retrieval step in the information retrieval method, when the sentence number and the appearance position information are stored in the index DB for each sentence, the sentence number and the appearance position information are stored based on the word obtained from the input character string. The index DB is searched, the obtained sentence number is searched from the document storage DB in which the document is stored, and the search result is output to the similarity calculation means only when the input character string exists. It may be that.
In the present embodiment, it is possible to provide an information retrieval program that causes a computer to function as each means for configuring the information retrieval device. It is also possible to provide a computer-readable recording medium in which this information retrieval program is stored.
Hereinafter, embodiments of the present invention will be described in detail.
In the embodiment of the present invention, it is assumed that an information retrieval program running on an Internet Web server is stored in a recording medium such as a hard disk and executed on the Web server. Therefore, the client side uses a Web browser or the like via the network to access the information retrieval device and searches for information through the client input reception unit and the output unit. This method can also be applied to a stand-alone system.
FIG. 6 shows a system configuration according to an embodiment of the present invention. As shown in FIG. 6, the system according to the embodiment of the present invention includes a client 20, a network 30, and an information retrieval device 10. The information retrieval device 10 is composed of a client input reception unit 1, a character string information retrieval unit 2, a character string index DB 3', a similarity calculation unit 4', and a client output unit 5. The configuration is almost the same as that of FIG. 1 described above, but the configuration of the character string index DB3'and the similarity calculation unit 4'is different from the character string index DB3 and the similarity calculation unit 4 shown in FIG.
In the configuration of FIG. 6, the information retrieval device 10 inputs a character string from the client 20 to the client input reception unit 1 via the network 30. When the character string is transferred from the client input reception unit 1 to the character string information retrieval unit 2, the character string is divided into words. The character string information retrieval unit 2 acquires the document containing each of the divided words and the appearance position of the word in the document from the character string index DB3'and transfers them to the similarity calculation unit 4'. The similarity calculation unit 4'calculates the similarity between the character string and each document based on the acquired word and the appearance position of the word. The division of functions in the configuration shown in FIG. 6 is not limited to those described below. For example, the character string information retrieval unit 2 may perform a part of the processing (calculation of the degree of proximity, calculation of the degree of co-occurrence, etc.) in the similarity calculation unit 4'.
When the transferred character string is a single word, the similarity calculation unit 4'calculates the similarity between the word and the document in which the word appears, for example, by the method of Non-Patent Document 1 described above. , The document ID is output in the order of documents with high similarity. The document itself may be output by searching another database from the document ID. When the word is composed of a plurality of words, the similarity calculation unit 4'calculates the similarity between the character string and the document in consideration of how close the multiple words are. As a result, the document ID and the like are transferred to the client output unit 5 in the order of documents having a high degree of similarity.
FIG. 7 shows the configuration of the similarity calculation unit 4'in the first embodiment of the present invention.
Compared with the prior art, the similarity calculation unit 4'shown in the figure has a word position similarity calculation unit 43 added in addition to the word importance similarity calculation unit 41 and the word frequency similarity calculation unit 42. .. The word position similarity calculation unit 43 calculates the similarity using proximity information or co-occurrence information.
Next, a series of processes according to the embodiment of the present invention will be described.
FIG. 8 is a flowchart of a series of operations of the information processing apparatus 10 according to the first embodiment of the present invention.
Step 101) When a character string is input from the client input reception unit 1, the character string information retrieval unit 2 divides the input character string into words. For example, when the character string "Tokyo" is input, it is divided into words by using morphological analysis processing. When divided by word, it is generally divided into the character strings "Tokyo" and "Miyako".
Step 102) The character string information retrieval unit 2 checks whether the character string is composed of two or more words. If it is two or more words, the process proceeds to step 103, and if it is a single word, the process proceeds to step 105.
Step 103) The character string information search unit 2 searches the character string index DB3'and acquires the document having the divided words and the word appearance position for each divided word.
Step 104) The similarity calculation unit 4'calculates the similarity using the proximity information, and proceeds to step 107. The method of calculating the similarity using proximity information will be described later.
Step 105) The similarity calculation unit 4'acquires a document having a single word which is an input character string and a word occurrence frequency from the character string index DB3'.
Step 106) In the similarity calculation unit 4', since the importance of a word in the case of one word is common to all documents, the word frequency similarity calculation unit 42 uses only the frequency of the word, and the word and the word. Calculate the degree of similarity with each document in which.
Step 107) In step 102 above, when the input character string is two or more words, the similarity calculation unit 4'sets the search result (for example, document ID) using the proximity information in the order of similarity (for example, similarity). Arrange in descending order of) and transfer to the client output unit 5. In step 102, when the input character string is a single word, the similarity calculation unit 4'sorts the search results in the order of similarity calculated using the word frequency information and transfers the search results to the client output unit 5. As a result, the client output unit 5 presents the search result to the client 20.
Next, a similarity calculation method using the position proximity information in step 104 will be described.
FIG. 9 is a flowchart of the similarity calculation process using the position proximity information in the first embodiment of the present invention.
Step 201) In the word position similarity calculation unit 43 of the similarity calculation unit 4', the document ID of the document including each word obtained by dividing from the character string and the appearance position of each word in the document ID are acquired. .. The information can be obtained from the information acquired from the character string index DB3'by the character string information retrieval unit 2.
Here, FIG. 10 shows an example of the structure of the data stored in the character string index DB3'in the present embodiment. In the example of FIG. 10, the word and the document ID in which the word appears are stored for each word, and the appearance position data of the word is stored for each document ID. Further, the appearance position of the word in the present embodiment is the appearance position of each sentence.
For example, if word1 is the string "Tokyo", "Tokyo" exists in the document id1 and its appearance position is the 1st, 3rd, 4th, and 8th sentences in the document id1. Indicates that it is a sentence. In addition, Fig. 11 shows an example in which the structure is provided by bitmap. In FIG. 10, the appearance position of each sentence is shown numerically, whereas in FIG. 11, it is represented by a bitmap. For example, "0010" for the document with id1 of word1 indicates that the word word1 exists in the third sentence. In the embodiment of the present invention, the position information of a word is set as a sentence unit, but the present invention is not necessarily limited to this, and any unit may be used for each section. For example, even if this position information is changed to the appearance position of a word unit or the appearance position of a paragraph unit, the similarity calculation processing based on the same proximity can be performed.
When the words obtained by dividing the input character string are "Tokyo" and "Miyako", in the process of step 201, the word position similarity calculation unit 43 has both "Tokyo" and "Miyako". The document ID of the document including, the appearance position of "Tokyo" in the document in the sentence unit, and the appearance position of the "city" in the document in the sentence unit are acquired. For example, in the example of FIG. 10, if word1 is "Tokyo" and word2 is "capital", in step 201, the word position similarity calculation unit 43 assigns the document ID 1 to "Tokyo". , The appearance position in it (1st sentence, 3rd sentence, 4th sentence, 8th sentence), and for the "city", the document ID1 and the appearance position in it (1st sentence, 5th sentence) , 6th sentence, 8th sentence) will be acquired.
In this example, in the following, in the document of document ID 1, how close the appearance position of "Tokyo" in the sentence unit and the appearance position of the sentence unit of "Tokyo" are in the document of the document ID1. A score indicating whether or not the document is used is calculated, and the score is used as the degree of similarity between the input character string (Tokyo) and the document with the document ID 1.
Step 202) The word position similarity calculation unit 43 sets the counters k = 0 and n. n is a numerical value for specifying the number of sentences before and after the proximity to be viewed. For example, if n = 1, the proximity similarity is calculated by focusing only on adjacent sentences. On the contrary, if a document is composed of m sentences and m is set to n, the proximity similarity of the entire document can be calculated.
Step 203) The word position similarity calculation unit 43 checks whether the value of k is n or less. If it is n or less, the process proceeds to step 204, and if it is greater than n, the process proceeds to step 207.
Steps 204 to 205) The word position similarity calculation unit 43 shifts (for example, subtracts) the appearance position of one of the appearance position group pairs obtained in step 201 by k in sentence units, and obtains the appearance. Calculate the degree of agreement score (Q, d, k) between the position and the other appearance position. The degree of agreement score (Q, d, k) is calculated by, for example, the formula shown below.
<maths num="2"><img file="JP5116775B2_D0003.tif" /></maths> In the above formula, Q is a query set (a set of words obtained by dividing a character string), d is a document, and Pos.<sub>d</sub>(q<sub>i</sub>) Is q in document d<sub>i</sub>Indicates the position of occurrence of (word, eg "Tokyo" in the above example), Pos<sub>dk</sub>(q<sub>j</sub>) Is q in document d<sub>j</sub>Represents a column obtained by subtracting k from the appearance position of (word, for example, "city" in the above example). For example, in the example where word2 in Fig. 10 is "city", if k is 0, Pos<sub>dk</sub>(q<sub>j</sub>) Is (1, 5, 6, 8). Also, in this example, if k is 1, Pos<sub>dk</sub>(q<sub>j</sub>) Is (0, 4, 5, 7).
Count (Pos, Pos) is a function that returns the degree of matching when two position data are given. For example, if the input is k = 0, the word q in the same sentence.<sub>i</sub>And q<sub>i</sub>Words other than q<sub>j</sub>Outputs the total number of times that appears. Also, when k = 1, Count (Pos, Pos) is the word q in the adjacent sentence.<sub>i</sub>And the word q<sub>j</sub>Outputs the total number of times that appears.
For example, the word q<sub>i</sub>Is "Tokyo" in the above example, q<sub>i</sub>Words other than q<sub>j</sub>Is the "capital" of the above example, then Pos in the case of k = 0<sub>d</sub>(q<sub>i</sub>) Is (1, 3, 4, 8) and Pos<sub>dk</sub>(q<sub>j</sub>) Is (1, 5, 6, 8), so Count returns 2. In other words, this indicates that "Tokyo" and "capital" appear twice in the same sentence (sentence 1 and sentence 8) in the target document.
α is a coefficient, for example 1. A large score is calculated when each word is present in a sentence closer to 1 / (αk + 1).
Step 206) The word position similarity calculation unit 43 increments the value of k by 1, and returns to step 203.
Step 207) The word position similarity calculation unit 43 outputs the total value of the score values calculated for each k value as the similarity between the input character string and the document. This is the degree of similarity considering the position information of words. More specifically, this is a similarity that takes into account the degree of proximity. This is calculated by the following formula.
<maths num="3"><img file="JP5116775B2_D0004.tif" /></maths> When using the location information using the bitmap shown in Fig. 11, Pos<sub>d</sub>And Pos<sub>dk</sub>The count can be calculated simply by taking the logical product with the information obtained by shifting the logical product by k bits and counting the number of 1s from the bits resulting from the logical product. FIG. 12 is a diagram for explaining the proximity calculation when the position information using the bitmap is used, and the calculation of the score when Q (word_2, word_1), d = 1, k = 1 is input. An example is shown. First, the character string information retrieval unit 2 acquires the position information of word1 and word2 from the character string index DB3'(the configuration shown in FIG. 11). The similarity calculation unit 4'shifts the position information of word_1 by 1 bit from the acquired position information, and ANDs the position information of word_2 with the logical product. The bit position of the result of the logical product is the position where word1 and word2 existed in the next sentence. In this example, the score is "1" because there is only one standing bit, the second bit. As a method of counting the number of 1 from this bit string, for example, there is a method of obtaining it at high speed by the SSE4 instruction POPCNT of the Intel CPU.
Another example of how to calculate the similarity between the character string, which is the search condition, and the document will be described below. In this example, the similarity (score) is calculated using the following formula.
<maths num="4"><img file="JP5116775B2_D0005.tif" /></maths> In the above formula (1), the similarity score between the set Q of the character string (query) q and the document d is the similarity w for each query.<sub>d</sub>It is shown that it is the sum of (q), and in equation (2), the query q is the set T of the morphemes t.<sub>q</sub>It is shown that it is composed of. Equation (3) shows the similarity w<sub>d</sub>(q) is the similarity w for each morpheme<sub>d</sub>Indicates that it is the sum of (t). Similarity w<sub>d</sub>(t) is calculated by Eq. (4).
This equation (4) is the following equation used in BM25 shown in Non-Patent Document 2.
<maths num="5"><img file="JP5116775B2_D0006.tif" /></maths>In, tf (t, d) (frequency of appearance of morpheme t in document d), coocc (T)<sub>q</sub>Corresponds to the one replaced with, d). Other parameters are the same as those in BM25. N indicates the total number of documents. df (t) is the number of documents including t, k<sub>1</sub>Is a coefficient. K is k<sub>1</sub>It is a value obtained from the document length and the like.
coocc (T<sub>q</sub>, d) is T<sub>q</sub>It is the number of co-occurrence in the sentence in the document d of. For example, T<sub>q</sub>However, in the case of "Tokyo" and "capital", when calculating the similarity, the similarity calculation unit 4'sees FIG. 13 as the document ID, the frequency of occurrence, and the sentence unit position information for each word. The indicated information is acquired from the character string index DB3'or from the character string information retrieval unit 2. In this example, the character string index DB3'has a word appearance frequency as well as a document ID and word appearance position information for each sentence. This frequency of occurrence is incremented by 1 if the word exists in a sentence, even if it appears in the same sentence more than once.
The similarity calculation unit 4', which has acquired the information shown in FIG. 13, detects that "Tokyo" and "capital" co-occur in sentence 1, sentence 34, and sentence 52 based on the position information of each sentence. And coocc (T<sub>q</sub>, d) (T in this case<sub>q</sub>Is "Tokyo" and "capital", and d is 133). Then, using the above equations (4) and (3), the similarity of the document 133 with respect to "Tokyo" can be obtained.
In the above example, a part of the formula of BM25 is used as the number of co-occurrence in the sentence coocc (T).<sub>q</sub>The similarity is calculated using the formula replaced with, d), but it is not always essential to be based on the BM25 formula. Number of co-occurrence in sentences coocc (T<sub>q</sub>If, d) is used, other equations may be used. Also, for example, the number of co-occurrence in sentences coocc (T)<sub>q</sub>It is also conceivable to use, d) itself as the degree of similarity.
As described above, in the present embodiment, the similarity is calculated based on the degree of proximity of words or the degree of co-occurrence of words in the divided sections (sentences in the above example) without performing the concatenation processing. Therefore, the processing cost is reduced as compared with the conventional technique of performing articulated processing. In addition, the reproducibility is improved as compared with the conventional technique. Furthermore, by constructing the character string index DB3'using the position information of the sentence unit instead of the position information of the word unit as the delimited section, the conventional character string index DB using the position information of the word unit is used. The amount of information can be reduced compared to.
[Second Embodiment] When the character string input above is two words (morphemes) or more, the method of calculating the similarity using the fact that they are close to each other has been explained. However, it cannot be said that close proximity leads to improvement in accuracy. For example, if the string "restaurant name phone number" is entered as a search condition, you are not looking for "restaurant name phone number", but the string "phone number" exists on a restaurant page. You may only need to show the page you are on. Therefore, in the present embodiment, it is determined whether or not the word input as the search condition considers proximity.
FIG. 14 shows the configuration of the information retrieval device 10 according to the second embodiment of the present invention. In the figure, the same components as those in FIG. 6 are designated by the same reference numerals, and the description thereof will be omitted.
The information retrieval device 10 shown in the figure has a configuration in which a character string selection unit 6 and a proximity unconsidered word list storage unit 12 are added to the information retrieval device 10 of FIG. In the proximity unconsidered word list storage unit 12, words that do not consider proximity are registered.
Based on the word input as the search condition, the character string selection unit 6 refers to the proximity-unconsidered word list storage unit 12, and if the word exists in the proximity-unconsidered word list storage unit 12, the proximity is not considered. Is determined to be a word that does not consider. When the character string information retrieval unit 2 determines in the character string selection unit 6 that the word does not consider proximity, the character string information retrieval unit 2 performs proximity processing assuming that the word exists at all positions. FIG. 15 shows a case where a word that does not consider proximity in the second embodiment of the present invention is input. Word_1 shown in the figure is an example of a word that does not consider proximity, and its appearance position is "0010". In the present embodiment, the appearance position of the word is converted to "1111" existing in all sentences, and the similarity calculation process is performed by the similarity calculation unit 4'. The processing by the similarity calculation unit 4'is the same as the processing in the first embodiment described above.
By converting the appearance position of the word that does not consider proximity to "1111", the appearance position of the word does not contribute to the similarity score.
[Third Embodiment] When the input character string is composed of two words, the proximity processing using only the word appearance position information for each sentence cannot discriminate compound words, which may reduce the search accuracy. Therefore, in the present embodiment, a method of realizing the concatenation processing of words composed of two words when the word appearance position information is stored in sentence units will be described.
FIG. 16 shows the configuration of the information retrieval device 10 according to the third embodiment of the present invention.
The information retrieval device shown in the figure has a configuration in which the document storage DB 14 is added to the configuration shown in FIG. In addition, the character string index DB3'holds the document ID and the position information of the sentence unit, not the word unit, for the word.
FIG. 17 is a diagram for explaining a method of realizing concatenation when position information is provided for each sentence in the third embodiment of the present invention.
As shown in the figure, the character string index DB3'stores the document ID (DOC_ID) in which the word exists and the word appearance position information for each sentence represented by bits in the character string index DB3'for each word. As a result, the character string information retrieval unit 2 morphologically analyzes the input character string, and if the character string is two or more words, the document ID having the word from the character string index DB3'and the word's The presence or absence of co-occurrence in the sentence is examined by acquiring the appearance position and taking the logical product of the position information.
Then, when there is a sentence in which the corresponding word co-occurs in the sentence, the character string information search unit 2 searches the document storage DB14 based on the document ID and the sentence number, acquires the corresponding sentence, and obtains the corresponding sentence. Check if there is a compound word in which the above multiple words are concatenated in the sentence. Then, if there is a compound word, a document containing the compound word can be output as a search result.
Further, also in this example, the word position similarity calculation unit 43 of the similarity calculation unit 4 is close to the document corresponding to the two words acquired from the character string information retrieval unit 2 in the same manner as in the first embodiment. The similarity may be calculated based on the information (location information), but the score may be added to the document in which the compound word is detected as described above. Further, the word position similarity calculation unit 43 may calculate the similarity only for the documents in which compound words are detected, and output the search results to the client output unit 5 in descending order of similarity.
For example, when the character string "Tokyo" is input as a search query, the sentence in which "Tokyo" and "Tokyo" exist is first specified. This is equivalent to the conventional proximity processing based on position information in sentence units. In the example of FIG. 17, the second sentence in Document 1 and the first sentence in Document 2 include "Tokyo" and "Miyako", respectively. The specified document ID and sentence number are referenced from the document storage DB14, and whether or not the character string "Tokyo" exists is investigated. This process can be done by scanning everything from the beginning of the sentence. As a result, the character string information retrieval unit 2 outputs only the document in which the character string exists (only the document in which the concatenation word exists) to the similarity calculation unit 4', and the similarity calculation unit 4'is the similarity degree. Can be calculated and the search result can be presented from the client output unit.
Further, it is possible to construct each operation of the component of the information retrieval device in the above embodiment as a program, install it on a computer used as the information retrieval device and execute it, or distribute it via a network. is there.
Further, the constructed program can be stored in a hard disk or a portable storage medium such as a flexible disk or a CD-ROM, and installed or distributed on a computer.
The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.
The present invention can be applied to information retrieval for acquiring desired information using a character string input from a large amount of documents such as a database.
This international application claims priority based on Japanese Patent Application No. 2007-299784 filed on November 19, 2007, the entire contents of which are incorporated into this international application.
32 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| JP2000242646A | Cites | Japan | Search report |
| JP2000242646A | Cites | Japan | Examiner |
| JP2004094387A | Cites | Japan | Examiner |
| JP2005215716A | Cites | Japan | Search report |
| JP2005215716A | Cites | Japan | Examiner |
| JP2007004458A | Cites | Japan | Search report |
| JP2007004458A | Cites | Japan | Examiner |
| JP2004094387A | Cites | Japan | – |
| JP2005215716A | Cites | Japan | – |
| JP2007004458A | Cites | Japan | – |
| JP2000242646A | Cites | Japan | – |
5 members in 3 offices
Priority claims11
| Document | Office | Kind | Date |
|---|---|---|---|
| 2007299784 | Japan | A | |
| 2007299784 | Japan | A | |
| 2007299784 | Japan | – | |
| 2008066356 | Japan | W | |
| 2008066356 | Japan | W | |
| 2009542497 | Japan | A | |
| 20072007299784 | – | – | – |
| 2008066356 | – | – | – |
| JP20070299784 | – | – | – |
| JP20090542497 | – | – | – |
| WO2008JP66356 | – | – | – |
Members5
| Document | Office | Kind | |
|---|---|---|---|
| WO2009066501A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US2010257159A1 | United States of America | A1 | |
| JPWO2009066501A1 | Japan | A1 | |
| JP5116775B2This record | Japan | B2 | |
| US8909654B2 | United States of America | B2 |
13 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Written notification of registration of transferJAPANESE INTERMEDIATE CODE: R350R350 | R350 | |
| Written request for registration of change of nameJAPANESE INTERMEDIATE CODE: R313533S533 | S533 | |
| Written notification of registration of transferJAPANESE INTERMEDIATE CODE: R350R350 | R350 | |
| Written request for registration of change of domicileJAPANESE INTERMEDIATE CODE: R313531S531 | S531 | |
| Renewal fee payment (event date is renewal date of database)FPAY | FPAY | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 |
Numbers
- Publication
- 5116775
- Publication, DOCDB
- 5116775
- Publication, EPODOC
- JP5116775B
- Application
- 2009542497
- Application, DOCDB
- 2009542497
- Application, EPODOC
- JP20090542497
Titles2
- Japanese
- 情報検索方法及び装置及びプログラム及びコンピュータ読取可能な記録媒体
- English
- Information retrieval methods, devices and programs, and computer-readable recording media
Classification
- CPC, 3
- G06F16/334
- G06F16/313
- G06F16/316
- IPC, 1
- G06F17 30
