Display apparatus having multi-touch recognizing function and driving method thereof

Display device having multi-touch recognition function and driving method thereof

1 is a diagram schematically showing a display device having a multi-touch recognition function according to an embodiment of the present invention;

FIG. 2 is a view showing the display device shown in FIG. 1 in detail;

FIG. 3 is a circuit diagram equivalently showing a part of the pixel array shown in FIG. 2;

4 is a flowchart showing step by step a control procedure of a multi-touch signal processing program according to an embodiment of the present invention.

Fig. 5 is a diagram for explaining window processing;

6 is a view showing a shadow of a touch object.

7 is a view showing an example of a malfunction of recognizing a shadow as in FIG. 6 as a touch object;

8 is a graph showing a histogram of a background image and average values and standard deviation values in the histogram;

9 is a graph showing histograms of a background image, a touch object, and a shadow, and average values in the histograms;

10 is a diagram illustrating a labeling process of each of the touch positions.

It is a figure for demonstrating the triangulation method.

12 is a view showing a process of removing a shadow from a touch image.

<Explanation of symbols for main parts of the drawing>

10 : LCD panel 11 : Source driver

12 : gate driver 20: touch & display module

30 : Control board 31 : Timing controller

32 : multi-touch processor 40 : system

21A to 21D: Camera 22: Blackened inner wall

<backgroundart><p>The present invention relates to a display device, and more particularly, to a display device having a multi-touch recognition function to prevent a malfunction by removing a shadow, and a driving method thereof.</p><p>A touch panel is generally attached to a display device and is one of the user interfaces that detect the touch position by changing electrical characteristics at the touch position in contact with the hand or pen, and its application range is expanding to small portable terminals, office equipment, etc. . Such a touch panel may malfunction when two or more multi-touches are generated at the same time, or any one may be selected by a preset program.</p><p>In order to overcome the limitations of multi-touch recognition in the existing touch panel, a multi-touch recognition device that simultaneously recognizes multiple touches has been developed. </p><p>Such a multi-touch recognition apparatus may misrecognize a touch position due to a shadow of a touch object or aberration of a lens when a camera having a lens is used.</p></backgroundart><abstractproblem><p>Accordingly, an object of the present invention relates to a display device having a multi-touch recognition function and a driving method thereof, which is an invention devised to solve the problems of the prior art and prevents malfunction by removing shadows.</p></abstractproblem>

<p>In order to achieve the above object, a display device having a multi-touch recognition function according to an embodiment of the present invention includes a display element; a plurality of cameras installed near the edge of the display element; A difference image is obtained by comparing a reference value that is a function of the average value and standard deviation value of the background image captured by the cameras and stored in advance with the touch image currently captured by the cameras, and the difference image is touched on the display element a touch control circuit for calculating the position of the touch object; and a display driving circuit for displaying the image of the touch object on the display device.</p><p>The background image is an image of the blackened sidewall of the display device and includes images captured by each of the cameras for 100 frames or more. </p><p>The average value and standard deviation value of the background image are analyzed as a histogram for each pixel of the background image and include average values and variance values of the histogram for each pixel calculated as a result of the analysis. </p><p>If the pixel value of the touch image is greater than the reference value, the touch control circuit stores the pixel value as the pixel value of the touch image, and if the pixel value of the touch image is less than or equal to the reference value, the pixel value of the background image pixel stored in advance It is replaced with the average value and stored as the pixel value of the touch image. </p><p>The touch control circuit obtains the difference image by calculating a difference between a pixel value of the background image and a pixel value of the touch image in units of pixels. </p><p>The method of driving a display device according to an embodiment of the present invention compares the touch image currently captured by the cameras with a reference value that is a function of the average value and standard deviation value of the background images captured by the cameras and stored in advance to compare the difference obtaining an image; calculating a position of a touch object touched on the display element from the difference image; and displaying the image of the touch object on the display device.</p><p>Hereinafter, preferred embodiments of the present invention will be described with reference to FIGS. 1 to 12 .</p><p>1 to 3 , a display device having a multi-touch recognition function according to an embodiment of the present invention includes touch &amp; ; display module 20; a control board 30 for controlling the touch &amp; display module 20 and performing an algorithm for recognizing a touch position; and a system 40 for supplying digital video data RGB to be displayed on a display element of the touch &amp; display module 20 together with a timing signal to the control board 30 .</p><p>The touch &amp; display module 20 is configured to supply a data voltage to the liquid crystal display panel 10 on which the pixel array 10A on which an image is displayed is formed, and the data lines D1 to Dm of the liquid crystal display panel 10 . The source driver 11 , the gate driver 12 for supplying scan pulses to the gate lines G1 to Gn of the liquid crystal display panel 10 , and a camera disposed near the four corners of the pixel array 10A, respectively and 21A to 21D. </p><p>The liquid crystal display panel 10 includes a thin film transistor (hereinafter, referred to as "TFT") substrate and a color filter substrate. A liquid crystal layer is formed between the TFT substrate and the color filter substrate. On the TFT substrate, the data lines D1 to Dm and the gate lines G1 to Gn are formed on the lower glass substrate to be orthogonal to each other. In cell regions defined by the data lines D1 to Dm and the gate lines G1 to Gn, the liquid crystal cells Clc are arranged in a matrix form. The TFT formed at the intersection of the data lines D1 to Dm and the gate lines G1 to Gn is supplied via the data lines D1 to Dm in response to a scan pulse from the gate lines G1 to Gn. The resulting data voltage is transferred to the pixel electrode of the liquid crystal cell Clc. To this end, the gate electrode of the TFT is connected to the gate lines G1 to Gn, and the source electrode is connected to the data lines D1 to Dm. The drain electrode of the TFT is connected to the pixel electrode of the liquid crystal cell Clc. A common voltage Vcom is supplied to the common electrode facing the pixel electrode. The color filter substrate includes a black matrix and a color filter formed on an upper glass substrate.</p><p>The common electrode is formed on the upper glass substrate in a vertical electric field driving method such as TN (Twisted Nematic) mode and VA (Vertical Alignment) mode, and horizontal electric field such as IPS (In Plane Switching) mode and FFS (Fringe Field Switching) mode It is formed on the lower glass substrate together with the pixel electrode in the driving method. </p><p>Reference numeral 'Cst' denotes a storage capacitor. The storage capacitor Cst may be formed by overlapping the gate line and the pixel electrode of the liquid crystal cell Clc, or may be formed by overlapping the separate common line and the pixel electrode.</p><p>The source driver 11 includes a plurality of data integrated circuits to convert digital video data (RGB) input from the control board 30 into positive or negative analog gamma compensation voltage under the control of the control board 30, The analog gamma compensation voltage is supplied to the data lines D1 to Dm as an analog data voltage. </p><p>The gate driver 12 includes a plurality of gate integrated circuits, and sequentially supplies scan pulses to the gate lines G1 to Gn under the control of the control board 30 . </p><p>The data integrated circuits of the source driver 11 and the gate integrated circuits of the gate driver 12 are formed by Tape Automated Bonding (TAB) using a Tape Carrier Package (TCP) or Chip on Glass (Chip). on glass, COG) method may be formed on the lower glass substrate. The gate integrated circuits of the gate drive 12 may be directly formed on the lower glass substrate simultaneously with the TFTs of the pixel array 10A and in the same process as the TFT process.</p><p>The cameras 21A to 21D are cameras using a complementary metal oxide semiconductor (CMOS) sensor and are disposed at four corners of the pixel array 10A on which an image is displayed in the liquid crystal display panel 10 . Each of these cameras 21A to 21D captures an image of the display surface of the pixel array and its vicinity. The touch image captured from each of these cameras 21A to 21D is supplied to the control board 30 .</p><p>A lens angle of view of each of the cameras 21A to 21D is preferably 90° as shown in FIG. 4 . The lens angle of view of the cameras 21A to 21D may vary depending on the distance between the liquid crystal display panel 10 and the cameras 21A to 21D or the size of the liquid crystal display panel 10, but 80 It should be in the range of °-90 °. This is because when the lens angle of view of the cameras 21A to 21D is narrowed to less than 80°, there are many shaded areas that the cameras 21A to 21D cannot capture, and the multi-touch cannot be accurately recognized, If it is greater than 90°, it is because the multi-touch cannot be accurately recognized including the part outside the liquid crystal display panel 10 .</p><p>The control board 30 is connected to the source driver 11 and the gate driver 12 through a flexible printed circuit (FPC) and a connector. The control board 30 includes a timing controller 31 and a multi-touch processor 32 .</p><p>The timing controller 31 controls the operation timing of the source driver 11 and the gate control signal for controlling the operation timing of the gate driver 12 using the vertical/horizontal synchronization signals V and H and the clock CLK. Generates a data control signal for controlling. Also, the timing controller 31 supplies digital video data RGB input from the system 40 to the source driver 11 .</p><p>The touch control circuit of the multi-touch processor 32 stores the average value of the background image, the standard deviation value of the background image, and the reference value compared for each pixel of the touch image, and executes the multi-touch signal processing program to implement the multi-touch recognition algorithm. carry out This algorithm includes a process of comparing the background image and the touch image, and removing the shadow from the touch image in order to prevent a malfunction of recognizing the shadow of the touch object as the touch object in the comparison process. The multi-touch processor 32 supplies the touch location coordinate information Txy obtained by performing an algorithm for recognizing each touch location to the system 32 . Since the multi-touch processor 32 shares timing signals such as vertical/horizontal synchronization signals V and H and a clock CLK with the timing controller 31 , the multi-touch processor 32 operates in synchronization with the timing controller 31 .</p><p>The inner walls 22 of the touch &amp; display module 20 exposed to the lenses of the cameras 21A to 21D are painted black. In order to remove the shadow of the touch object, which will be described later, the background image is captured by the cameras 21A to 21D for a period of 100 frames or more, and is obtained as an average value for each pixel. This background image is stored in the memory of the multi-touch processor 30 and compared with the touch image in the touch recognition process.</p><p>The system 40 synthesizes an image and a touch image to be displayed on the memory in which the application program is embedded, a central processing unit for executing the application program, and the liquid crystal display panel 10, and signal interpolation of the synthesized data It includes a graphic processing circuit that processes processing and resolution conversion. The system 40 receives the touch location information Txy from the multi-touch processor 32 and executes an application program related to the touch location information Txy. For example, if there is an icon of a specific program at the coordinates of the touch position, the system 40 loads the program from the memory and executes it. In addition, the system 40 generates digital video data RGB by synthesizing an image to be displayed on the liquid crystal display panel 10 and a touch image. The system 40 may be implemented as a personal computer (PC), and may transmit and receive data to and from the multi-touch processor 32 through a serial or Universal Serial Bus (USB) interface. </p><p>4 is a flowchart illustrating a control procedure of a multi-touch signal processing program executed in the multi-touch processor 32 step by step. </p><p>4, the multi-touch signal processing program receives a touch image captured by the cameras 21A to 21D, and removes an unnecessary spatial image on the effective touch area through window processing as shown in FIG. An effective touch image near the effective touch surface is extracted. (S1 and S2) The window processing uses an image extraction technique of extracting only a necessary image through a sub-matrix operation with respect to an input image signal.</p><p>Then, the multi-touch signal processing program converts R, G, and B included in the image of the effective touch area extracted by window processing into gray information. (S3) Image pickup by the cameras 21A to 21D The processed image includes color information of red (R), green (G), and blue (B), and the effective touch image extracted through window processing also includes red (R), green (G) and blue (B) color information. include information. The background image of the blackened sidewall 22 and the effective touch image are not compared with color information, but are compared based on gray including black and white and intermediate tones therebetween. For this reason, in step S3, the color information of the effective touch image is converted into grayscale information by Equation 1 below.</p><p><maths num="1"><df>Gray scale intensity = pR+qG+sB</df></maths></p><p>Here, 'p', 'q', and 's' are constants having different values. </p><p>For touch recognition, only the touch image needs to be extracted by comparing the background image with the touch image where the actual touch is made. To this end, the multi-touch signal processing program compares the previously stored background image with the touch image currently photographed by the cameras 21A to 21D as an average value for each pixel, and extracts the image of the actual touch object corresponding to the difference between the two images. (S4)</p><p>If the shadow of the touch object is reflected on the blackened sidewall 22 as shown in FIG. 6 , the image of the touch object extracted by the difference in the process of comparing the background image and the touch image is not only the touch object but also the shadow of the touch object is extracted. In this case, as shown in FIG. 7 , a erroneous operation occurs by recognizing the shadow of the touch object as the touch object. 7 is an example showing a difference image between the background image and the touch image when the background image, the touch image, and the shadow of the touch object are included in the touch image.</p><p>In order to prevent such a malfunction, the multi-touch signal processing program of the present invention calculates the cumulative function for each gradation of each pixel, that is, a histogram in the background image obtained by shooting by the cameras 21A to 21D for a period of 100 frames or more. After calculating, the histogram is divided to calculate the average values and standard deviation values for each pixel in the background image and store them in the memory. </p><p>The process of obtaining the background image will be described in detail as follows. 8 is an example of a histogram for the experimental pixels 140 and 2 of the background image. This histogram is the experimental result obtained from the experimental pixels of the background image for a period of 10,000 frames. Since the background image is an image of the blackened inner wall 22, a histogram of pixels other than the experimental pixel in the background image is also obtained as a histogram similar to FIG. The present invention calculates a histogram for all pixels of the background image. In FIG. 8 , the horizontal axis represents gradations, and the vertical axis represents cumulative values. Then, in the histograms for each pixel, an average value and a standard deviation value for each pixel of the background image for a period of 100 frames or more are calculated. In the histogram, the average value μ for each pixel is calculated by Equation 2, and the standard deviation σ is calculated by Equation 3.</p><p><maths num="2"><df><img file="KR20080098787A_D0001.tif" /></df></maths></p><p><maths num="3"><df><img file="KR20080098787A_D0002.tif" /></df></maths></p><p>The multi-touch signal processing program of the present invention compares the background image and the touch image in units of each pixel. 9 shows an example of a histogram of a background image and a touch image with respect to one pixel. As can be seen from FIG. 9, the average value μ of the shadow of the touch object<sb>S</sb>is the average value μ of the background image<sb>B</sb>has a value similar to , and the average value μ of the touch object<sb>O</sb>has a relatively larger difference than the shadow or background. According to the experimental results, the average values of the background image and the touch image in all pixels show a trend as shown in FIG. 9 .</p><p>The multi-touch signal processing program of the present invention is the average value μ of the shadow of the touch object in order to remove the shadow of the touch object.<sb>S</sb>and the average value μ of the touch object<sb>O</sb> The reference value is determined by the value in between. And the multi-touch signal processing program of the present invention compares the reference value with each pixel value of the currently captured touch image. As a result of this comparison, the multi-touch signal processing program of the present invention stores the pixel value as a pixel value of the touch image if the pixel value of the currently captured touch image is greater than the reference value, The pixel value is replaced with the average value of the previously stored background image pixels and stored as the pixel value of the touch image.</p><p>Next, the multi-touch signal processing program of the present invention compares the background image and the touch image in units of pixels to obtain a difference image. </p><p>Equation 4 below is the pixel value B of the background image<sb>p</sb>is an equation that defines, and Equation 5 is the pixel value O of the touch image.<sb>P</sb>is the formula that defines And Equation 6 is the difference image pixel value D between the background image and the touch image<sb>P</sb>is the formula that defines </p><p><maths num="4"><df><img file="KR20080098787A_D0003.tif" /></df></maths></p><p><maths num="5"><df><img file="KR20080098787A_D0004.tif" /></df></maths></p><p><maths num="6"><df><img file="KR20080098787A_D0005.tif" /></df></maths></p><p>In Equations 4 to 6, </p><p>μ<sb>B </sb>: Average value of histogram of background image calculated over a period of 100 frames or more </p><p>σ<sb>B </sb>: Histogram standard deviation value of the background image calculated over a period of 100 frames or more</p><p>α: adjustment factor (0α1), the reference value can be adjusted by this adjustment factor. </p><p>μ<sb>B</sb>+α·σ<sb>B </sb>: Standard value to distinguish between shadow and touch object</p><p>B<sb>P</sb> : Pixel value of the background image</p><p>I<sb>P</sb> : Pixel value of the touch image captured by the current camera</p><p>O<sb>P</sb> : Pixel value of touch image</p><p>D<sb>P</sb> : Difference image pixel value between background image and touch image</p><p>Next, the multi-touch signal processing program compares the gray scale intensity of the difference image calculated in step S4 with a preset threshold. In addition, the multi-touch signal processing program converts the grayscale information of the touch image into white data only for data exceeding the threshold value, and converts data below the threshold value into black data. (S5) White data is valid It means a touch position, and the black data means invalid data that is not actually touched among the touch images. The threshold is determined experimentally.</p><p>In step S6, the multi-touch signal processing program performs labeling processing. In the f-labeling process, unique identification codes ( to ) are given to each of white data, that is, valid touch location data, as shown in FIG. 10 in order to distinguish each touch location.</p><p>Then, the multi-touch signal processing program uses the angle calculation algorithm to find the position on the two-dimensional plane for each of the valid touch positions converted into white data, the angle of each of the cameras 21A to 21D looking at the valid touch positions. Measure (S7) </p><p>Each of the measurement angles between the effective touch positions and the camera measured in step S7 includes an amount of distortion equal to the lens aberration of the cameras 21A to 21D. Therefore, the multi-touch signal processing program registers the compensation value for each angle between the effective touch position and the camera in a look-up table and adds the compensation angle from the look-up table to the measurement angle, The amount of distortion due to the characteristic is compensated. (S8) A method for calculating a compensation angle and a compensation circuit using the same are described in detail in P07-024180 previously filed by the applicant of the present application.</p><p>In step S9, the multi-touch signal processing program calculates the position of each effective touch position using the triangulation method as in Equation 6 from the angles for which the lens distortion is compensated. Equation 7 is a calculation formula for calculating one touch position as a two-dimensional xy coordinate value, and as shown in FIG. It contains the distance (a, b, c) between the camera and the touch location. The angle C between the touch position and the cameras is calculated as "C=180-angle A-angle B". This triangulation is applied for each of multiple touch locations.</p><p><maths num="7"><df><img file="KR20080098787A_D0006.tif" /></df></maths></p><p>In this way, when touch location information Txy including x and y coordinate data for each touch location is calculated from the multi-touch signal processing program, the system 40 displays the touch location information Txy on the liquid crystal display panel 10 . Combined with the digital video data to be The touch position information Txy synthesized in the background image is transmitted to the ICs of the source driver through the timing controller 31 and displayed on the liquid crystal display panel 10. (S10) The touch position information synthesized in the background image includes various It can be shaped into a shape.</p><p>On the other hand, the liquid crystal display panel 10 and the cameras can be separated from the touch &amp; (10) and install a detachable mechanism to the camera module. The liquid crystal display panel 10 is another flat panel display panel, for example, an organic light emitting diode (OLED) display panel, a plasma display panel (PDP), a field emitting device (Field Emitting Display, FED) or a display panel of a stereoscopic image display device including a flat panel display panel.</p><p>12 is an example schematically illustrating an example in which a shadow is removed from a touch image obtained through comparison with a reference value. </p>

<p>As described above, a display device having a multi-touch recognition function and a driving method thereof according to an embodiment of the present invention remove a shadow from a touch image using a reference value between the average value of the actual touch object and the average value of the shadow of the touch object. It can prevent malfunction. </p><p>Those skilled in the art from the above description will be able to see that various changes and modifications can be made without departing from the technical spirit of the present invention. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.</p>