Method and system for providing haptic effects based on information complementary to multimedia content

(See Related Application) This application is entitled "Tactile Audio" and is incorporated by reference in its entirety to US Provisional Patent Application No. 61 / 874,933, filed September 6, 2013.

The entire application of U.S. Patent Application No. 14 / 078,438, filed November 12, 2013, under the name "Systems and Methods for Producing Tactile Effects Associated with Transitions to Audio Signals" To be used.

This application is the entire specification of U.S. Patent Application No. 14 / 078,442, filed November 12, 2013, under the name "Systems and Methods for Producing Tactile Effects Associated with Envelopes in Audio Signals." Invite.

This application is entitled "Systems and Methods for Producing Tactile Effects Associated with Audio Signals" and is incorporated by reference in its entirety to U.S. Patent Application No. 14 / 078,445, filed November 12, 2013. ..

The present invention generally relates to methods and systems for providing haptic effects based on multimedia data, and more specifically to methods and systems for providing haptic effects based on information supplementing multimedia content. ..

Advances in multimedia technology over the last few years have continued to play a role in emphasizing and providing users with a more immersive visual experience. Today's televisions have large screens, high resolution screen resolutions, and 3D (dimensional) performance. In addition, handheld devices devices) always have a high-resolution screen that expresses bright and brilliant colors, and also accesses huge content sources via the Internet. Similarly, advances in audio technology such as improved surround sound systems, high fidelity home theater speakers and high resolution audio further emphasize the user's multimedia viewing experience. In addition to advances in video and audio technology, tactile effects through the touch sensation of viewing devices are used to immerse users in the viewing experience. Tactilely operable video game console controllers that provide tactile effects in response to events in video games are a popular example of the use of tactile effects to emphasize the user's viewing experience. Similarly, most handheld mobile devices (eg, mobile phones, tablet computers) today have hardware that enables applications to provide tactile effects.

The use of haptic effects to enhance the playback of audio and / or video content is well known. In some cases, the tactile designer edits multiple tactile tracks. A tactile track is a specific set of tactile effects that are output at a particular time during playback of multimedia content, used by a system or device that plays a multimedia file to provide the tactile effects. The advantage of this traditional method is that it provides multimedia content with a specially adapted tactile track, but requires the use of a tactile designer to manually design the tactile effect for each multimedia content item. To do. Alternatives to manual design include an automated process that uses algorithms to analyze audio and / or video content to determine the tactile effect that is output. Specific examples of such techniques are described in U.S. Patent Application No. 14 / 078,445 and Application No. 14 / 078,445 in the related application references described above, Blind Source. Separation Includes Acoustic Event Detection ("AED") such as ("BSS")) and image pattern matching. However, these well-known automated tactification algorithms are subject to inaccurate and heavy processing demands. In addition, automated tactile algorithms tend to result in overly comprehensive tendencies, i.e. undesired and overly noisy tactile tracks (eg typing on the keyboard, yelling, music) and vice versa. It tends to be inadequately comprehensive, that is, it tends to fail to tactilely represent important events in multimedia content.

The present disclosure generally determines receiving multimedia data, including multimedia content and supplementary data describing the characteristics of the multimedia content, and determining tactile effects based on at least a portion of the supplementary data. And how to provide tactile effects while playing multimedia content. In another embodiment, the computer-readable medium comprises program code for causing the processor to perform such a method.

These exemplary embodiments are not mentioned to limit or define the invention, but to provide embodiments that aid in the understanding of the invention. Illustrative embodiments are discussed in the detailed description of the invention, and another description of the invention is also provided in the detailed description of the invention. The beneficial effects proposed by the various embodiments of the invention are further understood by reviewing the specification.

These or other features, aspects, and advantageous effects of the present disclosure will be better understood by examining the following invention details with reference to the drawings attached herein.

<figref num="1A">It is a block diagram of the apparatus which concerns on embodiment of this disclosure.</figref><figref num="1B">It is a block diagram of the apparatus which concerns on embodiment of this disclosure.</figref><figref num="2A">It is a block diagram of multimedia which concerns on embodiment of this disclosure.</figref><figref num="2B">It is a block diagram of multimedia which concerns on embodiment of this disclosure.</figref><figref num="2C">It is a block diagram of multimedia which concerns on embodiment of this disclosure.</figref><figref num="2D">It is a block diagram of multimedia which concerns on embodiment of this disclosure.</figref><figref num="2E">It is a block diagram of multimedia which concerns on embodiment of this disclosure.</figref><figref num="2F">It is a block diagram of multimedia which concerns on embodiment of this disclosure.</figref><figref num="2G">It is a block diagram of multimedia which concerns on embodiment of this disclosure.</figref><figref num="2H">It is a block diagram of multimedia which concerns on embodiment of this disclosure.</figref><figref num="3">It is a flowchart which concerns on embodiment of this disclosure.</figref><figref num="4">It is a flowchart which concerns on embodiment of this disclosure.</figref>

The embodiments according to the present disclosure relate to a method and a system for providing a tactile effect based on multimedia data, and more specifically, as information for supplementing multimedia contents (in the present specification, as "supplementary information, supplementary information"). With respect to methods and systems for providing tactile effects based on (referred to). As used herein, "multimedia data" includes both multimedia content and supplementary information.

<u style="single">Illustrated Embodiment</u> In one exemplary embodiment, a family has a home theater system that includes a television, an internet-enabled and tactile receiver, multiple speakers, and a tactile sofa / couch. Have. The family regularly watches movies using an internet-based movie streaming service where A / V receivers receive multimedia data streams. The multimedia data stream contains supplementary information (eg, metadata tags, closed captioning information) for the selected movie as well as for that movie. In one concrete example, the family chooses to watch a World War I action / drama film. When the data stream for the movie begins to arrive, the A / V receiver determines if there is a metadata container commonly known as a tag. By analyzing the tag, the A / V receiver determines if there is text "war" in the tag's genre field and further increases the intensity of the tactile effect output above the default level is applied. Determine if there is a tactile theme. In addition, the A / V receiver provides its genre for automated tactification algorithms. Automated tactile algorithms analyze movie audio and video tracks to pattern-identify for the generation of tactile effects. In this embodiment, the A / V receiver focuses on patterns commonly found in war movies, such as explosions, shootouts, and produces tactile effects based on tactile algorithms. In addition to analyzing audio and video tracks, the A / V receiver also analyzes closed caption text for keywords. For example, if an A / V receiver discovers the word "explosion" with a special type stamp in a closed caption, the A / V receiver inputs this information into an automated tactile algorithm. Automated tactile algorithms look for explosion patterns around the time stamp It also produces a corresponding tactile effect, such as an intense vibration that corresponds to an explosion. As the movie plays, the A / V receiver sends a signal to the tactile sofa (sofa). The tactile sofa outputs the tactile effect at the time corresponding to the event occurring in the film.

This exemplary embodiment is merely a embodiment and is performed as a plurality of other embodiments are described herein.

<u style="single">Illustrated device</u> Referring to drawings in which similar numbers are shown for the same element throughout several figures, FIGS. 1A and 1B are block diagrams of a system for providing a tactile effect based on supplementary information according to embodiments of the present disclosure. is there.

Figure 1A shows a system 100 with various components. In some embodiments, the system 100 shown in FIG. 1A includes a television, an A / V receiver, multiple speakers, a game console with a game controller, a universal remote controller, tactile furniture, and /. Alternatively, it is a home entertainment system equipped with a wearable tactile output device. In one specific embodiment shown in FIG. 1B, system 100 includes a television (display 116), A / V receiver 122, multiple speakers 120, remote controller 114, and a tactilely operable chaise longue. (Sofa) 124 is provided. In other embodiments, the system 100 comprises a handheld device such as a mobile phone, personal digital assistant (PDA), tablet computer, laptop computer, palmtop computer, or handheld navigation system. In yet another embodiment, the system 100 may include a kiosk, an automatic head unit, or a similar system.

The embodiments of the present disclosure are embodied in or include combinations of digital electronic circuits, computer hardware, firmware and software. The system 100 shown in FIGS. 1A and 1B includes a processor 110. Processor 110 receives the input signal and generates signals for communication, display and tactile feedback. Processor 110 includes or communicates with one or more computer-readable media, such as memory 112 with random access memory (RAM).

Processor 110 executes computer-executable program instructions, such as executing one or more computer programs stored in memory 112 for transmitting or generating tactile feedback. Processor 110 comprises a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), one or more user writable gate arrays (FPGAs) or state machines. The processor 110 may further include a programmable electronic device such as a PLC, a programmable interrupt controller (PIC), a programmable read-only memory (PROM), an electrically programmable read-only memory (EPROM or EEPROM) or other similar device. ..

The memory 112 includes a computer-readable medium that stores instructions that, when executed by the processor 110, cause the processor to perform various steps as described herein. Examples of embodying computer-readable media include, but are not limited to, electronic, optical, magnetic, or other storage or transmission devices capable of providing computer-readable instructions to the processor 110. Other specific examples of media are, but are not limited to, floppy (registered trademark) disks, CD-ROMs, magnetic disks, memory chips, ROMs, RAM, ASICs, configured processors, all. Optical media, all magnetic tapes or other magnetic media, or other media that can be read by a computer processor. In addition, various other devices include computer-readable media such as routers, private or public networks, or other transmitting devices. The processor 110 and process processing described may be distributed over one or more structures or even one or more structures.

System 100 communicates with and / or receives communications from a network that uses network interface 122. In some embodiments, network interface 122 is an Ethernet® card. In other embodiments, the network interface 122 is a cable TV and / or housed within an A / V receiver, cable TV set-top box, satellite TV set-top box, television, or other component of system 100. It is a module for receiving and decoding satellite TV signals. However, in embodiments of the present invention, the network interface 122 may be any component that allows communication with the network. For example, a network interface is a module and antenna that communicates with a wireless networking device, or mobile phone network or Wi-Fi network.

According to FIGS. 1A and 1B, the system 100 also includes a user input device 114 that communicates with the processor 110. For example, in some embodiments, the user input device 114 comprises a touch screen. In such an embodiment, the user input device 114 detects a user's interaction and the location of that interaction. One embodiment includes a capacitance-based touch screen. In other embodiments, the user input device 114 may include buttons, switches, sliders, or trackballs. In yet another embodiment, the system 100 may include both a touch screen and an additional user input device 114.

In some embodiments, the user input device 114 is a remote controller, game controller, or wired or wireless connection (eg, infrared, Bluetooth®, Zigbee®, or other wireless communication technology). Includes similar devices that communicate with processor 110 by. Such an embodiment comprises a remote controller for an audio / video receiver that communicates with an audio / video receiver using Bluetooth®.

System 100 also includes a display 116. The display 116 is configured to communicate with the processor 110 and display the output from the processor 110 to the user. In one embodiment, display 116 includes television. In another embodiment, the device 102 comprises a liquid crystal display (LCD) located below the user input device 114. In some embodiments, the display 116 and the user input device 114 include a single or integrated component such as a touch screen LCD. In some embodiments, device 102 may not include a display.

In the embodiments shown in FIGS. 1A, 1B and 2, the tactile output device 118 is configured to communicate with the processor 110 and provide one or more tactile effects. For example, in one embodiment, when an operation signal is supplied to the tactile device 118 by the processor 110, each tactile device 118 outputs a tactile effect based on the operation signal. For example, in the illustrated embodiment, the processor 110 is configured to transmit a tactile output signal, including an analog drive signal, to the tactile output device 118. In some embodiments, the processor 110 is configured to send instructions to the tactile output device 118. This instruction includes multiple parameters used to generate an appropriate drive signal for the tactile output device 118 to output a tactile effect. In another embodiment, different signals and different signal types are transmitted to each of one or more tactile output devices. For example, in some embodiments, the processor transmits a low level drive signal to drive a tactile output device that outputs a tactile effect. Such low drive signals are converted from digital to analog or analog to digital using a suitable processor or circuit that implements a particular tactile output device that is amplified or driven by an amplifier. Will be done.

A tactile output device, such as the tactile output device 118, is a component or aggregate of parts capable of outputting one or more tactile effects. For example, tactile output devices are, but are not limited to, eccentric rotational mass (ERM) actuators, linear resonant actuators (LRA), piezo electric actuators, voice coil actuators, electro-active polyma (EAP) actuators, storage shapes. Alloys, pagers, DC motors, AC motors, movable magnet actuators, E-core actuators, smart gels, electrostatic actuators, electric tactile actuators, deformable surfaces Surface), electrostatic friction (ESF) devices, ultrasonic friction (USF) devices, or collections of parts that perform the functions of other tactile output devices or tactile output devices, or a collection of parts capable of outputting tactile effects. The body. Multiple tactile output devices or tactile output devices of different sizes are used to provide a range of vibration frequencies that can be operated individually or simultaneously. In various embodiments, one or more tactile output devices communicate directly or indirectly with electronic devices via wireless or wired communication. In one embodiment, the electronic device can be placed in the vehicle or integrated into the vehicle, and one or more tactile devices are mounted in the vehicle. For example, one or more tactile output devices can be embedded in vehicle seats, steering wheels, pedals, and the like. In some embodiments, instead of having the tactile output device 118, or in addition to having the tactile output device 118, the system 100 has one or more other output devices. For example, system 100 may have one speaker and / or display. In one embodiment, the system 100 has one or more tactile output devices, one or more speakers, and even one or more displays. A number of other embodiments are disclosed herein and variations are within the scope of this disclosure.

In various embodiments, one or more tactile effects are produced by many methods or by synthesis of many methods. For example, in one embodiment, one or more vibrations are used to generate a tactile effect by rotating the eccentric mass or by oscillating the mass linearly. In some embodiments, the tactile effect is generated to transmit vibrations to all electronic devices, or to transmit vibrations to only one side of the electronic device, or to a limited portion of the electronic device. In other embodiments, of two or more parts, such as providing braking to a moving part, resisting the movement of the part, or applying torque. Friction between, friction between at least one part and at least one contact is used to generate the tactile effect. Many devices use some type of actuator and / or other tactile output device to generate the vibration effect. Well-known tactile output devices used for this purpose are eccentric rotating masses ("ERM") whose eccentric mass is driven by a motor, and linear resonant actuators ("LRA") whose spring-mounted mass is driven back and forth. "), Or a" smart material "such as a piezoelectric, electroactive polymer or shape memory alloy.

In other embodiment, deformation of one or more parts is used to produce a tactile effect. For example, one or more tactile effects are output by changing the shape of the surface or the coefficient of friction of the surface. In one embodiment, one or more tactile effects are generated by forming electrostatic and / or ultrasonic forces that are used to alter friction on the surface. In other embodiment, an array of transparent transforming elements, such as one or more areas containing smart gels, is used to generate the tactile effect. Tactile output devices are further devices, non-mechanical or non-vibrating devices, or tactile substrates and flexible or such that generate acoustic radiation pressure by electrostatic friction (ESF), ultrasonic surface friction (USF), or ultrasonic tactile converters. It includes a wide range of devices that use deformable surfaces or that provide a projected tactile effect, such as blown air with an air jet. In some embodiments, the tactile effect is a kinesthetic effect. U.S. Patent Application No. 13 / 092,484 describes how one or more tactile effects are produced, as well as a variety of tactile output devices. U.S. Patent Application No. 13 / 092,484 was filed on April 22, 2011 and is incorporated herein by reference.

Various types of input synthesis methods are used to generate interaction parameters for one or more tactile effect signals, including, but not limited to, the examples of synthesis methods shown in Table 2 below. Will be recognized. The drive signal is applied to the tactile actuator based on the interaction parameters. Numerous other embodiments are disclosed herein, and variations exist within the scope of this disclosure.<u style="single">Table 2-Synthesis method</u> -<u style="single">Additive synthesis</u>-Combine inputs, typically varying amplitude-<u style="single">Subtractive synthesis</u>-Filtering composite or multiple signal inputs-<u style="single">Frequency modulation synthesis</u>-Modulating a carrier signal by one or more operators-<u style="single">sampling</u>-Use recorded input as an input source that is subject to change-<u style="single">Mixed synthesis</u>-Use artificial and sampled inputs to establish "new" inputs as a result-<u style="single">Phase distortion</u>-Changing the speed of waveforms stored in multiple waveform tables during playback-<u style="single">Waveform shaping</u>-Intentionally transforming a signal to produce a modified result-<u style="single">Resynthesis</u>-Digitally change the sampled input before playback-<u style="single">Crude synthesis</u>-Combine several small input segments into a new input-<u style="single">Linear predictive coding</u>-Similar technology used for speech synthesis-<u style="single">Direct digital compositing</u>-Computer modification of the generated waveform-<u style="single">Vector composition</u>-Fading technology between an unspecified number of different input sources-<u style="single">Physical model</u>-Mathematical equations of physical features of virtual movement

In another embodiment, it is used to implement a tactile device (eg, wristband, belt, or shirt) to which one or more tactile output devices 118 can be worn. In other embodiments, one or more tactile output devices 118 are used to implement tactilely operable furniture such as chairs or chaise longues (sofas).

As described above, the system 100 of the embodiment of FIG. 1A is a home entertainment system. For example, in one embodiment depicted in FIG. 1B, system 100 includes a television (display 116), A / V receiver 122, multiple speakers 120, remote controller 114, and tactilely operable chaise longue 124. To be equipped. The A / V receiver 122 includes a processor 110, a memory 112, and a memory 122. User input device 114 includes a remote controller that communicates with processor 110 of A / V receiver 122. The A / V receiver operates to play multimedia content resident in memory 112 or multimedia content received via a network interface. In one embodiment, the present invention includes a processor 110 that transmits a video signal to a display 116 to display a video and transmits an audio signal to a speaker to reproduce the audio when playing multimedia contents. It is well known in the field. In another embodiment, when playing multimedia content, a tactile signal may be transmitted to the tactile output device 118 of the chaise longue 124 capable of tactile operation, and the chaise longue on which the viewer sits may output the tactile effect. ..

In another embodiment, the system 100 comprises a plurality of sensors communicating with the processor 110. In one embodiment, the system 100 is configured to use one or more sensors to detect viewer information. For example, system 100 is configured to detect the viewer's location, location, movement and / or viewer's biometric information. For example, the remote controller 114 comprises a proximity sensor configured to detect the distance from the A / V receiver 122 to the remote controller 114. In such an embodiment, the remote controller 114 is configured to transmit the detected distance information to the processor 110 of the A / V receiver 122. In other embodiments, the system 100 may further include software that determines viewer information based on cameras and videos captured by video cameras. In other embodiments, the wearable device (eg, wearable tactile device) detects the distance from the A / V receiver 122 to the viewer wearing the wearable device and A. It comprises one or more proximity sensors configured to transmit the detected distance information to processor 110 of the / V receiver 122. In some embodiments, the wearable device of system 100 detects motion and is further configured to transmit the detected motion to processor 110 of A / V receiver 122 (eg, acceleration). A total) may be provided. Based on such information, the processor 110 may determine the user behavior of the system 100 based on methods well known to those skilled in the art. In another embodiment, the wearable device of the system 100 comprises a plurality of sensors configured to detect biometric information such as heart rate or body temperature of the viewer. System 100 then analyzes biometric data to determine the viewer's stress level and / or the viewer's involuntary physical response when the multimedia content is played.

In another embodiment, one piece of furniture (eg, tactilely operable chaise longue 124) comprises multiple sensors for detecting the presence and movement of one or more users of the system 100. For example, pressure sensors are located at various locations on the tactilely operable chaise longue 124 and are configured to transmit the detected pressure information to the processor 110 of the A / V receiver 122, which is detected by the pressure sensor. Based on the change in pressure, let the system 110 detect whether the user is sitting on the chaise longue 124, how it is sitting, as well as the movement of the user, based on a method well known by those skilled in the art. It is composed of. Further, the present disclosure contemplates methods well known by all sensors and those skilled in the art for detecting the position and / or movement of a user of System 100.

<u style="single">Supplementary information</u> Here, supplementary information will be described with reference to FIG. 2A-2H. For the purposes of this disclosure, the supplemental information is information that corresponds to multimedia content (eg, audio and / or video signals, filters, or other formats of audio and / or video). In some embodiments, the supplemental information represents common multimedia content such as subtitles, closed captioning data (including subtitles), scripts, or metadata tags. In other embodiments, supplementary information is posted by annotations or viewer comments (eg, by members of an online video website) about the video that applies to the entire multimedia content or at a particular time. It may be a comment made). The supplemental information may take many embodiments in different embodiments and may further include other types of information, such as title or chapter information. Supplementary information is included (embedded) in the multimedia content in various embodiments referred to as a single multimedia file, A / V file, video file, or audio file.

FIG. 2A is a block diagram of multimedia data according to an embodiment of the present disclosure. In this embodiment, the multimedia data 200 includes a multimedia file 201. In the present embodiment, the multimedia file 201 is a video file including the multimedia content 202 (video) and supplementary information in the subtitle / closed caption data format corresponding to the multimedia content 202. In one embodiment, the subtitle / closed captioning data 204 is a text describing a plurality of scenes and / or a plurality of events visually depicted by the multimedia content 202, a sound heard in the audio of the multimedia content 202. Includes descriptive text and / or literal quotes from the speech heard in the audio of multimedia content 202.

FIG. 2B is a block diagram of multimedia data according to an embodiment of the present disclosure. In this embodiment, the multimedia data 200 includes a multimedia file 201. In the present embodiment, the multimedia file 201 is a video file containing the multimedia content 202 (video) and supplementary information in the form of tags 208 and script 206 corresponding to the multimedia content 202. Tags for multimedia content are titles, performers / writers, genre data, ratings, and / or other data that describes the characteristics, content, or origin of the multimedia content (eg, critics or other viewing). Includes impressions from people, a list of music used in the movie). One well-known example is the ID3 tag, which is a metadata container often used for MP3 files. In other embodiments, the tag data is defined by the distributor of the multimedia content. In yet other embodiments, tag data is defined by the purchaser and / or recipient of multimedia content. In such embodiments, tags are generated by writers, distributors, and / or recipients.

FIG. 2C is a block diagram of multimedia data according to an embodiment of the present disclosure. In this embodiment, the multimedia data 200 includes a multimedia file 201. In the present embodiment, the multimedia file 201 is an audio file including the multimedia content 202 (audio) and supplementary information in the format of the tag 208 corresponding to the multimedia content 202.

FIG. 2D is a block diagram of multimedia data according to an embodiment of the present disclosure. In this embodiment, the multimedia data 200 includes a multimedia file 201 and a supplementary information file 210. In the present embodiment, the multimedia file 201 is a video file including the multimedia content 202 (video) and supplementary information in the format of the tag 208 corresponding to the multimedia content 202. In the present embodiment, the supplementary information file 210 includes supplementary information in the form of subtitle / closed caption data 204 corresponding to the multimedia content 201. In one embodiment, the supplementary information file 210 resides in the same memory (eg, memory 112) in which the multimedia file 201 resides. In another embodiment, the supplemental information file 202 resides on a remote server accessible by system 100 via a network such as the Internet. In another embodiment, the supplementary information file 210 corresponding to the multimedia file 201 is the information held in the tag 208 (eg, the name of the movie, the name of the song, the identification code, or other information for identification). ) Is searched from the remote server.

FIG. 2E is a block diagram of multimedia data according to an embodiment of the present disclosure. In this embodiment, the multimedia data 200 includes a multimedia file 201 and a supplementary information file 210. In the present embodiment, the multimedia file 201 is a video file including the multimedia content 202 (video) and supplementary information in the format of the subtitle / closed caption data 204 corresponding to the multimedia content 202. In the present embodiment, the supplementary information file 210 includes supplementary information in the form of tag data 208 corresponding to the multimedia content 201. In one embodiment, the supplementary information file 210 resides in the same memory (eg, memory 112) in which the multimedia file 201 resides. In another embodiment, the supplemental information file 202 resides on a remote server accessible via a network such as the Internet by system 100. In one embodiment, the supplementary information file 210 corresponding to the multimedia file 201 is retrieved from the remote server based on the information (eg, metadata, file name) in the multimedia file 201.

FIG. 2F is a block diagram of multimedia data according to an embodiment of the present disclosure. In this embodiment, the multimedia data 200 includes a multimedia file 201. In the present embodiment, the multimedia file 201 is a video file including the multimedia content 202 (video) and supplementary information in the format of chapter data 212 corresponding to the multimedia content 202. In one embodiment, chapter data 212 includes the chapter title of multimedia content 202 and the time each chapter begins.

FIG. 2G is a block diagram of multimedia data according to an embodiment of the present disclosure. In this embodiment, the multimedia data 200 includes a multimedia file 201. In this embodiment, the multimedia file 201 is a video file containing multimedia content 202 (video). In the present embodiment, the supplementary information file 210 includes supplementary information in the form of user comment 214 corresponding to the multimedia content 202. In one embodiment, the supplementary information file 210 resides in the same memory (eg, memory 112) in which the multimedia file 201 resides. In another embodiment, the supplemental information file 202 resides on a remote server accessible via a network such as the Internet by system 100. In one embodiment, the supplementary information file 210 corresponding to the multimedia file 201 is retrieved from the remote server based on the information (eg, metadata, file name) in the multimedia file 201. In one embodiment, user comment 214 is posted by a viewer's comment (eg, by a member of an online video website) about a video that applies to the entire multimedia content 202 or at a particular time. Comment) may be used.

FIG. 2H is a block diagram of multimedia data according to an embodiment of the present disclosure. In this embodiment, the multimedia data 200 includes a multimedia file 201. In this embodiment, the multimedia file 201 is a video file containing multimedia content 202 (video). In the present embodiment, the supplementary information file 210 includes supplementary information in the form of tactile keywords / phrases 216 provided by the user corresponding to the multimedia content 202. In one embodiment, the supplementary information file 210 resides in the same memory (eg, memory 112) in which the multimedia file 201 resides. In another embodiment, the supplemental information file 202 resides on a remote server accessible via a network such as the Internet by system 100. In one embodiment, the supplementary information file 210 corresponding to the multimedia file 201 is retrieved from the remote server based on the information (eg, metadata, file name) in the multimedia file 201. In one embodiment, the tactile keyword / phrase 216 provided by the user is provided by the user of the website for improving or modifying the tactile track generated for the multimedia content 202. In one embodiment, the tactile keyword / phrase 216 provided by the user applies to chapters or parts of multimedia content 202. Further, in other embodiments, the tactile keyword / phrase 216 provided by the user is applied at a special time point during the playback of the multimedia content 202.

<u style="single">Example system behavior</u> FIG. 3 is a flowchart for explaining an example of embodying a method for providing a tactile effect based on supplementary information. FIG. 4, which is included in step 304 of FIG. 3, is a flowchart illustrating an example of embodying a method for providing a tactile effect based on supplementary information. In some embodiments, the steps 3 and 4 are performed in program code executed by a processor, such as a processor in a computer, mobile terminal, or server of general purpose. In some embodiments, these steps are performed by a group of processors, such as a mobile terminal processor, one or more, multiple processors of a general purpose computer such as a server. .. To help you understand how each step is performed, the following description is provided in connection with the exemplary block diagram of System 100 shown in Figures 1A and 1B. However, embodiments related to the present disclosure may be implemented in other embodiments.

When started in step 302, processor 110 in system 100 receives multimedia data 200. In one embodiment, the multimedia data 200 is a multimedia file 201 resident in memory 112, and the multimedia data 200 is an operating system operation (eg, open / open of commands issued to the file system). Received according to (read). In another embodiment, the multimedia data 200 is streamed through network coupling through network interface 122 and further buffered in system 110. Embodiments of the present disclosure include all network couplings (eg, Wi-Fi, Ethernet®, including streaming from any number of servers across the Internet and streaming from servers within a corporate intranet or home network. Wired modems, wireless modems, etc.) and sources and methods of streaming content well known by those of skill in the art. In another embodiment, the multimedia data 200 is downloaded over a network such as the Internet and further stored in memory 112.

In some embodiments, the multimedia content 202 and supplementary information 422 (collectively, "multimedia data") are held in a single file or data stream. However, in other embodiments, the multimedia content 202 and supplementary information 422 may be separate files. As mentioned above, in one such embodiment, the multimedia content 202 and the supplementary information 422 reside in different locations. For example, in one embodiment, the multimedia content 202 resides in the memory of system 100 and the corresponding supplemental information 422 resides in an accessible server on a network such as the Internet. As soon as the multimedia content 202 is accessed for analysis and / or playback, the system 100 searches for the corresponding supplementary information from an accessible server on the network and loads the searched supplementary information from the server into memory 112. To do. In another embodiment, the multimedia content 202 and supplementary information 422 may reside on separate servers accessible via a network such as the Internet. In yet another embodiment, the multimedia content 202 and supplementary information 422 may reside in different files on the same server. In such an embodiment, the system 100 receives the multimedia content 202 from one server in a streaming specification, and downloads the supplementary information 422 from the server on which the multimedia content 202 resides. In yet another embodiment, the multimedia file 201 includes supplementary information 422, and the corresponding supplementary information file 210 further includes additional supplementary information 422.

In step 304, during playback of the multimedia content 202, the output tactile effect is determined based on at least a portion of supplementary information 422. According to the present disclosure, supplementary information 422 is used to determine the tactile effect that is output during playback of multimedia content 202, using three main techniques: 1) Identifying tactile effects that are output based on multiple keywords or phrases, 2) Contextual to multiple algorithms / processes for automatically tactile audio and / or video images Information; to provide contextual information) and 3) to identify tactile themes based on supplementary information 422. In one embodiment, each of these techniques maps a plurality of words and / or phrases in supplementary information 422 to tactile effect identification information, context identification information and / or tactile theme information. Includes analyzing supplemental information 422 by comparing it to one or more lists or databases of keywords and / or frames (hereinafter referred to as the "keyword database"). In one embodiment, the list or database is information indicating whether a particular plurality of keywords and / or phrases are applicable to all types of supplemental information 422 (eg, subtitle / closed captioning data 204, It may include scripts 206 and / or tags 208), or a subset thereof. In other embodiments, a separate list or database of multiple keywords or phrases that apply to a particular type of supplemental information 422 is used. For example, in one embodiment, the first database of multiple keywords and / or phrases is used for each of the subtitle / closed caption data 204 and the plurality of scripts 206, and the second database is used for each of the plurality of tags 208.

<u style="single">Identifying predetermined tactile effects based on multiple keywords / phrases</u> In one embodiment, tactile effects are identified based on multiple keywords / phrases present in the subtitle / closed captioning data 204 or script 206. In one such embodiment, the subtitle / closed captioning data 204 is analyzed at element 412. In one embodiment, the subtitle / closed captioning data 204 and / or the tactile keyword / phrase 216 provided by the user is analyzed by comparing the word and / or phrase with the keyword database. For example, in such an embodiment, comparing the subtitle / closed captioning data 204 and / or the tactile keyword / phrase 216 provided by the user with the keyword database reveals that an "explosion" exists. When it becomes, it is mapped to tactile effect identification information (mimicking an explosion) for intense vibrating tactile sensations that disappears over time until the effect disappears. In another embodiment, when subtitle / closed captioning data 204 and / or user comment 214 is compared to a keyword database and it becomes clear that "shooting" exists, it is intense in a short time that mimics shooting. It is mapped to the tactile identification information of the shaking tactile effect. In another embodiment, the subtitle / closed captioning data 204 includes a song title / type indicating the characteristics or genre of the scene to be played in the background of different scenes, and further compares the song title / type with the keyword database. This reveals a mapping for each song title / type (or subset thereof) that maps to tactile identification information for a unique tactile effect. In some embodiments, these tactile effect mappings are selected to convey the features / genres of the scene to the user. One of ordinary skill in the art provides a plurality of words or phrases used by the present disclosure to identify the tactile effect output during playback of multimedia file 201.

For each keyword or phrase found in the keyword database, processor 110 searches the keyword database for the corresponding tactile effect identification information and associates it with the time stamp corresponding to the word or phrase in the subtitle / closed captioning data 204. In some embodiments, processor 110 fully analyzes supplemental information 422 prior to playback of multimedia content 202. In one such embodiment, the processor 110 collects the tactile effect identification information and the corresponding time stamp in the tactile track 414. The tactile track 414 is embedded in the multimedia file 201 containing the multimedia content 202 in a manner well known to those skilled in the art. In another embodiment, the tactile track 414 is stored in memory (eg, memory 112) as a single file separated from the multimedia file 201. In yet another embodiment, processor 110 receives the multimedia content 202 through streaming, and at element 412, processor 110 analyzes supplemental information 422 during playback of the multimedia content to provide a temporary tactile effect. Identification information (eg, part of tactile track 414) is temporarily stored in memory. In one such embodiment, the playback of the multimedia content 202 is delayed enough to cause the processor to perform an analysis of supplemental information 422, and in a timely manner during the playback of the multimedia content 202. And start the output of the tactile effect.

In some embodiments, a method of identifying tactile effects that are identified based on keywords present in subtitle / closed captioning data 204 and / or script data 206 is one technique used. In such an embodiment, the applicable elements of FIG. 4 are multimedia data 200, supplementary information 422, elements 412 and 414, and tactile track 418. As described in detail below, in some embodiments, the two techniques may be used in conjunction, and in other embodiment, a total of three techniques are linked. May be used.

<u style="single">Providing context for automated tactile algorithms</u> In some embodiments, supplemental information 422 is contextualized in an automated tactile algorithm (such as algorithm 406 for tactile one or more audios and / or algorithm 408 for tactile video one level above). The audio track 402 and / or the video track 404 of the multimedia content 202 is analyzed to identify the tactile effect that is used to provide and output. In one embodiment, the presence of an event, scene type, or element in one scene is determined based on the presence of keywords in supplementary information 422. For example, in one embodiment, at element 412, processor 110 searches one or more keyword databases for words or phrases contained in supplemental information 422 corresponding to multimedia content 202. For each keyword or phrase found in the keyword database, the processor 100 searches for context identification information corresponding to the corresponding word or phrase in supplementary information 422, if possible. Context identification information is provided to one or more automated tactile algorithms (406 and / or 408) by data structures, function calls, messages, or other means known by those skilled in the art. For example, in one embodiment, the word "bomb" in supplemental information 422 is mapped to contextual identification information used for all types of explosions, thereby an automated tactile algorithm (406) to look for an explosion. And / or notify 408). In other embodiments, the word "crime" in the genre field of tag 208 or the word "violence" in the rating field of tag 208 embedded in the multimedia content 202 is an event related to shooting recognition, car tracking. Point to automated tactile algorithms (406 and / or 408) to prioritize events related to (eg, keyy tires, collisions) and / or other crimes. Map to the context identification used to indicate. In other embodiments, the word "saxphone" in supplemental information 422 is a context used to direct an automated tactile sensation algorithm (406 and / or 408) to prioritize recognition of a saxphone or similar instrumental performance. Mapped to identification information. In other embodiment, the detected key provides context to an automated tactile algorithm (406 and / or 408) for a subset of multimedia content 202 (eg, a particular scene or movie scene). Used to do. Training automation algorithms on contextual information helps to handle both efficiency and accuracy by prioritizing what patterns or events to look for in audio and / or video. That is, based on the recognition of the algorithm of the event, the timing and accuracy of the corresponding tactile effects output by the algorithm are improved. Those skilled in the art will appreciate that the words or phrases used to provide context to the automated tactile algorithm are not limited by the embodiments provided by the present disclosure, but rather what event or element is the multimedia content 202. It can be recognized to extend to words or phrases that provide clues as to whether they are present in. Helps handle both efficiency and accuracy by prioritizing what patterns or events to look for in bio and / or video. That is, based on the recognition of the algorithm of the event, the timing and accuracy of the corresponding tactile effects output by the algorithm are improved. Those skilled in the art will appreciate that the words or phrases used to provide context to the automated tactile algorithm are not limited by the embodiments provided by the present disclosure, but rather what event or element is the multimedia content 202. It can be recognized to extend to words or phrases that provide clues as to whether they are present in. Helps handle both efficiency and accuracy by prioritizing what patterns or events to look for in bio and / or video. That is, based on the recognition of the algorithm of the event, the timing and accuracy of the corresponding tactile effects output by the algorithm are improved. Those skilled in the art will appreciate that the words or phrases used to provide context to the automated tactile algorithm are not limited by the embodiments provided by the present disclosure, but rather what event or element is the multimedia content 202. It can be recognized to extend to words or phrases that provide clues as to whether they are present in.

In some embodiments, processor 110 completely analyzes supplementary information 422 at element 412 to determine contextual identification information, and then uses the contextual identification information to perform automated tactile sensation algorithms 406 and /. Or 408 is used to process audio track 402 and / or video track 404 to generate tactile track 410 prior to playback of multimedia content 202. In one such embodiment, the processor 110 embeds the tactile track 410 generated by the automated tactile algorithm into the multimedia file 201 containing the multimedia content 202 by a method well known to those of skill in the art. In another embodiment, the tactile track 410 is stored in memory (eg, memory 112) as a file separated from the multimedia file 201. In yet another embodiment, especially when processor 110 receives multimedia content 202 via streaming, processor 110 analyzes supplemental information 422 and further, as described above, during playback of multimedia content 202. Executes an automated tactile sensation algorithm (406 and / or 408) and temporarily stores the generated portion of the tactile track 410 in memory such as memory 112. In such an embodiment, the reproduction of the multimedia content 202 has a tactile effect in a timely manner during the analysis of the supplementary information 422 on the processor 110 and the reproduction of the multimedia content 202 as described above. It is delayed enough to cause the automated tactile algorithm (406 and / or 408) to run in order to generate a portion of the tactile track 410 at the time to start the output.

In some embodiments, the contextual information found in Supplementary Information 422 should be avoided or tactile in response to events where tactile sensation is not desired. It is used to attenuate the intensity of the tactile effect. For example, in some embodiments, the presence of "laughter" in the subtitle / closed captioning data 204 is associated with one or more automated tactile algorithms (406 and / or 408) to look for laughing or laughing. It is used to inform and avoid the tactile effect being output for the event. Similarly, in other embodiments, the presence of "music" in the subtitle / closed captioning data 204 is one or more automated tactile algorithms to look for playing and / or singing various musical instruments. Used to inform 406 and / or 408) and avoid causing the output of the corresponding tactile effect. In other embodiments, the list of songs at tag 208 is used to signal an automated tactile algorithm 406 to look for and to avoid tactile reproduction of the listed songs.

In some embodiments, a method for analyzing supplemental information 422 to provide contextual identification information to one or more automated haptic algorithms (406 and / or 408). This is one technology used. In such an embodiment, the applicable elements of FIG. 4 are multimedia data 200 (including audio track 402 and / or video track 404), auxiliary information 422, and one or more automated tactile algorithms (406). And / or 408), element 412, and tactile track 410.

In some embodiments, a technique for identifying tactile effects that are output directly based on keywords and tactile information for contextual information found in auxiliary information 422 to an automated tactile sensation algorithm (406 and / or 408). Both of the techniques provided for the automatic generation of tracks are used. In one such embodiment, the processor 110 automatically synthesizes the generated tactile track 410 and the tactile track 414 to generate the combined tactile track 418. In one such embodiment, the processor 110 operates to insert the tactile effect found from the tactile track 414 directly into the tactile track 410, and if necessary, a contradictory auto-generated tactile sensation. Removes the effect from the automatically generated tactile track 410 and turns it into a synthetic track 418.

<u style="single">Tactile theme</u> In some embodiments, supplementary information 422 applies to a tactile theme applied to a tactile effect output during playback of multimedia content 202, or to a scene or portion of multimedia content 202. Used to determine tactile themes. For example, in one embodiment of the processor 110, element 412 searches one or more keyword databases for words or phrases contained in supplementary information 422 for multimedia content 202. For each keyword or phrase found in the keyword database, processor 110 searches the keyword database for haptic theme information and associates it with the entire multimedia content or a specific scene, thereby applying one or more haptic themes 416. Occurs. For example, in one embodiment, comparing the genre field of tag 208 with the keyword database reveals that each "action" exists, and also the overall tactile effect output during video playback. Make it clear that the "action" is mapped to the tactile theme information that indicates that the intensity or library is increased beyond the default level. In another embodiment, comparing Script 206 with the keyword database reveals that the phrase "I love you" exists in each, and also the overall tactile effect output during video playback. Reveal that the phrase is mapped to tactile theme information that indicates that the intensity of is reduced below the default level. In another embodiment, comparing the chapter data 212 with the keyword database reveals that the phrase "praise" exists in the chapter data 212 and the keyword database, and the tactile sensation output during video playback. It is clear that the phrase is mapped to haptic theme information that indicates that the overall intensity of the effect is slightly above the default level during the playback of Chapter 1 of Multimedia Content 202. To. In another embodiment, the identified tactile theme 416 is applied to the synthetic tactile track 418 to generate the final tactile track 420. In yet another embodiment, one or more identified tactile themes 416 may be applied to tactile tracks generated using only one of the two techniques described above. In such an embodiment, one or more identified tactile themes 416 are applied to the tactile track 414 or the tactile track 410 to generate the final tactile track 420. In one embodiment, processor 110 embeds the final tactile track 420 into multimedia data 200, including multimedia content 202, by methods well known to those of skill in the art. In another embodiment, the tactile track 420 is stored in memory (eg, memory 112) as a file separate from the multimedia file 201. In some embodiments, including streaming, the identified tactile theme 416 is applied to a portion of the tactile track when generated.

The identification and application of haptic theme 416 according to the present disclosure assists automated haptic algorithms (406 and / or 408) so that haptic effects are more accurately applied to multimedia content. However, in some embodiments, a given keyword may be prioritized beyond the identified tactile theme. For example, in one embodiment, the intensity of the tactile effect output is increased above the default intensity based on the presence of an "explosion" in the closed captioning data of the multimedia file, thereby embedding it in the multimedia content. Prioritizes tactile effect applications that soften the intensity of tactile intensity based on the presence of "romantic comedy" in the genre field of metadata tags. Those skilled in the art do not limit the embodiment in which the words or phrases used to determine the appropriate tactile theme are provided by the present disclosure, but rather any event or element is present in the multimedia content. Understand that it extends to words or phrases that provide clues.

The first two techniques for determining the tactile effect based on the supplementary information 422 described above may be used alone or in combination, and the technique using the tactile theme may be either of the first two techniques. Or in connection with both, or in connection with any other technique that identifies the tactile effect output during playback of multimedia content 422. For example, in one embodiment, supplemental information 422 is generally used to provide context for automated tactile algorithms (406 and / or 408) that use pattern recognition techniques for audio and / or video tracks. However, due to the small set of keywords present in the close and caption data, the tactile effect in particular is output solely based on the presence of these keywords in the supplementary information.

At step 306, the tactile effect is output during playback of the multimedia content 202. In one embodiment, the tactile track generated in the previous step (eg, 410, 414, 418 or 420) comprises a data structure that defines the tactile effect output by the tactile output device 118. In other embodiments, the tactile track (eg, 410, 414, 418 or 420) comprises a tactile effect code, further the processor 110 defines data in the tactile effect database stored in memory 112. Use a tactile effect code to search for structures. In such an embodiment, the processor 110 transmits a signal to drive the tactile output device 118 based on the tactile effect definition in such a data structure. In other embodiments, a tactile accessory that combines one or more tactile output devices 118 (eg, tactile furniture, wearable tactile devices) is a tactile effect identification in a processor and a data structure that defines the tactile effect. It has a memory that contains a database that maps code. In one such embodiment, the processor 110 sends a tactile effect code to the processor of the tactile accessory, which then uses the tactile effect code to retrieve the data structures that define the tactile effect, and further its A signal for driving the tactile output device 118 is transmitted based on the tactile effect definition in such a data structure.

<u style="single">Audio track commentary</u> In some cases, it has a commentary audio track that is played at the same time as a video in which a movie, documentary and other video content is associated, with directors, actors and / or others discussing the video. In one embodiment of the present disclosure, the commentary audio track is of a different type than the supplementary information 422 on which the determination of the tactile track 420 is based. In such an embodiment, the relevant commentary audio track has its own subtitle / closed captioning data. The unique subtitle / closed caption data is used to determine the tactile effect provided in a manner similar to that described for the subtitle / closed caption data 204 described above. In other embodiments, the unique subtitle / closed captioning data for the commentary audio track is not available and the system 100 is to generate a textured version of the commentary audio track in a manner well known to those skilled in the art. It is used to process a commentary audio track and then determine the tactile effect provided in a manner similar to that described above for subtitle / closed captioning data 204.

<u style="single">Advanced ways to provide tactile effects based on supplemental information</u> With supplementary information 422 of multiple keywords or phrases to identify a predetermined tactile effect, to provide context to an automatic tactile algorithm, and / or to identify one or more tactile themes to be applied. In addition to the multiple embodiments of the present disclosure described above that generally describe matching, another technique may be associated with or independently of the methods described above to improve the accuracy of the generated tactile track 420. It may be used. In one embodiment, in element 412, the presence of a keyword / phrase in the subtitle / closed captioning data 204 is audible, the presence of which corresponds to the audio of the multimedia content 202. If it is present in the textual citation of speech), it is ignored, but if its presence is present in the text describing the event visually portrayed by the multimedia content 202, in the section above. As described, it is relied on to identify predetermined tactile effects, to provide context for automated tactile algorithms, and / or to identify tactile effects. For example, in one embodiment, when a person is speaking the word "explosion", it is impossible to have an explosion, so the presence of the word "explosion" in the subtitle / closed captioning data 204 is ignored. However, if the "explosion" is present in the subtitle / closed captioning data 204 that describes the scene or event in the multimedia content 202, its presence is a predetermined tactile sensation, as described in the section above. Used to identify effects, to provide context for automated tactile algorithms, and / or to identify tactile themes. What type of keyword / phrase of Supplementary Information 422 described herein is to be ignored or dependent on it will be determined by methods well known to those of skill in the art. For example, the database of keywords / phrases mentioned above is the presence of specific keywords / phrases for a particular type and / or subtype of supplemental information 422 (eg, textual descriptions and citations of subtitle / closed captioning data 204). May be mentioned when ignoring and depending on a particular keyword / phrase for a particular type and / or subtype of supplemental information 422.

In other embodiments, the presence of keywords / phrases in supplemental information 422 may be weighted based on the type or subtype of supplementary information 422 found. In one embodiment, when the above-mentioned processors output a plurality of tactile effects at the same time, or when a plurality of tactile effects overlap or interfere with each other in a short period of time, one tactile effect is provided on the tactile track 414. Weighting is used by processor 110 to prioritize overlap insertion. For example, the presence of a "bomb" in a textual citation in subtitle / closed captioning data 204 is given a low weight, and the presence of "encouragement" in a textual description of a scene / event in subtitle / closed captioning data 204 is given a high weight. Given. In one such embodiment, supplementary information 422 includes subtitle / closed captioning data 204 containing the textual description "encouraging crowd" and further textual citation "Dan Marino bombs" subtitle / closed captioning data 204 at the same time. Throw, Dolphins touch down! "including. Processor 110 identifies a predetermined tactile effect, provides context to an automated tactile algorithm, and / or identifies a tactile theme, based on "bomb" and "encouragement," as described above. It works for, but prioritizes tactile effects, contexts, and / or tactile themes related to "encouragement" in the textual description in the subtitle / closed captioning data 204 based on the high weight of "encouragement". The weights assigned to the presence of various types of keywords / phrases in Supplementary Information 422 described herein may be determined by one of ordinary skill in the art in a manner well known. For example, the aforementioned database of keywords / phrases weights for specific keywords / phrases in a particular type and / or subtype of supplemental information 422 (eg, textual descriptions and citations of subtitle / closed captioning data 204). Prescribe.

In other embodiments, well-known natural language analysis techniques are used by the processor 110 to improve the correlation of the tactile track 420 to the events of the multimedia content 202. For example, a well-known algorithm for natural language analysis techniques in one embodiment takes into account the syntax and meaning of phrases in subtitle / closed captioning data 204, making sure that the actual meaning is very accurate, and subtitle / closed. Used to interpret the phrase in captioning data 204. For example, processor 110 applies natural language analysis technology to analyze the phrase "no bomb" in subtitle / closed captioning data 204 and predetermines for "bomb" by understanding the true meaning of that phrase. Determine that the presence of a "bomb" should not be compared to a database of keywords and phrases to identify the tactile effects that have been made. However, the processor 110 determines that the presence of the "bomb" description in the subtitle / closed captioning data 204 is a representation of a genre of multimedia content such as war or action, and selects a more appropriate tactile theme. You may judge. In another embodiment, the processor 110 analyzes the phrase "3, 2, 1 ... the bomb explodes" in the subtitle / closed captioning data 204, recognizing the countdown in that phrase. To apply natural language analysis technology and insert a predetermined tactile effect corresponding to the "explosion" into the tactile track 420 at the exact time, or at the exact time based on that phrase, automated tactile sensation It works to apply context identification information to the analysis algorithm (406 and / or 408). The above are just a few examples of using natural language analysis techniques to analyze supplemental information 422. Those skilled in the art will recognize numerous applications for the use of natural language analysis techniques in the context of this disclosure.

In some examples, keywords or keywords to provide context to automated tactile algorithms to identify predetermined tactile effects and / or to identify one or more tactile themes to be applied. Matching a phrase with supplemental information 422 may be affected by being inaccurate due to some words or phrases that have multiple possible meanings in a particular language. In one embodiment, the subtitle / closed captioning data 204 includes subtitles and / or closed captioning information in multiple languages (eg, English, Spanish, and even French) and in a first language (eg, English). Such uncertainties in words / phrases generally mean that a word / phrase in one of the other languages has one acceptable meaning, one or more other available languages (eg Spanish, French). ) Is solved by analyzing the word / phrase. In one embodiment, this disambiguation method is performed by marking (flagging) an ambiguous word / phrase in each ambiguous language in the keyword database for that language. In such an embodiment, by determining a keyword database that indicates that a word / phrase in one language is ambiguous, System 100 attempts to resolve the ambiguity in the corresponding word / phrase in a second language. Will analyze the phrase. For example, in one embodiment, the subtitle / closed captioning data 204 includes closed captions in English, French, and Spanish. In this embodiment, processor 110 compares the English closed caption text with the keyword database and finds that the word "tie" exists in the English closed caption text and keyword database, but with the word "tie". Is flagged as an ambiguous word. Based on "tie" flagged for ambiguity Then, in order to determine the meaning of the "tie", the processor 110 analyzes the French closed caption text near the same type stamp in the multimedia content 202 in order to search for such a "tie". In one embodiment, this analysis involves searching for French synonyms for the various meanings of "tie", and once identified by the French closed captions, the meaning of the word "tie" is correct. Show that. For example, the analysis may identify the French "cravate", which means a tie, or the French "match nul", which means a tie game. Once the "tie" disambiguation is complete, the methods of the invention are to identify a predetermined tactile effect, to provide context to an automated tactile algorithm, and / or to apply one or more. Used to identify the tactile theme of. For example, a tactile theme appropriate for a sporting event is applied based on the presence of a "match nul". Those skilled in the art will recognize that this is just one of many possible examples. May identify "nul". Once the "tie" disambiguation is complete, the methods of the invention are to identify a predetermined tactile effect, to provide context to an automated tactile algorithm, and / or to apply one or more. Used to identify the tactile theme of. For example, a tactile theme appropriate for a sporting event is applied based on the presence of a "match nul". Those skilled in the art will recognize that this is just one of many possible examples. May identify "nul". Once the "tie" disambiguation is complete, the methods of the invention are to identify a predetermined tactile effect, to provide context to an automated tactile algorithm, and / or to apply one or more. Used to identify the tactile theme of. For example, a tactile theme appropriate for a sporting event is applied based on the presence of a "match nul". Those skilled in the art will recognize that this is just one of many possible examples.

<u style="single">Remote occurrence of tactile truck</u> The embodiments of the present disclosure described above generally include processor 110 of system 100 for analyzing multimedia content 202 and corresponding supplemental information for generating tactile tracks (eg, 410, 414, 418 or 420). However, in other embodiments, the analysis and generation of tactile tracks (eg, 410, 414, 418 or 420) is performed on another computer or other computer that communicates with processor 110 through a network such as the Internet. It may be run by the server. In one such embodiment, the remote computer or server embeds a tactile track (eg, 410, 414, 418 or 420) in multimedia file 201. The processor 110 of the system 100 then downloads the multimedia file 201 containing the tactile track while playing the multimedia content 202 described above in connection with step 306 of FIG. In another embodiment, the remote computer stores a tactile track (eg, 410, 414, 418 or 420) in the memory of the remote computer as a file locally separated from the multimedia file 201. In another embodiment, the processor 110 of the system 100 downloads a tactile track (eg, 410, 414, 418 or 420) from a remote computer in which the tactile track is stored, in connection with step 306 of FIG. As described above, the tactile effect is output while the multimedia content 202 corresponding to the tactile track is being played.

<u style="single">Pre-Generation of tactile tracks</u> The embodiments described above describe that the tactile effect occurs when the user is playing the multimedia content, but in other embodiments, the multimedia content is downloaded to the system. , Tactile effects may occur before being regenerated. In other embodiments, the tactile track may be generated by the present disclosure by the creator or distributor of multimedia content prior to enabling the content for use.

<u style="single">Providing tactile effects based on viewer information</u> It has been described that the embodiments of the present disclosure described above provide a tactile effect based on supplementary information 422 for multimedia content 202, but in other embodiments the generated tactile track 420 is It may be analyzed at element 412 based on the viewer's position, orientation, response, and even biostatistics (shown as additional information 424). Viewer information is determined for System 100 using one or more of the sensors described above.

In one embodiment, the tactile theme that adjusts the output level of the tactile effect to be above or below the default level is an output source for playback of multimedia content 202 (eg, A / V reception of system 100). Judgment is based on viewers in close proximity to the machine, display 116 and / or speaker 120). For example, in one embodiment, if the viewer proximity is 10 feet (10 feet; 304.8 cm) from the output source for playing multimedia content, the tactile theme is not selected. However, if the viewer's proximity is 9 feet (9 feet; 274.3 cm) from the output source for multimedia content playback, a haptic theme is selected that defines the output level for haptic effects above the default level. .. The viewer's approach to the output source gradually increases the adjustment of the output level of the tactile effect defined by the selected tactile theme. On the other hand, the viewer's proximity to the output source is approximately 9 feet (9 feet; 274.3). cm) selects a haptic theme that defines the output level for haptic effects below the default level. The distance from the viewer's output source gradually reduces the adjustment of the output level of the tactile effect defined by the selected tactile theme.

In a similar embodiment, a tactile theme that adjusts the output level of the tactile effect to be above or below the default level is viewed on an output source for playback of multimedia content 202 (eg, display 116 of system 100). Judgment is based on the viewing angle of the person. For example, in one embodiment, if the viewer's viewing angle is about 90 ° (directly facing) with respect to the output source for playing multimedia content, the tactile effect is at maximum level according to the tactile track 420. It is output. However, as the viewer's angle to the output source approaches 0 °, the tactile effect defined on the tactile track 420 is adjusted to a lower output level.

In other embodiments, the viewer's position, posture and / or movement is used to determine the tactile theme. For example, in one embodiment, the tactile theme for a horror or action movie is selected based on covering the viewer's eyes. In other embodiments, tactile effects for romantic comedies are selected based on couples sitting close to each other in a relaxed and smiling mood.

In yet other embodiments, the viewer's biometric information is used to select a theme. For example, in one embodiment, the tactile effect on an action movie is selected based on a viewer with an elevated heart rate indicating stress and / or excitement. In other embodiments, a tactile theme that adjusts the tactile effect to a lower level is selected based on the viewer's heart rate, which indicates the viewer's relaxation.

<u style="single">video game</u> In one embodiment, the multimedia content 202 includes audio and / or video output during playback of the video game. For example, in one embodiment, the video game contains a video sequence and supplementary information 422 (eg, subtitle / closed caption data 204 corresponding to the video sequence) including any of the aforementioned types associated with multimedia content including video. Includes multimedia content 202. That is, the embodiment that provides the tactile effect based on the supplementary information 422 as described above is further applied to the video game. However, in other embodiments, other information (indicated by additional information 424) is used independently or is output in addition to the supplementary information 422 considered in the aforementioned embodiments. It is used to identify the tactile effect.

In one embodiment, the system 100 is configured to run a video game program and further provides an application program interface that prompts the video game program to start playing an audio file ("audio API"). In one such embodiment, the textual identifier (eg, filename) of the audio file that is required to be played back via the audio API ("filename data") is provided to the audio API. In one embodiment, at element 412, system 100 1) identifies tactile effects output based on keywords; 2) algorithms for automatically tactile audio and / or video images. / To provide contextual information to the processor; and 3) For various types of supplemental information 422, filename data may be analyzed to identify tactile effects in a similar manner as described above. For example, in one embodiment, the filename data is analyzed by comparing the words in the filename data with the keyword database at element 412. In such an embodiment, the file name data contains the text "thunder.mp3", and comparing "thunder" in the keyword database means that "thunder" exists in the keyword database and pulsates and vibrates. Clarify that the intensity of the tactile effect (mimicking lightning) is mapped to the tactile effect identification information. In another embodiment, the file name data is the text "scream. Comparing "scream" in the keyword database, including "mp3", means that "scream" is present in the keyword database and screams, screams, and fears the automated tactile algorithms (406 and / or 408). It is mapped to contextual identification information used to indicate prioritization of recognition of other sounds that are commonly associated with the scene. In other embodied examples, comparing "screams" (from "scream.mp3") to the keyword database reveals that each "scream" exists, and also during video playback. Maps to tactile theme information that describes that the overall intensity of the output tactile effect or library increases appropriately beyond the default level for horror video sequences.

In another embodiment, System 100 provides an application program interface that encourages the video game program to output the text displayed on the screen as audio ("Text-to-Speech API") during the video game. provide. In one such embodiment, the text displayed on the screen (eg, instructions, signs or other text displayed in the video game scene, comic-style action slogans such as "BAM" and "THWACK"). Is provided for the Text-Speech API. In one embodiment, the system 100 uses element 412 to 1) identify tactile effects output based on keywords; 2) algorithms for automatically tactile audio and / or video images. / To provide contextual information to the processor; and 3) Different types of supplemental information 422 (eg, subtitle / closed captioning data 204) are identified to identify tactile effects in a similar manner as described above. Analyze that.

<u style="single">Summary</u> The above-mentioned statements of some embodiments of the present invention are shown for purposes of illustration and illustration only, and the above-mentioned statements limit the present invention to exhaustive or disclosed scrutiny forms. Is not intended. Numerous variations and their applications will be apparent to those skilled in the art without departing from the spirit and scope of the invention.

As used herein, "one embodiment" or "one embodiment" includes the unique properties, structures, behaviors or other features described in connection with an embodiment in at least one implementation of the invention. It means that it may be. The present invention is not limited to the particular embodiments so described. The appearance of the frames "in one embodiment" or "in one embodiment" in the present specification does not necessarily refer to the same embodiment. The unique properties, structures, behaviors or other features associated with "one embodiment" described herein relate to other properties, structures, behaviors or other features described for other embodiments. Will be integrated.