CN115105293A - Object detection, analysis and prompting system for providing visual information to the blind - Google Patents

Abstract

The present disclosure provides an object detection, analysis and prompting system for providing visual information to a blind person. More specifically, a landmark detection system is provided. A portable closed loop system for acquisition, analysis and feedback uses a headset that includes a small, unobtrusive camera and a control computer that wirelessly communicates with a wireless network and/or a remote platform. The headset may also contain user controls, audio feedback components, batteries, interconnect circuitry, cables, and connections for the intraoral device. The camera component of the headset captures images during an activity to be analyzed (e.g., walking or viewing a room) and sends data (e.g., visual data) to the controller. The controller transmits the data to a database on a remote platform that includes software for instant analysis of the image information represented in the data, and then provides instant feedback to the headset. The controllers may process the data independently.

Description

Object detection, analysis and prompting system for providing visual information to the blind

technical field

The present invention generally relates to methods and apparatus for providing visual information to visually impaired or blind persons. More particularly, the present invention relates to methods and apparatus designed to provide a completely blind person with the ability to detect and identify landmarks and navigate within their surroundings. The device includes a portable closed-loop system of acquisition, analysis, and feedback using a headset that includes an unobtrusive camera and a control computer that communicates wirelessly with a wireless network and/or a remote platform. The camera component of the headset captures images during the activity to be analyzed (eg, walking or viewing a room) and sends data (eg, visual data) to the controller. The controller transmits the data to a database on a remote platform that includes software that analyzes the image information presented in the data, and then provides feedback to the headset. Controllers can process data independently. The headset provides feedback to the user in the form of tactile means (eg, electrical tactile stimulation of the user's tongue via an attached intraoral device) and/or auditory means (eg, via a speaker).

Background technique

The American Foundation for the Blind (AFB) estimates that there are currently approximately 1.3 million identified blind people in the United States. This number is a small fraction of the world's estimated blind population of approximately 40 million people. Nearly half of the world's identified blind people live in China.

Blind patients have traditionally relied on a cane to guide them (eg, when walking down a street or corridor, or when moving around a room or store). However, conventional walking sticks only provide very limited information about the user's surroundings (often about objects that can be physically touched by the stick).

Other devices have been developed to provide a blind or visually impaired person with information about his or her surroundings beyond the physical reach of a conventional cane. For example, vocal canes provide information through acoustic feedback (echolocation). When the vocal wand is used, it emits an audio signal that reflects or echoes from objects within the user's surroundings. The user interprets the echoes to identify the layout of the environment. Other devices emit light signals that reflect off objects around the user. The reflections are then converted into auditory signals, such as clicks or beeps of variable pitch, to convey information about surrounding objects back to the user.

US Patent Application Serial No. 10/519,483 (Publication No. US2006/0098089A1 ) discloses an apparatus including an electro-optical device for detecting and identifying objects. The control unit is used to receive and process information from the device. The sound representation unit is then used to receive instructions from the control unit for the purpose of aurally describing the object to the user.

U.S. Patent Application Serial No. 12/354,266 (Publication No. US2010/0177179) discloses a device that includes components similar to those in U.S. Patent Application Serial No. 10/519,483, but also includes a A monitor on a device on which to view its surroundings.

US Patent Application Serial No. 11/925,393 (Publication No. US2009/0312817 ) discloses a visual aid and/or enhancement device that uses electrical tactile stimulation to provide visual imagery on a user's tongue.

However, such devices have significant limitations in that they provide a profoundly blind user with little information about the user's distal environment. For example, devices that rely on monitors to provide information about the surrounding environment to the blind do not provide usable information to the person. Additionally, the use of audio signals alone to convey information about the surrounding environment to the user is not suitable for noisy environments, eg, busy streets, or for deaf-blind people who cannot hear audio signals. Furthermore, for deeply blind users, these and other existing devices cannot identify landmarks (eg, signs or navigational cues) for the blind in the personal environment beyond the distance that can be swept by and touched with the cane.

SUMMARY OF THE INVENTION

The present invention addresses the problems of the prior art approaches by providing an apparatus and method that provides a blind user with the ability to scan her or his near and far environment to detect and identify landmarks (eg, signs or other navigational indications), and the ability to view the environment via electrotactile stimulation of the user's tongue.

Accordingly, in one embodiment, the present invention generally relates to an apparatus and method for providing visual information to a visually impaired or blind person. More particularly, the present invention relates to devices and methods designed to provide a completely blind person with the ability to detect and identify landmarks and navigate within their surroundings. The device includes a portable closed-loop system of acquisition, analysis, and feedback using a headset that includes an unobtrusive camera and a control computer that communicates wirelessly with a wireless network and/or a remote platform. The camera component of the headset captures images during the activity to be analyzed (eg, walking or viewing a room) and sends data (eg, visual data) to the controller. The controller transmits the data to a database on a remote platform that includes software that analyzes (eg, instant analysis) the image information represented in the data, and then provides feedback (eg, instant feedback) to the headset. The headset controller can process data independently. The headset provides feedback to the user in the form of tactile means (eg, electrical tactile stimulation of the user's tongue via an attached intraoral device) and/or auditory means (eg, via a speaker).

Accordingly, in one embodiment, it is an object of the present invention to provide methods and apparatus to provide blind persons with the ability to detect and identify landmarks in their surroundings. Another object is to provide increased versatility, efficiency, adaptability and/or economy in a device comprising a portable closed-loop system of acquisition, analysis and feedback using an unobtrusive camera and a wireless network and/or a control computer that wirelessly communicates with a remote platform for a headset for landmark detection and recognition.

Another object of the present invention is to provide an algorithmic method and apparatus for detecting and recognizing objects within a camera's field of view worn by a blind person. Another object is to provide a method and apparatus for algorithms for analyzing transmitted data/information within a database on a platform remote from the headset controller. In another embodiment, the algorithm is based on generating feedback information related to the analyzed transmitted data/information, wherein the feedback information is transmitted (eg, via a wireless network) to the headset for delivery to the blind. The present invention is not limited by means of delivering feedback information to the blind. Exemplary means include delivering feedback information (eg, including visual information) to the blind via tactile (eg, electrical tactile stimulation of the human tongue) and auditory (eg, via speakers, earphones, or bone conduction) means.

In one embodiment, the present invention provides a method and apparatus for detecting, identifying, and highlighting landmarks within an environment of a blind user (eg, as a way to guide the user to landmarks (eg, signs (eg, exit signs, toilet signs, means of pedestrian crossing signs or other signs commonly used for navigation in the environment))). The present invention is not limited to any particular means of highlighting landmarks. Several non-limiting examples include methods and apparatuses of algorithms that highlight landmarks on a user's tongue using electrical tactile stimulation, and methods and apparatuses of algorithms that audibly highlight landmarks to a user using electrical stimulation as they scan the environment Provides visual information on the user's tongue.

In one embodiment, the present invention provides a feature that assists blind persons in detecting, recognizing, and/or moving toward landmarks (eg, while circumventing, passing, and/or passing through obstacles or buildings within a person's environment). equipment and methods. In one embodiment, the device includes a headset including a control computer including an unobtrusive camera and communicating with a wireless network and/or a remote platform, user controls, audio feedback components, batteries, interconnects Circuits, cables, and connections for intraoral devices. In one embodiment, the camera component of the headset collects visual information of the environment during the user activity to be analyzed (eg, walking or looking at a room) and sends the visual information to the controller, whereby the controller sends the visual information data (eg, all or part of the total visual information captured by the camera) is transmitted to a database on a remote platform that includes software that analyzes the image information represented in the visual information data, and then converts information about the visual information data provides feedback to the headset, which in turn provides feedback to the blind user (e.g., via tactile (e.g., electrical tactile stimulation of a person's tongue) and/or hearing (e.g., via speakers, headphones, or bone conduction) )means).

Another object of the present invention is to provide a method and apparatus for detecting and removing shadows from visual information collected by a camera. In one embodiment, detecting and removing shadows from visual information (eg, a digital image stream) captured by the camera includes control by a headset controller and/or a remote platform (eg, using a processor, algorithm, and/or other computer) component) to process the visual information to remove shadows from the visual information (eg, before providing feedback to the blind via the headset regarding the visual information (eg, digital image stream) captured by the camera).

In one embodiment, the present invention provides a method for blind persons to detect, identify, and/or move toward landmarks (eg, when going around, over, and/or through obstacles or buildings within a person's environment) ), comprising the steps of: receiving visual information of the user's environment, transmitting visual information data (eg, all or part of the total visual information captured by the camera) to a remote platform (eg, to a database on the remote platform), Analyzing the visual information data on the remote platform and sending feedback related to the visual information data from the remote platform to the user, thereby enabling the user to detect, identify, and/or move towards the landmark (e.g., when going around, over and/or when passing through obstacles or buildings within a human environment).

In one embodiment, the present invention provides methods and apparatus for detecting, identifying, and/or highlighting landmarks within the environment of a blind user (eg, as a way to guide the user to landmarks (eg, signs (eg, exit signs, toilet signs) , pedestrian crossing signs or other signs commonly used for navigation in the environment))). In one embodiment, the apparatus includes a headset including a camera and a control computer in wireless communication with a wireless network and/or a remote platform, and a remote platform including a processor component, a memory component, and a software component, wherein data obtained from the camera Visual information is processed on a remote platform. In one embodiment, the remote platform includes algorithms to detect, identify and/or highlight landmarks present in the visual information. In another embodiment, the remote platform includes algorithms for detecting and reducing and/or eliminating shadows in visual information. In yet another embodiment, the remote platform includes an algorithm to detect objects in the visual information. In one embodiment, the apparatus includes means for communicating information about landmarks, shadows and/or objects to a blind user.

Another object of the present invention is to provide a method and apparatus for detecting range and distance information of objects in visual information collected by a camera. In one embodiment, the range and distance information of objects in the visual information (eg, digital image stream) captured by the camera includes information generated by the headset controller and/or the remote platform (eg, using a processor, algorithm, and/or other computer components) processing the visual information to calculate range and distance information of objects in the visual information (eg, before providing feedback via the headset to the blind person about visual information (eg, digital image streams) captured by the camera) ).

Another object of the present invention is to provide a method and apparatus for detecting color and/or contrast information of an object in visual information collected by a camera. In one embodiment, color and/or contrast information of objects in visual information (eg, a digital image stream) captured by the camera is included by a headset controller and/or a remote platform (eg, using a processor, algorithms, and Process the visual information to calculate color and/or contrast information for objects in the visual information (eg, when passing feedback related to visual information (eg, digital image streams) captured by the camera via the headset) before the blind).

Another object of the present invention is to provide a method and apparatus for detecting gesture-based instructions in visual information collected by a camera. In one embodiment, gesture-based instructions (eg, digital image streams) captured by the camera include processing of visual information (eg, using processors, algorithms, and/or other computers) by the headset controller and/or the remote platform components) to detect gesture-based instructions in visual information.

These objects and others not indicated above are achieved by exemplary embodiments of the present invention, wherein the system of the present invention is designed to assist blind persons in detecting, recognizing, and/or highlighting landmarks within the environment of a blind user (eg, as an Means to lead to landmarks (eg, signs (eg, exit signs, toilet signs, crosswalk signs, or other signs commonly used for navigation in an environment)). The device of the present invention is compact and lightweight, and can be mounted to blind persons The present invention, along with its additional features and advantages, may be best understood by reference to the following description taken in conjunction with the accompanying drawings.

Description of drawings

Figure 1 shows a non-limiting example of a visual aid device (VAD) for the blind of the present invention.

2A-2G illustrate additional views of non-limiting examples of visual VADs of the present invention.

2H-2N show schematic diagrams of a device incorporating the VAD of the present invention.

3 illustrates an exemplary VAD of the present invention including a number of interconnection subsystems including printed circuit assemblies (PCAs) interconnected by cables.

4A and 4B illustrate exemplary control buttons implemented as membrane switch assemblies in accordance with embodiments of the present invention.

Figure 5 shows a sensor subsystem located in the front of the headgear frame of the VAD, according to an embodiment of the present invention.

6 illustrates a VAD battery housing connected to a VAD headset with a power cable, according to an embodiment of the present invention.

Figure 7 shows an exemplary VAD of the present invention.

Figure 8 illustrates an exemplary user control integrated into a VAD according to an embodiment of the present invention.

Figure 9 shows what a sighted companion would see regarding images and status information obtained by a user of the VAD of the present invention.

Figure 10 shows sample exit sign detection. (A) Successful assay. (B) Rectangles show false negatives (missed detections), indicating candidates detected by the first (Adaboost) classifier stage that were incorrectly rejected by the second (SVM) classifier stage. (C) False positive (false) detections showing textured areas in the building facade.

Figure 11 shows a sample toilet sign detection. (A) Successful assay. (B) Successful assay. (C) The two rectangles show two false negatives, ie, two toilet icons that were not detected.

Figure 12 shows the receiver operating characteristic (accuracy versus recall) curve for the toilet sign detector: the curve shows the result without tracking, the X shows the result with tracking. Tracking improves recall with only a modest decrease in precision.

Figure 13 shows the receiver operating characteristic (accuracy versus recall) curve for the exit sign detector. The curve shows the result without tracking, the X shows the result with tracking.

To facilitate understanding of the present invention, various terms and phrases are defined as follows:

As used herein, the term "amplifier" refers to a device that produces an electrical output as a function of a corresponding electrical output parameter and increases the magnitude of the input (ie, it introduces gain) by means of energy absorbed from an external source. "Amplification" refers to the reproduction (usually at an increased intensity) of an electrical signal by an electronic device. "Amplification means" refers to the use of an amplifier to amplify a signal. Desirably, the amplifying means also include means for processing and/or filtering the signal.

As used herein, the term "receiver" refers to a portion of a system that converts transmitted waves into an output in a desired form. The frequency range over which a receiver operates at a selected performance (ie, a known sensitivity level) is the "bandwidth" of the receiver.

As used herein, the term "transducer" refers to any device that converts a non-electrical parameter (eg, sound, pressure, or light) into an electrical signal or vice versa.

As used herein, the terms "stimulator" and "actuator" refer herein to a component of a device for delivering stimulation (eg, tactile vibrations, electrical haptics, heat, etc.) to a subject's tissue. As used herein, the term stimulator provides an example of a transducer. Unless described to the contrary, the embodiments described herein using stimulators or actuators may also employ other forms of transducers.

The term "circuit" as used herein refers to the complete path of electrical current.

As used herein, the term "resistor" refers to an electronic device that possesses resistance and is selected for that use. The term is intended to encompass all types of resistors, including but not limited to fixed value or adjustable carbon, wire wound and thin film resistors. The term "resistance" (R; ohm) refers to the tendency of a material to block the passage of electrical current and convert electrical energy into thermal energy.

The term "magnet" refers to a body (eg, iron, steel, or alloy) that has the property of attracting iron and producing a magnetic field outside of it, which, when freely suspended, points toward the magnetic poles of the Earth.

As used herein, the term "magnetic field" refers to the area around a magnet in which magnetic forces can be detected.

As used herein, the term "electrode" refers to a conductor used to establish electrical contact with non-metallic parts of electrical circuits, particularly parts of biological systems (eg, human skin on the tongue).

The term "housing" refers to a structure that encloses or encloses at least one component of the device of the present invention. In a preferred embodiment, the "housing" is made of a "biocompatible" material. In some embodiments, the housing includes at least one hermetically sealed guide hole through which leads extend from components inside the housing to locations outside the housing.

As used herein, the term "biocompatible" refers to any substance or compound that has minimal (ie, no significant difference is seen compared to a control) or no irritant or immunological effect on surrounding tissue. It is also intended that the term be used to refer to substances or compounds used to minimize or avoid immunoreactivity with the capsid or other aspects of the invention. Particularly preferred biocompatible materials include, but are not limited to, titanium, gold, platinum, sapphire, stainless steel, plastics, and ceramics.

As used herein, the term "hermetic seal" refers to a device or object such that liquids or gases located outside the device are at least somewhat prevented from entering the interior of the device. "Totally hermetic seal" refers to a device or object that is sealed in such a way that no detectable liquid or gas located outside the device enters the interior of the device. Desirably, sealing is accomplished by various means, including but not limited to mechanical, glue or sealants, and the like. In a particularly preferred embodiment, the hermetically sealed device is manufactured such that it is completely leak-proof (ie does not allow liquids or gases to enter the interior of the device at all).

As used herein, the term "processor" refers to a device capable of reading a program from computer memory (eg, ROM or other computer memory) and performing a set of steps in accordance with the program. The processor may include non-algorithmic signal processing components (eg, for analog signal processing).

As used herein, the terms "memory component," "computer memory," and "computer memory device" refer to any storage medium readable by a computer processor. Examples of computer memory include, but are not limited to, RAM, ROM, computer chips, digital video disks (DVDs), compact disks (CDs), hard disk drives (HDDs), and magnetic tapes.

As used herein, the term "remote platform" refers to any remote computer, phone, tablet, personal computer, or computer containing a processor and memory components (eg, for storing a database) separate from the headset controller of the present invention. other devices.

As used herein, the term "computer-readable medium" refers to any device or system for storing and providing information (eg, data and instructions) to a computer processor. Examples of computer-readable media include, but are not limited to, DVDs, CDs, hard drives, magnetic tapes, flash memory, and servers on a network for streaming media.

As used herein, the terms "multimedia information" and "media information" are used interchangeably to refer to information that encodes and represents audio, video, and/or text (eg, digitized information and analog information). Multimedia information can also carry information that does not correspond to audio or video. Multimedia information may be transmitted from one location or device to a second location or device by methods including, but not limited to, electrical, optical, and satellite transmission, among others.

As used herein, the term "Internet" refers to any collection of networks using standard protocols. For example, the term includes a collection of interconnected (public and/or private) networks linked together by a set of standard protocols (eg, TCP/IP, HTTP, and FTP) to form a global, distributed network. Although the term is intended to refer to the network generally referred to as the Internet, it is also intended to cover possible future variations, including changes and additions to existing standard protocols or integration with other media (eg, television, radio, etc.) . The term is also intended to cover non-public networks, such as private (eg, enterprise) intranets.

As used herein, the term "security protocol" refers to an electronic security system (eg, hardware and/or software) that restricts access to a processor, memory, to specific users authorized to access the processor. For example, a security protocol may include a software program that locks out one or more functions of the processor until the correct password is entered.

As used herein, the term "resource manager" refers to a system that optimizes the performance of a processor or another system. For example, a resource manager may be configured to monitor processor or software application performance and manage data and processor allocation, perform component failure recovery, optimize data reception and transmission, and the like. In some embodiments, the resource manager includes a software program provided on the computer system of the present invention.

As used herein, the term "electronic communication" refers to electrical devices (eg, computers, processors, communication devices) that are configured to communicate with each other through direct or indirect signaling. For example, a conference bridge is connected to the processor by a cable or wire so that information can be passed between the conference bridge and the processor, such conference bridges being in electronic communication with each other. Likewise, a computer configured to communicate information (eg, via a cable, wire, infrared signal, telephone line, etc.) to another computer or device is in electronic communication with the other computer or device.

As used herein, the term "transfer" refers to moving information (eg, data) from one location to another using any suitable means (eg, wireless communications (eg, WIFI, Internet, cloud, etc.) and wired communications) (eg, moving from one device to another).

As used herein, the term "electrical haptics" refers to the means by which sensing channels (eg, nerves) responsible for sensing functions are stimulated by electrical current. In some embodiments, the term refers to the means by which the sensing channels (eg, nerves) responsible for a person's tactile (and/or taste) perception are stimulated by electrical current (applied via surface (or implanted) electrodes). The term electrohaptic is used interchangeably with the terms "electrocutaneous" and "electrodermal".

Detailed ways

The present invention addresses the problems of the prior art approaches by providing a method and apparatus that provides a blind user with the ability to scan her or his near and far environment to detect and identify landmarks (eg, landmarks) or other navigational indicators), and the ability to view the environment via electrical tactile stimulation of the user's tongue.

Accordingly, in one embodiment, the present invention generally relates to an apparatus and method for providing visual information to a visually impaired or blind person. More particularly, the present invention relates to a device (eg, a visual aid device (VAD) 100, such as Figure 1, Figure 2H- Figure 2N, Figure 6 and Figure 7) and method. The device comprises a portable closed loop system of acquisition, analysis and feedback using a headset 1 comprising an unobtrusive camera 10 and a control computer (or controller) 8 in wireless communication with a wireless network and/or a remote platform. The camera 10 of the headset 1 acquires images during the activity to be analyzed (eg, walking or viewing a room) and sends data (eg, visual data) to the controller 8 . The controller 8 transmits the data to a database on a remote platform that includes software for instant analysis of the image information represented in the data, and then provides instant feedback to the headset 1 . The controller 8 can process the data independently. The headset 1 provides feedback to the user via the attached intraoral device 2 in the form of electrical stimulation of the user's tongue.

In one embodiment, the visual aid device 100 (VAD, also referred to herein as "V200" or "BRAINPORT V200") of the present invention converts images of objects captured by the digital camera 10 into images provided to the user electrical tactile signals of the tongue. The user interprets the electrotactile signals to perceive visual information including, but not limited to, the shape, size, position, and motion of the object. In further embodiments, the VAD 100 of the present invention includes components that allow for wireless connection/communication with one or more remote platforms (eg, for exchanging image data, status and/or control information). This functionality significantly expands the capabilities of the VAD 100 of the present invention (eg, by utilizing computer processing capabilities for processing one or more remote platforms (eg, including access to the Internet and related services)) over conventional devices.

In some embodiments, the VAD 100 of the present invention augments other visual assistance technologies (eg, canes or guide dogs). In other embodiments, the VAD 100 of the present invention completely replaces other visual assistance technologies (eg, canes or guide dogs).

A non-limiting example of a visual aid device 100 (VAD) for the blind of the present invention and various components therein is shown in FIGS. 1-2 and 4-8. In one embodiment, the VAD 100 of the present invention may include a fully wearable, battery-operated device with no physical connection to external devices (eg, during normal operation). The device is intended for portable use. As shown in Figure 1, the VAD 100 of the present invention may include a headset 1, an intraoral device (IOD) 2, a battery housing 4 and/or a charger. The headset 1 provides image input and output functions of the device. IOD2 contains stimulating electrodes (eg, arranged in an array (eg, an array of 20 rows by 20 columns with several electrodes removed on both edges to better conform to the tongue)). The IOD 2 can be placed on the user's tongue with the electrodes 37 in contact with the tongue. Stimulus patterns derived from camera images (or other sources such as remote platforms) are output to the array. The user perceives these patterns and translates them into visual information (eg, thereby perceiving information about the scene and environment in front of and around the user).

The present invention is not limited by the type of battery 5 . In one embodiment, the battery housing 4 houses a lithium polymer rechargeable battery 5 (eg, a battery that can be removed or replaced by a user). The battery housing 4 is attached to the headgear 1 via an adjustable strap and can be worn behind the head. This allows the user to have complete hands-free operation of the VAD100.

The headset 1 may include a camera unit 10, user controls 9, a control computer (controller, eg, a computer on a module (COM) or a system on a module) 8, interconnecting circuitry, and associated cables (eg, all housed in a housing similar in size and shape to an eyeglass frame). Simple head movements of the user direct the camera 10 field of view to the scene of interest (eg, scanning the environment sideways and/or horizontally). The camera 10 captures the seen scene as a digital image, where the image is forwarded to the controller 8 (eg, for processing and/or relaying to a remote platform). The electrode array on IOD2 presents the stimulation pattern representing the camera image to the user's tongue. Although the present invention is not limited by the number of electrodes present on the IOD2, in one embodiment the IOD2 contains 396 electrodes arranged in a 20x20 grid 37 (3 electrodes on each front corner are not installed so that the corners can be rounded). The electrodes are placed on the top surface of the tongue during use. In one embodiment as shown in Figure 1, the IOD2 is attached to the headset 1 with a flexible cable 3, which allows easy repositioning of the IOD2 on the tongue and removal of the IOD2 from the oral cavity. In another embodiment, the IOD2 is connected to the headset 1 wirelessly. User controls 9 and feedback are located on the headset 1 (eg, see Figure 1, push button membrane switch assembly 9). The present invention is not limited by the means of providing tactile (electrical) stimulation to the user's tongue. In one embodiment, the device described in US Patent Application Serial No. 11/925,393 (Publication No. US2009/0312817), which is incorporated herein by reference in its entirety, is used.

Additional features may be incorporated into the headset 1 (eg, see Figure 1). These features include audio feedback devices (eg, audio speakers 12, audio connection plugs 13), wireless (eg, WIFI) system-on-module (SOM) 8, light sensors 27, proximity sensors 24, including 3-axis accelerometer, 3-axis gyroscope Meter, Motion Tracking Unit (MTU) for 3-axis magnetometer and temperature sensor, and printed circuit assembly for each part. In a certain embodiment, the headset 1 also includes a connection or port for the IOD cable 14, a connection or port for the power cord 15, and/or a nose bridge clip 16 for adapting and/or conforming the headset 1 to the user's Face.

In one embodiment, using simple head movements, the user points the camera 10 at the scene of interest. Camera 10 captures light reflected by objects in the scene, creating its equivalent digital image. The controller receives the digital image. In one embodiment, a portion of the camera image (eg, the portion of the image selected by the user, or identified by an algorithm residing on the remote platform) is rendered by the controller as a 20x20 pixel equivalent tongue representation image. In one embodiment, for wider fields of view, appropriately sized regions of the camera image are spatially averaged to establish corresponding pixel values in the tongue image. For very narrow fields of view, camera pixels repeat. The rendered tongue representation image was then raster scanned onto a 20x20 matrix of electrodes on IOD2. The spatial relationship that exists in the camera image is maintained by using a one-to-one mapping between the positions of the electrodes on the array and the area in the user-selected field of view. The intensity of stimulation at each electrode is proportional to the brightness of the corresponding area of the image captured by the camera 10 . The present invention is not limited by the frame rate of the system. In a preferred embodiment, the frame rate of the system is fast enough that the user perceives the visual signal as a continuous stream.

User controls 9 may be located on the headset 1 (see Figures 1 and 8). In one embodiment, the placement and/or shape of the controls 9 provides the tactile information needed to differentiate between the controls 9 because the user has limited or no natural vision available. A number of different types of controls 9 can be integrated into the VAD 100 of the present invention. For example, one or more controls 9 (eg, control buttons) may provide the following core functions: power on/off 36; system status 33; stimulus intensity (eg, 0 to 100% = 0 to 16 volts); camera image zoom/ Amplitude (e.g., 3° to 48°); stimulus reversed; contrast normal (grayscale) or high (black/white); edge enhancement; volume for adjusting auditory volume (default, low, mute); WiFi enabled/ Deactivate; test, a test pattern presented to the electrode array that allows the user to verify correct stimulation performance. In one embodiment, the camera housing 50 on the headset 1 can be tilted down to 45 degrees to minimize the burden on the user's neck when looking down.

In one embodiment, the feedback provided to the user (eg, the state of the VAD 100 after pressing a control button (see FIG. 8, numerals 30-36), or feedback from an algorithm located on a remote platform) may be non-visual. For example, feedback may be provided via haptic and/or auditory subsystems within the controller. Similar to commercial devices such as cell phones, the VAD 100 of the present invention can provide synthesized speech and tones to notify the user of status/changes and/or visual information.

Additional exemplary drawings of the VAD 100 of the present invention are shown in Figures 2A-2N. FIG. 2A is a three-quarter perspective view of VAD 100. FIG. FIG. 2B is a front view of VAD 100. FIG. Figure 2C is a rear view of VAD100. Figure 2D is a right side view of VAD100. Figure 2E is a left side view of VAD 100. FIG. 2F is a top view of VAD 100 . Figure 2G is a bottom view of VAD 100. 2H-2N show a device with the headset 1 mounted thereon. Figure 2H is a three-quarter perspective view of the device. Figure 2I is a front view of the device. Figure 2J is a rear view of the device. Figure 2K is a right side view of the device. Figure 2L is a left side view of the device. Figure 2M is a top view of the device. Figure 2N is a bottom view of the device.

FIG. 1 shows the headset 1 of the VAD 100 of the present invention. In one embodiment, the headset 1 contains printed circuit assemblies (PCAs) 6, 8, 23 and flex cables 18, 20, 29 for the components of the headset 1, including, but not limited to, membrane switch assemblies 9; Camera PCA; Sensor PCA23; Ambient Light PCA27; SOM Carrier PCA8; SOM2Sensor (SOM to Sensor) Cable PCA (rigid-flex cable 18 with connectors at each end); Antenna PCA; and/or Audio PCA13. The headset 1 may also include temples (eg, left and right), audio flex cables 29, plastic housing components, strap clips, and/or arm inserts.

An exemplary VAD 100 of the present invention may include multiple interconnected subsystems that work in conjunction with software components to provide the core functionality of the device. As shown in Figure 3 and described in the following paragraphs, these subsystems include printed circuit assemblies (PCAs) interconnected by custom cables.

processing subsystem. The processing subsystem is located in the headset 1 housing. In one embodiment, as described and shown in Figure 3, the present invention provides a housing specifically designed to accommodate circuit boards and cables.

TorpedoSOM. TorpedoSOM is System 8 on the LogicPDDM3730Torpedo+ Wireless Module. SOM8 is mounted on the SOM carrier PCA8. The antenna (antenna PCA) is connected to the WiFi module integrated with the SOM8. The SOM8 is powered via mains power on the battery PCA. SOM8 monitors and/or controls other subsystems in VAD100 via the SOM8 carrier PCA. TorpedoSOM8 is a computer running the Linux operating system and customized VAD100 application software. For example, SOM8 functions may include: configure device settings at startup; modify device settings during operation; manage camera image acquisition and processing; monitor and respond to user controls 9, including but not limited to power on/off, status requests and mode control, Battery status, volume, WiFi, test image related status/mode controls, intensity, zoom, contrast, inversion and/or edge enhancement; generate audio output (eg, provide feedback to the user); create and send stimulus patterns into the mouth device (IOD2); monitor ambient light sensor 11 (eg, to track lighting conditions); monitor proximity sensor 24 to determine when headset 1 is being worn; monitor inertial measurement unit sensors (accelerometer, gyroscope, compass) to track head movement and orientation of the wearable device 1; monitoring battery status (eg, via a battery fuel gauge); setting and monitoring a real-time clock/calendar component; allowing remote connections (eg, via a wireless connection (eg, via WiFi (802.11a/b/g) /n) interface and antenna)); and/or streaming images, commands, and/or status data to/or from remote platforms.

SOM vector (not shown in the figure). The SOM carrier PCA provides the electromechanical interface between the SOM 8 and other headset hardware components. The TorpedoSOM8 plugs into the receiving connector on the SOM carrier PCA. In addition to the SOM8 connector, the SOM8 carrier PCA may include a push-button power on/off controller (eg, in the off state, a momentary press of the power button (eg, see user control 9 button in Figure 1) will enable the battery PCA mains power on and the device will go to the on state; while in the on state, holding down (eg, 1 to 3 seconds) will deactivate the mains power and the device will go to the off state); voltage converters (e.g. for devices that convert logic voltage levels between devices operating at different voltages); real-time clock/calendar (eg, once set by SOM8 (eg, the device can communicate with SOM8 via an I2C bus), retains the current day time and date of the device); IOD2VSTIM power supply (e.g., for supplying power (e.g., 17V to 100mA) to stimulating electrodes (e.g., power can be turned on/off by TorpedoSOM8)); SOM8 power supply (e.g., for applying clean 3.3 V power supply to TorpedoSOM8 (e.g., the supply can be enabled at mains power-up); LED circuitry (e.g., TorpedoSOM8 lights up a green LED when the software boot process starts and an amber LED when the IODVSTIM power supply is enabled). In one embodiment, the SOM carrier is connected to the battery PCA via a 6-conductor power cable 7 with an inner jacket. The sheath is attached to the SOM carrier. In one embodiment, the SOM carrier is connected to the sensor PCA by a custom flex cable 18 (eg, a 30-wire flex cable). In one embodiment, the SOM carrier is connected to the IOD addressing PAC via an IOD cable 3 with inner jacket (eg, a 6-conductor cable with jacket). A sheath may be attached at the IOD end.

Antenna PCA. An antenna PCA includes an antenna subsystem, is interconnected, and can be used to broadcast radio frequency (RF) signals. It can be connected directly to the SOM8 via a custom length cable and is designed according to the rules specified by LogicPD to ensure that the SOM's FCC and ICID are used without changes. It can be placed within the VAD100 housing at a location that ensures compliance with the specific absorption rate (SAR) limits of the body-worn device.

electricity. The power subsystem may also be located in the battery housing 4, which is connected to the SOM carrier PCA by a power cable 7 (eg, a 6-conductor cable). The battery housing 4 is designed to accommodate the battery 5 , the battery PAC 6 and the power cable 7 .

Battery. Any type of rechargeable battery 5 can be used with the VAD 100 of the present invention. In one embodiment, the VAD 100 uses a VARTAEASYPAKXL battery or any other type of battery that can provide 220mAh at 3.7V (eg, in accordance with IEC63133).

Battery PCA. Battery PCA 6 may include: a battery connector (eg, providing '+' and '-' connection terminals for battery 5); a battery fuel gauge (eg, monitoring battery state of charge (eg, may be directly connected to the battery for monitoring purposes) , and, via the I2C bus (on power cable 7) to the SOM8 (e.g. to allow the SOM8 to query the fuel gauge for current battery status))); and/or mains power (e.g. to convert battery power to constant 4.1V supply (max 1A)). In the preferred embodiment, the battery PCA 6 is designed to fit within the battery housing 4 .

User Controls - Hardware. In one embodiment, the user controls 9 are located on the top piece at the front of the frame of the headset 1 (see Figures 1 and 8). In one embodiment, the control buttons 30 , 31 , 32 , 33 , 34 , 35 , 36 are implemented as membrane switch assemblies 9 (see FIGS. 4A and 4B ) to provide seven single pole/ Single throw momentary normally open switch. The flexible tap 17 is connected to the sensor PCA 23 . Each control button is implemented as a metal dome switch with an actuation force, eg, an 8mm diameter metal dome switch with an actuation force of 180g.

sensor subsystem. In one embodiment, the sensor subsystem (eg, integrating multiple VAD100 sensors) is located in the front of the frame of the headset 1 (see Figure 5).

Sensor PCA. Sensor subsystem PCA 23 (see FIG. 5 ) serves as an integration center for the sensors in headset 1 , user controls 9 , and audio subsystem 12 . The sensor PCA23 is custom designed to fit within the VAD100 headset 1 front housing and provides: power and signal connections between the sensor assembly and the SOM carrier 8 and battery PCA6; Jumper circuit; separate power supply (eg, 3.3V power supply) for ambient light detector 27/proximity detector 24 and proximity sensor LED; and/or for membrane switch flex tap 17, audio flex cable 29, camera Connection point for flex tap 20, and/or SOM2 sensor flex cable 18. Sensor subsystem PCA 23 may include camera connector 19, SOM carrier connector 22, audio connector 26, and/or membrane switch connector 28.

Camera PCA (not shown). The camera PCA is mounted within the camera housing 50 (see Figure 5). The PCA is a rigid-flex design and includes a digital image sensor and lens 21 . The PCA's flex circuit extension 20 allows the camera PCA (within its housing) to tilt up or down up to 45 degrees.

Image Sensor. In one embodiment, the camera image sensor of the VAD100 of the present invention has the following features:

Optical Fmt 1/3inch Image size, Horz(mm) 4.51 Image size, Vert(mm) 2.88 Image size, Diag(mm) 5.35 Effective Pixels 752×480 Pixel size (μm) 6 fuzzy circle 0.012 shutter global Responsiveness (V/lux-sec) 4.8 Dynamic range (dB) 80 SNR(dB) 45

Table 1. Exemplary features of the camera image sensor of the VAD100 of the present invention.

Any sensor that handles these characteristics can be used, including but not limited to the APTINAMT9V024 digital image sensor.

lens. In one embodiment, the lens 21 used in conjunction with the image sensor has the following characteristics: Effective Focal Length (EFL): 3.3 (eg, EFL providing at least a 45 degree field of view of the camera); lens height; 4.5mm +/- 10% (eg , which can fit on the VAD100 camera housing/lens holder); Image Circle: >4.0mm; and/or IR Filter: 645nm.

The electrodes of the IOD2 array of the VAD 100 of the present invention are arranged in a grid (eg, a 20x20 space square grid). To match the aspect ratio of the IOD2 array, the VAD100 crops the image sensor data so that the spatial arrangement of pixels used for image processing is also a square, with the centers of the pixel groups centered on the image sensor. The 'image circle' of the lens 21 must cover at least the selected set of pixels. The present invention is not limited to any particular lens 21 . In one embodiment, a 14033MPF lens with the following specifications is used: Size: 1/4", EFL3.3mm, F2.8, M7*0.35 Mounting lens with IR filter.

Ambient Light Sensor PCA. Ambient Light Sensor PCA27 includes a light-to-digital conversion ambient light light sensor that converts light intensity into a digital signal output capable of direct I2C interface. This digital output is monitored by the TorpedoSOM8, where the brightness in lux (ambient light level) is derived using an experimental formula to approximate the human eye response. The ambient light sensor is connected to the TorpedoSOM8 via the I2C communication bus. The present invention is not limited to any particular light sensor 25 . In one embodiment, an APDS-9301 miniature ambient light light sensor with digital (I2C) output is used.

Proximity sensor. The proximity detector 24 of the sensor PCA is located on the rear side of the sensor PCB, collinear with the opening in the headset 1 and the corresponding protective lens, allowing it to detect when the user is wearing the headset 1 . By monitoring this signal, once the headset 1 is removed, the TorpedoSOM 8 can enter a low power or power down mode, thereby significantly extending battery life (eg, no user action to power down or power up the device is required). The proximity sensor 24 can detect objects up to 100mm away. Proximity sensors do this by enabling the IRLED transmitter and then measuring the energy reflected off the nearest object and received by the IR detector. Proximity sensor 24 is connected to TorpedoSOM8 via an I2C communication bus. The present invention is not limited to any particular proximity sensor 24 . In one embodiment, Avago APDS-9130 is used.

Motion Tracking Unit. Sensor PCA 23 includes a motion tracking unit (MTU), which may include a 3-axis accelerometer, a 3-axis gyroscope, a 3-axis magnetometer, and a temperature sensor. By monitoring the data output of the MTU, the TorpedoSOM 8 can determine the orientation of the headset 1, whether the headset 1 is moving, and the direction of the movement. The MTU is connected to the TorpedoSOM8 via the I2C communication bus. The present invention is not limited to any particular motion tracking unit or its components. In one embodiment, an InvenSense MPU-9250 multi-chip module is used.

Tongue irritation. The IOD2 is placed on the user's tongue, and stimulation occurs through electrodes 37 on the underside of the IOD2. The electrical current between the electrodes and the tongue is used to stimulate nerves on the tongue. Users describe the irritation as a slight tingling, buzzing, or blistering feeling. In one embodiment, no more than four electrodes are active simultaneously. In another embodiment, the active electrodes are separated by at least 4 inactive electrodes. In another embodiment, all 396 inactive electrodes are used as a common return for 4 active electrodes.

The intraoral device (IOD) 2 assembly includes an IOD electrode array PCA, an IOD addressing PCA, and an IOD cable 3 for connecting the IOD assembly 2 to the headset 1 . In one embodiment, the IOD electrode array 37 is a custom PCA that contains one switch circuit per electrode (eg, 394 switch circuits). The electrodes are arranged in a grid (eg, a square grid of 20 rows by 20 columns) that is evenly spaced (eg, at 1.32 mm (0.52 in) center to center). Electrode array 37 is connected to the IOD addressing PCA via high density connectors. Electrode row and column activation signals are received from the addressing board via high density connectors. These activation signals enable and disable switching circuits on the array. When the switch is enabled, the electrode array 37 gates the analog voltage from the addressing plate to the active electrode.

The IOD2 addresses PCA as a custom printed circuit assembly. Addressed PCA receives stimulus patterns from TorpedoSOM8. It uses this data to drive electrode row and column activation signals. The row/column activation signals are implemented such that the electrode array is activated in a raster scan manner. The magnitude of the voltage signal is proportional to the brightness of the pixels in the IOD2 image (eg, a 20x20 representation of a camera image). IOD2 image pixels correspond to electrodes activated by row and column signals. During the manufacturing process of the electrode array 37, the addressing plates and cables 3 are joined and then encapsulated in a biocompatible epoxy resin. Epoxies protect electronic devices and provide mechanical rigidity to the complete assembly. After encapsulation, the epoxy is polished to fully expose the electrodes and remove any rough edges. A silicon sleeve placed over the flex cable is butted against the edge of the epoxy and glued in place with silicone to complete the subassembly.

Audio PCA. An audio PCA includes speakers, audio controllers, power supplies, and amplifiers. Additionally, the supercapacitors used to provide long-term power to the real-time clock can be located on the PCA. In one embodiment, the present invention provides audio feedback to the user using a speaker driven by a TorpedoSOM8 or MSP340 audio controller with the following characteristics: Frequency range: 300Hz-17kHz; Impedance: 8Ohm; Sound pressure level: 73.5dB; Rated power : 600mW; maximum power: 1.2W.

In one embodiment, the MSP340 audio controller executes embedded firmware and has a single digital input from TorpedoSOM8. When the digital input is turned off, the MSP340 audio controller uses the audio sequence to drive the speakers. When the digital input is on, the MSP340 audio controller releases control of the speaker to the TorpedoSOM8, which can then drive the speaker 12 with its own audio sequence. Additionally, the audio PCA includes a 2-channel audio mixer so that when a headphone plug is inserted into the headphone jack, all audio output is routed to the headphone instead of the speaker 12 .

V200 battery case. In one embodiment, the battery case 4 is a power source for the VAD100 of the present invention, and includes the following: V200-3V7P-PowerPCA (PowerPCA); VARTEAasyPackXL 3.7V, 2260maH lithium-ion battery pack. In one embodiment, as shown in FIG. 6 , the VAD100 battery housing 4 is connected to the VAD100 headset 1 using a power cable 7 (as shown in FIG. 6 ).

An exemplary VAD 100 of the present invention is shown in FIG. 7 . Several visible components are shown in Figure 7, including:

1. The camera 10 is used to capture the scene in front of the wearer

2. Speaker 12 provides audio feedback

3. The battery case 4 contains a rechargeable battery. Mounted to the rear of the headset 1 using an adjustable strap.

4. The proximity sensor 24 detects when the headset 1 is being worn. If headset 1 is removed, the system will shut down after a few minutes

5. The IOD2 contains electrodes that deliver the stimulation pattern to your tongue.

An exemplary user control 9 that may be integrated into the VAD 100 of the present invention is shown in FIG. 8 . As shown in Figure 8, the control buttons can be configured to control

Power (36) (eg, device on/off button (eg, to turn the device on or off, press the button));

System (33) (eg, button scrolls through system features (eg, up (34) and down (35) buttons next to the system, selects a specific action for that feature)). In one embodiment, system features can be configured as follows:

Status: Up/Down will cycle through the following status reports, posting information at each stop,

- battery charge level,

- the lighting conditions detected by the device,

- the version of the device;

Volume: Up/Down will cycle through the following volume levels, changing the volume to the currently selected characteristic

WiFi: Up or Down button to enable or disable WiFi (for example, disabling WiFi will help preserve battery life); and/or

Test: Up and Down buttons select a test mode (eg, for troubleshooting device operation).

Imaging (32) (eg, imaging buttons 32 scroll through image features (eg, use up (31) and down (30) buttons to select the desired level for each feature)). Exemplary image features include, but are not limited to:

Intensity: stimulus intensity control (eg, use up 31 and down 30 buttons to (respectively) increase or decrease the intensity of the stimulus on the tongue (eg device will be at stimulus limits (eg max=100, min=0) beeps). Stimulus intensity is always reset to zero when power is increased and must be increased to a comfortable level).

Zoom: Camera Field of View (FOV) control (for example, use the up and down buttons to zoom out (smaller FOV) or zoom in (larger FOV)). Press the upper button to increase the camera zoom level (decrease the camera's effective field of view). Press the down button to decrease the camera zoom level (increase the camera's field of view). The device will beep at the zoom limit (eg, widest = 48 degrees, narrowest = 3 degrees).

Invert: (eg, invert stimulus intensity values, where strongest becomes weakest, and vice versa (eg, use up and down buttons to toggle between bright or dark objects in the field of view to stimulate the tongue array)).

Contrast: Image contrast controls (for example, the up and down buttons toggle between normal contrast (default) and high contrast mode). High contrast will accentuate the difference between light and dark areas in the camera image.

Edge Enhancement: Enable/disable edge enhancement (eg, use the up and down buttons to enable or disable the feature (eg, in this mode, edges in camera images are enhanced to make them easier to identify)).

Additional components of the VAD 100 of the present invention are also shown in FIG. 8 . For example, as described herein, the camera ( 10 ) of the headset 1 can be adjusted to point directly outward from the headset 1 or tilt downward (to approximately 45 degrees) to reduce neck fatigue.

In one embodiment, the VAD 100 of the present invention includes an accompanying observer. For example, trainers or sighted companions can use a web browser to view VAD100 camera images and basic status information. Using a WiFi-capable mobile device (eg, a laptop, tablet, or smartphone), a sighted companion can establish a WiFi connection with the VAD 100 and display a web page with images and status information (eg, see Figure 9 ).

Remote platform access. As described herein, in a preferred embodiment, the present invention provides a VAD 100 that includes not only a controller located in the headset 1, but also components that enable a connection to a remote platform (eg, a wireless (eg, WIFI) connection) and antenna). Thus, in one embodiment, the remote platform is connected to the VAD 100 of the present invention via WiFi (or other wireless connection scheme). Using a communication protocol, applications on the remote platform can exchange data with the VAD100. Data exchanged may include, but is not limited to, image streams, status information, and/or command/control sequences. Furthermore, the data exchange can be bidirectional. For example, in one embodiment, VAD 100 may transmit visual information (eg, recorded by a camera (eg, an image stream)) to a remote platform (eg, whereby the remote platform processes the image stream (eg, detects, identifies, and/or generates) feedback on the image stream) and communicate information (eg, visual information (eg, processed image stream)) to VAD 100 (eg, to augment or replace information presented to the user (eg, via IOD2 and/or auditory signals) )). In one embodiment, the remote platform has connections to multiple VADs 100 . In another embodiment, the VAD 100 of the present invention has connections to more than one remote platform (eg, two, three, four, five or more remote platforms).

In another embodiment, the present invention provides algorithms and/or software for use in conjunction with the methods and/or apparatuses of the present invention (eg, software remotely located on TorpedoSOM8 and/or in connection with any method or apparatus described herein) on the platform). As described herein, the present invention is not limited to any particular remote platform. Indeed, a variety of remote platforms can be used in the methods and devices of the present invention, including, but not limited to, smartphones (eg, iOS and Android-based tablets), tablet devices (eg, iOS and Android-based tablets) device), desktop PC (eg, any operating system running any operating system that can be connected (eg, wirelessly or hardwired) to the headset 1 components of the VAD 100 of the present invention). Furthermore, any software algorithm can be coded into hardware and/or software to improve performance, reduce cost, etc.

Landmark detection and recognition. For blind users, being able to locate landmarks of interest (eg, signs, crosswalks, buildings, geographic locations, etc.) significantly improves the user's quality of life. As described in detail herein, the VAD 100 of the present invention provides blind users with the ability to detect, identify, highlight, and/or move toward landmarks not available above (eg, while bypassing, passing, and/or passing through the user's obstacles or structures in the environment). These newly acquired capabilities are significant improvements over those provided by other devices available in the art. For example, the VAD 100 of the present invention allows a blind user to accurately locate a toilet or exit (eg, via detecting, recognizing and/or guiding toward toilet and/or exit signs) without the need for a sighted individual (eg, who may not be available) ) assistance.

For example, in one embodiment, using the camera 10 component of the VAD100 device, visual information of the environment (eg, a digital image stream) is captured, relayed to a controller and/or remote platform, inspected, and/or processed (eg, by Software and/or hardware algorithms for landmarks of interest (eg, for exit signs, women's restroom signs, or men's restroom signs). If the landmark is in the field of view of the camera 10, the VAD 100 alerts the user (eg, by prompting means (eg, tactile means, auditory means, etc.)) that the landmark is present. In another embodiment, VAD 100 guides the user to a landmark by highlighting the landmark in visual information provided to the user via IOD2.

In one embodiment, the landmark detection algorithm is based on a sliding window approach, where a small window is translated (eg, "slid") across the image. For each type of target landmark to be detected, the corresponding sliding window has a fixed aspect ratio, and multiple scales are used to capture landmarks of different apparent sizes in the image. For example, for exit signs, these window sizes range from 18x12 to 216x144 pixels, while for toilet signs, the sizes range from 12x32 to 120x320 pixels. In one embodiment, each image patch is converted into a visual descriptor, which is fed into a classifier, which determines whether the image patch is classified as containing a marker of interest. The search is performed over multiple scales to accommodate the range of viewing distances (eg, adjacent scales separated by a factor of 1.5, although this factor may be higher or lower). In one embodiment, this yields approximately -10 candidate image patches for each image classified as "marker" (marker present) or "no marker" (marker not in field of view).

In one embodiment, the overall classifier for each block is based on a cascade of filters in a raised paradigm form, where the filters in each stage remove blocks from subsequent consideration (if they are classified as unmarked); At each successive layer, fewer image patches need to be analyzed. In a further embodiment, finally, a more discriminative (but computationally intensive) classifier is used to make the final flag/no flag decision on the remaining candidate image patches, typically a much smaller number (eg, each image dozens of candidates).

In one embodiment, a region of interest (ROI) encompassing the image region containing the detected landmark may be used to highlight the corresponding region on the display, thereby assisting the user in keeping the landmark in the field of view (and navigating towards the landmark).

In one embodiment, the landmark detection algorithm is local (eg, on the VAD 100) or remotely (eg, on a remote platform (eg, on a smartphone, tablet, PC, or similar device) (eg, using WiFi or other Wireless or wired connection))) execution. In one embodiment, for remote execution, a data exchange protocol is used between the VAD 100 and the remote platform. In one embodiment, the remote platform sends audio/haptic feedback to the user.

In one embodiment, as described herein, landmark detection operations are coupled with shadow removal such that landmarks occluded by shadows can be detected.

Empirical data generated during the development of embodiments of the present invention identified deficiencies in the VAD 100 according to the present invention. Specifically, it was determined that the distance at which the relevant landmarks could be detected was approximately 7m, and in practice, due to pixel density limitations of the camera/imaging system, the user reliably used the device for landmarks detected at 3 to 4m. Specifically, this limitation is due to the pixel density of the imaging system, which for VAD100 units over 7m, the height of the logo in the image is only 2 to 3 pixels (or less). Therefore, it is difficult to detect. Therefore, additional experiments performed during the development of embodiments of the present invention addressed this issue by using cameras with higher pixel densities and/or by implementing pixel-enhancement algorithms that 8m, greater than 9m, greater than 10m, greater than 15m, greater than 20m, greater than 25m, greater than 30m, greater than 35m, greater than 40m, greater than 45m or greater than 50m) to enhance the pixel density. Thus, by using cameras with higher pixel density and/or pixel enhancement algorithms, the range over which landmarks can be detected is increased. In another embodiment, when a landmark is detected, the software (eg, SOM8 or remote) is configured to take specific actions to improve the accuracy of the detection. For example, in one embodiment, upon detection, the camera is commanded to 'zoom out' to the detection position, and/or the camera is commanded to increase the image resolution.

Shadow detection and removal. When using the VAD 100 of the present invention, shadows in the image scene can be confusing to blind users because the user may have to determine whether the lack of stimulation (eg, based on brightness) is due to the presence of holes or other objects that absorb light, or that shadows are cast by objects difficulties caused. Accordingly, the present invention provides methods and systems for detecting and reducing and/or eliminating shadows in image streams (eg, to improve visual information relayed to and/or perceived by a user). For example, in one embodiment, the camera 10 of the VAD 100 is used to inspect the digital image stream (eg, by software and/or hardware algorithms residing in the VAD 100 controller 8 and/or on a remote platform) to detect images in the scene. Shadow-like features. In one embodiment, if the shadow-like feature is in the camera's field of view, a shadow removal algorithm is applied to the suspicious area (eg, to allow the user to experience and/or evaluate their environment/scene without shadows; alternatively, If VAD 100 determines that the shadow-like area is not a shadow, VAD 100 provides information to the user about shadow-like features in the field of view (eg, thereby allowing the user to avoid shadow-like features (eg, objects))).

In one embodiment, the shadow removal algorithm is performed locally (eg, on the VAD 100 by the VAD 100 controller 8). In another embodiment, the shadow removal algorithm is performed remotely (eg, on a remote platform (eg, on a smartphone, tablet, PC, or similar device (eg, using WiFi or other wireless or wired connection))) . In one embodiment, the shadow removal algorithm is executed locally (eg, by the VAD 100 controller 8) and remotely (eg, on a remote platform). For remote execution, a data exchange protocol is used between the VAD100 and the remote platform.

In one embodiment, the VAD 100 of the present invention includes a headset 1 motion tracking unit (MTU) that monitors user data (eg, movement, location, orientation, etc.). In one embodiment, user data is used to correlate images that are temporally continuous (eg, to mimic parallax achieved by using multiple cameras). Thus, in one embodiment, scene features that depend on lighting (eg, the direction of the lighting) are identified and appropriately classified as shadowed or non-shadowed. In an alternative embodiment, the head-mounted device 1 of the VAD 100 includes two or more cameras 10, thereby allowing the difference in parallax to be directly calculated according to the corresponding image scene. In another embodiment, along with or in addition to a software/hardware algorithm, an active transducer is coupled to and synchronized with the VAD100 camera 10 image stream to detect features in shadow areas. In one embodiment, active transducers include, but are not limited to, light (eg, any wavelength) based time-of-flight ranging sensors (eg, single point, imaging arrays, etc.) and/or ultrasonic rangefinders.

Obstacle detection and collision avoidance. In one embodiment, the user of the VAD 100 of the present invention learns to translate stimulation patterns provided to the tongue. This translation task takes time, which can be improved by practice and/or instruction. In one embodiment, the VAD 100 of the present invention provides detection of obstacles, thereby assisting the user in avoiding collisions and/or helping to reduce the user's translation burden. For example, in one embodiment, the digital image stream is inspected (eg, by software and/or hardware algorithms) using the VAD 100's camera to infer whether obstacles are in the user's path. In one embodiment, the user is prompted by one or more means (eg, audio means or haptic means) if the obstacle is in the camera's field of view. In one embodiment, a region of interest (ROI) comprising an image region containing an obstacle is used to highlight the corresponding region on the tongue display, thereby helping the user avoid the obstacle. In another embodiment, the obstacle detection algorithm is implemented locally (eg, on the VAD 100 ) and/or remotely (eg, on a remote platform (eg, on a smartphone, tablet, PC, or similar device) (eg , using WiFi or other wireless or wired connection ))) to perform. For remote execution, a digital exchange protocol is used between the VAD100 and the remote platform.

In further embodiments, alone or in combination with software/hardware algorithms, the headset 1 tracking unit (MTU) is used to monitor user data (eg, to aid in identification and avoid collisions with objects). In another embodiment, in addition to a software/hardware algorithm, an active transducer is coupled to and synchronized with the VAD100 camera image to directly detect objects in the camera's field of view and in the vicinity of its user (eg, by determine the distance of the object from the user). Exemplary active transducers include, but are not limited to, light (any wavelength) based time-of-flight ranging sensors (eg, single point imaging arrays, etc.) and ultrasonic rangefinders.

Pedestrian assistance. In one embodiment, the present invention provides VAD 100 and methods of using the same to assist blind persons in identifying crosswalks and/or crossing streets controlled by traffic and/or pedestrian signals. For example, in one embodiment, using the VAD 100 of the present invention, the user enters a "crosswalk mode" and, once activated, the VAD 100 user directs the camera toward the area where the traffic signal is believed to be located. Using the video image stream captured by the VAD100's camera, a connected mobile app application (eg, located on a remote platform or running locally on the VAD100 controller) locates the signal in the image field and sends feedback to the user to help maintain the signal in the center of the image. In one embodiment, the mobile application analyzes the image (eg, determines whether the signal indicates permission to cross) and indicates the user related to the status of the crosswalk (eg, provides guidance to the user).

Range detection and filtering. In one embodiment, when using the VAD 100 of the present invention, the 3-dimensional world is captured by a 2D image sensor, where the 2D image sensor has image processing that processes 2-dimensional data. In these situations, depth or distance information is difficult to obtain by blind users. Accordingly, the present invention provides methods and apparatus for implementing means for a user to determine a distance to an object. In addition to distance detection and reporting, the user can filter image data based on distance in order to reduce the amount of useless information (eg, to eliminate any objects beyond 20 feet (eg, to allow closer analysis of information within a set distance) For example, identification of obstacles within a set distance))).

Color detection. With grayscale images derived from luminance data, blind users cannot identify colors, even though the raw camera data is in color. Furthermore, the current stimulation waveform uses a fixed pattern (pulse frequency) to present luminance data to the user's tongue. Thus, in one embodiment, the present invention provides VAD 100 that assigns a unique waveform pattern to a specific color, thereby allowing the user to perceive a different sensation for each color (eg, allowing the user to associate a specific, unique sensation with associated with a specific color).

Contrast detection. It has been noted that in grayscale images derived from luminance data, features with the same contrast cannot be distinguished by blind users. Thus, in one embodiment, the present invention provides a VAD 100 by using color images from a camera and applying filters (eg, edge enhancement) to those images, and then overlaying (adding) the filtered data to the luminance data , features with the same contrast can be distinguished by the user.

Gesture-based controls. In one embodiment, the present invention provides a VAD 100 that, by monitoring MTU data, TorpedoSOM 8 or software on a remote platform connected to the device, can determine the motion of the headset 1 . In one embodiment, in a user-selected mode "MTU Gesture Control", the unit will respond to certain body movements by adjusting settings. For example, in pose control mode, leaning forward has the effect of 'shrinking' the camera's field of view, effectively making objects in the scene appear larger. Leaning back has the opposite effect of 'zooming in'. The rate and angle of the tilt can affect the size of the zoom action.

In one embodiment, similar gesture control actions are used for any parameters that can be set by the user. In addition to leaning, postural movements include turning or bending the head (wearing the device), bouncing/jumping, etc. Additionally, in "gesture control" mode, software on the TorpedoSOM8 or remotely connected platform can examine camera image data to detect hand motion, and translate the motion into user input to adjust parameters. For example, in gesture control mode, moving the hand from the bottom to the top of the camera's field of view increases stimulus intensity. The speed of hand motion can affect the rate of change of the parameter. Similar gesture control actions can be used for any parameter that can be set by the user. In one embodiment, gesture control is used separately from MTU gesture control. In another embodiment, hand and MTU gesture controls are used simultaneously.

In another embodiment, the present invention provides the use of gesture control to activate one or more electrodes of an intraoral device. For example, gestures can be used as training tools for users to detect and sense (eg, via electrical tactile stimulation of the tongue) letters, shapes, or other objects recognized by gestures and recognized by the systems of the present invention. In one embodiment, gesture recognition is used to assist the user in learning about letters or objects (eg, icon language (for example, Chinese)). In one embodiment, the system of the present invention activates electrodes on IOD2 to represent on the user's tongue the letter or object being tracked by the user (eg, the user is using his or her finger to track the letter or object and "sees" the letter or object on the tongue). In one embodiment, the system of the present invention guides the user in learning to track letters or objects via the actuation of electrodes on IOD2 (eg, the system is programmed to move the user in the correct direction, shape, or route when tracking letters or objects). Electrodes are activated when/her finger (eg, to assist the user in knowing what a shape or object looks like (eg, the system of the present invention is used as a training tool))).

In another embodiment, the VAD 100 of the present invention includes a remote platform and a touch screen (eg, on a tablet device, smartphone, etc.), and means for presenting an array of IOD electrodes on the touch screen (eg, software executed by the VAD 100 to display on the touch screen) IOD electrode array is shown). In another embodiment, when the user touches an electrode location on the screen, the corresponding electrode on IOD2 is activated (eg, with an intensity based on touch pressure, or with a preset intensity). When the user moves her/his finger around on the touch screen (eg, touches an additional electrode), the corresponding electrode is activated on IOD2. In one embodiment, an electrode that is activated has a duration such that it remains activated for a certain period of time (eg, a selectable and/or programmable amount of time (eg, milliseconds, seconds, or two seconds, seconds, 10 seconds) , 20, 30, 40, 50, 60 or more seconds, or until the user deactivates the signal)). Accordingly, in one embodiment, the present invention provides a VAD 100 that can be used by blind users to learn to draw letters and/or objects/shapes, and/or play games (eg, to provide users with recognition of the appearance of letters, shapes, and/or objects) known games). In another embodiment, and as described above, as an alternative to a touch screen, the VAD 100 of the present invention includes a gesture-based control system that provides the user with motion as the user moves his/her hand through the space in front of the camera The ability to stimulate electrodes on the IOD2 worn by the user (e.g., allow the user to learn to draw letters and/or objects/shapes, and/or play games (e.g., provide the user with awareness of the shape of letters, shapes, and/or objects) game)),

In one embodiment, the software is configured to run independently of other software. In other embodiments, the software is configured to run within or with other software, including but not limited to WINDOWS (eg, WINDOWS 10 (or earlier) or other WINDOWS-based operating systems), JAVA, Mobile operating systems or other types of software. In some embodiments, visual information and/or data is collected, recorded and/or stored locally (eg, by a controller located in the headset 1) or remotely (eg, on a remote platform). In one embodiment, the stored visual information is used by the same user from which the stored visual information originated. In another embodiment, the stored visual information is used by a different user than the user from which the stored visual information originated. In one embodiment, the stored information is communicated to software configured to track and/or manage such information (eg, via the Internet, cloud, or other wireless communication (eg, via Bluetooth, ZIGBEE, infrared, FM, AM, Cellular, WIMAX, WIFI or other types of wireless technology)). In one embodiment, the information and/or data collected, recorded and/or stored by the VAD 100 of the present invention is made available over a network (eg, TCP/IP, SANS, ZIGBEE, wireless, wired, USB, and/or other types of networks ) or via a mobile information recording device (eg, flash card, memory stick, disk, flash drive, etc.). In one embodiment, the network is configured to comply with certain government protocols and/or regulations. In one embodiment, the software configured to interact with the VAD 100 of the present invention includes mobile resources for the user of the VAD 100 in the field of view. For example, in some embodiments, the software is configured to provide various information to the user of the VAD 100 of the present invention, including but not limited to location, surrounding landmarks, landmarks within the user's field of view, GPS coordinates, weather, traffic status, user known obstacles in the field of view, or other types of information.

Experiments were conducted during the development of embodiments of the present invention to test and characterize systems, methods and algorithms for detection marker generation. Specifically, the systems and methods of the present invention are used and implemented to test the ability to detect signs (eg, exit signs and signs of men's and women's restrooms). The tested systems and methods use desktop environments and standard libraries for output to tablet devices (eg, Android tablets), using streaming video/images from remote video feeds (eg, from VAD100 or streaming from the Internet) , or an algorithm implemented by a video feed from a controller (eg, a camera housed within a controller (eg, tablet, smartphone, etc.)).

detection algorithm. Landmark detection algorithms are based on a sliding-window approach (eg, see Wei and Tao, 2010 IEEE Conference, Jun. 13-18, 2010, pp. 3003-3010), where a small window is translated (eg, slid) over the entire image. For each type of target landmark to be detected, the corresponding sliding window has a fixed aspect ratio, and multiple scales are used to capture landmarks of different apparent sizes in the image. For example, by way of non-limiting example, for exit signs, these window sizes range from 18x12 to 216x144 pixels, and for toilet signs, from 12x32 to 120x320 pixels. Each image patch is converted into a visual descriptor (see, for example, Freund and Schapire, Journal of Computer and System Sciences, 1997 55(1), pp. 119-139), which is fed into a classifier which determines that the image patch is classified as Include the logo of interest or not. The search is performed at multiple scales to accommodate a range of viewing distances (eg, where adjacent scales are separated by a factor of 1.5). For each image, this yields about -10 candidate image patches classified as flags or no flags.

The overall classifier for each block is based on a cascade of filters in the form of an elevated paradigm (see, for example, Hastie et al., The Elements of Statistical Learning, 2nd ed. 2009, Springer; Schapire and Singer, MachineLearning, 1999, pp. 80-91) , where filters in each stage remove blocks from subsequent consideration (if they are classified as flagless); at each successive layer, fewer image blocks need to be analyzed. Finally, a more discriminative (eg, more computationally intensive) classifier is used to make the final flag/no flag decision on the remaining candidate image patches, usually in much smaller numbers (eg, dozens of candidates per image).

Different types of flags. The VAD 100, methods and algorithms use detected exit signs and men's and women's restroom signs when the user selects what type of signs the user wants to detect. However, and as detailed herein, the present invention is not limited to exit signs and men's and women's restroom signs. Indeed, the systems, methods, and algorithms described herein can be used to detect any type of desired landmark. In addition, it is also possible to use systems, methods and algorithms to simultaneously detect multiple types of markers (eg, 1, 2, 3, 4, 5, 10, 15, 20, 25, 50, 100 or more different types) (eg, if the user desires (eg, the user may want to be notified whenever an exit sign or a gender-specific toilet sign is detected))). Additional computation (eg, additional computational processing bandwidth and power consumption) is required when the system is configured to detect multiple types of markers. Thus, in one embodiment, using a separate mode for each flag reduces computational load, allowing real-time performance and improved responsiveness, and also potentially extending battery life of the VAD 100 (eg, of a tablet device). However, in another embodiment, when the system of the present invention is configured to detect multiple types of markers simultaneously, additional computation (eg, additional computational processing bandwidth and power consumption) is performed on a remote processor (eg, may Access via a connection to a server/processor accessible on the Internet (eg, a wireless connection). Having additional computations performed on a remote server reduces the computational load on the VAD 100 itself, allowing real-time performance and improved responsiveness, and extending the battery life of the VAD 100 (eg, tablet device).

first-level classifier. The first-level cascade uses a GentleAdaboost (see, for example, Nonlinearestimation and classification by Schapire and Robert, Springer New York, 2003, pp. 149-171) classifiers that use the Local Binary Pattern (LBP) descriptor (see, for example, Ojala et al., p. 12 Proceedings of the IAPR International Conference on Pattern Recognition (ICPR), 1994, Vol. 1, pp. 582-585; Wang et al. International Conference on Computer Vision (ICCV), 2009) to describe images. The implementation of cascaded classifiers uses an OpenCV implementation that uses a set of simple decision tree classifiers as weak classifiers and combines them to learn a single strong classifier that is trained to minimize the number of actual landmarks that are lost , which sacrifices precision (eg, possibly including some non-marker blocks) to achieve this high recall. This ensures that the flag of interest is not eliminated at this stage, but passed on to the next stage responsible for finding the flag (eg, if it exists) in the remaining detections.

Typically, a single object in an image will cause multiple detections at similar locations and with similar dimensions, since the Adaboost classifier is more robust to small translations and dimensional changes of objects in sliding windows. Since these multiple detections are redundant, a clustering step is performed at the end of the first stage, which identifies clusters of rectangles with similar positions and dimensions, and for each cluster only a single detection candidate is selected (e.g., rectangle). This reduces the number of detection candidates that must be processed in the second-level classifier, which is more selective and computationally intensive.

Second-level classifier. At the output of the first-stage cascaded classifiers, the number of candidates is reduced to about tens per image. The second layer in the cascade uses a Histogram of Orientation of Gradients (HoG) (see, for example, Dalal and Triggs, IEEE Computer Society Conference, Vol. 1, 2005, pp. 886-893) as a visual descriptor, which complements the LBP description used by the first layer symbol. Note that ^HoG is too computationally intensive to apply to all ~105 original image patches (eg, the original image patches are analyzed by the cascaded first layer), but the first layer filters out the vast majority of these patches. This descriptor is used as input to a support vector machine (SVM) with an RBF kernel (see, for example, Intelligent Data Analysis by Cristianini and Shawe-Taylor, M. Berthold and DJ Hand, Eds. Springer Berlin Heidelberg, 2007, pp. 169-197).

The SVM layer classifies all remaining blocks as flagged or unflagged. Each classification is also assigned a confidence value between 0 and 1 corresponding to the likelihood that the block is an identity, where 1 is very likely and 0 is very unlikely. Of the blocks classified as containing the flag of interest, only those whose likelihood exceeds the set threshold are returned. If no block is classified as a flag with a confidence value above this threshold, no detection is reported. For example, the basic restroom sign detector responds equally to men's and women's restroom signs, but additional processing stages are used to differentiate between men's and women's restrooms; the second and last SVM layer is applied after the toilet sign is detected, in order to determine whether it is a men's restroom sign or a women's restroom sign.

track. No detection algorithm is perfectly reliable, which means that valid target markers may not be detected in some frames, while false detections may occur in others. Furthermore, detection performance is often compromised by camera motion blur, which can occur anytime the camera is moving, and is especially problematic in low light conditions (eg, indoor environments). These issues raise questions for the development of an effective landmark recognition system in conjunction with the VAD systems and methods described herein for use by blind and visually impaired persons (eg, requiring coherent information about the presence and location of each object of interest). challenge.

To address and overcome these problems, in one embodiment, a temporal integration stage (eg, eg, motion tracking) is applied after the classifier stage. For example, a means to combine static shape cues (eg, obtained using a classifier in separate video frames) with motion cues (eg, obtained by integrating information over multiple video frames) is tested. Finally, motion tracking is used to combine stationary shape cues with motion cues, however, any other means known in the art for combining stationary shape cues with motion cues may also be used in the present invention. Therefore, in one embodiment, a motion tracking algorithm is implemented. The motion of each candidate is tracked and verified via optical flow through consecutive frames, and the valid sign is only on the next fifteen consecutive video frames (eg, corresponding to slightly less than thirty frames per second of video) Validation delay of half a second) is issued after successive detections (eg, from a classifier) in three of them. The selection of this parameter is done heuristically; less stringent criteria (eg, requiring two out of every fifteen frames) will reduce latency (which may be preferable at low frame rates), and more stringent criteria A standard (eg, requiring three out of every ten frames) will reduce false positives at the expense of greater latency.

Objects are tracked in subsequent frames, where the stationary shape-based criteria for selecting object candidates based on the classifier are loosened (e.g., allowing the possibility of tracking objects that are temporarily more indistinguishable due to motion blur); e.g., the system Configured such that only another flag of successful verification needs to occur every 10 frames (eg, but parameters can be adjusted for any resolution environment). If the flag is not validated for 10 consecutive frames of the track, the target is removed from the tracker.

Thus, in some embodiments, the tracking algorithm has the effect of eliminating false positives (eg, false detections) and false negatives (eg, missed detections) that occur with the classifier. In another embodiment, simultaneous tracking of multiple targets is allowed. Additionally, in one embodiment, by locking onto the target, the user is prompted only once for each sign upon detection (eg, thus reducing potential confusion for blind users (eg, where it may be unclear to the user that detection corresponds to condition of the same object)).

The systems, methods, and algorithms described herein were tested in exit sign detection experiments and successfully demonstrated how the tracking algorithm eliminates noise detection. Detection experiments and procedures were attempted in both cases with tracking turned on and tracking not turned on. When tracking is off, there are many false negatives (eg, missed detections), even when exit signs are clearly resolved by a video capture device (eg, a VAD100's camera). In strong contrast, when tracking is on, exit signs are detected continuously after a short delay while the tracker needs to lock onto the target. Accordingly, in one embodiment, the present invention provides a VAD 100 including hardware and algorithms that allow a blind or visually impaired person to track a target (eg, continuously while the target remains in view (eg, of the VAD100 camera), thereby significantly improving the accuracy of the provided location estimate).

Figures 10 and 11 show sample detections in the acquired images, as well as some missed and erroneous detections. The missed detections (rectangles) shown in Figure 10B are examples of landmarks that were correctly picked up by the first classifier stage (Adaboost) but incorrectly picked up by the second classifier stage (SVM). Although only local shape-based evidence may be available for a landmark in a particular image, motion continuation cues (eg, for a tracking algorithm) are used to enhance the evidence for the landmark and yield an overall successful detection.

Systems and methods including algorithm performance are objectively measured using ROC curves showing how accuracy and recall can be traded off against each other. The term "precision" as used herein refers to the fraction of correct detections, while "recall" refers to the fraction of detected markers. The performance results with the tracker compared to when the tracker is turned off are shown in Figures 12 and 13. Recall and precision calculations measure the performance of the entire detector (toilet or exit) using a different video feed than the images used to train the detector.

Accordingly, in one exemplary embodiment, the present invention provides a VAD100 system and method for marker detection. In one embodiment, a user uses the VAD 100 system and method in conjunction with an application (eg, a windows app, a MAC app, or an app for other operating systems described herein) to detect landmarks (eg, signs). In one embodiment, upon installation, the app allows the user to turn on or off the tracking functionality of the VAD100 system. The user can then select a video source (eg, a remote video feed (eg, from the VAD 100 or streamed from the Internet) or a video feed from the controller (eg, a camera housed within the controller (eg, tablet, smartphone) Wait))). The user can then select an object acquisition mode (eg, select a particular type of object to search for (eg, exit or toilet signs), or select search and acquire multiple objects). In one embodiment, each detection is highlighted (eg, shown as a rectangle (highlighted in a particular color)) and superimposed (eg, obtained at VGA resolution) on the original video image. The present invention is not limited to such detection notifications. Indeed, additional means of informing the user that a desired object (eg, a landmark) has been acquired, including those disclosed herein, may be used.

All publications and patents mentioned in the above specification are incorporated herein by reference. Various modifications and variations of the described methods and systems of the present invention will become apparent to those skilled in the art without departing from the scope and spirit of the present invention. While the present invention has been described in connection with certain preferred embodiments, it should be understood that the claimed invention should not be unduly limited to these specific embodiments. Indeed, various modifications of the described modes for carrying out the invention that are obvious to those skilled in the relevant arts are intended to be within the scope of the following claims.

Claims (20)

1. A landmark detection system, comprising: a) an apparatus for obtaining visual information from the environment, the apparatus comprising a digital camera located in a headset; b) means for detecting and/or identifying landmarks in said visual information, said means comprising a processor located in a controller residing in said headset and/or a processor located on a remote computer to for analyzing the visual information; and c) means for providing feedback relating to the detection and/or identification of landmarks within said visual information, said means comprising haptic and/or auditory means; The landmarks are signs used for navigating in the environment. 2. The system of claim 1, wherein the controller receives the visual information captured by the digital camera. 3. The system of claim 1 or 2, wherein the visual information is a stream of digital images. 4. The system of claim 1 or 2, wherein the controller communicates with the remote computer via a wireless network. 5. The system of claim 1 or 2, wherein the controller communicates with the remote computer via a wired network. 6. The system of claim 1 or 2, wherein the landmark is a sign. 7. The system of claim 6, wherein the sign is selected from an exit sign and a toilet sign. 8. The system of claim 1 or 2, wherein the landmark is a pedestrian crossing. 9. The system of claim 1 or 2, wherein a processor on the remote computer executes software; the software analyzes the visual information in order to detect and/or identify landmarks. 10. The system of claim 1 or 2, wherein a processor in the headset executes software that analyzes the visual information in order to detect and/or identify landmarks. 11. The system of claim 1 or 2, wherein the haptic and/or auditory device signals the detection and/or identification of landmarks within the visual information. 12. The system of claim 1 or 2, wherein the haptic device comprises electrical haptic stimulation via an intraoral device. 13. The system of claim 1 or 2, wherein the hearing device comprises an audio speaker and/or a headphone plug. 14. The system of claim 1 or 2, wherein the system further provides for providing and interfacing with obstacles and/or obstacles in the visual information existing between the headset and the landmark A device for building-related feedback. 15. The system of claim 1 or 2, wherein the landmark is highlighted using electrical haptic stimulation. 16. The system of claim 1 or 2, wherein shadowing is performed by a processor running on the processor located in a controller residing in the headset and/or on a remote computer Algorithms remove from the visual information. 17. The system of claim 1 or 2, wherein the camera comprises a complementary metal oxide semiconductor CMOS digital image sensor. 18. The system of claim 1 or 2, further comprising a motion tracking unit MTU in communication with the controller, wherein the MTU comprises a 3-axis accelerometer, a 3-axis gyroscope, a 3-axis magnetometer, and/or Temperature Sensor. 19. The system of claim 1 or 2, further comprising one or more components in communication with the controller, the one or more components selected from the group consisting of proximity sensors, intraoral devices, time-of-flight-based Distance sensor, ultrasonic rangefinder and/or ambient light sensor. 20. The system of claim 19, wherein the intraoral device comprises stimulation electrodes configured to provide stimulation patterns related to the visual information acquired by the digital camera.

Images