DE102021105869A1 - CORRECTION AND CALIBRATION OF A SPECTRAL SENSOR OUTPUT - Google Patents

Abstract

A spectral sensor system includes an array of optical sensors disposed on an integrated circuit, an interface between the plurality of optical sensors and a first processing device, a plurality of sets of optical filters configured as a layer overlying a plurality of optical sensors are arranged, each set of optical filters including a plurality of optical filters, each optical filter configured to pass light in a different wavelength range. The spectral sensor system includes a memory configured to communicate with the first processing device, the memory configured to store calibration data associated with the plurality of sets of optical sensors. The spectral sensor system further includes a second processing device including an artificial neural network configured to correct a spectral response generated by the plurality of optical sensors, and an interface between the first processing device and the second processing device is configured to pass information therebetween transfer.

Description

CROSS REFERENCE TO RELATED PATENTS

This US patent application also claims priority under 35 U.S.C. Section 119 (e) of U.S. Provisional Application # 62 / 988,759 entitled "CORRECTION AND CALIBRATION OF SPECTRAL SENSORS," filed 03-12-2020 and hereby incorporated by reference in its entirety and incorporated herein. Patent application for all purposes is.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Inapplicable.

Record material on a compact disc

Inapplicable.

BACKGROUND OF THE INVENTION

TECHNICAL FIELD OF THE INVENTION

This invention relates generally to spectroscopy and, more particularly, to spectral correction for spectral sensors using interference based filters.

Spectroscopy devices have proven useful for applications in a variety of industries, including health, biometrics, agriculture, chemistry, and fitness, for example. In general, spectroscopy devices function by detecting and / or detecting incident light with respect to multiple ranges of wavelengths and extracting spectral information. It has been shown that interference-based filters such as Fabry-Perot filters in conjunction with spectral sensors can deliver controlled wavelengths of light.

As is further known, light traveling through interference-based filters is subject to various non-ideal conditions and non-ideal sensor performance, each of which can adversely affect the performance of a particular spectroscopic device.

DESCRIPTION OF RELATED ART

Figure list

• 1A Figure 10 provides a top-down view of an exemplary optical sensor overlaid with filters in accordance with the present invention;
• 1B Figure 12 is a side view of an exemplary optical sensor overlaid with filters in accordance with the present invention;
• 2 Figure 10 illustrates the exemplary filter responses of a CMOS sensor overlaid with filters;
• 3 provides an illustration of a comparison of raw output data and the ideal output data of an exemplary spectral sensor;
• 4th Figure 3 is a schematic block diagram of one embodiment of a spectral sensor in accordance with the present invention;
• 5A provides a schematic block diagram of one embodiment of a spectral sensor incorporating an artificial neural network in accordance with the present invention;
• 5B provides a schematic block diagram of another embodiment of a spectral sensor incorporating an artificial neural network in accordance with the present invention;
• 6A provides a simplified representation of an artificial neural network incorporated into a spectral sensor in accordance with the present invention;
• 6B provides a simplified representation of a multilayer artificial neural network in accordance with the present invention;
• 7A Figure 4 is a logic diagram illustrating an exemplary method for correcting raw spectral output into corrected spectra in accordance with the present invention;
• 7B Figure 4 is a logic diagram illustrating an exemplary method for adaptively correcting the spectral output from a sensor array in accordance with the present invention;
• 7C provides a logic diagram illustrating another exemplary method for adaptively correcting the spectral output from a sensor array in accordance with the present invention;
• 8A FIG. 10 is an illustration of an exemplary spectral sensor in which a sweep of monochromatic light and input power is provided in accordance with the present invention;
• 8B provides an exemplary spectral sensor incorporating an artificial neural network in accordance with the present invention;
• 8C FIG. 10 provides a representation of an example of a physical scene provided to a spectral sensor and according to FIG present invention is output to a trained artificial neural network;
• 8D provides an exemplary spectral sensor in which metadata of a physical scene is made available to the artificial neural network according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In various embodiments, spectral sensors are combined with interference filters in order to provide spectral information about a scene and / or light source. Interference-based filters such as Fabry-Perot filters can have a non-ideal filter response to an underlying sensor. For example, the filter response may include crosstalk between filters, unwanted second order responses to incident light, angle dependencies on incident light, and incident light may contain light from sources that are not intended to be evaluated. Sensors themselves can also have less than ideal performance, including non-linearities, crosstalk, electronic noise, etc. 1A Figure 10 shows an exemplary integrated optical sensor from top to bottom 100 that with filters 110 , 120 and 130 is superimposed, which are optimized for one of three spectral bands. As shown, the filters repeat 110 , 120 and 130 as an array over the surface of the optical sensor 100 . Alternatively, the filters could 110 , 120 and 130 repeat using a different pattern or even in a random pattern to route filter responses to sensors underlying the filter array. In one example (not shown) spectral bands greater than 3 could be used to overlay sensors in any practical orientation as desired. In one embodiment the optical sensor is 100 an example of a spectral sensor useful for diffuse optical spectroscopy, with arrays of spectral filters associated with optical sensors to provide diffuse spectral detection. 1B Figure 12 shows a side view of an exemplary optical sensor overlaid with a filter array. In the example, incident light is 180 through the filter assembly 160 on the sensor array 170 directed.

Ideally, the filters of the integrated optical sensor in FIGS. 1A and 1B would only pass incident light within the desired spectral wavelengths to the sensor below, so that an ideal sensor could then output filter responses that exactly represent the spectrum of the incident light. In practice, neither the interference filters nor the sensors are ideal when therefore the output of a given optical sensor, such as that in FIG 1A and 1B is measured, the resulting spectrum is unlikely to correctly represent the spectrum of incident light. Indeed, the resulting spectrum can have a non-linear relationship to the intensity of the input light, to the spectrum of an object to be measured and / or to the illumination spectrum of the input light. In addition, manufacturing related errors such as manufacturing tolerances and uniformity between integrated optical sensors, such as the integrated optical sensor 100 , contribute to sub-ideal performance.

2 shows the filter responses of a CMOS sensor (in this case using Fabry-Perot filters) exhibiting crosstalk from neighboring sensors. In an ideal sensor, the filter response of a particular sensor pixel, such as that of pixel # 13, would only come from pixel # 13 without contributing to other pixels within the array. As shown, neighboring pixels # 8, # 12, # 14, and # 18 exhibit at least one undesirable response from the wavelength centered on pixel # 13 along with other non-idealities. Reconstructing a "clean" output spectrum (one that adequately represents the spectrum of light incident on the optical sensor) is not trivial, even if the filter response shapes shown are known. In the representation of 2 the responses do not reflect the perfect "Fabry-Perot" responses, and the generation of crosstalk from the neighboring filters is problematic and contributes to the response failure. Given that each pixel may have non-ideal responses and crosstalk errors, it can be readily understood that the correction and / or calibration of a given optical sensor can become extremely difficult and / or inaccurate. In addition, the material properties of the filters themselves, quality differences and part-to-part differences add to the complexity.

3 represents a representation of the output data 190 a reflection measurement (filter response) from approximately 740 to 1020 nm of an exemplary optical sensor together with an ideal or "corrected" spectrum 192 ready for measurement. In the example, the raw output data cannot be adjusted with a complex correction exercise to convert it into a “real” spectrum (ie a spectrum that is representative of the spectrum of the incident light).

In one example, the raw power of a spectral sensor ORAW can be measured as a matrix of [1 × N] values. It can be corrected by a matrix multiplication so that:
O = O _RAWX C
where O is the corrected spectrum and C [NxM] is a correction matrix, where M is the desired output wavelength response (s) (in a linear correction step). C is a "correction matrix" made up of measured and / or derived knowledge about the sensor and the filter or the factory measurement / calibration of the filter properties of the sensor or a combination of both.

In one example, O _{RAW is constructed} from N (e.g. with N = 64 in a 64-channel spectrometer that is made up of N light-sensitive elements and N filters over the elements) values from the filter outputs. In the example of 3 the 64 outputs are limited to the sensor outputs and represent the fully integrated response of each individual filter on each photodiode. Referring to 1A For example, a spectral sensor may include a plurality of optical sensors having a plurality of sets of optical filters configured as a layer having a top surface and a bottom surface located on top of the plurality of optical sensors. In the example, a set of optical filters of the plurality of sets of optical filters includes a plurality of optical filters, each optical filter of the plurality of optical filters configured to pass light in a different wavelength range. In one example, a plurality of optical filters configured to pass light in the same wavelength range may be provided in an array so that each wavelength range is redundantly represented in a given array. There are countless layout options for the optical filters. In a specific example, the optical filters of each set of optical filters can be arranged in a pattern that repeats across a sensor array. In another example, each set of optical filters can be configured in several different patterns within the same sensor array. Accordingly, it may be due, at least in part, to the non-idealities referred to with reference to the 1A , 1B and 2 have been discussed, each of the plurality of optical filters configured to pass light in the same wavelength range behave differently.

Referring again to FIG 3 a given sensor output does not necessarily include the full influence of the spectral responses from each filter on the sensor, but this behavior (as in 2 shown above) is extremely complex and cannot be adequately captured in only 64 sampled data points. In addition, the other non-idealities associated with the sensor are not detected in the output, including, for example, its inherent non-linearities and the non-linearities that affect crosstalk, among other things. Accordingly, providing a single correction matrix that can provide a corrected spectrum over all input power, illumination spectral profile, and scene spectral profile conditions can be difficult, if not mathematically impossible. In one example, the correction matrix C from the above equation is “optimized” using a least squares approach in order to provide desired and / or expected spectral profiles in spectral measurements. More complex mathematical correction matrices can be created, but as shown, they can present an extremely complex mathematical problem.

4th Figure 3 is a schematic block diagram of one embodiment of a spectral sensor 10 , which is a spectral sensor array 220 , an analog-to-digital converter (ADC) 240 , a control machine 230 and calibration data 250 contains. The spectral sensor 10 can be configured as a single integrated circuit, multi-chip module, chipset, or contained elements contained in a single integrated circuit. In one example, the spectral sensor array 220 during the manufacturing and / or testing process of the spectral sensor array 220 calibrated to correct for deviations inherent in manufacturing processes, such as inter-die variance between sensors and inter-die variance. In one example, the calibration data is stored in memory (such as flash memory) residing in the spectral sensor 10 is contained, or in a memory that is in the same integrated circuit as the spectral sensor array 220 is included.

In one embodiment, the analog output of the spectral sensor array is 220 through an analog-to-digital converter (ADC) 240 for input into the control machine 230 converted. In one example, calibration data obtained during manufacture and / or testing of the spectral sensor array 220 were collected as calibration data 250 stored in memory, and the control machine 230 uses calibration data 250 to get those from the ADC 240 to "correct" received output. In another example, calibration data 250 also from the implementation of the spectral sensor 10 collected and / or changed heuristically. In another example, calibration data 250 during manufacture, testing, or use of a user device in which the spectral sensor 10 activated, collected and / or changed.

In a further example, the spectral sensor consists of a plurality of Single photon detectors, such as single photon avalanche diodes (SPADS) or other microphoton devices configured in a spectral detection array or pseudo array. The digital information from the resulting spectral acquisition array can then be fed directly into the control machine 230 can be entered. Examples of SPADS include, but are not limited to, single pixel silicon, InGaAs detectors, and two-dimensional arrays of CMOS detectors.

In the example of 4th become calibration data 250 as above in relation to 3 discussed, used to output the spectral sensor array 220 correct over a variety of input / operating conditions. In the example, the control machine represents 230 a calibrated sensor output from the spectral sensor array 220 using calibration data 250 ready. Calibrating the output of the spectral sensor array 220 this way involves a mathematical correction that requires a matrix that requires comparatively complex mathematics. For example, matrix inversion can be used to construct a correction matrix for the example, but the resulting correction matrix would likely be relatively imperfect while also being extremely complex to design and / or implement.

5A Figure 10 shows a schematic block diagram of one embodiment of a spectral sensor 10 ready, with hardware and / or software that can provide artificial neural network functions, called an artificial neural network 370 are designated in the spectral sensor 10 are included. In one example, this is the artificial neural network 370 in a spectral sensor 10 contained in a module or in one or more elements of the spectral sensor 10 integrated and configured to provide a corrected output based on a coordinated output of the artificial neural network 370 provide. In one example, calibration data 350 directly to the artificial neural network 370 to be delivered. In another example, calibration data 350 from the control machine shown in FIG 230 used to get a calibrated output from the sensor array 320 to the artificial neural network 370 provide.

In another example, the artificial neural network 370 implemented by an external computing device. In another example, the artificial neural network 370 implemented in another physical layer or element of a “stacked sensor” using 3D integration techniques. In one example, the artificial neural network can be massively connected in parallel to the spectral sensor by means of 3D stacking. For example, each pixel or filter field can be configured to contain a direct connection to one or more artificial neural network nodes.

The artificial neural network 370 comprises an interconnected structure of "artificial neurons" that act as a path for data transmission. A conventional computer system can consist of a number of simple, highly interconnected processing elements that process information with their dynamic state response to external inputs. In contrast, a neuron can be in the context of the artificial neural network 370 produce a linear or a non-linear response. In one example, a non-linear artificial network is created by connecting non-linear neurons; such a non-linear system may include inputs that are not proportional to the outputs.

In one example, the artificial neural network 370 "loaded" with weights or coefficients derived from a training process such as the one in 8B illustrated and described in the accompanying text. The coefficients can be fed directly into the artificial neural network 370 loaded or with calibration data 350 for input into the artificial neural network 370 get saved.

5B FIG. 11 shows a schematic block diagram of an embodiment of a spectral sensor 10 , wherein an artificial neural network temperature data is provided to improve the accuracy and / or precision of a corrected output. In one embodiment, the artificial neural network 370 Temperature data 390 provided. In one example, temperature data 390 from a temperature sensing device external to the spectral sensor 10 and can be transmitted to the sensor array continuously or intermittently using a duty cycle with the temperature signal only active for a fraction of a certain period of time. The temperature data 390 can also be done continuously or using a duty cycle for the artificial neural network 370 to be provided.

The temperature data 390 can be stored in dedicated memory or in memory shared with other spectral sensor elements. In one example, temperature data 390 are stored as numerical values representative of the actual temperature, such as a voltage or resistance value. In another example will be the temperature data 390 converted into a temperature value before being sent to the artificial neural network 370 be transmitted.

6A provides a simplified representation of an artificial neural network such as an artificial neural network 370 in the context of a spectral sensor such as a spectral sensor 10 . In the example, a measured output response 410 like the calibrated sensor output 260 , on the in 4th Reference is made to an input layer 400 of the artificial neural network 470 entered, the input layer 400 artificial neurons (nodes) 450 contains that with one or more nodes of the hidden layer 402 and an output layer 420 connected by nodes. In another example there is no hidden layer 402 , where nodes of the input layer 400 connected to nodes of the output layer without a hidden layer of nodes. In a further example of what is known as a single-layer neural network, the inputs into the artificial neural network 470 using a series of weights directly fed to the nodes of the output layer, calculating the sum of the products of the weights and the inputs in each node, and if the value is above a threshold value (typically 0) the neuron is triggered and takes the activated one Value (typically 1); otherwise the disabled value (usually -1) is used.

6B provides a simplified representation of an artificial neural network, sometimes referred to as a multilayer perceptron, with each node in one layer having connections directed to the nodes of the subsequent layer, with multiple layers of computing units interconnected in a feedforward arrangement. For example, if a value within the Perceptron node 510 (Node) represents the explicit threshold of each node (which can be subtracted so that all nodes have the same thresholds with arrows showing the weight of the inputs. This network assumes that if the threshold is not met , Zero (not -1), accordingly, in the example, a two-layer neural network can compute XOR.

In a simplified example, an input 500 for the neural network and a required target response is provided on the output 520 is set, and an error is obtained from the difference in the desired response together with the output of the real system.

The error information is reported back to the system and makes many adjustments to its parameters in a systematic order. After this process has been repeated for a sufficiently large number of training cycles, the network can converge to a state in which the error in the calculations is small and the network has "learned". In one example, a general approach to nonlinear optimization is used, where the network calculates the derivative of the error function with respect to the network coefficients or weights and changes the coefficients so that the error decreases.

Various forms of neural networks can be used to provide spectral correction including, but not limited to, artificial neural feedback networks, artificial neural feedforward networks, artificial neural networks with classification prediction, and others.

7A Figure 13 is a logic diagram illustrating an exemplary method for training an artificial neural network to provide corrected spectra. In step 600 is the spectral output from a sensor array, such as the sensor array 220 , collected over a range of input power and wavelengths, as shown in the table that 7A assigned. In one example, the sensor array may be presented with a sweep of monochromatic light over a desired range of wavelengths while the input power is fixed; the input power can then be incrementally increased and swept at any desired input power setting for the desired wavelength range. In an alternative example, the input power can be increased in fixed steps for each fixed wavelength of a desired wavelength range. In another alternative example, selected wavelength and input power pairs such as those in the table can be selected 7A randomly or randomly (or pseudo-randomly) to fill in a matrix. In another example, a matrix like the one in 7A can be modified using a fractional factorial design to reduce training runs and resources.

In the example of 7A light wavelengths and input power are taken into account. In other examples, input factors can be many. Exemplary input factors may include, for example, a signal or other indication of a type of lighting source (such as laser, natural light, LED (Light Emitting Diode) light, etc.), a type of object being sensed (e.g., human skin, outdoor scene, etc.), and / or an intended use for contain the spectral output (such as health analyzes, white balance, etc.). Further input factors can be found in and in the accompanying text. In one example, multidimensional matrices can be used to accommodate a variety of input factors. In a related example, multidimensional matrices designed for multiple input factors can use full and fractional factorial designs.

In the example, the matrix of Figure 7A contains monochromatic light at wavelengths from 720 nm to xxx nm in 10 nm increments for each of the 1-y input power units. In practice, wavelengths could be incremented in smaller increments such as 1 nm or in increments> 10 nm, and the input power can be incremented in a similar manner.

The procedure of 7A leads to a characterization model of a hypothetical spectrometer M with the dimensions L × S × N, where L is the number (#) of input power increments, S is the # of wavelength increments and N is the number of filter "patches". In the example, a filter patch is the number of filter elements across the sensor array that are selected for the same wavelength transmission. In one example, this spectrometer “model” can provide full resolution spectra (single wavelength increments of monochromatic light) and output N values. For example, for a 64 channel spectrometer there would be 64 outputs. The procedure starts in step 600 , where an output is collected for multiple pairs of input power and wavelength, and continues in step 610 where a large number of synthetic input spectra are generated. In a specific example, 100 million spectra covering essentially all conceivable permutations to which the spectrometer could be exposed. In another specific example, the synthetic input spectra can be selected for targeted permutations of the exposure conditions of the spectrum. These synthetic input spectra can then be used to train a corrected spectrum, as discussed above.

In step 620 the artificial neural network is trained (i.e. the coefficients or weights of the neural network are determined) so that when synthetic spectra are provided to the spectrometer, the artificial neural network outputs the artificial spectra (i.e. the error in the spectrometer is reduced to an acceptable one Amount minimized). In step 630 the coefficients or weights are put into the artificial neural network (like the artificial neural network 370 from 5 ) loaded. The spectrometer can then correct any light input into the spectral array to produce a corrected spectrum.

7B Figure 13 is a logic diagram illustrating an exemplary method for adaptively correcting the spectral output from a sensor array. The procedure starts in step 640 where the scanned spectrum is recorded from an object or scene. In one example, recording may include one or more of storing the scanned spectrum in memory, holding the scanned spectrum in a buffer, and providing readings as received to a processing module. The procedure is in step 642 continues, analyzing the recorded spectrum. In one example, the analysis may include analyzing the recorded spectrum to determine one or more of intensities at selected wavelengths, total energy, ratios for recorded peaks in the spectrum (such as peak / valley ratios or ratios between peaks), relative smoothness, or to determine relative spikiness signal, etc.).

In step 644 the method continues, wherein a calibration method is determined based on the analyzed spectrum and one or more additional input factors. Additional input factors can include, for example, a signal or other indication of a type of lighting source (such as laser, natural light, LED light emitting diode light, etc.), a type of object being scanned (such as human skin, outdoor scene, etc.) and / or a intended use for spectral output (such as health analysis, white balance, etc.). Please refer 8D and the accompanying text for further input factors. In one example, the calibration method may include a matrix such as that in FIG 7A shown matrix. Other calibration methods include one or more calibrations using one or more of two wavelengths (such as two wavelengths useful for measuring peripheral oxygen saturation (SpO2)), calibrations using principal component analysis, and calibrations using other schemes. In step 644 the particular calibration procedure is applied to the scanned spectrum. In step 648 when the correction is complete, the corrected sampled spectrum is analyzed to determine if it is within predetermined limits, and if the corrected sampled spectrum is not within predetermined limits, step 642 repeated and / or another calibration can be used until the corrected scan spectrum is within predetermined limits. In one example, the predetermined limits may be based on one or more input factors, such as one or more metadata associated with an object or scene, a classification limit, a look-up table, a formula or a calibration method used.

7C provides a logic diagram illustrating another exemplary method for adaptively correcting the spectral output from a sensor array. The procedure starts in step 650 having an optical sensor array associated with the spectral sensor system that receives light having N input powers for each light wavelength of M light wavelengths to produce N × M wavelength and input power pairs. In one example, each wavelength of light of the M wavelengths of light and each input power of light of the N input powers of light is received by the optical sensor array in a predetermined sequence. In another example, each optical sensor of the optical sensor array is associated with an optical filter from a plurality of sets of optical filters, and each optical filter of a set of optical filters is configured to pass light in a different wavelength range. The procedure is in step 652 continued, wherein the spectral sensor system generates a plurality of signals to produce a generated signal for each wavelength and input power pair, each of the plurality of signals being representative of the received light at each wavelength and input power pair of the N × M wavelength and input power pairs. In step 654 the spectral sensor system determines a plurality of error coefficients, each of the plurality of error coefficients being representative of the difference between a generated signal and a target value for a wavelength and input power pair. In step 656 the sensor system uses the plurality of error coefficients to train an artificial neural network configured to adjust a spectral response generated by the optical sensor array, and in FIG 658 For example, the output of the trained artificial neural network can be used to adjust the spectral response generated by another optical sensor array.

In a special implementation example, this can be done in 7C said artificial neural network can be configured to receive information associated with a scene that is scanned by another spectral sensor. In another example, the output of the trained artificial neural network is provided to another artificial neural network that is assigned to a different optical sensor array (in a different spectral sensor system), the other artificial neural network configured to receive information that a Scene are assigned, which is scanned by the other spectral sensor system.

In one example, this can be done in 7C mentioned artificial neural network can be trained with coefficients for N × M wavelength and input power pairs together with corrections based on other input factors such as the input factors referred to in FIG 7A Is referred to. In another example, a trained artificial neural network can be used to provide weighting for one or more artificial neural networks associated with other spectral sensor systems that can themselves be trained based on various input factors.

8A shows an exemplary spectral sensor, with the input parameters 310 , such as scanning monochromatic light over selected wavelengths at selected input power units, at the spectral sensor 10 and an uncorrected output response 500 is collected (like L × S × N from step 600 from 7A) .

In one example, includes a spectral sensor 10 Calibration data obtained during testing and / or manufacture of the spectral sensor 10 were collected, with the calibration data specific to the spectral array 220 of the spectral sensor 10 are. In one example, the calibration data is in one of the spectral sensor 10 allocated flash memory.

In the example is the uncorrected output response 500 a measure of how the spectral sensor is 10 when swept with monochromatic light at incremental input power levels. In an “ideal” spectral sensor, each sensor or pixel in combination with an associated filter element generates a signal that is an essentially perfect representation of the input power and that of the spectral sensor 10 provided light, so that a linear sequence of wavelengths at a given input power leads to a spectral output response that essentially corresponds to the input power and that at the spectral sensor 10 provided light matches (ie the spectral output response would be "ideal"). Since no spectral sensor works in such an ideal way, the uncorrected output response gives way 500 depends on the ideal spectral output response and requires a correction to the input parameters 310 to represent exactly.

In 8B an exemplary spectral sensor using an artificial neural network is shown. In one example, raw spectral data, such as the uncorrected output response 500 the end 8A , used to model a spectrometer 510 for the spectral sensor 10 to create, so that the spectrometer model 510 the non-ideality of the spectral sensor 10 for all input parameters 310 that reflects the spectral sensor 10 provided, as in the example of 8A .

The example uses artificial spectra 560 on the spectrometer model 510 received, its output then into an artificial neural network such as the artificial neural network 370 entered and with a desired or expected spectral output for artificial spectra 560 is compared. In one example, the error is 530 repeated for the artificial spectra 560 determined and in the artificial neural network 370 input to generate coefficients representing the difference between the artificial spectra 560 and the spectrometer model 510 reflect. The generated coefficients can then be stored and then loaded into a generic artificial neural network in order to correct the spectrum in an associated spectral sensor. In one example, a number of artificial spectra 560 based on the desired precision and accuracy for the spectral sensor 10 and can, for example, exceed 100 million spectra.

In one embodiment, the spectral sensor 10 Filter patterns over the spatial areas of a sensor array (such as the spectral sensor array 100 the end 1 ) provide a corrected output spectral response 520 to optimize. For example, the filter pattern can be via the spectral sensor array (such as the spectral sensor array 100 the end 1 ) encrypted and / or repeated several times across the spectral array. For example, the repeated patterns can reduce shading effects and other effects due to non-uniformities that can be the result of other elements affecting the spectral sensor 10 are assigned, such as collimation optics.

In another embodiment, the effect of changes in the lighting and / or the inclination of the scene can be achieved by the artificial neural network 370 fixed and / or changed. The slope of the scene can be described as the relationship between a scene or an object on a vertical and / or horizontal axis relative to an imager such as a spectral imager. For example, a scene that is relatively perpendicular to the line of sight of an imaging system (such as a camera) in a vertical and horizontal axis could be described as having a tilt of zero degrees ( 0 Degree). If the line of sight is described as a line extending from an imager's point of view to a scanned object or area (e.g., a stage), then an incline of 20 degrees could indicate that the scanned object is 20 degrees away from normal to the line of sight .

In one example, include synthetic spectra, such as artificial spectra 560 , measured parameters for various incline and lighting increments that can be used in training to minimize the effects of these parameters. In one example, correcting the tilt and / or lighting can reduce the requirement to collimate elements in the sensor array. In a specific example, measured parameters such as a scene inclination or a change in lighting could be made with an artificial neural network 370 configured to allow a user to impose an effect on an image, such as a scene slant. In one example, a tilt shift effect could be provided so that an image with zero degrees of tilt could reflect a variable amount of tilt to provide a tilt shift effect. In a specific related example, a measured scene slope could be with an artificial neural network 370 can be used to enable a user to manipulate the focal plane of an image to correct for distortion due to tilt. In another specific related example, a measured scene slope could be with an artificial neural network 370 can be used to enable a user to manipulate the focal plane of an image to mitigate the convergence of vertical and / or horizontal lines of another feature in the image.

In a further embodiment, the effect of changes in ambient light and / or temperature changes can be achieved by the artificial neural network 370 be resolved. In one example, the synthetic spectra, such as the artificial spectra, contain 560 , measured parameters for various ambient light conditions and temperatures that can be used in training to minimize the effects of these parameters.

In a specific example of implementation and operation, a spectral sensor system includes a plurality of optical sensors disposed on an integrated circuit. wherein the plurality of optical sensors are arranged in an array and an interface between the plurality of optical sensors and a first processing device configured to communicate information between one another. In one example, multiple sets of optical filters are configured as a layer that is on top of the plurality of optical sensors, with one set of optical filters of the plurality of sets of optical Filtering includes a plurality of optical filters, and each optical filter of the plurality of optical filters is configured to pass light in a different wavelength range. Included in the example is a memory configured to interface and communicate with the first processing device and to store calibration data associated with the plurality of sets of optical sensors, a second processing device comprising an artificial neural network configured to correct a spectral response generated by the plurality of optical sensors, along with an interface between the first processing device and the second processing device.

8C shows an exemplary spectral sensor in which a physical scene 160 a spectral sensor 10 is provided, which is then sent to a trained artificial neural network 370 is output to a corrected output spectral response 520 for the physical scene 560 provide. 8D shows an exemplary spectral sensor in which metadata of a physical scene for the artificial neural network such as the artificial neural network 370 to be provided. In one example is metadata 570 the physical scene for use by the artificial neural network 370 predetermined. In one example, when a particular plant is the subject of analysis, known spectral responses for that plant can be used to assess the efficiency of the artificial neural network 370 to increase. This could be achieved by optimizing one or more of the measurement areas, increasing the granularity of the measurements in a part of the physical scene, or analyzing based on the metadata 570 manipulated in other ways. The metadata 570 for physical scenes may include, but are not limited to, data provided by additional sensors such as gyroscopes, magnetometers, accelerometers, temperature sensors, and image sensors.

In a specific implementation and operation example, the metadata 570 Information about a physical scene 560 that contain the artificial neural network 370 either cannot correct or a corrected output spectral response 520 would produce outside acceptable limits. In one example, the artificial neural network 370 be configured to provide a notification, such as a signal, indicating that a correction is not within acceptable limits. In another example, the artificial neural network 370 configured to provide a notification, such as a signal, indicating that a correction is within acceptable limits, thereby effectively informing a user that the corrected output spectral response 520 can be trusted. Examples of metadata 570 Information that the artificial neural network 370 either cannot correct or would be outside acceptable limits include one or more of insufficient input light, sensor saturation, or the angle of incidence for incident light to pass through the artificial neural network 370 to be corrected.

In another specific example of implementation and operation, the artificial neural network 370 can be configured with a variety of operating modes. In one example, the artificial neural network 370 can be configured with a high accuracy, higher performance mode that produces a spectral output of relatively high quality but consumes relatively more power, or the artificial neural network 370 can be configured with a lower accuracy, lower performance mode with fewer calculations to reduce power consumption on a relative basis. In another specific example of implementation and operation, a reduced complexity operation can be configured to correct one or more scenarios. For example, the artificial neural network 370 be configured to select a condition or set of conditions, such as an average light intensity and angles of incidence for incident light collimated at an angle of 6 degrees, before correction by the artificial neural network 370 is performed. For example, in one example, the reduced set of conditions can provide reduced power consumption. In a related example, a specific correction can be made by an artificial neural network 370 for use in a non-neural based processor to reduce the relative performance or cost of spectral correction.

In another specific example of implementation and operation, the artificial neural network 370 can be configured to provide a confidence or accuracy value with a corrected output spectral response 520 provide. For example, if a red spot (or red tile) is measured at 50% input power, the resulting corrected output spectral response 520 for the measurement an absolute / relative accuracy of 1% at the wavelength of 700 nm and an absolute / relative accuracy of 5% at 500 nm. For example, if a red spot (or a red tile) is measured at 50% input power, the resulting corrected output spectral response 520 for the measurement an absolute / relative accuracy of 1% at the wavelength of 700 nm and an absolute / relative accuracy of 5% at 500 nm. If in an example the corrected spectral output response 520 has a relatively low confidence value between 500 nm and 550 nm, to be precise, the subsequent application layer might assign less importance to this part of the corrected spectrum.

In one example, a neural network correction algorithm may include one or more weighted loss functions (sometimes referred to as weighted cost functions) associated with an area and / or area of interest, such as a portion of the physical scene 560 or one or more identified objects or areas from the physical scene 560 . In one example, a weighted loss function can be used to measure the sensitivity of a spectral acquisition system, such as the spectral sensor 10 to increase for a particular area and / or area being scanned. In a specific implementation and operation example, one or more weighted loss functions may enable the hardware of the spectral sensor system (such as the spectral sensor 10 ) is adaptable for use in areas with different requirements and performance expectations. In one example, the one or more weighted loss functions are established at manufacture. In another example, the one or more weighted loss functions can be selected manually. In a further example, the one or more weighted loss functions are determined based on an algorithm, for example a pattern recognition algorithm.

In a specific related example, the measurement of hydration, such as the degree of hydration in the skin, can be implemented using data from spatially separated areas of a sensor to determine the accuracy and / or precision at 5 specific sections of the spectrum used for the hydration analysis raise. In the example, the accuracy and / or precision of the spectrum can be relatively less relevant in other spectral ranges, so the spectral sensor system can reduce or eliminate the samples from these spectral ranges.

In another example, the spectral sensor 10 in combination with the artificial neural network 370 can be used to generate a confidence image, the confidence image providing information sufficient to determine whether the actual spectrum measured is adequate. For example, the artificial neural network 370 provide an indication of the deviation of the measured spectrum from the already determined training data. The deviation can then be used to indicate abnormal spectra (spectra that cannot normally occur) that indicate an incorrect measurement.

In another example, the spectral sensor can be used in combination with the artificial neural network 370 can be configured to provide a correction of the spectral response as an intermediate step before a corrected spectrum is output. In the example, an artificial neural network-corrected spectral response is passed on to the subsequent layer, and an algorithm or the artificial neural network is used to determine a property or properties which are derived from the measured spectrum. For example, a property such as the water content and / or the oxygen content in a medium to be analyzed can be determined, the determination being based on the corrected spectral response provided by the artificial neural network without the need for a corrected spectrum.

In one embodiment, physical scene metadata 570 and / or other data that is in the artificial neural network 370 are input, provided essentially in real time (live), so that the artificial neural network 370 can constantly revise the corrected output. In another embodiment, metadata 570 the physical scene and / or other data previously stored and an artificial neural network 370 provided as required and / or feasible.

It is noted that terminologies as may be used herein such as bit stream, stream, signal sequence, etc. (or their equivalents) have been used interchangeably to describe digital information the content of which corresponds to any of a number of desired types ( e.g. data, video, voice, text, graphics, audio, etc., each of which can be broadly referred to as "data").

As may be used herein, the terms “substantially” and “approximately” offer an industry-accepted tolerance for the corresponding term and / or the relativity between elements. For some industries, an industry accepted tolerance is less than one percent and for other industries the industry accepted tolerance is 10 Percent or more. Other examples of industry tolerance range from less than one percent to fifty percent. Tolerances accepted by the industry correspond, but are not limited to, component values, integrated circuit process variations, temperature variations, rise and fall times, thermal noise, dimensions, signaling errors, dropped packets, temperatures, pressures, material compositions, and / or performance metrics. Within an industry, tolerance deviations of accepted tolerances can be more or less than a percentage level (e.g. dimensional tolerance of less than +/- 1%). Some relativity between elements can range from a difference of less than a percentage to a few percent. The other relativity between elements can range from a difference of a few percent to the size of the differences.

As may also be used herein, the terms "configured to", "operably coupled to", "coupled to" and / or "coupling" include a direct coupling between elements and / or an indirect coupling between elements via an intermediate element, (e.g. an object comprises a component, an element, a circuit and / or a module, but is not limited to this) if, for example for an indirect coupling, the element in between does not change the information of a signal, but rather its current level, voltage level and / or adjust power levels. As may be further used herein, inferred coupling (i.e., when an element is inference coupled to another element) includes direct and indirect coupling between two elements in the same way as "coupled to".

As can be used further here, the term “configurable to”, “operable to”, “coupled to” or “operable coupled to” indicates that an object has one or more of power connections, inputs / inputs, outputs / outputs etc., to perform one or more of its respective functions when activated, and may further include inferred coupling to one or more other elements. As may be further used herein, the term "associated with the direct and / or indirect coupling of separate elements and / or an element that is embedded in another element."

As may be used herein, the term “compare favorably” indicates that a comparison between two or more elements, signals, etc. provides a desired relationship. For example, if the desired relationship is that signal 1 has a larger size than signal 2, a favorable comparison can be obtained if the size of signal 1 is larger than that of signal 2 or if the size of signal 2 is smaller than that of signal 1 is. As may be used herein, the term “compare poorly” indicates that a comparison between two or more elements, signals, etc. does not provide the desired relationship.

As may be used herein, one or more claims in a specific form of this generic form may include the term “at least one of a, b and c” or of this generic form “at least one of a, b or c” with more or fewer elements included as "a", "b" and "c". The phrases are to be interpreted identically in both formulations. In particular, “at least one of a, b and c” is synonymous with “at least one of a, b or c” and means a, b and / or c. As an example it means: only "a", only "b", only "c", "a" and "b", "a" and "c", "b" and "c" and / or "a", "B" and "c".

As may also be used herein, the terms “processing module,” “processing circuit,” “processor,” “processing circuit,” and / or “processing unit” can be a single processing device or multiple processing devices. Such processing device may be a microprocessor, microcontroller, digital signal processor, microcomputer, central processing unit, field programmable gate array, programmable logic device, state machine, logic circuit, analog circuit, digital circuit, and / or any Be a device that manipulates signals (analog and / or digital) based on hard coding of the circuit and / or operating instructions. The processing module, the module, the processing circuit, the processing circuit and / or the processing unit can be a memory and / or an integrated memory element or furthermore comprise a memory and / or an integrated memory element, which is a single memory device, multiple memory devices and / or an embedded one Circuit of another processing module, module, a processing circuit, a processing circuit and / or a processing unit can be. Such a storage device may be read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, and / or any device that stores digital information saves. It should be noted that if the processing module, the module, the processing circuit, the processing circuit and / or the processing unit contains more than one processing device, the processing devices can be arranged centrally (e.g. directly connected to one another via a wired and / or wireless bus structure ) or distributed (e.g. cloud Computing via indirect coupling via a local network and / or a wide area network). It should also be noted that if the processing module, the module, the processing circuit, the processing circuit and / or the processing unit implements one or more of their functions via a state machine, an analog circuit, a digital circuit and / or a logic circuit, the memory and / or the memory element in which the corresponding operating instructions are stored can be embedded in or outside the circuit which comprises the state machine, the analog circuit, the digital circuit and / or the logic circuit. It should also be noted that the memory element can store and the processing module, module, processing circuit, processing circuit and / or processing unit executes hard-coded and / or operating instructions which correspond to at least some of the illustrated steps and / or functions in one or more of the figures. Such a storage device or such a storage element can be contained in an article of manufacture.

One or more embodiments have been described above in terms of method steps that illustrate the performance of specified functions and relationships thereof. The limits and the order of these functional building blocks and process steps have been arbitrarily defined here to simplify the description. Alternative boundaries and sequences can be defined as long as the specified functions and relationships are properly performed. Such alternative limits or sequences are thus within the scope and spirit of the claims. Furthermore, the limits of these functional building blocks have been arbitrarily defined to simplify the description. Alternative limits could be defined as long as the significant functions identified are adequately performed. Similarly, flowchart blocks may also have been defined arbitrarily herein to illustrate certain significant functions.

To the extent used, the block boundaries and the sequence of the flowchart could have been defined differently and still perform the particular significant functionality. Such alternative definitions both of function blocks and of flowchart blocks and sequences are thus within the scope and spirit of the claims. One of ordinary skill in the art will also recognize that the functional building blocks and other illustrative blocks, modules, and components herein can be implemented as shown or by discrete components, application specific integrated circuits, processors running appropriate software, and the like, or any combination thereof.

In addition, a flowchart can contain a “Start” and / or “Continue” indication. The “Start” and “Next” indicators indicate that the steps shown can optionally be incorporated into, or otherwise used in conjunction with, one or more other routines. In addition, a flowchart may contain an “End” and / or “Continue” indicator. The indications “End” and / or “Next” reflect that the steps presented end as described and shown or, optionally, can be integrated into one or more other routines or otherwise used in connection therewith. In this context, “Start” indicates the beginning of the first step presented and can be preceded by other activities that are not specifically shown. Furthermore, the indication “Next” reflects that the steps shown can be carried out multiple times and / or can be followed by other activities that are not specifically shown. While a flowchart shows a particular order of steps, other orders are possible provided the principles of causality are maintained.

The one or more embodiments are used herein to illustrate one or more aspects, one or more features, one or more concepts, and / or one or more examples. A physical embodiment of an apparatus, article of manufacture, machine, and / or method may include one or more of the aspects, features, concepts, examples, etc. described with reference to one or more of the embodiments discussed herein. Further, from figure to figure, the embodiments may include the same or similarly named functions, steps, modules, etc., which may use the same or different reference numbers, and as such, the functions, steps, modules, etc. may have the same or similar functions, steps , Modules etc. or different.

Unless expressly stated otherwise, signals to, from and / or between elements in a figure can be analog or digital, continuous time or discrete time, and single-ended or differential. For example, when a signal path is displayed as a single-ended path, it also represents a differential signal path. Similarly, when a signal path is displayed as a differential path, it also represents a single-ended signal path. While one or more particular architectures are described herein, other architectures may also be implemented using one or more data buses not expressly shown, direct connectivity between elements, and / or indirect coupling between other elements, as would be recognized by one of ordinary skill in the art.

The term “module” is used in the description of one or more of the embodiments. A module implements one or more functions through a device such as a processor or other processing device or other hardware that may contain or operate in conjunction with memory in which operating instructions are stored. A module can operate independently and / or in conjunction with software and / or firmware. As also used herein, a module can include one or more sub-modules, each of which can be one or more modules.

As can be further used herein, computer readable memory includes one or more storage elements. A storage element can be a separate storage device, multiple storage devices, or a set of storage locations within a storage device. Such a storage device may be read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, and / or any device that stores digital information saves. The storage device can be in the form of solid-state storage, hard disk storage, cloud storage, USB stick, server storage, computer storage, and / or other physical medium for storing digital information.

While certain combinations of various functions and features of the one or more embodiments have been specifically described herein, other combinations of these features and functions are also possible. The present disclosure is not limited to the particular examples disclosed herein and expressly includes these other combinations.

Claims (20)

Spectral sensor system comprising: a multitude of optical sensors that are arranged on an integrated circuit, wherein the plurality of optical sensors are arranged in an array; an interface between the plurality of optical sensors and a first processing device configured to transfer information therebetween; a plurality of sets of optical filters configured as a layer disposed on the plurality of optical sensors, wherein one set of optical filters of the plurality of sets of optical filters comprises a plurality of optical filters, each optical filter of the A plurality of optical filters configured to pass light in a different wavelength range; a memory configured to interface with and communicate with the first processing device, the memory configured to store calibration data associated with the plurality of sets of optical sensors; a second processing device including an artificial neural network configured to correct a spectral response generated by the plurality of optical sensors; and an interface between the first processing device and the second processing device configured to transfer information therebetween. Spectral sensor system according to Claim 1 wherein the plurality of optical filters of each set of the plurality of sets of optical filters are arranged in a pattern, the pattern being repeated for at least some of the sets of the plurality of sets of optical filters across the array. Spectral sensor system according to Claim 1 or Claim 2 wherein the optical filters in at least some of the sets of the plurality of sets of optical filters are interference filters. Spectral sensor system according to Claim 1 or Claim 3 , the interference filters being Fabry-Perot filters. Spectral sensor system according to Claim 1 wherein the second processing device is configured to receive information associated with a scene scanned by the spectral sensor system. Spectral sensor system according to Claim 5 wherein the information associated with the scene includes metadata associated with the scene. Spectral sensor system according to Claim 5 or Claim 6 wherein the information associated with the scene comprises at least one of the following elements: 1) a temperature; 2) a measure of ambient light; 3) a measure of the lighting of the scene; 4) a slope for the scene, the slope being a deviation of the scene on at least one vertical or horizontal axis from normal relative to a line of sight of the spectral sensor system; 5) an alignment of the spectral sensor system; 6) an orientation of the scene; 7) an angular velocity of the spectral sensor system; 8) an angular velocity of one or more objects in the scene; 9) a magnetic field associated with the scene; 10) a magnetic field associated with the spectral sensor system; and 11) a magnetic dipole moment of the spectral sensor system. Spectral sensor system according to Claim 5 or Claim 6 wherein the information associated with the scene is provided by a user, the information provided by the user includes information sufficient to identify an object in the scene, the second processing device being configured to correct the spectral response generated by generating the plurality of optical sensors based at least in part on the object. Spectral sensor system according to Claim 1 or Claim 5 , further comprising: a user notification mechanism, the second processing device configured to provide a representation of a corrected spectral response to a user using the user notification mechanism. A method for execution by one or more devices of a spectral sensor system, the method comprising: Receiving light with N input powers for each light wavelength of M wavelengths of light by an optical sensor array associated with the spectral sensor system to produce N × M wavelength and input power pairs, each wavelength of the light of the M wavelengths of light and each input of light of the N Input powers of the light is received by the optical sensor array in a predetermined sequence, each optical sensor of the optical sensor array being associated with one of a plurality of sets of optical filters and each optical filter of a set of optical filters being configured to capture light in a different wavelength range to let through; Generating a plurality of signals, each of the plurality of signals being representative of the received light at each wavelength and input power pair of the N x M wavelength and input power pair to generate a generated signal for each wavelength and input power pair; Determining a plurality of error coefficients, each error coefficient of the plurality of error coefficients being representative of a difference between a generated signal and a target value for a wavelength and input power pair; Using the plurality of error coefficients to train an artificial neural network, the artificial neural network configured to adjust a spectral response generated by the optical sensor array. Procedure according to Claim 10 wherein the optical filters in each set of the plurality of sets of optical filters are arranged in a pattern, the pattern being repeated for at least some of the sets of the plurality of sets of optical filters across the array. Procedure according to Claim 10 or Claim 11 wherein the optical filters in at least some of the sets of the plurality of sets of optical filters are interference filters. Procedure according to Claim 12 , the interference filters being Fabry-Perot filters. Procedure according to Claim 10 or Claim 12 wherein the artificial neural network is configured to receive information associated with a scene that is scanned by another spectral sensor. Procedure according to Claim 14 wherein the information associated with the scene includes metadata associated with the scene. Procedure according to Claim 14 or Claim 15 , wherein the information associated with the scene comprises at least one of the following properties: 1) a temperature; 2) a measure of ambient light; 3) a measure of the lighting of the scene; 4) a slope for the scene, the slope being a deviation of the scene on at least one vertical or horizontal axis from normal relative to a line of sight of the spectral sensor system; 5) an alignment of the spectral sensor system; 6) an orientation of the scene; 7) an angular velocity of the spectral sensor system; 8) an angular velocity of one or more objects in the scene; 9) a magnetic field associated with the scene; 10) a magnetic field associated with the spectral sensor system; and 11) a magnetic dipole moment of the spectral sensor system. Procedure according to Claim 14 or Claim 15 , wherein the information assigned to the scene is provided by a user, wherein the user-provided information includes information sufficient to identify an object in the scene, the artificial neural network configured to correct the spectral response generated by the optical sensor array that is based at least in part on the object. Procedure according to Claim 10 further comprising: transmitting the plurality of signals to a first processor associated with the spectral sensor; The first processor calibrating the plurality of signals to produce the generated signals, wherein the calibration of the plurality of signals is based on calibration data, the calibration data being based on measured variations associated with manufacturing the optical sensor array. Procedure according to Claim 18 wherein the artificial neural network is associated with a second processor, the first processor being configured to transmit the generated signals to the second processor. Procedure according to Claim 19 , further comprising: a user notification mechanism, the second processor configured to provide a representation of a corrected spectral response to a user using the user notification mechanism.

Images