CA2331121A1

CA2331121A1 - Device for recording information in different modes

Info

Publication number: CA2331121A1
Application number: CA002331121A
Authority: CA
Inventors: Christer Fahraeus
Original assignee: Individual
Current assignee: Anoto Group AB
Priority date: 1998-04-30
Filing date: 1999-04-30
Publication date: 1999-11-11
Also published as: AU4303999A; JP2002514021A; EP1073991A1; CN1303504A; BR9910579A; EP1073989A1; WO1999060516A1; AU4303199A; AU758236B2; IL139106A0; CN1138227C; WO1999057678A1; KR20010052280A

Abstract

A device for recording information by means of imaging with the aid of a light-sensitive sensor with a two-dimensional sensor surface is described. The device is adjustable between a first mode and a second mode. In the first mode, the imaging is effected by means of a plurality of images with partially overlapping contents. Each of the images depicts a small area on an information carrier and the images are put together into a composite image. In the second mode, the imaging is effected by means of at least one image of a larger area of the information carrier. The device has means for indicating the size of the imaging area.

Description

DEVICE FOR RECORDING INFORMATION IN DIFFERENT MODES.
Field of the Invention The present invention relates to a device for recording information according to the preamble to the appended claim 1.
Background of the Invention There are many situations where someone wishes to combine selected parts of text or image information into a document which can be edited by means of suitable soft-ware in a computer. A known way of feeding text and image information into a computer is to use a scanner. Scanners of both the stationary and the portable type are avail-able. A stationary scanner is used for scanning whole pages of text and image information, the scanner being passed over the page automatically at a constant speed.
This type of scanner is not suitable for scanning select-ed parts of information on a page. On the other hand, a portable scanner may be suitable for this purpose.
US 5 301 243 discloses a hand-held scanner for scan ning characters from a character sequence on a substrate.
The scanner, which is passed over the characters which are to be read in contact with the substrate, has an optical system which "sees" a small part of the sub-strate. The optical system comprises a CCD type line sen-sor, which has a plurality of light-sensitive elements arranged in a row. When the scanner is passed over the characters on the substrate, a sequence of vertical "slices" of the characters and of the spaces between them is recorded. The "slices" are stored in the scanner as a digital bitmap. Subsequently, OCR software (OCR = Optical Character Recognition) is used to identify the scanned characters and to store them in character-coded format, e.g. with the aid of ASCII code, in a memory. The charac-ter recognition can be performed either in the scanner or WO 99/5767$ PCT/SE99/00716 in an external computer to which the bitmapped characters have been transferred.
Another type of hand-held scanner for inputting text is described in US 4 949 391. Unlike the one described above, this scanner has a two-dimensional sensor, which records images of the underlying surface when the scanner is passed over the same. The scanner can only be moved in a direction determined by a wheel abutting against the surface. Redundant information is removed from the recorded images, which are subsequently combined into a larger image. The larger image can be analysed in a com-puter for identifying characters.
A drawback of the above scanner and similar hand held scanners is that their "field of vision" is rela Lively small. Consequently, in order to record a larger amount of information, such as a text section comprising several lines, a user must pass the scanner back and forth repeatedly over the surface, which is often regard-ed as boring and a waste of time.
Summary of the Invention The object of the present invention is to provide a solution to the problem described above.
This object is achieved according to the invention by the device having the features recited in the append-ed claim 1, with preferred embodiments in the appended claims 2-19.
More specifically, the device is adjustable between a first operating mode and a second operating mode, in which images of an area with a first small and a second large size of the information carrier are recorded so that the information on the information carrier can be recorded at different degrees of selectivity. In the first operating mode, the device is adapted to be passed over the information while recording a plurality of images with partially overlapping contents, which are put together into a composite image of the information and which each are related to an area with the first small size of the information carrier. In the second operating mode, the device is adapted to image the information by means of at least one image which relates to an area with the second large size of the information carrier.
The device thus has a first mode for recording information at a high degree of selectivity, e.g. indivi-dual words or character sequences, and a second mode for recording larger amounts of information at a lower degree of selectivity, e.g. several lines of text or an image, at the same time.
An advantage of this device is that the user can choose whether he or she wishes to record information at a high or low degree of selectivity. This can be achieved by a simple adjustment of the device. The user can thus first use the device in the first mode for recording a sentence in a text and subsequently adjust it to the second mode and use it for fast recording of an illu-stration which takes half a page.
The information carrier can be any two- or three-dimensional object. Of course, it will usually be a sheet of paper. The information on the surface may be composed of characters in the form of letters, numbers, or other written characters or symbols, or of various types of images.
The recording is carried out with the aid of a light-sensitive sensor having a two-dimensional sensor surface. In this context, a two-dimensional sensor sur-face refers to the fact that the sensor surface must be able to image a surface with a matrix of pixels so that images with overlapping contents can be recorded. The light-sensitive sensor is preferably of the CCD or CMOS
type. It can record images in grey scale or in colour.
The images comprise a plurality of pixels, each having an intensity value.
The device can be adjusted by the user or automati-cally by the device. The adjustment involves the adapta-tion of the device in some manner, e:g, physically or in WO 99/57678 PCT/SE99/0071b terms of software, to make it suitable for use in the mode in question.
Since areas of different sizes are imaged in the different modes, it may be difficult for the user to know exactly which information he is recording. To solve this problem, the device is preferably provided with indicat-ing means for indicating the size of the,imaging area. It is particularly important that the user knows the size of the imaging area in the second mode since this area is larger in that mode. However, the size of the imaging area is preferably indicated in the first mode as well.
Advantageously, the size of the imaging area can be indicated on the information carrier itself so that the user can see exactly which information will be recorded.
It can be indicated by, fox example, dots, lines, or colour, etc. In the first mode, the indicating can be effected by means of a single luminous spot or by means of a line showing where on a line the recording will com-mence. What is important is that the user gets an idea of the size of the imaging area.
In a preferred embodiment, the indicating means com-prise an illumination means, e.g. a light-emitting diode or a laser diode.
Preferably, at least in the second mode, the imag-ing area is indicated by a plurality of luminous spots between which the imaging area is located. A less complicated optical design of the device as well as more accurate image recording are achieved in this way, since no coloured light is recorded in the image. This is par-ticularly important in connection with the use of colour-sensitive sensors.
Another way of indicating the imaging area can be to show what is located within the field of vision of the device on a display which in this case constitutes the indicating means. In some cases, the indicating means may comprise a display as well as an illumination means for indicating on the information carrier.

In a preferred embodiment, the device can be acted upon for continuous adjustment of the size of the area which is to be imaged between the first small size in the first mode and the second large size in the second mode.

5 In this way, the user can himself determine, e.g. with the aid of a slider control, the exact size of the imaging area, something which can be advantageous, for example, in connection with the recording of images.
Preferably, the indicating means is also acted upon in connection with the continuous adjustment so that the user can see how the size of the imaging area changes.
The change of the size of the imaging area between the first and the second mode can advantageously be effected by utilising a smaller part of the sensor sur-face in the first mode and a larger part of the sensor surface in the second mode. This can be achieved by only saving information from the desired part of the sensor surface. The continuous adjustment of the imaging area can also be achieved in this way. In connection with an adjustment of the sensor surface, the indicating means advantageously consist of a display.
An alternative way of changing the size of the imaging area is to provide the device with a lens means, which is adapted to project the images of the information carrier onto the sensor surface and which is adjustable for changing the imaging area. The lens means can com-prise one or more lenses. The advantage of this alterna-tive is that it provides better resolution.
One way of changing the size of the imaging area with the aid of the lens means is to change the focal distance of the lens means. This can be achieved by the lens means being adjustable between a first position in the first mode and a second position in the second mode.
The lens means can also be continuously adjustable for continuous adjustment of the position of the lens means ' between the first and the second position, so that an optional size of the imaging area can be obtained.

In another embodiment, the device comprises a fur-ther light-sensitive sensor with a two-dimensional sensor surface, the one light-sensitive sensor being used in the first mode and the other light-sensitive sensor being used in the second mode. In this embodiment, the device thus has two different beam paths and two different open-ings, for instance one at each end of the casing, in which case the user is allowed to choose which opening is to be directed at the information to be recorded depend-ing on which mode he or she wants to use. The advantage of this embodiment is that it is possible to have a sta-tionary beam path for each sensor.
The device can advantageously be adapted to be held at a distance from the information carrier in both the first and the second mode. This has the advantage that the information carrier does not need to be plane nor have only a two-dimensional extent, which is the case in connection with the utilisation of prior art scanners.
By means of a device according to the invention, informa tion upon or in the form of three-dimensional objects, or parts thereof, can also be imaged.
Preferably, the device is adapted to be held at the same distance from the information carrier in both modes so that the user is not required to change the position of his hand when changing the size of the imaging area. A
suitable distance may be 5-15 cm.
In order for the device to be easy to handle in every type of recording situation, it is suitably hand-held.
In a preferred embodiment, the device further com-prises a signal-processing unit which is adapted to uti-lise the partially overlapping contents of the images for putting together the images into a composite image in the first mode. By virtue of the fact that the overlapping contents of the images are used for putting together the images, the device need not contain any means, e.g.
wheels, for recording the position of the device relative to the information carrier or for measuring the distance between the images. This results in considerably improved user-friendliness.
The putting-together is preferably carried out both vertically and horizontally. This has the advantage that the device can be held at different angles and even be turned while being passed over the information carrier and yet the images can be put together well enough that the characters in the composite image can be identified and OCR processed.
In order to enable as much information as possible to be stored in the device, the signal-processing unit preferably comprises software for identifying characters in the composite image and for storing the same in the device in character-coded format, e.g. ASCII code. The character identification can, for example, be carried out with the aid of a neural network.
Moreover, the device may advantageously comprise a transceiver for wireless communication with an external unit. In this way, information can be transferred between the device and, for example, an external computer. The transceiver can be an IR transceiver, a mobile radio transceiver, or some other suitable transceiver.
As mentioned above, the device can advantageously operate as a scanner also in the second operating mode, in which case it is adapted to be passed over the infor-mation while recording a plurality of images with par-tially overlapping contents, which images are put toge-ther into a composite image of the information and which each relate to an area of the second large size.
Brief Description of the Drawincrs The present invention will be described below by way of an example showing how the invention can be implemented. The description refers to the accompanying drawings, in which Fig. 1 schematically shows a first embodiment of a device according to the invention;

Fig. 2 is a block diagram of the electronic cir-cuitry in a first embodiment of the device according to the invention;
Fig. 3 schematically shows the indicating area in connection with the use of the device in the first mode and the second mode respectively;
Fig. 4 is a flowchart showing how the device is intended to be used in the first mode;
Fig. 5 is a flowchart showing how the device is intended to be used in the second mode; and Fig. 6 schematically shows a second embodiment of a device according to the invention.
Description of Preferred Embodiments In the embodiment of the device according to the invention shown in Fig. 1, it comprises a casing 1 hav-ing approximately the same shape as a conventional high-lighter pen. One short side of the casing has a window 2, which is intended to be directed at an area on an infor-mation carrier which a user wishes to image. The informa-tion carrier can, for example, consist of a sheet of paper or some other type of storage medium, from which a user wishes to record information in character or image format .
The window 2 is somewhat recessed in the casing 1 in order to reduce the risk of scratches or other damage when the device is not in use.
The casing 1 essentially contains an optics part 3, an electronic circuitry part 4, and a power supply part 5.
The optics part 3 comprises four light-emitting diodes 6 jointly constituting an indicating means, but of which only two are visible in the Figure, as well as an adjustable lens system 7 and a light-sensitive sensor 8 which constitutes the interface with the electronic circuitry part 4.

The LEDs 6 are used for indicating the area on the information carrier under the device which can be imaged by means of the device.
The lens system 7 fulfils two tasks simultaneously.
One of its tasks is to project light from the LEDs 6 onto the information-carrying surface towards which the window of the device is directed for defining the imaging area.
The light from the LEDs can be seen on the surface in the form of luminous spots.
The other task of the lens system 7 is to image the area indicated by the device on the light-sensitive sen-sor 8 as accurately as possible. In order to achieve imaging of areas of different sizes when the device is held at a given distance from the information carrier, the lens system 7 is adjustable between two positions.
The adjustment of the lens system can be effected by means of the same technique as is used in cameras.
Fig. 3 schematically shows how the device is held over a surface in the first mode and the second mode respectively and how, as a result, two imaging areas of different sizes are provided. Moreover, the lens system 7 is also schematically indicated in the Figure in two dif-ferent positions corresponding to the first mode and the second mode respectively.
In this example, the light-sensitive sensor 8 com-prises a two-dimensional, square CCD unit (CCD = charge coupled device) with a built-in A/D converter. Such sen-sors are commercially available. In this case, the sensor 8 is mounted on its own printed circuit board 11.
The power supply to the device is obtained from a battery 12 which is mounted in a separate compartment 13 in the casing.
The block diagram in Fig. 2 schematically shows the electronic circuitry part 4. This comprises a processor 20, which by the intermediary of a bus 21 is connected to a ROM 22, in which the programs of the processor are stored, to a read/write memory 23, which constitutes the working memory of the processor and in which the images from the sensor as well as the characters identified and interpreted are stored in the first mode, to a control logic unit 24, to the sensor 8, as well as to the lens 5 system 7 and the LEDs 6.
The control logic unit 24 is in turn connected to a number of peripheral units, comprising a display 25, which is mounted in the casing, an IR transceiver 26 for transferring information to/from an external computer, 10 buttons 27, by means of which the user can control the device, as well as an operation indicator 28 comprising, for example, a couple of additional LEDs indicating which mode the device is in and whether or not recording is taking place. Control signals to the memories, the sen-sor, and the peripheral units are generated in the con-trol logic unit 24. The control logic also handles gene-ration and prioritisation of interrupts to the processor.
The buttons 27, the IR transceiver 26, the display 25, and the LEDs 6 are accessed by the processor writing and reading in the register in the control logic unit. The buttons 27 generate interrupts to the processor 20 when they are activated.
An example of how the device is intended to be used will now be described. Suppose that a user wishes to record a section of text and an image from a sheet of paper. In this case, he holds the device at a comfortable distance from the sheet. Naturally, the distance will vary from person to person, but it is usually within the 5-15 cm range. In the first mode, the device functions as follows. First, the user wishes to record the section of text. He starts the device by pressing a button 27, whereby two of the LEDs 6 are turned on projecting two luminous spots onto the sheet of paper. These luminous spots show the height of the imaging area, thereby indi-Gating its size since the imaging area is square. The user places these spots to the left of the first letter in the section of text he wishes to record. Subsequently, he presses the button 27 to activate the device and moves the device so that the luminous spots move across the text to be recorded, in the same way as when one reads the text. When the user activates the device, the proces-s sor 20 commands the device to begin recording images at a predetermined frequency, e.g. 25 Hz, whereupon the images recorded by the sensor are stored in the read/write memory 23. However, text which is stored in image format requires a large amount of memory space. Consequently, to save memory space in this mode, the characters in the images are identified and stored with the aid of ASCII
code. When the user has come to the end of a line or has let the luminous spots move across the selected section of text, he releases the activating button, whereupon the processor 20 turns off the image recording. When the selected section of text has been recorded, the user can control the device to show the recorded text on the dis-play 25 or to transfer the text to an external computer by the intermediary of the IR transceiver 26. The possi-bility of showing the scanned information directly on the screen has proven very important since a user often wishes to verify that the correct information has been scanned.
The flowchart in Fig. 4 shows how the device is adapted to operate in the first mode. In step 401, the device indicates the imaging area with the aid of lumi-nous spots from the LEDs 6. In step 402, the user moves the luminous spots across the area he wishes to record, whereupon the device records images with overlapping contents and stores them in a current image area in the read/write memory 23. The images are stored as images, i.e. with the aid of a plurality of pixels, each having a grey scale value in a range from white to black.
As soon as an image has been stored in the current image area, the putting-together of the image with the previous image is suitably commenced, step 403, if such an image is available. If there is no previous image, the current image is entered directly into a line image area in the read/write memory.
In order to determine how the current image is to be put together with the previous image so that the best match is achieved between the contents of the images, every possible overlap position between the images is examined, at the pixel level, and an overlap measurement is determined as follows:
1) For each overlapping pixel position, the grey scale values of the two relevant pixels are added up if the latter are not white. Such a pixel position in which none of the pixels are white is designated a plus posi-tion.
2) The grey scale sums for all the plus positions are added up.
3) The neighbours of each pixel position are examin-ed. If an overlapping pixel position is not a neighbour of a plus position and consists of a pixel which is white and a pixel position which is not white, the grey scale value of the non-white pixel is subtracted, possibly mul-tiplied by a constant, from the sum in point 2).
4) The overlap position providing the highest over-lap measurement as stated above is selected. In the com-posite image the mean value of the grey scale value of the overlapping pixels is used. In this way, noise can be suppressed in the overlap area. The putting-together is thus carried out both vertically and horizontally. If it is detected that, when being put together, the images do not end up on a horizontal line, the composite image is suitably adjusted so that it becomes horizontal, for example by turning the composite image.
Our Swedish patent application No: 9704924-1 and the corresponding US Application No. 024 641 describe an alternative way of matching the images in order to find the best overlap position. The content of these applica-tions is herewith incorporated by reference.

The composite image gradually develops in the line image area in the read/write memory. It is preferred that the line image area be large enough to store an A4 line of normal typewritten text.
In step 404, the software of the processor 20 divides the composite image in the line image memory area into sub-images each containing only one character. The purpose of this is to create input signals to the neural network software which is to interpret the characters.
The division is effected by adding up the grey scale values of the pixels for each pixel row and each pixel column in the composite image. By studying the local intensity minima for the row sums and column sums thus obtained, boundaries can be determined for the extent of each character in the image.
Subsequently, in step 405, each character in the composite image of the character sequence imaged is interpreted. The grey scale values of the pixels which together constitute a sub-image containing only one char-acter are fed as input signals to a neural network. Each output from the neural network represents a character which the network can identify. The output from the net-work which has the highest output signal is chosen and the character thus chosen is stored in step 406 using a predetermined character code format, for example ASCII
code, in the read/write memory 23 in a memory area for interpreted characters. When the character identification and storing in character coded format is completed, the processor activates the operation indicator 28 to inform the user that it is ready to record a new character sequence, step 407. Subsequently, it goes back to step 401.
In the first mode, the steps described above are thus carried out by the processor 20 with the aid of the associated units and suitable software. Such software can be created by the skilled person with the aid of the above instructions. The character recognition is perform-ed with the aid of neural network software, which is trained in a suitable manner. Neural network software is commercially available from, for example, MATLAB Neural Network Toolbox, The MathWorks Inc., 24 Prime Park Way, Natick, MA 01760, USA.
In this example, the user also wishes to record the image included on the sheet of paper. In order to do this, he presses a button 27, whereby the device changes modes and the processor sends a signal to lens system 7.
The lens system 7 is moved from its position in the first mode to the position required for the second mode. Fur-ther, the two LEDs 6 which were previously switched off are turned on. In connection with the adjustment, the focal distance of the lens system 7 is changed and the four luminous spots move diagonally outwards from the centre of the imageable section, so that a larger image-able section is provided, something which is also shown in Fig. 3. Subsequently, the user directs the device so that the luminous spots projected onto the paper surround the image he wishes to record. In this connection, he can adjust the size of the imaging area by changing the dis-tance to the paper. He then presses the button 27 to activate the device which then records an image of the section between the luminous spots. When the image has been recorded, the user can control the device to either show the recorded image on the display 25 or transfer the image to an external computer by the intermediary of the IR transceiver 26. If the image on the paper which the user wishes to record is too big to fit within the sec-tion defined by the luminous spots, the user can record a number of sub-images, which are put together into a larger image in the same way as in connection with the use of the device in the first mode. Subsequently, the text and image information recorded by the user can be shown either on the display 25 of the device or on an external computer.

The flow chart in Fig. 5 shows how the device is adapted to function in the second mode. In step 501, the extent of the imageable section is indicated by four luminous spots from the LEDs 6 which are projected onto 5 the paper. When the user is satisfied that the right section of the paper has been marked out, he presses a button 27 and, in step 502, the image is then recorded.
The image is recorded in the current image area of the read/write memory with the aid of a plurality of pixels, 10 which can have either grey scale values or colour values.
The user can now choose whether or not he wishes to keep the current image. If the user decides to keep the image, the process continues along the solid line to step 503, in which the image is stored in the memory 23. When the 15 image has been stored, the device indicates, in step 504, that it is ready to scan a new image. If the user does not wish to keep the image, the process continues from step 502 along the dashed line back to step 501 in order for a new image to be recorded.
In the embodiment described above, the device has a single light-sensitive sensor which is used in the first as well as in the second mode. As mentioned above, how-ever, the device can alternatively have a sensor for each mode. Fig. 6 shows schematically how the embodiment in Fig. 1 could be modified to have two sensors. In the embodiment in Fig. 6, the device has a second window 2' in the side of the casing, a second light-sensitive sensor 8' with a two-dimensional sensor surface, and a second lens means 7' which can have a variable focus. The electronic circuitry part is the same as in the embodi-ment according to Fig. 1. When adjusting from one mode to the other, an adjustment of which sensor is activated for imaging takes place as well as an adjustment of from which sensor the electronic circuitry part collects images.

Claims

1. A device for recording information available on an information carrier by means of imaging of the information with the aid of a light-sensitive sensor with a two-dimensional sensor surface, characterised in that the device is adjustable between a first operating mode and a second operating mode, in which images of an area with a first small and a second large size of the information carrier are recorded so that the information on the information carrier can be recorded at different degrees of selectivity, the device in the first operating mode being adapted to be passed over the information while recording a plurality of images with partially overlapping contents, which are put together into a composite image of the information and which each relate to an area with the first small size of the information carrier, and the device in the second operating mode being adapted to image the information by means of at least one image which relates to an area with the second large size of the information carrier.

2. A device according to claim 1, further comprising an indicating means (6) for indicating, at least in the second mode, the size of the area to be imaged.

3. A device according to claim 2, wherein the indicating means are adapted to carry out said indication on the information carrier.

4. A device according to claim 2 or 3, wherein the indicating means (6) further comprise an illumination means which is adapted to indicate the imageable area on the information carrier.

5. A device according to any one of claims 2-4, wherein the indicating means (6) are adapted to generate the indication of the area on the surface which is imageable and consist of a plurality of illuminous spots on the information carrier.

6. A device according to any one of claims 2-5, wherein the indicating means comprise a display.

7. A device according to any one of the preceding claims, wherein said device is capable of being acted upon for continuous adjustment of the size of the area to be imaged between the first small area in the first mode and the second large area in the second mode.

8. A device according to any one of the preceding claims, wherein said device is adapted to utilise a smaller part of the sensor surface in the first mode and a larger part of the sensor surface in the second mode.

9. A device according to any one of claims 1-7, further comprising a lens means (7) which is adapted to project the images of the information carrier onto the sensor surface and which is adjustable for changing the size of said area.

10. A device according to claim 9, wherein the lens means (7) is adjustable between a first position in the first mode and a second position in the second mode.

11. A device according to any one of the preceding claims, further comprising an additional light-sensitive sensor with a two-dimensional sensor surface, one light-sensitive sensor being used in the first mode and the other light-sensitive sensor being used in the second mode.

12. A device according to any one of the preceding claims, wherein the device is adapted to be held at a distance from the information carrier in both the first mode and the second mode.

13. A device according to claim 12, wherein the device is adapted to be held at essentially the same distance from the information carrier in both the first mode and the second mode.

14. A device according to any one of the preceding claims, wherein the device is of the hand-held type.

15. A device according to any one of the preceding claims, further comprising a signal-processing unit (20), which is adapted to utilise the partially overlapping contents of the images for putting together the images into a composite image, no recording being required of the position of the device relative to the surface which is being imaged.

16. A device according to claim 15, wherein the signal-processing unit (20) is adapted to carry out the putting-together of the images horizontally as well as vertically.

17. A device according to claim 13 or 14, wherein the signal-processing unit (20) further comprises software for identifying characters in the composite image and storing the same in the device in character-coded format.

18. A device according to any one of the preceding claims, further comprising a transceiver for wireless communication with an external unit.

19. A device according to any one of the preceding claims, wherein the device also in this second operating mode is adapted to be passed over the information while recording a plurality of images with partially overlapping contents, said images being put together into a composite image of the information and each relating to an area of the second larger size.