JP2003514451A - Digital image processing method - Google Patents

Abstract

SUMMARY The present invention relates to mobile video transmission of documents by facsimile transmission and provides dedicated parameter settings for on-board imaging devices to capture and process characters and / or graphics. The present invention includes the steps of data pipelining RAW image data into fax format data, and transmitting facsimile data as facsimile transmission. This makes it possible to reduce processing power and memory requirements.

Description

Detailed Description of the Invention

TECHNICAL FIELD The present invention relates to digital image processing, and more particularly to digital image processing in portable wireless communication devices.

Background Mobile digital video communications is an area of telecommunications technology that is currently at the center of much research work. Digital video offers numerous advantages over traditional analog systems and supports services such as video telephony and multimedia applications. However, a significant problem with digital video when compared to analog systems is the demand on communication and storage resources. For example,
A bandwidth of about 160 Mbps is required to transmit broadcast quality video comparable to a bandwidth of about 5 MHz for comparable analog video quality. Therefore, in order to enable digital video, it is necessary to reduce the amount of data in digital signals. Data reduction is achieved by using compression techniques to remove redundant data while retaining sufficient information to allow the original image to be reproduced at an acceptable quality level.

DISCLOSURE OF THE INVENTION An application such as a videophone is concerned with capturing, compressing, and transmitting image information such as a scene around a mobile videophone.
Address specific requirements for the capture, processing, and transmission of digital image data representing documents, including, for example, text and graphics.

Accordingly, the present invention provides a method for transmitting an image of a document between a portable wireless communication device comprising an imaging device and a network in which the portable wireless communication device operates, the method comprising the image of the transmitted document. Providing dedicated parameter settings to the imaging device for capturing and processing the character and / or graphics information of the device, converting the image to facsimile format data, and transmitting the facsimile data as a facsimile transmission. Including and

Unlike the images of normal scenes, the images of documents consist mainly of text and / or graphics, which are usually displayed in white and black in documents. The inventors of the present invention have recognized. Moreover, there are generally moving elements within it, eg, in the context of a document, an image has a fixed letter to it when compared to a scene. Thus, the inventor has realized that contrast rather than color is an important aspect for imaging a document for transmission.

From a review of existing digital video imaging devices, the inventor's attention is that such devices are always set up to capture, process, and transmit images of natural landscapes, and thus such devices Means that it is configured to capture and process images that have very different properties than the image of the document. For example, existing digital video imaging devices include hardware and software designed to perform image compression and other processing steps. When recognizing this fundamental difference between scene imaging and document imaging, the inventor has found that it is more appropriate to use different image processing approaches in the context of document imaging. I made a suggestion.

Moreover, since the characters and graphics are displayed in high contrast white and black, many of the processing steps used in color scene imaging are inherently necessary in document imaging. There is no such thing. Therefore,
The inventor concludes that there is room for processing steps and power savings when imaging a document rather than a scene. From this idea,
The inventor has devised a video imaging device concept that configures the video imaging device, in particular to receive and transmit images of documents. More specifically, video imaging devices have a dedicated setup optimized for imaging and transmitting documents by the facsimile standard.

Accordingly, the present invention provides the advantages of saving memory usage and processing power by providing a portable wireless telephone apparatus and method that uses a dedicated setup for an on-board imaging device for document imaging. Be done. For example, the present invention may omit common image compression techniques by optimizing black and white contrast, which may reduce the RAM usage of the device. The present invention takes advantage of the observation that fewer variables are needed to image a document of text and / or graphics as compared to natural scene imaging.

Applicant 'allows remote facsimile transmission from a mobile terminal operating in a cellular network to a facsimile machine having a fax number.
There are also currently available products such as s Communicator. However, fax transmissions from such devices are limited to those either typed directly into a computer file or transferred. The present invention allows a paper document to be captured by an imaging device and sent to a fax number.

In the preferred embodiment, the imaging device is provided with a switch that allows the imaging device to be switched to a state for processing document images. The present invention will now be described in more detail by way of example with reference to the accompanying drawings.

BEST MODE FOR CARRYING OUT THE INVENTION First, FIG. 1 shows basic building blocks of a preferred embodiment of a digital video imaging device according to the present invention. Video imaging device
It has an optical head 100 located at the forefront of the figure, which is linked to the image sensor and the A / D converter 105. The sensor and A / D converter connect to a buffer and interface 110 for the control bus of the video imaging device. It links to the image processing algorithm block 115 which is controlled by the microprocessor of the digital video imaging device. The image processing algorithm block 115 connects to a block 120 containing image setup parameters. The image setup parameter block 120 also links directly to the buffer 110 and then the sensor 1
05 is controlled. The output of the image processing algorithm block 115 connects to a fax document storage memory 125 containing a fax viewer 130. A facsimile engine 135 is connected to the storage memory 125, and the facsimile engine 135 then outputs to the transmission block 140.

The operation of the system is as follows. When a document sent by fax is sent to the video imaging device of a mobile wireless communication device,
The video imaging device detects that it is a page of a document and switches the imaging device to fax transmission mode. Or
The user can switch the imaging device to fax transmission mode. When switched to fax transmission mode, the image sensor is configured based on one or more parameters from the image setup parameter block.

To achieve this, the optimized document imaging parameters are first transferred from the setup parameter block to the buffer and then from there to the high resolution sensor. Dedicated setup parameters are, for example, gain control,
Includes sensitivity adjustment and gamma correction. Therefore, the sensor can be tuned for optimal contrast in black and white, and other parameters can be sacrificed.
For example, color reproducibility may be compromised to increase sensitivity to contrast.

It is usually the pixel exposure time and pixel gain that can be adjusted at the sensor level of a CMOS sensor. Here is a simple example. The black and white text document is
It is a high-contrast image consisting of a large area of white and a sharp transition to dark text. Long exposure times result in pixel saturation from bright pixels to dark pixels. However, short exposure times lead to a reduction in the signal-to-noise ratio. A high pixel gain is required to correct this. Since this is complicated by the conversion of RAW image data (Bayer pattern) to other formats, a greater effect is achieved at that level when combined with sensor level parameter adjustment.

The document that is captured and faxed from the mobile wireless communication device is provided to an optical head of a digital video imaging device of the mobile wireless communication device. The optical head projects the page image of the document onto the image sensor and captures the image. This image is converted from an analog signal to a digital signal by an A / D converter, and the RAW image data is transferred from the sensor to the buffer. The sensor provides image information in an 8x8 pixel format.

The RAW image data is then transferred from the buffer to the image processing algorithm block. In this block, the RAW data is converted from RAW image data to fax format data (more on this process later). Once the image data is converted to fax data, this data is sent to fax storage. The fax viewer can view the fax data of the RAW image data in the fax storage device. When a fax is sent,
The fax is loaded into the fax engine, which prepares to send the fax document. As a result, it becomes possible to transmit from the block labeled with transmission.

From block 140, the document image is sent as a fax transmission to a network such as GSM. The network consists of the public switched telephone network (PSTN), the Internet, or any kind of network group, and in the case of standard facsimile, the fax transmission finally arrives in analog form at the receiving terminal. In the case of a personal computer or a communicator, it will be a digital form.

Returning to block 115 of FIG. 1, details of the image processing algorithm will be described in connection with FIG. Here, in order to convert the RAW image data into fax format data, all color data is first removed from the RAW image data. The contrast is then adjusted (usually enhanced) to increase brightness. It is then converted to bitmap format and then to fax format. The fax format remains in memory and is added in 16 kB increments until the complete image is saved. Once complete, it is transmitted to fax storage transmission memory. The present invention performs data pipelining of RAW image data into a fax format because the data is processed as it comes directly from the camera sensor. Using a data pipelining scheme from the sensor to the RAW format directly to the fax format does not require any form of independent compression step or large amount of intermediate memory.

Therefore, the entire process in the preferred embodiment can be summarized as including the following sequence of steps: 1. 1. The user selects the fax document application from the mobile phone user interface; 2. a step of selecting an image scanning function, 3. Launching a parameter setting application and extracting fax designated parameters. 4. Performing the hardware level setting of the sensor, 5. Scanning the image, 6. Pipelining from RAW format to fax without color interpolation, compression, or features commonly used when taking normal landscapes with the same sensor. 7. Manually add characters, lines, figures, etc. from the user interface if necessary. Select a recipient from the user interface to trigger the sending of the document.

In contrast, currently available methods for sending images of documents by fax require the use of digital cameras and personal computers as follows. In this case the document is photographed and the camera
It automatically converts from a RAW data format, such as Bayer Matrix, to some storage format, such as JPEG, commonly used today, through some intermediate steps such as YUV420. The image is then transferred to the PC via the connecting cable. Now, use a PC software tool such as Coral or PaintShop to convert the image to a bitmap image format, and then use a PC facsimile software tool such as WinFax to convert it to a fax compatible format. Fax.

In a preferred embodiment of the present invention, RAW (such as Bayer matrix)
By including an algorithm that performs the direct conversion from the data format to the fax format, many of the above steps are eliminated. In this way, it is possible to reduce the processing steps. Since this algorithm uses pipelining techniques, when the data is produced by an imaging device, the data is transformed into blocks of data that are much smaller than the entire image. The required memory size is one or two blocks, not the whole image.

A camera that can be used for scanning a document is generally a universal type color camera. Many of these cameras today are equipped with image sensors containing so-called Bayer color mosaic RGB patterns. Thus, a color image is created by interpolating different colors from the RAW mosaic image. However, this interpolation loses some spatial resolution, ie sharpness. Therefore, the optimal solution is to use a RAW Bayer image at the starting point when producing a black and white document image. A simple way to do this is in document imaging mode, assuming the target is white and black, and then providing different white / black thresholds for each color of pixel (R, G, and B). Is.

Further, the sensor exposure time should be adjusted to give maximum contrast to the RAW Bayer image. Therefore, since the data flows directly from the imaging device to the conversion process, the image occupies only the same amount as the facsimile file, thus allowing minimal use of RAM. Prior art methods store the image once in RAW format and then convert it. This requires a maximum of 1,300,000 × 12 bits = 2 Mbytes for one A5 sheet. Fax images are only about 20 kilobytes. In another conventional technique, J
The image is once stored by a common compression method such as PEG, and then converted into a fax. However, this method significantly degrades the image quality of the fax, because it compresses even the most useful information, the high spatial frequency components.

Since the transfer of the whole image is handled on a stage-by-stage basis, it is optimized without user interaction. Photos on white pages generally look very dark and gray. this is,
It can be adjusted by post-processing means, adding brightness and contrast until the image looks like fax quality. In this case, the input is a document and the output format is known to be fax, ie black and white, so the image can be adjusted automatically.

Regarding the conversion of RAW image data into facsimile-ready data, there are several issues that need to be addressed that are related to hardware compliance. This will be described with reference to FIG. In the fax format, the pixels are rectangular, but in the most popular digital still cameras today, image pixel data is based on square pixels. Pixel resolution varies between modes, and this varies only in one dimension, height. For example, one A4 page has a width of 1725 × height 1077 in the standard fax mode, and a width of 17 in the fine fax mode.
25 x height 2156. In the super fine fax mode, the width is 1725 × the height 4301. Furthermore, the aspect ratio of the image varies between modes and is not equal to the general sensor aspect ratio of 1.25: 1 that is prevalent in camera sensors. In addition, in facsimile transmission, the document is presented to the imager in a controlled manner, whereas the camera is carried by the user and is subject to various changes. That is, the document can be provided to the imager at any angle and distance.

A solution to these problems can be obtained by increasing the resolution of image data. High resolution data, higher than any of the facsimile formats, can be sampled or filtered by many techniques to obtain the required fax resolution. This high resolution image can be generated by a high resolution sensor.
This high resolution sensor is already available now. Or, in each case, by scanning the document multiple times, either using a low-resolution sensor to present different parts of the document at different times, or by scanning the entire document several times. , It is possible to generate high resolution images by the combining process.

For example, in standard fax mode, one A5 page (1/2 of A4
) Requires 929K pixels. For example, to enable a 10% image border, 1021K pixel image data is required. If the image is scanned with a 2.3M pixel sensor (which is available today), then more data is available to produce the required fax resolution. However, using a 110K pixel image sensor, approximately 100 multiple images would be needed in this case to produce the required high resolution image. Therefore, in order to store these 100 images and process them all, a relatively large capacity memory (~ 11 Mbytes) and extended processing time are required.

Therefore, from such a background, the effect of the data pipelining solution of the present invention is clear. When an image is generated, it involves the processing of the data for each of the images (approximately 100 as described above), and is concerned with resolving the high resolution image or actually the facsimile image. By storing only the required results resulting from, only the data needed to store one A5 fax page is needed. This is about 20 Kbytes.

An advantage of the present invention is that a business user carrying a mobile device equipped with the present invention can:
It is possible to make a copy of a document at any location, and to save, send or file it immediately with little effort. The present invention may be embodied in other specific forms without departing from its essential characteristics. Digital imaging devices can be installed only when needed,
It can also be an optional removable accessory. Therefore, reference should be made to the appended claims and other general description within this specification, rather than the above description, as to the scope of the invention.

Furthermore, each feature disclosed in this specification (which term includes the claims) and / or shown in the drawings may be incorporated into the invention without regard to other disclosed and / or exemplary features. good. In this regard, the invention is hereby expressly set forth regardless of whether it is related to the claims, or whether it mitigates any or all of the related problems. , Or any new feature or combination of features disclosed in any general form. The accompanying abstract filed herein is hereby incorporated by reference.

[Brief description of drawings]

FIG. 1 is a block diagram showing components of an image processing apparatus according to a preferred embodiment of the present invention.

FIG. 2 schematically shows a flowchart for controlling the image processing apparatus of FIG.

[Figure 3] It is a figure showing the comparison of the resolution of A4 page and A5 page.

─────────────────────────────────────────────────── ─── Continued front page    (81) Designated countries EP (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, I T, LU, MC, NL, PT, SE, TR), OA (BF , BJ, CF, CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG), AP (GH, G M, KE, LS, MW, MZ, SD, SL, SZ, TZ , UG, ZW), EA (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), AE, AG, AL, AM, AT, AU, AZ, BA, BB, BG, BR, BY, B Z, CA, CH, CN, CR, CU, CZ, DE, DK , DM, DZ, EE, ES, FI, GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS, J P, KE, KG, KP, KR, KZ, LC, LK, LR , LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, MX, MZ, NO, NZ, PL, PT, R O, RU, SD, SE, SG, SI, SK, SL, TJ , TM, TR, TT, TZ, UA, UG, US, UZ, VN, YU, ZA, ZW (72) Inventor Janne Haabist             Finland Ruutana FIN-36110               Kebatoni Tinkya 10 A 2 F-term (reference) 5C062 AA02 AA11 AA12 AB38 AC24                       AC29 AC41 BA04 BB03                 5K067 BB04 BB21 DD52 GG01

Claims (5)

[Claims] 1. A method for transmitting an image of a document between a portable wireless communication device having an imaging device and a network in which the portable wireless communication device operates, wherein the characters and / or graphics of the document image to be transmitted. Providing dedicated parameter settings to the imaging device for capturing and processing information, converting the image to facsimile format data, and transmitting the facsimile data as a facsimile transmission. And how to. 2. The method of claim 1, wherein said step of providing dedicated parameter settings to said imaging device for capturing and processing character and / or graphics information comprises at least one for black and white imaging. A method comprising optimizing a sensor parameter. 3. The method of claim 2, wherein the at least one sensor parameter comprises pixel exposure time. 4. The method of claim 2 or 3, wherein the at least one sensor parameter comprises pixel gain. 5. An imaging device, a hardware setup mode for the imaging device dedicated to processing black and white characters and / or graphics to provide image data, and a fax capable of wireless transmission. And a means for pipelining the image data directly in a format.