JP2000322602A - Image processing apparatus and method, and medium - Google Patents

Abstract

(57) [Summary] [Problem] To express a visual effect that fuses a real space and a virtual space. SOLUTION: In step S21, a 3D application receives a 2D code ID as an argument supplied from an FA program. In step S23, 3
The D application reads three-dimensional object data and the like corresponding to the received 2D code ID. Step S
At 24, the 3D application receives the information of the coordinates of the 2D code on the finder window supplied from the FA program. In step S25, the 3D
The application calculates the coordinates on the finder window for displaying the 3D object using the two-dimensional coordinates of the received 2D code. In step S26, the 3D application displays a 3D object corresponding to the calculated coordinates in the finder window coordinate system.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, a method, and a medium, and more particularly to identification information obtained from a two-dimensional barcode and recognition information such as information on a position where the two-dimensional barcode is attached. The present invention relates to an image processing apparatus and method, and a medium, which are capable of acquiring various information and executing various processes based on such information.

[0002]

2. Description of the Related Art The present applicant has already disclosed a system for reading a two-dimensional barcode (hereinafter abbreviated as 2D code) pattern as shown in FIG. It has been proposed as Hei 10-184351.

In the example shown in FIG. 1, a 2D code 201 is attached (including printing) to the right side of a business card card (object) 200. As shown in FIG. 2, the 2D code 201 includes cells within a rectangular area having a length of 9.5 blocks in the vertical direction and a length of 7 blocks in the horizontal direction, with one block as one unit. The part A and the logo part B are arranged one block apart. In the cell part A, both the vertical direction and the horizontal direction are within the range of a square having a length of 7 blocks,
Rectangular cells are two-dimensionally patterned and arranged. In the logo portion B, a large rectangular cell having a length of 1.5 blocks in the vertical direction and a length of 7 blocks in the horizontal direction is arranged, for example, for the name of an object or for advertising. Mark (logo) is written.

The personal computer 1 is a notebook computer, and a CCD is provided at the center of the upper end of the display unit 3.
(Charge Coupled Device) A video camera 23 is provided. The 2D code 201 on the object 200 is
An image is captured by the CCD video camera 23 of the personal computer 1, the pattern of the 2D code 201 is recognized from the image data obtained as a result, and a predetermined process corresponding to the pattern is executed.

[0005]

Conventionally, the above-described system for identifying a 2D code has not been used in combination with an application for displaying a three-dimensional image.

[0006] The present invention has been made in view of such a situation. By superimposing and displaying a three-dimensional image at the position of a 2D code identified by imaging, a visual space that combines a real space and a virtual space is provided. The effect can be expressed.

[0007]

An image processing apparatus according to claim 1, wherein said storage means stores three-dimensional image data in association with a two-dimensional barcode pattern, and an object to which said two-dimensional barcode is attached. Imaging means for imaging the object, detection means for detecting a two-dimensional barcode from the image of the object imaged by the imaging means,
Detecting means for detecting the position of the two-dimensional barcode; identifying means for identifying the pattern of the two-dimensional barcode detected by the detecting means; and two-dimensional bar identified by the identifying means in the three-dimensional image data stored in the storage means. Display control means for controlling the display corresponding to the code pattern so as to be superimposed and displayed on the position of the two-dimensional barcode detected by the detection means of the image of the object imaged by the imaging means. .

[0008] The image processing apparatus according to a third aspect of the present invention is characterized in that the image processing apparatus further comprises a reproducing means for reproducing a sound file corresponding to the two-dimensional barcode pattern identified by the identifying means.

According to a fourth aspect of the present invention, there is provided an image processing method comprising: a storage step of storing three-dimensional image data in association with a two-dimensional barcode pattern; and an imaging step of capturing an object to which the two-dimensional barcode is attached. A detecting step of detecting a two-dimensional barcode from an image of the object captured in the imaging step, and a detecting step of detecting a position of the two-dimensional barcode detected in the detecting step;
An identification step of identifying a pattern of the two-dimensional barcode detected in the detection step; and a three-dimensional image data stored in the storage step corresponding to the two-dimensional barcode pattern identified in the identification step in the imaging step. A display control step of performing control so as to superimpose and display the two-dimensional barcode detected at the detection step of the image of the captured object image.

According to a fifth aspect of the present invention, there is provided a program for a medium, comprising:
A storage step of storing two-dimensional image data in association with a two-dimensional barcode pattern, an imaging step of imaging an object to which the two-dimensional barcode is attached, and a two-dimensional bar out of the image of the object imaged in the imaging step A detecting step of detecting a code, a detecting step of detecting a position of the two-dimensional barcode detected in the detecting step, an identifying step of identifying a pattern of the two-dimensional barcode detected in the detecting step, One of the two-dimensional image data corresponding to the pattern of the two-dimensional barcode identified in the identification step is superimposed and displayed on the position of the two-dimensional barcode detected in the detection step of the image of the object imaged in the imaging step. And a display control step of performing control.

An image processing apparatus according to claim 1, wherein
In the image processing method described in the above, and the medium program described in the claim 5, the three-dimensional image data is stored in association with the pattern of the two-dimensional barcode, and the object to which the two-dimensional barcode is attached is imaged. Then, a two-dimensional barcode is detected from the captured image of the object. Further, the position of the detected two-dimensional barcode is detected, the detected two-dimensional barcode pattern is identified, and the detected two-dimensional barcode pattern corresponds to the identified two-dimensional barcode pattern in the stored three-dimensional image data. The object is controlled to be displayed so as to be superimposed on the position of the detected two-dimensional barcode in the image of the captured object.

[0012]

3 to 8 show examples of the configuration of a notebook personal computer to which the present invention is applied. The personal computer 1 is a mini-notebook type personal computer.
The display unit 3 is configured to be openable and closable with respect to the main body 2. 3 is an external perspective view showing a state in which the display unit 3 is opened with respect to the main body 2, FIG. 4 is a plan view of FIG.
Is a left side view showing a state in which the display unit 3 is closed with respect to the main body 2, FIG. 6 is a right side view showing a state in which the display unit 3 is opened 180 degrees with respect to the main body 2, and FIG. 8 is a bottom view of FIG.

On the upper surface of the main body 2, there is arranged a keyboard 4 which is operated when inputting various characters and symbols. At a substantially center of the keyboard 4, when moving a mouse cursor or the like, like a joystick. Operated,
A stick-type pointing device (hereinafter, abbreviated as a stick) 5 is provided. The left button 31 and the right button 33 are operated in the same manner as the left and right buttons of a mouse in a normal personal computer. The center button 32 is used when operating the scroll bar without moving the cursor to the scroll button. Further, a speaker 8 for outputting sound and a CCD video camera 23 provided on the display unit 3 are provided on the upper surface of the main body 2.
Is provided with a shutter button 10 which is operated when capturing an image with.

A claw 13 is provided at an upper end portion of the display section 3. As shown in FIG. 5, when the display section 3 is closed with respect to the main body 2, the main body 2 is located at a position opposed to the claw 13. Is provided with a hole 6 into which the claw 13 is fitted. A slide lever 7 is provided on the front surface of the main body 2 so as to be movable in parallel with the front surface. The slide lever 7 engages with a claw 13 fitted in the hole 6 to lock and unlock. I can do it. By releasing the lock, the display unit 3 can be rotated with respect to the main body 2. Microphone 2 next to claw 13
4 is attached. As shown in FIG. 8, the microphone 24 can also collect sound from the back.

A programmable power key (PPK) 9 is also provided on the front of the main body 2. An exhaust hole 11 is provided on the right side surface of the main body 2 as shown in FIG.
As shown in FIG.
Is provided. Further, on the right side of the exhaust hole 11, a PC
MCIA (Personal Computer Memory Card InternationalAs
A slot 12 for inserting a sociation card (PC card) is provided.

An LCD for displaying an image is provided on the front of the display unit 3.
A (Liquid Crystal Display) 21 is provided, and an imaging unit 22 is provided at the upper end thereof so as to be rotatable with respect to the display unit 3. That is, the imaging unit 22 is
It is possible to rotate to any position within a range of 180 degrees between the same direction as 21 and the opposite direction (backward direction). A CCD video camera 23 is attached to the imaging unit 22.

A power lamp is provided on the lower body side of the display unit 3.
PL, battery lamp BL, message lamp ML, other LEDs
(Light emitting diode).
As shown in FIG. 5, a power switch 40 is provided on the left side surface of the main body. As shown in FIG. 7, an adjustment ring 25 for adjusting the focus of the CCD video camera 23 is provided on the upper side surface of the display unit 3. Further, as shown in FIG. 8, a lid 2 for covering an opening for mounting an additional memory in the main body 2 is provided on the bottom of the main body 1.
6. A small hole 41 is provided for inserting a pin for removing the lock nail of the lid 26.

FIG. 9 shows the internal configuration of the personal computer 1. CPU (Central
A processing unit 52, a PC card 53 inserted as needed, a RAM (Random Access Memory) 54, and a graphic chip 64 are connected. Internal bus 51
Is connected to an external bus 55, which includes a hard disk drive (HDD) 56, an I / O (input / output) controller 57, a keyboard controller 58,
A stick controller 59, a sound chip 60, a modem 61, an LCD controller 62, and the like are connected.

The CPU 52 is a controller that controls various functions, and the PC card 53 is appropriately mounted when adding optional functions. In the RAM 54, for example, at the time when the activation is completed, for example, the OS 56C (basic program software), application program software, and the like are transferred from the HDD 56 and stored.

The OS 56C (basic program software) controls the basic operation of a computer represented by Windows 98 (trademark).

On the other hand, as shown in FIG. 10, an e-mail program 56A and an auto-pilot program 5
6B, operation system software (OS) 56
C, two-dimensional barcode database (hereinafter abbreviated as 2D code database) 56D, finder application program (hereinafter abbreviated as FA program)
56E, a two-dimensional barcode associate application program (hereinafter abbreviated as 2D code program) 56F, and a management tool program 56G are stored.

The 2D code database 56D has a 2D code database.
In addition to the code identification number (hereinafter referred to as 2D code ID), an executable file (program) (for example, a file having an extension such as .exe, .com) or a document file set in association with the 2D code (For example, a file having an extension such as .txt, .html, or .jpg) is stored. This executable is 2
An execution file (hereinafter, referred to as a 2D code-related execution file) set to be automatically started when the D code is identified. The document file is a document file set to be automatically opened when a 2D code is identified (hereinafter referred to as a 2D code-related document file). In the following, 2
When it is not necessary to particularly distinguish a D-code related execution file and a 2D code-related document file, both are collectively described as a 2D code-related file.

The 2D code database 56D also includes
Information describing the contents of the 2D code related file (hereinafter referred to as
Information associated with the 2D code, such as 2D code memo information) and the title of a 2D code related file, is stored. In this case, the 2D code database 56D is a DDL (Dynamic Link Library).

The FA program 56E extracts, for example, image data of a 2D code portion from the image data stored in the VRAM 81A of the graphic chip 81, and automatically executes a 2D code related file corresponding to the extracted 2D code. It is a program that causes FA Program 56E
Also developed a 2D code program 56F.
It has an API (Application Programming Interface).

The 2D code program 56F is a 2D code-related executable file that can exchange data with the FA program 56E via the API of the FA program 56E.

The management tool program 56G is a program for registering data necessary for executing the FA program 56E and the 2D code program 56F. The management tool program 56G or a predetermined
It has a GUI (Graphical User Interface).

OS 56 in hard disk drive 56
C, FA program 56E, etc.
0 is operated and the OS 56C is started (boot-up),
Further, in the process of starting the FA program 56E,
These programs are sequentially transferred and stored in the RAM 54.

Returning to FIG. 9, the I / O controller 57
Microcontroller, I / O interface, CPU, R
It is composed of AM, ROM, etc. The I / O controller 57 drives lamps such as a power lamp PL, a battery lamp BL, and a message lamp ML. The I / O controller 57 includes a power switch 40, a programmable power key 9, a half-press switch 67, and a full-press switch 6.
8. The operation of the reversing switch 69 or the like is detected.

The power switch 40 is operated when the power is turned on or off. The half-press switch 67 is turned on when the shutter button 10 is half-pressed, and the full-press switch 68 is turned on when the shutter button 10 is fully pressed. The inversion switch 69 is connected to the imaging unit 2
2 is rotated 180 degrees (CCD video camera 23
It is turned on when it is rotated in the direction of imaging the opposite side of the CD 21). An RTC (Real-Time Clock) 70 always performs a clocking operation and outputs the current time to the I / O controller 57. The I / O controller 57 is backed up by a battery 71.

A keyboard controller 58 connected to the external bus 55 controls an input from the keyboard 4. The stick controller 59 controls input of the stick 5. The sound chip 60 receives an input from the microphone 24,
To supply an audio signal. The modem 61 can be connected to a communication network 92 such as the Internet, a mail server 93, a WWW server 94, and the like via a public telephone line 90 and an Internet service provider 91.

The image data captured by the CCD video camera 23 is input to the graphic chip 64 connected to the internal bus 51 after being processed by the processing unit 66. The graphic chip 64 receives the CC through the processing unit 66.
The video data input from the D video camera 23 is stored in the built-in VRAM 65, and is read out as appropriate.
Output to LCD controller 62. LCD controller 62
Outputs the image data supplied from the graphic chip 64 to the LCD 21 for display. Backlight 63
Illuminates the LCD 21 from behind.

Next, 2D code recognition processing by the FA program 56E will be described with reference to the flowchart of FIG. In the 2D code recognition process, the FA program 56E stored in the hard disk drive 56 executes the RA
The processing is started by being transferred to and stored in the M54 and executed by the CPU 52.

After the user executes the program starting operation, the surface of the object 200 on which the 2D code 201 is attached is turned to the CCD video camera 23, and the image is captured by the CCD video camera 23, and a predetermined Is performed. In step S1, the FA program 56E superimposes the normal mode image (the aiming image indicating the position of the 2D code 201) on the image of the image data processed by the processing unit 66 and drawn on the VRAM 65 of the graphic chip 64. The displayed image is displayed on the LCD 21 as a finder window (FIG. 13, but FIG. 13 shows a hyper search mode).

In step S2, the FA program 56
E determines whether or not a predetermined program end operation has been performed by the user. If it is determined that the end operation has not been performed, the process proceeds to step S3.

In step S3, the FA program 56
E captures the image data of the entire finder window (320 × 240 pixels) (FIG. 12A) (transfers the image data from the VRAM 65 to the RAM 54).

In step S4, the FA program 56
E determines whether or not the shutter button 10 has been pressed (whether or not the hyper search mode has been set). If it is determined that the shutter button 10 has been pressed, the process proceeds to step S5. In step S5, the FA program 56E displays an image indicating that the hyper search mode has been set in the finder window. That is, as shown in FIG. 13, the FA program 56E enlarges the image in the center of the finder window and
A rectangular frame (80 × 60 pixels, FIG. 12 (B)) indicating the range to detect 1 and the character “Hyper Search” are displayed in the lower right part. By displaying such a rectangular frame, the user can easily adjust the 2D code 201 to an appropriate position.

In step S6, the FA program 56
E generates an image (320 × 240 pixels, FIG. 12C) obtained by enlarging the image data of the central portion (80 × 60 pixels) of the finder window four times vertically and horizontally.

In step S7, the FA program 56
E is an image of 320 × 240 pixels (the image captured in step S3 or the image enlarged in step S6)
Of the 2D code 201 is detected.

In step S8, the FA program 56
E determines whether or not the 2D code 201 has been detected in step S7. If it is determined that the 2D code 201 has not been detected, a message to that effect is displayed and the CCD video camera of the object 200 on which the 2D code 201 is attached is displayed to the user. The user is prompted to adjust the distance and angle with respect to 23. Then, step S1
And the subsequent processing is repeated. Thereafter, if it is determined that the 2D code 201 has been detected, the process proceeds to step S9.
Proceed to.

In step S9, the FA program 56
E identifies the pattern of the 2D code 201, the ID indicated by the pattern (2D code ID), and the four vertices P _i (i = 0, i = 0, 2) of the 2D code 201 in the finder window.
Two-dimensional coordinates ( _Pix , _Piy ) indicating the position of (1, 2, 3) and
The focal length of the CCD video camera 23 (hereinafter, also appropriately referred to as focal) is acquired. However, the upper left vertex of the 2D code 201 is P ₀ , the upper right vertex is P ₁ , the lower left vertex is P ₂ , and the lower right vertex is P ₃ .

In step S10, the FA program 5
6E determines whether or not the acquired 2D code ID is already registered (whether or not the 2D code ID acquired in step S9 is in the 2D code database 56D stored in the hard disk drive 56). If it is determined that the information has already been registered, the process proceeds to step S11. In step S11, the FA program 56E executes a registered process. For example, FA program 56E
Is a 2D code program 56 corresponding to the 2D code ID.
A three-dimensional object display application program (hereinafter abbreviated as 3D application) as F is started.

The 3D application is an application program for displaying a three-dimensional object on a finder window, and displays the three-dimensional object data (used in Microsoft Direct3D) in association with a 2D code ID supplied as an argument. X file which is described as a standard file format), sound data to be reproduced (WAVE file), XYZ coordinates for displaying a three-dimensional object on the finder window (2D code 20 on the finder window)
1, a coordinate system having a predetermined portion as the origin, and information such as animation speed.

The processing of the 3D application will be described with reference to the flowchart of FIG. Step S
At 21, the 3D application receives the 2D code ID as an argument supplied from the FA program 56E. In step S22, the 3D application generates a 3D object display window on the finder window.

In step S23, the 3D application reads three-dimensional object data or the like corresponding to the 2D code ID supplied in step S21. In step S24, 3D application receives a focal length of the two-dimensional coordinates (P _ix, P _iy) and CCD video camera 23 of the 2D code 201 on the finder window supplied (focal) from FA program 56E.

In step S25, the 3D application uses the two-dimensional coordinates of the 2D code 201 on the finder window received in step S24 to perform 3D application.
The coordinates on the finder window for displaying the D object are calculated.

Specifically, 2D in a real three-dimensional coordinate system
The coordinates of the four vertices of the code 201 are ω _i, and the 2D code 20 in the camera coordinate system having the origin at the lens surface of the CCD video camera 23 facing the −Z axis direction of the real three-dimensional space
1, the coordinates of the four vertices of the 2D code 201 on the finder window (hereinafter referred to as the finder window coordinate system) are P _{i
When i} (i = 0, 1, 2, 3), the coordinates ω _{i in} the real three-dimensional coordinate system and the coordinates C _i in the camera coordinate system are expressed by the following equation (1).
Are related as follows.

C _i = R * ω _i + T (1) where R is a rotation matrix representing the rotation of the camera coordinate system with respect to the real three-dimensional coordinate system, and T is the movement vector of the CCD video camera 23 ( (Distance between the origin of the real three-dimensional coordinate system and the origin of the camera coordinate system). First, the rotation matrix R and the movement vector T are calculated.

The coordinates P _{i in the} finder window coordinate system and the coordinates C _i in the camera coordinate system have the relationship shown in the following equation (2).

[0049] _{P ix = -focal * C ix /} C iz P iy = -focal * C iy / C iz P iz = -focal * C iz / C iz = -focal ··· (2) Here, the vector ω ₃₀ , ω ₂₀ and ω ₁₀ are defined as shown in the following equation (3) using the coordinates ω _i of the real three-dimensional coordinate system.

Ω ₃₀ = ω ₃ −ω ₀ ω ₂₀ = ω ₂ −ω ₀ ω ₁₀ = ω ₁ −ω ₀ (3) Note that the coordinates ω ₀ , ω ₁ , ω ₂ , and ω ₃ are on the same plane. S and t that satisfy ω ₃₀ = sω ₁₀ + tω ₂₀ (4) However, in this case, s = t = 1 since the coordinates ω ₀ , ω ₁ , ω ₂ , and ω ₃ are coordinates of four vertices of a rectangle.

The vectors ω ₃₀ , ω ₂₀ , ω ₁₀ are the coordinates C _i of the camera coordinate system and the vectors C ₃₀ , C ₂₀ , C ₁₀
Can be represented by using

Ω ₃₀ = C ₃ −C ₀ = C ₃₀ ω ₂₀ = C ₂ −C ₀ = C ₂₀ ω ₁₀ = C ₁ −C ₀ = C ₁₀ (5) By the way, the coordinates C in the camera coordinate system _i can be expressed as k _i (scalar amount) times the coordinate P _i of the finder window coordinate system as shown in the following equation (6).

C _i = k _i * P _i (i = 0, 1, 2, 3) (6) Here, equation (5) is substituted into equation (4), and the following equation (7) is obtained. obtain.

C ₃ −C ₀ = s (C ₁ −C ₀ ) + t (C ₂ −C ₀ ) (7) By substituting the equation (6) into the equation (7), the following equation (8) is obtained. obtain.

K ₃ * P ₃ −k ₀ * P ₀ = s (k ₁ * P ₁ −k ₀ * P ₀ ) + t (k ₂ * P ₂ −k ₀ * P ₀ ) (8) (8) is solved for _{P 3, P 0 k 0 (} 1-s-t) / k 3 + P 1 s * k 1 / k 3 + P 2 t * k 2 / k 3 = P 3 ··· (9 ).

When equation (9) is expressed by a determinant, the following equation (1) is obtained.
0).

[0057]

(Equation 1)

(10) Here, (P ₀ P ₁ P ₂ ) on the left side of the equation (10) is set to V, and its inverse matrix is multiplied by V−1 to both sides of the equation (10) from the left side. Then, the following equation (11) is obtained.

[0059]

(Equation 2)

(11) Further, V-1 * P ₃ on the right side of the equation (11) is expressed by (a ₀ a ₁ a
₂ ), the following equation (12) is obtained.

K ₀ = a ₀ * k ₃ / (1− _st ) k ₁ = a ₁ * k ₃ / s k ₂ = a ₂ * k ₃ / t (12) By the way, | ω ₃₀ Is the length of the diagonal line of the 2D code 201, and | ω ₃₀ | = | k ₃ * P ₃ −k ₀ * P ₀ | (13) If Equation (12) is applied to ₀ , | ω ₃₀ | = | k ₃ * P ₃ −a ₀ * k ₃ * P ₀ / (1−st) | = k ₃ | P ₃ −a ₀ * P ₀ / (1−st) | (14) is obtained.

| Ω ₃₀ | and | P ₃ −a ₀ * in equation (14)
Since the value of P ₀ / (1−st) | is known, the scalar quantity k ₃ can be obtained. Further, the obtained scalar quantity k ₃ is substituted into equation (12), and the scalar quantities k ₀ , k ₁ , k ₂
Get.

The resulting scalar quantity k _i (i = 0, 1, 2,
Substituting 3) into equation (6), the coordinates C _i (i
= 0, 1, 2, 3).

Here, the rotation matrix R is a vector ω₃₀, Ω
_TenTo the vector C₃₀, C _TenAnd the matrix to convert to
Can be calculated. Further, the movement vector T is given by the following equation (1).
And T = 演算 (Ci−R * ωi) / 4.

Using the rotation matrix R and the movement vector T obtained as described above, the coordinate ω in the real three-dimensional coordinate system is obtained.
_i is converted to three-dimensional coordinates in the camera coordinate system, and coordinates P _i in the finder window coordinate system are calculated.
The process proceeds to step S26 in FIG.

In step S26, the 3D application, as shown in FIG. 15, uses the coordinates corresponding to the coordinates of the finder window coordinate system calculated in step S25.
Display the D object.

By such 3D application processing, a composite image can be displayed as if a 3D object exists in the camera image displayed in the finder window.

The 3D object to be displayed is Mi
Since it is a standard file format (X file) used in crisoft Direct3D, it can be created by the user using any software. Also, 3D
Animations are also supported. In addition, at the same time, a sound file (WAVE file) can be specified and played back.
It is possible to realize an expression as if the object is talking.

Returning to the description of FIG. If it is determined in step S10 that the acquired 2D code ID is not already registered, the process proceeds to step S12, and the FA program 56E executes a process of receiving an input of a setting for associating the 2D code ID with a predetermined process. And step S
Return to 1.

In step S2, if the FA program 56E determines that the end operation has been performed by the user, the FA program 56E ends the 2D code recognition process.

If it is determined in step S4 that the shutter button 10 has not been pressed, the FA program 56E skips steps S5 and S6. In this case, in step S7, a 2D code is detected from the image captured in step S3.

Although the above description does not refer to whether the shutter button 10 is half-pressed or full-pressed in step S4, for example, the shutter button 10 is set to a half-pressed state. , The hyper search mode may be set, and the 2D code may be detected when fully pressed.

As described above, by detecting the 2D code 201 by enlarging the central portion of the captured image,
The distance from the CCD video camera 23 capable of recognizing the D code 201 can be increased from about 10 cm when the image is not enlarged to twice as large as about 20 cm.

In the present embodiment, since the process of enlarging the captured image places a heavy burden on the CPU 52, the enlarging process is executed only when the shutter button 10 is pressed. This makes it possible to improve the recognition rate only when necessary (when the 2D code cannot be recognized in the normal mode).

By the way, the following can be considered as an application of the 3D application processing.

For example, a trading card on which a picture and a 2D code of a popular character of a television broadcast or a comic magazine are printed, and animation data of the character are sold as a set. When the 2D code of the trading card is recognized, the animation is displayed.

For example, a picture book for children in which a 2D code is printed at the corner of a page, and 3D image data of characters in the picture book and sound files such as dialogue are sold as a set. When the 2D code is recognized, the character 3 is displayed on the page of the picture book.
By causing the D image to be displayed and the dialogue to be reproduced, a visual effect is created such that the characters of the picture book jump out of the picture book. Furthermore, the story of the picture book may be advanced by displaying an animation of the characters.

For example, the 2D code may be recognized at various angles, and the angle at which the D object is displayed may be changed according to the angle. Utilizing such features, it is also possible to apply the present invention to a teaching material for displaying a molecular structure model in chemistry.

For example, the 2D code may be applied to a New Year's card or a greeting card. That is, an arbitrary message 3
A D object is created by an arbitrary application program, the data is supplied in advance to a destination to which a New Year's card or the like is sent, and then a New Year's card or the like printed with a 2D code is sent.
When the 2D code such as the New Year's card is recognized, the 3D object is displayed.

The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processes is executed by software, a program constituting the software is installed in a computer incorporated in the image processing apparatus as dedicated hardware, or various functions are installed by installing various programs. Is installed in, for example, a general-purpose personal computer or the like that can execute the program.

Next, with reference to FIG. 16, a description will be given of a medium used to install a program for executing the above-described series of processes in a computer and make the computer executable.

The program is, as shown in FIG.
The hard disk 102 (corresponding to the hard disk built in the hard disk drive 56 in FIG. 9) as a recording medium built in the computer 101 or the semiconductor memory 103 can be provided to the user in a state where it is installed in advance.

Alternatively, the program is executed as shown in FIG.
As shown in FIG.
1, CD-ROM (Compact Disc-Read Only Memory) 112,
MO (Magneto Optical) disc 113, DVD (Digital V
ersatile Disc) 114, a magnetic disk 115, a semiconductor memory 116, or other such storage media, which can be temporarily or permanently stored and provided as package software.

Further, as shown in FIG. 16C, the program is transmitted from the download site 121 to the satellite 1 by radio.
The data can be transferred to the computer 123 via the network 22 or can be transferred to the computer 123 via a network 131 such as a local area network or the Internet by wire or wirelessly, and stored in a built-in hard disk or the like.

The medium in this specification means a broad concept including all these media.

In this specification, the step of describing a program provided by a medium is not limited to processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. This also includes processes executed individually.

In this specification, the system is
It represents the entire device composed of a plurality of devices.

[0088]

As described above, according to the image processing apparatus according to the first aspect, the image processing method according to the fourth aspect, and the program of the medium according to the fifth aspect, the image processing apparatus can select one of the captured images. A two-dimensional barcode is detected, its pattern is identified, and a three-dimensional image corresponding to the pattern is displayed so as to be superimposed on the image of the captured object. Can be expressed.

[Brief description of the drawings]

FIG. 1 is a diagram for describing a system for recognizing a two-dimensional barcode.

FIG. 2 is a diagram illustrating specifications of a two-dimensional barcode.

FIG. 3 is a perspective view of a personal computer 1 to which the present invention is applied.

FIG. 4 is a plan view of the personal computer 1 of FIG.

FIG. 5 is a side view of the personal computer 1 of FIG.

FIG. 6 is a side view of the personal computer 1 of FIG.

7 is a front view of the personal computer 1 of FIG.

8 is a bottom view of the personal computer 1 of FIG.

FIG. 9 is a block diagram showing an electrical configuration of the personal computer 1 of FIG.

10 is a diagram showing data recorded on an HDD 56 in FIG.

FIG. 11 is a flowchart illustrating a 2D code recognition process.

FIG. 12 is a diagram for explaining a hyper search mode.

FIG. 13 is a photograph of a halftone image in which a finder window in the hyper search mode is displayed on the LCD 21.

FIG. 14 is a flowchart illustrating 3D application processing.

FIG. 15 is a photograph of a halftone image in which an image obtained by synthesizing a 3D object on a finder window is displayed on an LCD 21;

FIG. 16 is a diagram for describing a medium used to install a program in a computer and make the program executable.

[Explanation of symbols]

1 personal computer, 10 shutter button, 23 CCD video camera, 52 CPU, 56
HDD, 56E finder application program, 56F 2D barcode associate application program, 200 objects, 20
1 2D barcode

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Junichi Koraimoto 3-14-13 Higashi Gotanda, Shinagawa-ku, Tokyo In-house within Sony Computer Science Laboratory (72) Inventor Takahiko Sueyoshi 6 Kita-Shinagawa, Shinagawa-ku, Tokyo 7-35 Chome Sony Corporation F-term (reference) 5B035 AA00 BB00 BB01 BC05 5B050 BA08 BA09 BA11 DA06 EA19 FA02 FA10 5B057 CA12 CB13 CE08 DA07 DA17 DB02 DB08 DC08 5B072 AA07 CC24 DD15 DD22 DD23 GG09

Claims (5)

[Claims] 1. An image processing apparatus for recognizing a pattern of a two-dimensional bar code, comprising: storage means for storing three-dimensional image data in association with the pattern of the two-dimensional bar code; Imaging means for imaging the object detected by the imaging means; detecting means for detecting the two-dimensional barcode from the image of the object imaged by the imaging means; and detecting the position of the two-dimensional barcode detected by the detecting means. Detecting means; identifying means for identifying the pattern of the two-dimensional barcode detected by the detecting means; and the two-dimensional barcode identified by the identifying means among the three-dimensional image data stored in the storage means. The secondary corresponding to the pattern corresponding to the pattern detected by the detection unit of the image of the object imaged by the imaging unit. A display control means for controlling the display so as to be superimposed on the position of the original barcode. 2. The method according to claim 1, wherein the three-dimensional image data is a Microsoft
2. An X file corresponding to an application program interface of rect3D.
An image processing apparatus according to claim 1. 3. The apparatus according to claim 1, further comprising a reproducing unit that reproduces a sound file corresponding to the pattern of the two-dimensional barcode identified by the identifying unit.
An image processing apparatus according to claim 1. 4. An image processing method for an image processing apparatus for recognizing a pattern of a two-dimensional barcode, comprising: a storage step of storing three-dimensional image data in association with the pattern of the two-dimensional barcode; An imaging step of imaging an object to which a code is attached; a detection step of detecting the two-dimensional barcode from an image of the object imaged in the imaging step; and a detection step of the two-dimensional barcode detected in the detection step. A detecting step of detecting a position; an identifying step of identifying the pattern of the two-dimensional barcode detected in the detecting step; and the identifying step of the three-dimensional image data stored in the storing step. An image corresponding to the pattern of the two-dimensional barcode is captured in the image capturing step A display control step of performing control so as to superimpose and display the image of the object on the position of the two-dimensional barcode detected in the detection step. 5. A program for image processing for recognizing a pattern of a two-dimensional bar code, comprising: a storage step of storing three-dimensional image data in association with the pattern of the two-dimensional bar code; An imaging step of imaging an object to which a code is attached; a detection step of detecting the two-dimensional barcode from an image of the object imaged in the imaging step; A detecting step of detecting a position; an identifying step of identifying the pattern of the two-dimensional barcode detected in the detecting step; and the identifying step of the three-dimensional image data stored in the storing step. The two-dimensional barcode corresponding to the pattern was imaged in the imaging step A display control step of performing control so as to superimpose and display the two-dimensional barcode on the position of the two-dimensional barcode detected in the detection step of the image of the object.