JP6701940B2 - Remote control system, image processing device, remote control method, and remote control program - Google Patents

Description

  The present invention relates to a remote control system, an image processing device, a remote control method, and a remote control program, and more particularly, to display an operation screen for controlling the image processing device on the portable information device to remotely control the image processing device with the portable information device. The present invention relates to a remote control system for controlling, an image processing apparatus thereof, a remote control method and a remote control program executed by the image processing apparatus.

  2. Description of the Related Art In recent years, a technique for remotely operating a multifunction peripheral (hereinafter referred to as “MFP”) with a mobile information device such as a smartphone or a tablet terminal has been known. It is possible to display an operation screen for operating the MFP on the portable information device and remotely operate the MFP with the portable information device. For example, in Japanese Unexamined Patent Application Publication No. 2012-93940, the operated terminal receives from the display terminal a display screen control unit that creates a mobile terminal screen in which the operated terminal screen is corrected based on the display attribute information of the mobile terminal. The mobile terminal includes an encoding unit that encodes the screen of the mobile terminal by the encoding method described above, and an operation processing unit that executes the operation specified by the position information received from the mobile terminal. When the decoding unit that decodes the terminal screen and outputs it to the touch screen and the touch operation on the touch screen that displays the mobile terminal screen are detected, the position information indicating the touch position on the mobile terminal screen is displayed as the display attribute information of the self. And a position information correcting unit that corrects the position information on the operated terminal screen based on the display attribute information of the operated terminal received from the operated terminal.

On the other hand, the direction of the display surface of the MFP is constant, and the operation screen is often displayed in a direction in which the longitudinal direction is horizontal. The user of the portable information device uses the portable information device in a state where the longitudinal direction of the display surface of the portable information device is vertical, and uses the portable information device in a state where the longitudinal direction of the display surface is horizontal. There is. When displaying the operation screen of the MFP in a state where the longitudinal direction of the display surface of the portable information device is horizontal, when the operation screen of the MFP is displayed in the vertical or horizontal direction, the area other than the operation screen on the display surface of the portable information device On the other hand, the size in the vertical direction is enlarged or reduced until the size becomes the same as the size in the vertical direction on the display surface of the portable information device. In the portable information device, when displaying the operation screen of the MFP in a state where the longitudinal direction of the display surface is vertical, in order to display the entire operation screen on the display surface, the horizontal size of the operation screen is The display is enlarged or reduced until the size becomes the same as the horizontal size of the display surface of the portable information device. Therefore, since the horizontally long rectangular operation screen is displayed on the vertically long rectangular display surface, there is a problem that an area other than the operation screen becomes large on the display surface.
JP2012-93940A

  The present invention has been made to solve the above problems, and one of the objects of the present invention is to enable an operation screen to be efficiently displayed on a portable information device whose aspect ratio of the display surface changes. It is to provide a remote control system.

  Another object of the present invention is to provide an image processing device capable of efficiently displaying an operation screen on a portable information device whose aspect ratio of a display surface changes.

  Still another object of the present invention is to provide a remote control method capable of efficiently displaying an operation screen on a portable information device whose aspect ratio of a display surface changes.

  Still another object of the present invention is to provide a remote control program capable of efficiently displaying an operation screen on a portable information device whose aspect ratio of a display surface changes.

According to an aspect of the present invention for achieving the above-mentioned object, a remote control system is a remote control system including an image processing device and a portable information device capable of remotely controlling the image processing device. The apparatus includes an operation acceptance unit that accepts an operation by a user, a first display unit that can display an image, and a first aspect that indicates a ratio of a vertical length of a display surface of the first display unit to a horizontal length. If the ratio is smaller than the second aspect ratio indicating the ratio of the length in the vertical direction to the length in the horizontal direction of the operation screen , at least a part of which is displayed on the first display means, it is at least a part of the operation screen. When the portion having the first aspect ratio is determined as the display portion and the first aspect ratio is not less than the second aspect ratio, the display portion determining means for determining the entire operation screen as the display portion and the display portion determining means are determined. A third aspect ratio indicating the ratio of the length in the vertical direction to the length in the horizontal direction of the display surface of the second display unit included in the portable information device and the display control unit for displaying the image of the display portion on the first display unit. When the first aspect ratio is smaller than the second aspect ratio and when the third aspect ratio is less than or equal to the first aspect ratio, the aspect ratio detection means acquired from the portable information device is secondly displayed. The image of the display portion is transmitted to the portable information device for displaying on the means, and when the third aspect ratio is larger than the first aspect ratio, the image on the operation screen is displayed on the portable information device for displaying on the second display means. The mobile information device includes a transmission means for transmitting an image of the operation screen, and the portable information device includes a gravity direction detection means for detecting a gravity direction, and a long side and a short side of the display surface of the second display means. Aspect ratio determining means for determining the third aspect ratio with one side having a large angle with the detected gravity direction as the horizontal direction, and at least a part of the operation screen when the image of the operation screen is received from the image processing device. And a portable display control means for displaying on the second display means a horizontal size equal to the horizontal size of the second display means.

  According to this aspect, when the third aspect ratio is less than or equal to the first aspect ratio, the image processing device displays the display portion of the first aspect ratio on the display surface of the third aspect ratio for displaying the display portion of the first aspect ratio on the portable information device. When transmitting the portion, when the third aspect ratio is larger than the first aspect ratio, the operation is performed to display the operation screen having the second aspect ratio larger than the first aspect ratio on the display surface having the third aspect ratio on the portable information device. Send the screen. Therefore, since the portable information device displays the operation screen having the second aspect ratio larger than the first aspect ratio on the display surface having the third aspect ratio, the display portion having the first aspect ratio is displayed on the display surface having the third aspect ratio. The display surface of the portable information device can be used more effectively than in the case of displaying. As a result, it is possible to provide a remote control system capable of efficiently displaying an operation screen on a portable information device in which the aspect ratio of the display surface changes.

Preferably, the image processing apparatus stores the image of the display portion in the storage means in response to the display portion being determined by the display portion deciding means, and the remote control by the portable information device. Display control means for displaying the image stored in the storage means on the first display means, and an operation acceptance means when the first aspect ratio is smaller than the second aspect ratio in a state where the portable information device is not remotely operated. Scrolling control means for moving the display portion in the operation screen in accordance with the received operation by the user, and the transmitting means is stored in the storage means in response to the image of the display portion being stored in the storage means. The first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is the first aspect ratio in a state in which the scroll control means is remotely operated by the portable information device. In the following cases and in the case where the third aspect ratio is larger than the first aspect ratio , the image stored in the storage unit is transmitted by the transmitting unit until the transmitting unit transmits the entire operation screen to the portable information device. In response to this, the display part is moved to a part in the operation screen which is not transmitted to the portable information device by the transmitting means, and the portable information device combines the plurality of images received from the image processing device. A synthesizing unit for generating an image of the operation screen is further provided.

  According to this aspect, the image of the operation screen can be displayed on the portable information device.

According to another aspect of the present invention, an image processing apparatus is an image processing apparatus remotely operated by a portable information device, and an operation receiving unit that receives an operation by a user, and a first display unit that can display an image. A first aspect ratio indicating the ratio of the vertical length to the horizontal length of the display surface of the first display means, and a second aspect ratio indicating the ratio of the vertical length to the horizontal length of the operation screen. When the ratio is smaller than the ratio, at least a part of the operation screen and having the first aspect ratio is determined as the display part, and when the first aspect ratio is not less than the second aspect ratio, the entire operation screen is determined as the display part. With respect to the horizontal length of the display surface of the display portion determining means, the display control means for displaying the image of the display portion determined by the display portion determining means on the first display means, and the second display means included in the portable information device. Aspect ratio detecting means for acquiring a third aspect ratio indicating the ratio of the length in the vertical direction from the portable information device, and when the first aspect ratio is the second aspect ratio or less, the third aspect ratio is the first aspect ratio or less. In the case of transmitting the image of the display portion to the portable information device in order to display the image of the determined display portion on the second display means, and when the third aspect ratio is larger than the first aspect ratio, the image of the operation screen And transmitting means for transmitting the image of the operation screen to the portable information device in order to display on the second display means.

  According to this aspect, when the third aspect ratio is less than or equal to the first aspect ratio, the display portion is transmitted to display the display portion having the first aspect ratio on the display surface having the third aspect ratio, and the third aspect ratio is transmitted. Is larger than the first aspect ratio, the operation screen is transmitted in order to display the operation screen having the second aspect ratio larger than the first aspect ratio on the display surface having the third aspect ratio. Therefore, since the operation screen having the second aspect ratio larger than the first aspect ratio is displayed on the display surface having the third aspect ratio, it is possible to display the operation screen having the first aspect ratio on the display surface having the third aspect ratio. The display surface having the third aspect ratio can be effectively used. As a result, it is possible to provide an image processing apparatus capable of efficiently displaying the operation screen on the display surface with the changed aspect ratio.

Preferably, in response to the display portion being determined by the display portion determining means, a storage control means for storing the image of the display portion in the storage means, and the storage means being stored in the storage means in a state where it is not remotely operated by the portable information device. Display control means for displaying an image on the first display means, and a user's operation accepted by the operation accepting means when the first aspect ratio is smaller than the second aspect ratio in a state where the portable information device is not remotely operated. And a scroll control unit for moving the display portion in the operation screen according to the above, wherein the transmission unit stores the image stored in the storage unit in accordance with the portable information in response to the storage unit storing the image of the display unit. When the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is less than or equal to the first aspect ratio while being remotely operated by the portable information device , In the case where the third aspect ratio is larger than the first aspect ratio, depending on that the transmission means transmits the image stored in the storage means until the transmission means transmits the entire operation screen to the portable information device. Then, the display portion is moved to a portion which is not transmitted to the portable information device by the transmitting means in the operation screen.

  According to this aspect, the image of the operation screen can be displayed on the portable information device.

  Preferably, the transmission unit transmits partial position information indicating the position in the operation screen of the display portion corresponding to the image to be transmitted together with the image.

  According to this aspect, it is possible to cause the portable information device to generate an image of the operation screen from a plurality of images.

Preferably, the position detecting means for detecting the position indicated by the user on the display surface of the first display means, and the position indicated by the user are detected by the position detecting means while not being remotely operated by the portable information device. Accordingly, the position determining means for determining the position in the operation screen based on the position in the display portion determined by the display portion determining means corresponding to the detected position, and the transmitting means for displaying the image of the operation screen. And a remote operation receiving means for receiving a remote operation including in-screen position information indicating a position in the operation screen from the portable information device after transmitting the partial position information and the position determining means. The position in the operation screen is determined based on the in- screen position information included in the operation.

  According to this aspect, the remote operation received from the portable information device can be processed in the same manner as the detection of the position designated by the user on the display surface of the first display means.

  Preferably, the position detecting means for detecting the position indicated by the user on the display surface of the first display means, and the position indicated by the user are detected by the position detecting means while not being remotely operated by the portable information device. Accordingly, the position determining means for determining the position in the operation screen based on the position in the display portion determined by the display portion determining means corresponding to the detected position, and the transmitting means for displaying the image of the operation screen. And a remote operation receiving means for receiving a remote operation including in-screen position information indicating a position in the operation screen from the portable information device after transmitting the partial position information and the position determining means. The position in the operation screen is determined based on the in-screen position information included in the operation.

  According to this aspect, the remote operation received from the portable information device can be processed in the same manner as the detection of the position designated by the user on the display surface of the first display means.

Preferably, the storage control means for storing the image of the operation screen in the storage means, and the image of the determined display portion among the images of the operation screen stored in the storage means in the state of not being remotely operated by the portable information device, Display control means for displaying on one display means, and the transmission means, in addition to the case where the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is larger than the first aspect ratio, When the first aspect ratio is equal to or higher than the second aspect ratio , the image of the operation screen stored in the storage unit is transmitted to the portable information device.

Preferably, the position detecting means for detecting the position indicated by the user on the display surface of the first display means, and the position indicated by the user are detected by the position detecting means while not being remotely operated by the portable information device. Accordingly, the position determining means for determining the position in the operation screen based on the position in the display portion determined by the display portion determining means corresponding to the detected position and the transmitting means transmit the operation screen. After that, the remote operation receiving means for receiving the remote operation including the in-screen position information indicating the position in the operation screen from the portable information device, and the position determining means receives the remote operation by the remote operation receiving means. Accordingly, the position in the operation screen is determined based on the in-screen position information included in the received remote operation.

  According to this aspect, the remote operation received from the portable information device can be processed in the same manner as the detection of the position designated by the user on the display surface of the first display means.

According to still another aspect of the present invention, a remote control method is a remote control method executed by an image processing device remotely operated by a portable information device, wherein the image processing device receives an operation accepted by a user. Means and a first display means capable of displaying an image, and a first aspect ratio indicating a ratio of a vertical length to a horizontal length of a display surface of the first display means has a first aspect ratio in the horizontal direction of the operation screen. Is smaller than the second aspect ratio indicating the ratio of the length in the vertical direction to the length of the first aspect ratio, the portion having at least a part of the operation screen and having the first aspect ratio is determined as the display portion, and the first aspect ratio is set to the second aspect ratio. When the aspect ratio is equal to or larger than the aspect ratio, a display portion determining step of determining the entire operation screen as a display portion, a display control step of displaying an image of the display portion determined in the display portion determining step on the first display means, and a portable information device. An aspect ratio detecting step of acquiring from the portable information device a third aspect ratio indicating a ratio of a vertical length to a horizontal length of a display surface of the second display means, and the first aspect ratio is the second aspect. If it is smaller than the ratio, and if the third aspect ratio is less than or equal to the first aspect ratio, the image of the display portion is transmitted to the portable information device for displaying the image of the determined display portion on the second display means, When the aspect ratio is larger than the first aspect ratio, a transmitting step of transmitting the operation screen image to the portable information device in order to display the operation screen image on the second display means is included.

  According to this aspect, it is possible to provide a remote control method capable of efficiently displaying the operation screen on the display surface of which the aspect ratio changes.

According to still another aspect of the present invention, the remote control program is a remote control program executed by a computer that controls an image processing device remotely operated by a portable information device, wherein the image processing device is operated by a user. And a first display unit capable of displaying an image, the first aspect ratio indicating the ratio of the length in the vertical direction to the length in the horizontal direction of the display surface of the first display unit being operated. When the ratio is smaller than the second aspect ratio indicating the ratio of the vertical length to the horizontal length of the screen, at least a part of the operation screen having the first aspect ratio is determined as the display part, and the first aspect ratio is determined. When the ratio is equal to or more than the second aspect ratio, a display portion determining step of determining the entire operation screen as a display portion, and a display control step of displaying the image of the display portion determined in the display portion determining step on the first display means. An aspect ratio detecting step of acquiring from the portable information device a third aspect ratio indicating a ratio of a vertical length to a horizontal length of a display surface of the second display means included in the portable information device, and a first aspect ratio. Is less than or equal to the second aspect ratio, and if the third aspect ratio is less than or equal to the first aspect ratio, the display portion image is transmitted to the portable information device in order to display the determined display portion image on the second display means. If the third aspect ratio is larger than the first aspect ratio, the computer is caused to execute a transmitting step of transmitting the operation screen image to the portable information device in order to display the operation screen image on the second display means. ..

  According to this aspect, it is possible to provide a remote control program capable of efficiently displaying the operation screen on the display surface whose aspect ratio changes.

It is a figure showing the whole remote control system outline in one of the embodiments of the present invention. FIG. 3 is a perspective view showing the external appearance of the MFP. FIG. 3 is a block diagram showing an outline of a hardware configuration of an MFP. It is a top view showing an example of an operation panel. It is a block diagram which shows the outline of the hardware constitutions of a portable information device. 3 is a block diagram illustrating an example of functions of a CPU included in the MFP according to the first embodiment. FIG. It is a block diagram which shows an example of the function of CPU with which the portable information device in 1st Embodiment is equipped. 6 is a first flowchart showing an example of a flow of remote control processing in the first embodiment. 9 is a second flowchart showing an example of the flow of remote control processing in the first embodiment. 6 is a flowchart showing an example of the flow of a direct operation execution process according to the first embodiment. It is a flowchart which shows an example of the flow of a screen transition process. 7 is a flowchart showing an example of a flow of remote operation processing according to the first embodiment. 9 is a block diagram illustrating an example of functions of a CPU included in the MFP according to the second embodiment. FIG. It is a block diagram which shows an example of the function of CPU with which the portable information device in 2nd Embodiment is equipped. 11 is a first flowchart showing an example of the flow of remote control processing according to the second embodiment. 11 is a second flowchart showing an example of the flow of remote control processing in the second embodiment. 11 is a flowchart showing an example of the flow of direct operation execution processing according to the second embodiment. 11 is a flowchart showing an example of the flow of remote control processing according to the second embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are designated by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

<First Embodiment>
FIG. 1 is a diagram showing an overall outline of a remote control system according to one of the embodiments of the present invention. Referring to FIG. 1, remote control system 1 includes an MFP (Multi Function Peripheral) 100 that functions as an image processing device, a wireless station 5, and portable information devices 200, 200A, and 200B. The MFP 100 is connected to the network 3. The portable information devices 200, 200A, 200B are connected to the network 3 through the wireless station 5.

  The network 3 is a local area network (LAN), and the connection form may be wired or wireless. The network 3 is not limited to a LAN, but may be a wide area network (WAN), a public switched telephone network (PSTN), the Internet, or the like.

  The portable information devices 200, 200A, and 200B are computers such as smartphones and PDAs (Personal Digital Assistants) that a user carries and uses. Since the portable information devices 200, 200A, and 200B have the same hardware configuration and function, the portable information device 200 will be described here as an example unless otherwise specified. Here, the portable information device 200 is a smartphone and has a wireless LAN function and a call function. Therefore, the mobile information device 200 can connect to the mobile phone network by wirelessly communicating with the mobile phone base station and make a call.

  In remote control system 1 in the present embodiment, mobile information devices 200, 200A, and 200B each remotely control MFP 100. In this case, each of portable information devices 200, 200A, and 200B functions as a remote operation device that remotely operates MFP 100, and MFP 100 functions as a remote control device that is remotely operated by each of portable information devices 200, 200A, and 200B. Here, a case where portable information apparatus 200 remotely controls MFP 100 will be described as an example. In this case, a remote control program for remotely controlling MFP 100 is installed in portable information device 200.

  FIG. 2 is a perspective view showing the external appearance of the MFP. FIG. 3 is a block diagram showing an outline of the hardware configuration of the MFP. 2 and 3, MFP 100 includes a main circuit 110, a document reading unit 130 for reading a document, an automatic document feeder 120 for feeding the document to document reading unit 130, and a document reading unit. An image forming unit 140 for forming an image on a sheet or the like based on image data read and output by a document 130, a sheet feeding unit 150 for supplying the sheet to the image forming unit 140, and an image are formed. A post-processing unit 155 for processing paper and an operation panel 160 as a user interface are included.

  The post-processing unit 155 executes a sorting process of rearranging and discharging one or more sheets on which images are formed by the image forming unit 140, a punching process of punching holes, and a stapling process of driving a staple.

  The main circuit 110 is equipped with a CPU 111, a communication interface (I/F) unit 112, a ROM 113, a RAM 114, a hard disk drive (HDD) 115 as a mass storage device, a facsimile unit 116, and a CD-ROM 118. External storage device 117, The CPU 111 is connected to the automatic document feeder 120, the document reading unit 130, the image forming unit 140, the paper feeding unit 150, the post-processing unit 155, and the operation panel 160, and controls the entire MFP 100.

  The ROM 113 stores a program executed by the CPU 111, or data necessary for executing the program. The RAM 114 is used as a work area when the CPU 111 executes the program. The RAM 114 also temporarily stores read data (image data) continuously sent from the document reading unit 130.

  Communication I/F unit 112 is an interface for connecting MFP 100 to network 3. The CPU 111 communicates with the portable information devices 200, 200A, 200B via the communication I/F unit 112 to send and receive data. The communication I/F unit 112 can communicate with a computer connected to the Internet via the network 3.

  Facsimile unit 116 is connected to the public switched telephone network (PSTN) and transmits facsimile data to PSTN or receives facsimile data from PSTN. The facsimile unit 116 stores the received facsimile data in the HDD 115 or outputs it to the image forming unit 140. The image forming unit 140 prints the facsimile data received by the facsimile unit 116 on paper. Facsimile unit 116 also converts the data stored in HDD 115 into facsimile data and transmits the facsimile data to a facsimile device connected to PSTN.

A CD-ROM (Compact DiSk ROM) 118 is attached to the external storage device 117. The CPU 111 can access the CD-ROM 118 via the external storage device 117. The CPU 111 loads the program stored in the CD-ROM 118 mounted on the external storage device 117 into the RAM 114 and executes the program. The medium for storing the program executed by the CPU 111 is not limited to the CD-ROM 118, and may be an optical disc (MO (Magnetic Optical Disc)/MD (Mini Disc)/DVD (Digital Versatile Disc)), an IC card, an optical card, It may be a semiconductor memory such as a mask ROM, an EPROM (Erasable Programmable ROM), or an EEPROM (registered trademark) (Electrically EPROM).

Further, the program executed by the CPU 111 is not limited to the program stored in the CD-ROM 118, and the program stored in the HDD 115 may be loaded into the RAM 114 and executed. In this case, another computer 300 connected to the network 3 may rewrite the program stored in the HDD 115 of the MFP 100, or may additionally write a new program. Further, MFP 100 may download the program from another computer 300 connected to network 3 and store the program in HDD 115. The program mentioned here includes not only a program directly executable by the CPU 111, but also a source program, a compressed program, an encrypted program, and the like.

  The operation panel 160 includes a display unit 161, a VRAM (Video RAM) 162, and an operation unit 163. The display unit 161 is, for example, a liquid crystal display device (LCD), and displays an instruction menu for the user, information regarding acquired image data, and the like. The display unit 161 may use an organic ELD (Electroluminescence Display). The VRAM 162 is controlled by the CPU 111 and stores image data to be displayed on the display unit 161. The display unit 161 is controlled by the CPU 111 and displays the image data stored in the VRAM 162. In the first embodiment, the storage capacity of the VRAM 162 corresponds to the image displayed by the display unit 161. In other words, the display unit 161 displays all the images stored in the VRAM 162.

  The operation unit 163 includes a touch panel 165 and a hard key unit 167. The touch panel 165 is provided on the upper surface or the lower surface of the display unit 161 so as to overlap the display unit 161. The hard key unit 167 includes a plurality of hard keys. The hard key is, for example, a contact switch. The touch panel 165 detects a position designated by the user on the display surface of the display unit 161.

  FIG. 4 is a plan view showing an example of the operation panel. The dotted line in the figure is added to show a hard key portion 167 described later, and does not actually exist. Operation panel 160 is provided on the upper surface of MFP 100. When the user operates the MFP 100, the user often takes an upright posture, and therefore the display surface of the display unit 161, the operation surface of the touch panel 165, and the hard key unit 167 are arranged facing upward. This is because the user can easily visually recognize the display surface of the display unit 161, and the user can easily point the operation unit 163 with a finger.

  FIG. 5 is a block diagram showing an outline of the hardware configuration of the portable information device. Referring to FIG. 5, portable information device 200 according to the present embodiment has a CPU 201 for controlling the entire portable information device 200, a camera 202, a flash memory 203 for storing data in a nonvolatile manner, and a call unit. A wireless communication unit 204 connected to 205, a display unit 206 for displaying information, an operation unit 207 for receiving user's operation, a wireless LAN I/F 208, an acceleration sensor 210, and an external storage device 211 are included.

  The display unit 206 is a display device such as a liquid crystal display device (LCD) or an organic ELD, and displays an instruction menu for the user, information regarding acquired image data, and the like. The operation unit 207 includes a main key 207A and a touch panel 207B. Further, when the user instructs the display surface of the display unit 206, the operation unit 207 outputs the position of the display surface detected by the touch panel 207B to the CPU 201. The CPU 201 detects the position designated by the user on the screen displayed on the display unit 206 based on the position detected by the touch panel 207B. The CPU 201 accepts input of various instructions, characters, numbers, and other data by a user's operation based on the screen displayed on the display unit 206 and the position detected by the touch panel 207B. For example, when displaying a screen including a ten-key image on the display unit 206, the number corresponding to the key displayed at the position detected by the touch panel 207B is accepted.

  The camera 202 includes a lens and a photoelectric conversion element, images the light condensed by the lens on the photoelectric conversion element, and the photoelectric conversion element photoelectrically converts the received light and outputs image data to the CPU 201. The photoelectric conversion element is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like.

  The wireless communication unit 204 wirelessly communicates with a mobile phone base station connected to a telephone communication network. The wireless communication unit 204 connects the portable information device 200 to a telephone communication network and enables a call using the call unit 205. The wireless communication unit 204 decodes the audio signal obtained by demodulating the wireless signal received from the mobile phone base station and outputs the decoded audio signal to the call unit 205. In addition, the wireless communication unit 204 encodes the voice input from the call unit 205 and transmits it to the mobile phone base station. The call unit 205 includes a microphone and a speaker, outputs voice input from the wireless communication unit 204 from the speaker, and outputs voice input from the microphone to the wireless communication unit 204. Further, the wireless communication unit 204 is controlled by the CPU 201, connects the portable information device 200 to an electronic mail server, and transmits/receives electronic mail.

  The wireless LAN I/F 208 is an interface for communicating with the wireless station 5 and connecting the portable information device 200 to the network 3. By registering the IP (Internet Protocol) address of each MFP 100 in mobile information device 200, mobile information device 200 can communicate with MFP 100 and can send and receive data. In the present embodiment, the case where portable information device 200 communicates with MFP 100 using wireless LAN I/F 208 is described as an example, but communication may be performed using another communication method. Specifically, when the portable information device 200 and the MFP 100 are equipped with a short-range wireless device such as Bluetooth (registered trademark), the portable information device 200 is configured to communicate with the MFP 100 on a one-to-one basis. Good.

  The flash memory 203 stores a program executed by the CPU 201 or data required for executing the program. The CPU 201 loads the program stored in the flash memory 203 into the RAM included in the CPU 201 and executes the program.

  The acceleration sensor 210 is a triaxial acceleration sensor that measures acceleration in three directions of the X axis, the Y axis, and the Z axis. The acceleration sensor 210 outputs the detected acceleration to the CPU 201.

  The external storage device 211 is removable from the portable information device 200, and a CD-ROM 210A storing a remote control program can be mounted therein. The CPU 201 can access the CD-ROM 210A via the external storage device 211. The CPU 201 can load the remote operation program stored in the CD-ROM 210A mounted on the external storage device 211 into the RAM included in the CPU 201 and execute it.

As the program executed by the CPU 201, the program stored in the flash memory 203 or the CD-ROM 210A has been described, but another computer 300 connected to the network 3 rewrites the program stored in the flash memory 203. Alternatively, it may be a new program additionally written. Further, the portable information device 200 may be a program downloaded from another computer 300 connected to the network 3. The program mentioned here includes not only a program directly executable by the CPU 201, but also a source program, a compressed program, an encrypted program, and the like.

The medium for storing the program executed by the CPU 201 is not limited to the CD-ROM 211A, but a semiconductor memory such as an optical disk (MO/MD/DVD), an IC card, an optical card, a mask ROM, an EPROM, or an EEPROM (registered trademark). May be

  FIG. 6 is a block diagram showing an example of the functions of the CPU included in the MFP according to the first embodiment. The functions illustrated in FIG. 6 are functions formed in the CPU 111 by the CPU 111 included in the MFP 100 executing the remote control program stored in the ROM 113, the HDD 115, and the CD-ROM 118. Referring to FIG. 6, CPU 111 includes operation accepting unit 51 that accepts an operation by a user, operation screen determining unit 53 that determines an operation screen, display control unit 55 that controls display unit 161, and at least one of the operation screens. A display portion determination unit 57 that determines the display portion of the unit, a storage control unit 59 that controls the VRAM 162, and a communication control unit 60 that controls the communication I/F unit 112 are included.

  The operation receiving unit 51 receives an operation input by the user. The operations accepted by operation accepting unit 51 include a direct operation that the user inputs to operation unit 163 and a remote operation that the user inputs to portable information device 200 when MFP 100 is remotely controlled by portable information device 200. Here, a case where the operation receiving unit 51 directly receives an operation will be described as an example, and a case where a remote operation is received will be described later. The operation reception unit 51 controls the operation unit 163 and receives a direct operation input by the user to the operation unit 163.

  The operation reception unit 51 includes an operation determination unit 61, a position determination unit 63, and a position detection unit 65. The operation determination unit 61 determines a direct operation input by the user. When the user presses any one of the plurality of hard keys included in the hard key unit 167, the operation determining unit 61 determines the direct operation assigned to the pressed hard key. In response to determining the direct operation, the operation determining unit 61 outputs the direct operation to the operation screen determining unit 53 and the display portion determining unit 57. The direct operation assigned to the hard key includes, for example, an operation of setting one of a plurality of setting modes of the MFP 100. The plurality of setting modes are stored in, for example, a copy mode for executing a process of copying a document, a scan mode for executing a process of reading a document, a facsimile mode for executing a process of transmitting a facsimile, and the HDD 115. It also includes a data management mode for causing the processed data to perform a predetermined process.

  The position detection unit 65 controls the touch panel 165 to detect the position on the display surface of the display unit 161 designated by the user with a finger or a stylus pen. The position detection unit 65 outputs the position on the display surface of the display unit 161 detected by the touch panel 165 to the position determination unit 63.

  When the position on the display surface of the display unit 161 is input from the position detection unit 65, the position determination unit 63 displays the operation screen displayed on the display unit 161 based on the position on the display surface of the display unit 161. Determine the position. The position determining unit 63 receives the position of the display portion in the operation screen from the display portion determining unit 57 described later. The display unit 161 may display the entire operation screen or a part of the operation screen. Here, the display portion is a portion displayed on the display unit 161 in the operation screen. The position determination unit 63 determines the position in the operation screen by identifying the position in the display portion corresponding to the position in the display surface of the display unit 161 input from the position detection unit 65. The position determination unit 63 outputs the determined position in the operation screen to the operation determination unit 61.

  When the position in the operation screen is input from the position determination unit 63, the operation determination unit 61 determines the direct operation assigned to the position input from the position determination unit 63 in the operation screen. In response to determining the direct operation, the operation determining unit 61 outputs the direct operation to the operation screen determining unit 53 and the display portion determining unit 57. The operation screen includes buttons to which operations are assigned. The buttons included in the operation screen include, for example, a transition operation for transitioning the operation screen to another operation screen, a scroll operation for moving a display portion in the operation screen, and a setting operation for setting a set value.

  When the direct operation input from the operation determining unit 61 is an operation for setting the setting mode or a transition operation, the operation screen determining unit 53 determines the operation screen based on the direct operation. If the direct operation is an operation for setting the setting mode, a default operation screen predetermined for the setting mode is determined. If the direct operation is a transition operation, the operation screen defined by the transition operation is determined. The operation screen determination unit 53 outputs screen identification information for identifying the determined operation screen to the display portion determination unit 57 and the storage control unit 59.

  The display portion determination unit 57 determines at least a part of the operation screen specified by the screen identification information as the display portion in response to the screen identification information input from the operation screen determination unit 53. Specifically, the display portion determination unit 57 determines that the first aspect ratio, which indicates the ratio of the length in the vertical direction to the length in the horizontal direction of the display surface of the display unit 161, is in the vertical direction with respect to the horizontal length of the operation screen. When the ratio is equal to or larger than the second aspect ratio indicating the length ratio, the entire operation screen is determined as the display portion. When the first aspect ratio of the display surface of the display unit 161 is smaller than the second aspect ratio, the display portion determination unit 57 determines the display portion for displaying a part of the operation screen on the display unit 161. The aspect ratio of the display portion is the same as the first aspect ratio of the display surface of the display unit 161. When the screen identification information is input from the operation screen determination unit 53, the display portion determination unit 57 determines the display portion including the uppermost portion of the operation screen. In response to determining the display portion, the display portion determining unit 57 outputs partial position information indicating the position of the display portion in the operation screen to the storage control unit 59 and the position determining unit 63.

  The display portion determination unit 57 includes a scroll control unit 69. When the first aspect ratio is smaller than the second aspect ratio, and the direct operation input from the operation determining unit 61 is a scroll operation, the scroll control unit 69 displays the display portion by the scroll amount specified by the scroll operation. Move in the direction determined by scroll operation. When the scroll control unit 69 moves the display portion, the display portion determination unit 57 outputs partial position information indicating the position of the moved display portion in the operation screen to the storage control unit 59 and the position determination unit 63.

  The operation screen may include a fixed part and a variable part. The fixed part is a part corresponding to the header of the operation screen. The display portion determination unit 57 determines the fixed portion of the operation screen as the fixed display portion, determines at least a part of the changed portion of the operation screen as the variable display portion, and combines the fixed display portion and the variable display portion with each other. To decide. Therefore, the vertical length of the variable display portion is a value obtained by subtracting the vertical length of the fixed display portion from the vertical length of the display portion. When the direct operation input from the operation determining unit 61 is a scroll operation, the scroll control unit 69 moves the variable display portion by the scroll amount specified by the scroll operation in the direction determined by the scroll operation on the operation screen.

  The storage control unit 59 receives the screen identification information from the operation screen determination unit 53 and the partial position information from the display portion determination unit 57. The storage control unit 59 specifies the display part in the operation screen specified by the screen identification information in response to the input of the partial position information from the display part determining unit 57, and stores the image of the specified display part in the VRAM 162. Remember. The storage control unit 59 outputs a display instruction to the display control unit 55 in response to storing the image of the display portion in the VRAM 162.

  The display control unit 55 controls the display unit 161 in response to the display instruction input from the storage control unit 59, and causes the display unit 161 to display the image stored in the VRAM 162.

  When the operation screen determination unit 53 determines the operation screen, the display portion is determined to be the uppermost portion of the operation screen, and thus the image of the display portion arranged at the uppermost end of the operation screen is displayed on the display unit 161. .. After that, when the scroll operation is determined, the display portion moves in the scroll direction by the scroll amount, so that the image of the display portion after the movement is displayed on the display unit 161. When the operation screen includes the fixed portion and the variable portion, the image of the display portion arranged at the uppermost end of the operation screen is displayed on the display unit 161, and when the scroll operation is determined thereafter, the scroll direction is the scroll amount. Since the variable display portion moves to, the image of the variable display portion after the movement is displayed on the display unit 161 in a state where the image of the fixed display portion of the operation screen is fixed above the display unit 161.

  The communication control unit 60 controls the communication I/F unit 112 to establish a communication path with any of the portable information devices 200, 200A, 200B. As will be described later, when the user operates portable information device 200 and points to MFP 100 as a target for remote operation, portable information device 200 transmits a connection request to MFP 100. When the communication I/F unit 112 receives the connection request from the portable information device 200, the communication control unit 60 establishes a communication path with the portable information device 200 by performing a predetermined negotiation with the portable information device 200. To do. The communication control unit 60 is a remote control unit that indicates that the display control unit 55 and the display portion determination unit 57 are being remotely operated by the portable information device while establishing a communication path with the portable information device 200. Output an operation signal.

  The display control unit 55 controls the display unit 161 to prevent the display unit 161 from displaying the image stored in the VRAM 162 while the remote control signal is input from the communication control unit 60. For example, the display control unit 55 may display a message on the display unit 161 for notifying that the remote control is being performed.

  The communication control unit 60 includes an aspect ratio detection unit 71, a transmission unit 73, and a remote control reception unit 75. The aspect ratio detection unit 71 detects the third aspect ratio of the display unit 206 included in the portable information device 200. Since the portable information device 200 is carried by the user, the direction in which the display unit 206 faces changes. There are a first state in which the longitudinal direction of the display surface of the display unit 206 of the portable information device 200 is vertical and a second state in which the longitudinal direction of the display surface of the display unit 206 is vertical. Here, the third aspect ratio of the display surface of the display unit 206 is the ratio of the vertical length to the horizontal length of the display surface of the display unit 206. Since the display surface of the display unit 206 is rectangular, the third aspect ratio differs between the first state and the second state. In the first state, the longitudinal direction of the display surface of the display unit 206 is vertical, and thus the third aspect ratio is larger than 1. In the second state, the longitudinal direction of the display surface of the display unit 206 is horizontal, so the third aspect ratio is larger. The aspect ratio is smaller than 1. Portable information device 200 transmits the aspect ratio to MFP 100 in response to the change of the third aspect ratio.

  When the communication I/F unit 112 receives the third aspect ratio from the portable information device 200, the aspect ratio detection unit 71 acquires the received third aspect ratio. The aspect ratio detection unit 71 outputs the acquired third aspect ratio to the scroll control unit 69 of the display portion determination unit 57.

  The transmission unit 73, in response to the communication I/F unit 112 establishing a communication path with the portable information device 200, the image stored in the VRAM 162 by the storage control unit 59 and the display portion corresponding to the image. A pair with partial position information indicating the position in the operation screen of is transmitted to the portable information device 200, and a transmission completion signal is output to the storage control unit 59 in response to the completion of the transmission. When the operation screen includes a fixed part and a variable part, the image is divided into a fixed part and a variable part and transmitted. The storage control unit 59 outputs the transmission completion signal to the scroll control unit 69 in response to the transmission completion signal input from the transmission unit 73.

  While the remote control signal is being input from the communication control unit 60, the scroll control unit 69 has a third aspect ratio input from the aspect ratio detection unit 71 when the first aspect ratio is smaller than the second aspect ratio. Is less than or equal to the first aspect ratio, a new display portion different from the display portion determined at that time is determined in response to the transmission completion signal being input from the storage control unit 59. The scroll control unit 69 determines a new display portion that does not include the display portion determined when the third aspect ratio is input.

  The storage control unit 59 stores the image of the new display portion in the operation screen in the VRAM 162 in response to the new display portion being determined by the scroll control unit 69, and rewrites the VRAM 162 with the image of the new display portion. The transmission unit 73 is notified of the rewrite signal indicating that the change has been made. The transmission unit 73 controls the communication I/F unit 112 in response to the input of the rewrite signal from the storage control unit 59, and the image of the display portion stored in the VRAM 162 by the storage control unit 59 and the image thereof. A pair of partial position information indicating the position in the operation screen of the display portion corresponding to is transmitted to the portable information device 200, and a transmission completion signal is output to the storage control unit 59 in response to the completion of the transmission.

  In the operation screen, the scroll control unit 69 determines all the portions other than the display portion determined when the third aspect ratio is input from the aspect ratio detection unit 71 as a new display portion, Even if a transmission completion signal is input from the storage controller 59, a new display portion is not decided.

  Here, the operation of the portable information device 200 will be described. FIG. 7 is a block diagram showing an example of the functions of the CPU included in the portable information device according to the first embodiment. The functions shown in FIG. 7 are functions formed in the CPU 201 by the CPU 201 included in the portable information device 200 executing the remote operation program stored in the flash memory 203 or the CD-ROM 211A. Referring to FIG. 7, CPU 201 includes a mobile communication control unit 251, a posture detection unit 253, a combining unit 255, a screen display unit 257, and an operation reception unit 259.

  When the user operates the operation unit 207 and inputs an instruction to remotely operate the MFP 100, the mobile communication control unit 251 controls the wireless LAN I/F 208 to send a connection request to the MFP 100 and communicate with the MFP 100. Establish a communication path. The mobile communication control unit 251 includes an aspect ratio transmission unit 261, a screen reception unit 263, and a remote operation transmission unit 265.

  The attitude detection unit 253 detects the attitude of the mobile information device 200 based on the triaxial acceleration output from the acceleration sensor 210, and the detected attitude of the mobile information device 200 is transmitted to the aspect ratio transmission unit 261 and the screen display unit. Output to 257. When the detected posture changes in a state where a communication path is established with MFP 100, posture detection unit 253 detects the changed posture and then transmits the detected posture to the aspect ratio transmission unit. It outputs to 261. Here, the attitude of the portable information device 200 is shown in the direction of the gravity direction of the display unit 206. As for the attitude of the portable information device 200, since the shape of the display surface of the display unit 206 is rectangular, the longitudinal direction of the display unit 206 is closer to the gravity direction than the lateral direction and the lateral direction of the display unit 206 is longitudinal. Either the lateral direction, which is closer to the direction of gravity than the direction, or. The posture detection unit 253 detects a posture in which the display surface of the display unit 206 is horizontally long if the short-side direction of the display surface of the display unit 206 forms a small angle with the gravity direction, and the long-side direction forms an angle with the gravity direction. If it is small, the posture in which the display surface of the display unit 206 is vertically long is detected.

  The aspect ratio transmission unit 261 calculates the aspect ratio in response to the posture being input from the posture detection unit 253, controls the wireless LAN I/F 208, and sets the calculated aspect ratio to the MFP 100 as the third aspect ratio. Send. Since the size of the display surface of the display unit 206 is predetermined, when the orientation is the vertical direction, the aspect ratio is calculated by dividing the length in the longitudinal direction by the length in the lateral direction, and the orientation is changed to the horizontal direction. In the case of the direction, the aspect ratio is calculated by dividing the length in the lateral direction by the length in the longitudinal direction.

  The screen receiving unit 263 controls the wireless LAN I/F 208 to receive the set of the image and the partial position information from the MFP 100. In response to receiving the set of the image and the partial position information from MFP 100, screen receiving section 263 outputs them to combining section 255.

  When a plurality of sets of images and partial position information are input from the screen receiving unit 263, the combining unit 255 combines a plurality of images based on the partial position information of each set. Since the partial position information indicates the position of the image on the operation screen, the relative positional relationship between the plurality of images can be known. Therefore, the synthesizing unit 255 synthesizes a plurality of images based on their relative positions to generate a synthesized image. The image after being combined by the combining unit 255 is an image of the operation screen. When the operation screen includes a fixed part and a variable part, the image is divided into the fixed part and the variable part and transmitted. Therefore, by combining a plurality of variable parts with the image of one fixed part, the operation is performed. Generate a screen image. The combining unit 255 outputs the combined image to the screen display unit 257 and also outputs a plurality of partial position information to the remote operation transmitting unit 265. When one set of an image and partial position information is input from the screen receiving unit 263, the synthesizing unit 255 outputs the image to the screen display unit 257 and also outputs the partial position information to the remote operation transmitting unit 265.

  The screen display unit 257 determines the direction of the display unit 206 in response to the posture input from the posture detection unit 253. If the posture is horizontal, the longitudinal direction of the display surface of the display unit 206 is determined to be the horizontal direction, and if the posture is vertical, the lateral direction of the display surface of the display unit 206 is determined to be the horizontal direction. decide. The screen display unit 257 displays the input image on the display unit 206 according to the determined direction in response to the image input from the combining unit 255. By enlarging or reducing the image of the operation screen, the horizontal length of the image of the operation screen is made the same as the horizontal length of the display surface of the display unit 206, and the image of the operation screen after enlargement or reduction is displayed. It is displayed on the display unit 206. When the vertical length of the image of the operation screen after enlargement or reduction is longer than the vertical length of the display surface of the display unit 206, the image of the operation screen is displayed in a scrollable state. The screen display unit 257 outputs area information indicating the position of the portion displayed on the display unit 206 in the image input from the combining unit 255 to the remote operation transmission unit 265.

  The operation reception unit 259 detects the position designated by the user in the image of the operation screen. Specifically, the operation reception unit 259 controls the touch panel 207B to detect the position designated by the user on the display surface of the display unit 206, and specifies the detected position on the display surface. The operation reception unit 259 outputs the specified position on the display surface to the remote operation transmission unit 265.

  The remote operation transmitting unit 265 receives one or more pieces of partial position information from the combining unit 255, area information from the screen display unit 257, and a position on the display surface from the operation receiving unit 259. When a plurality of pieces of partial position information are input from the combining unit 255, the remote operation transmitting unit 265 regards the image generated by the combining unit 255 as an image of the operation screen, and displays the region information and the operation input from the operation receiving unit 259. The position on the operation screen corresponding to the position on the display screen input from the operation reception unit 259 is determined based on the position on the screen. The remote operation transmission unit 265 specifies the display portion including the determined position in the operation screen based on the determined position in the operation screen and the plurality of partial position information. Further, the remote operation transmission unit 265 identifies the position in the display portion corresponding to the determined position in the operation screen. The remote operation transmission unit 265 controls the wireless LAN I/F 208 to transmit the remote operation including the partial position information of the display portion including the position in the operation screen and the position in the display portion to the MFP 100. When one piece of partial position information is input from combining unit 255, remote operation transmitting unit 265 sends a remote operation including the partial position information and the position designated by the user on the display surface of display unit 206 to MFP 100. To do.

  Returning to FIG. 6, when the communication I/F unit 112 receives a remote operation from the portable information device 200, the remote operation receiving unit 75 outputs the remote operation to the position determining unit 63.

  The position determination unit 63 determines the position in the operation screen according to the remote operation input from the remote operation reception unit 75. The position determination unit 63 outputs the determined position in the operation screen to the operation determination unit 61. The remote control includes the partial position information and the position on the display surface. The position determination unit 63 specifies the display portion in the operation screen from the partial position information, and determines the position in the specified display portion with respect to the position on the display surface as the position in the operation screen.

  The operation determining unit 61 determines the position while being remotely operated by the portable information device 200, in other words, while the communication I/F unit 112 establishes a communication path with the portable information device 200. When the position on the operation screen is input from the unit 63, the remote operation assigned to the position input from the position determination unit 63 on the operation screen is determined. In response to determining the remote operation, the operation determining unit 61 outputs the remote operation to the operation screen determining unit 53 and the display portion determining unit 57.

  The operation screen determination unit 53 determines the operation screen defined by the transition operation when the remote operation input from the operation determination unit 61 is the transition operation. The operation screen determination unit 53 outputs screen identification information for identifying the determined operation screen to the display portion determination unit 57 and the storage control unit 59.

  The display portion determination unit 57 is at least a part of the operation screen specified by the screen identification information in response to the screen identification information being input from the operation screen determination unit 53 while being remotely operated by the portable information device 200. To the display part. Specifically, when the first aspect ratio of the display surface of the display unit 161 is equal to or higher than the second aspect ratio of the operation screen, the display portion determination unit 57 determines the entire operation screen as the display portion. When the first aspect ratio of the display surface of the display unit 161 is smaller than the second aspect ratio of the operation screen, the display portion determination unit 57 determines a part of the operation screen as a display portion for displaying on the display unit 161. The aspect ratio of the display portion is the same as the first aspect ratio of the display surface of the display unit 161. When the screen identification information is input from the operation screen determination unit 53, the display portion determination unit 57 determines the display portion including the uppermost portion of the operation screen. In response to determining the display portion, the display portion determining unit 57 outputs partial position information indicating the position of the display portion in the operation screen to the storage control unit 59 and the position determining unit 63.

  The scroll control unit 69 is input from the operation determination unit 61 when the first aspect ratio of the display surface of the display unit 161 is smaller than the second aspect ratio of the operation screen while being remotely operated by the portable information device 200. When the remote operation is a scroll operation, the display portion is moved in the direction determined by the scroll operation on the operation screen by the scroll amount specified by the scroll operation. When the scroll control unit 69 moves the display portion, the display portion determination unit 57 outputs partial position information indicating the position of the moved display portion in the operation screen to the storage control unit 59 and the position determination unit 63.

  FIG. 8 is a first flowchart showing an example of the flow of remote control processing according to the first embodiment. FIG. 9 is a second flowchart showing an example of the flow of the remote control process in the first embodiment. The remote control process is a process executed by the CPU 111 when the CPU 111 included in the MFP 100 executes the remote control program stored in the ROM 113, the HDD 115, and the CD-ROM 118. Referring to FIGS. 8 and 9, CPU 111 executes the direct operation execution process in the first embodiment (step S01), and advances the process to step S02. Although the details of the straightness operation execution process in the first embodiment will be described later, it is a process executed according to an operation input by the user to the operation unit 163.

  In step S02, communication I/F unit 112 determines whether or not a connection request has been received. If communication I/F unit 112 receives a connection request from any of portable information devices 200, 200A, and 200B, the process proceeds to step S03. If not, the process returns to step S01. Hereinafter, a case where a connection request is received from the portable information device 200 will be described as an example.

  In step S03, a communication path is established with the device that has transmitted the connection request. Specifically, the communication I/F unit 112 is controlled to negotiate with the portable information device 200 that has transmitted the connection request, and establish a communication path with the portable information device 200. In the next step S04, the display on the display unit 161 is turned off. The display unit 161 stops displaying the image stored in the VRAM 162. In this case, for example, a message indicating that remote operation is being performed may be displayed on the display unit 161.

  In step S05, it is determined whether the third aspect ratio has been acquired. The third aspect ratio is the aspect ratio of the display unit 206 included in the portable information device 200. In step S03, portable information device 200 transmits the third aspect ratio when a communication path is established with MFP 100. If the communication I/F unit 112 receives the third aspect ratio from the mobile information device 200, the communication I/F unit 112 waits until the communication I/F unit 112 receives the third aspect ratio from the mobile information device 200 (NO in step S05). If (YES in step S05), the process proceeds to step S06.

  In step S06, the partial position information and the image of the display portion are transmitted, and the process proceeds to step S07. The partial position information indicates the position in the operation screen of the display part corresponding to the image stored in the VRAM 162. The image of the display portion is the image stored in the VRAM 162. Using the communication path established in step S03, the partial position information and the image of the display portion stored in VRAM 162 are transmitted to portable information device 200.

  In step S07, it is determined whether the third aspect ratio is larger than the first aspect ratio. If the third aspect ratio is larger than the first aspect ratio, the process proceeds to step S08. If not, the process proceeds to step S12. In step S08, it is determined whether or not there is an untransmitted portion. When the process proceeds from step S07, the image of the display portion has been transmitted to portable information device 200 in step S06. When the process proceeds from step S11 described below, the image of the display portion is transmitted to the portable information device 200 in step S11. In step S08, it is determined whether or not there is a display portion which has not been transmitted in the operation screen. If there is an untransmitted portion, the process proceeds to step S09. If not, the process proceeds to step S12.

  In step S09, the untransmitted portion in the operation screen is determined as the display portion, and the process proceeds to step S10. In step S10, the image of the display portion determined in step S09 on the operation screen is stored in VRAM 162, and the process proceeds to step S11. In step S11, the partial position information and the image of the display portion are transmitted, and the process returns to step S08. The partial position information indicates the position on the operation screen of the display portion determined in step S09. The image of the display portion is the image of the display portion stored in the VRAM 162 in step S10. By executing steps S09 to S11, the image of the entire operation screen is divided into a plurality of display portions and transmitted to portable information device 200 at the stage when the process proceeds to step S12.

  In step S12, it is determined whether a remote operation has been received. The communication I/F unit 112 determines whether or not a remote operation has been received via the communication path established in step S03. If the remote operation is received, the process proceeds to step S13. If not, the process proceeds to step S22. In step S22, it is determined whether the third aspect ratio has been acquired. If communication I/F unit 112 receives the third aspect ratio from portable information device 200, the process returns to step S07, but if not, the process returns to step S12.

  In step S13, the display portion to be processed is determined. Of the operation screen, the portion specified by the partial position information included in the remote operation received in step S12 is determined as the display portion to be processed. In the next step S14, the position in the operation screen is specified. In the display portion determined in step S13, the position in the operation screen corresponding to the position in the display portion included in the remote operation is specified. In the next step S15, the remote operation is specified from the position on the operation screen, and the process is branched by the specified remote operation. If the remote operation is a setting operation, the process proceeds to step S16; if it is an execution instruction operation, the process proceeds to step S17. If the screen operation is a screen transition operation, the process proceeds to step S18. If the remote operation is a scroll operation, the process proceeds to step S20. If the remote operation is an end operation, the process proceeds to step S21.

  In step S16, according to the setting operation, the setting value that has been set so far is changed to the setting value determined by the setting operation, and the process returns to step S12. In step S17, the process determined by the execution instruction operation is executed according to the execution instruction operation, and the process returns to step S12. In step S18, the screen transition process shown in FIG. 11 is executed, and the process proceeds to step S19. In step S19, if the process proceeds from step S18, the image of the display portion determined in step S18 is stored in VRAM 162, and the process returns to step S06.

The display portion is determined before the process proceeds to step S20. In step S20, a new display portion is determined, and the process proceeds to step S19. For example, a new display portion is determined by moving the display portion that has already been determined in the operation screen in the direction determined by the scroll operation by the amount determined by the scroll operation. When the remote operation of the scroll operation is received, the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is less than or equal to the first aspect ratio. Since the third aspect ratio is greater than the first aspect ratio, since the overall operation screen is transmitted to the portable information device 200, Ru can be scrolled such an operation screen in the portable information apparatus 200., No remote control of scrolling is received. When the process proceeds from step S20 in step S19, the image of the display portion determined in step S20 is stored in VRAM 162, and the process returns to step S06. In step S21, the communication path established with portable information device 200 in step S03 is disconnected, and the process returns to step S01.

  FIG. 10 is a flowchart showing an example of the flow of direct operation execution processing according to the first embodiment. Referring to FIG. 10, CPU 111 determines whether or not a direct operation has been accepted (step S31). If the operation unit 163 receives the operation by the user, it is determined that the operation is directly received. If the direct operation is accepted, the process proceeds to step S32. If not, the process returns to the remote control process.

  In step S32, the process is branched by a direct operation. If the direct operation is a screen transition operation, the process proceeds to step S33, if the direct operation is a scroll operation, the process proceeds to step S36, and if the direct operation is a setting operation for setting a set value, the process proceeds to step S37, and If the operation is an execution instruction operation for instructing execution of the process, the process proceeds to step S38.

  In step S33, a screen transition process is executed, and the process proceeds to step S34. Although details of the screen transition process will be described later, it is a process of determining at least a part of the operation screen after the transition as a display part. In step S34, the image of the display portion is stored in VRAM 162, and the process proceeds to step S35. In step S35, display unit 161 is controlled to display the image stored in VRAM 162, and the process is returned to the remote control process. The display portion has been determined before the process proceeds to step S36. In step S36, a new display portion is determined, and the process proceeds to step S34. For example, a new display portion is determined by moving the already determined display portion in the operation screen in the direction determined by the scroll operation by the amount determined by the scroll operation. In step S37, according to the setting operation, the setting value determined by the setting operation is changed, and the process is returned to the remote control process. In step S38, according to the execution instruction operation, the processing determined by the execution instruction operation is executed, and the processing is returned to the remote control processing.

  FIG. 11 is a flowchart showing an example of the flow of screen transition processing. The screen transition process is a process executed in step S33 of FIG. 10 or step S17 of FIG. Referring to FIG. 10, CPU 111 identifies an operation screen to be processed (step S41). In the next step S42, the first aspect ratio of the display surface of the display unit 161 is compared with the second aspect ratio of the operation screen. If the first aspect ratio is smaller than the second aspect ratio of the operation screen, the process proceeds to step S43. If not, the process proceeds to step S44. In step S43, a part of the operation screen is determined as a display part, and the process is returned to the direct operation execution process or the remote control process. For example, the portion including the uppermost end of the operation screen is determined as the display portion. In step S44, the entire operation screen is determined as the display portion, and the process is directly returned to the operation execution process or the remote control process.

  FIG. 12 is a flowchart showing an example of the flow of remote control processing according to the first embodiment. The remote operation process is a process executed by the CPU 201 when the CPU 201 included in the portable information device 200 executes the remote operation program stored in the flash memory 203 or the CD-ROM 211A. Referring to FIG. 12, CPU 201 determines whether or not a remote operation instruction has been received (step S51). If the operation unit 207 receives a remote operation instruction input by the user, the remote operation instruction is accepted. The process waits until the remote operation instruction is received (NO in step S51), and if the remote operation instruction is received (YES in step S51), the process proceeds to step S52.

  In step S52, a connection request is transmitted to MFP 100. The wireless LAN I/F 208 is controlled to send a connection request to establish a communication path with the MFP 100. Then, it is determined whether or not the communication path is established (step S53). A communication path is established by negotiating with the MFP 100. Step S52 is repeated until the communication path is established (NO in step S53), and if the communication path is established (YES in step S53), the process proceeds to step S54.

  In step S54, the posture is detected. The posture of the portable information device 200 is detected based on the triaxial acceleration output by the acceleration sensor 210. As for the attitude of the portable information device 200, the longitudinal direction in which the longitudinal direction of the display unit 206 is closer to the gravity direction than the lateral direction, and the lateral direction in which the lateral direction of the display unit 206 is closer to the gravity direction than the longitudinal direction are. Is either. Then, the third aspect ratio is transmitted, and the process proceeds to step S56. When the posture is in the vertical direction, the third aspect ratio is calculated by dividing the length in the longitudinal direction by the length in the lateral direction. When the posture is in the lateral direction, the length in the lateral direction is calculated. The third aspect ratio is calculated by dividing by the length in the longitudinal direction. The third aspect ratio is transmitted to the MFP 100 via the communication path established in step S53.

  In the next step S56, the process waits until the partial position information and the image are received from MFP 100 (NO in step S56), and if the partial position information and the image are received (YES in step S56), the process is performed. Proceed to S57. In step S57, the received image is displayed on display unit 206, and the process proceeds to step S58.

  In step S58, it is determined whether partial position information and an image have been received from MFP 100. If the partial position information and the image have been received, the process proceeds to step S59. If not, the process proceeds to step S61. In step S59, the image displayed on display unit 206 is combined with the image received in step S58, and the process proceeds to step S60. The relative position of the two images is determined from the partial position information received with the image, and the two images are combined according to the relative position. In step S60, the combined image is displayed on display unit 206, and the process returns to step S58.

In step S61, it is determined whether an instruction has been accepted. When the position where the user points the display surface of the display unit 206 is detected by the touch panel 207B, it is determined that the instruction has been accepted. If the instruction from the user is accepted, the process proceeds to step S62. If not, the process proceeds to step S65.
In step S62, the display portion is determined. In step S61, the position in the image displayed on the display unit 206 is determined based on the position on the display surface detected by the touch panel 207B. When the composite image is generated in step S59, the image displayed on the display unit 206 is a part of the composite image, and thus the display part is determined from the one or more partial position information received together with the image in step S58. To do. The composite image generated in step S59 is an operation screen image. In the image of the operation screen, the portion displayed on the display unit 206 is specified, and the position in the operation screen is determined from the position detected in step S61. Then, a display portion including the determined position in the operation screen is determined from the plurality of display portions respectively specified by the plurality of partial position information (step S62). In the next step S63, the position in the display portion is determined by converting the position in the operation screen into the position in the display portion determined in step S62.

  In step S62, when the composite image is not generated in step S59, the image received in step S56 is displayed on the display unit 206, and thus corresponds to the partial position information received together with the image in step S56. Determine the display area. Then, in step S63, the position detected in step S61 is determined as the position in the display portion.

  In the next step S64, the remote operation is transmitted, and the process returns to step S58. The remote operation is transmitted to the MFP 100 via the communication path established in step S52. The remote operation includes the partial position information of the display portion determined in step S62 and the position in the display portion determined in step S63.

  In step S65, it is determined whether the communication path established with MFP 100 in step S53 has been disconnected. If the communication path is disconnected, the process is terminated, and if not, the process proceeds to step S66. In step S66, the posture is detected as in step S54. Then, it is determined whether or not the posture has changed. If the posture has changed, the process returns to step S55. If not, the process returns to step S58.

<Modification>
In the remote control system according to the first embodiment described above, the remote operation transmitted from portable information device 200 includes the partial position information and the position in the display part, and in MFP 100, the position in the operation screen is displayed according to the remote operation. The case has been described as an example. In the remote control system according to the modification, portable information device 200 determines the position in the operation screen and transmits the remote operation including the position in the operation screen to MFP 100. In this case, since the operation screen is displayed on the display unit 206 when the plurality of partial position information is input from the combining unit 255, the remote operation transmitting unit 265 included in the portable information device 200 is input from the operation receiving unit 259. The position in the operation screen is determined from the position designated by the user detected by the touch panel 207B. Further, when one piece of partial position information is input from the synthesizing unit 255, the remote operation transmitting unit 265 displays the display portion specified by the partial position information on the operation screen on the display unit 206. The position in the operation screen is determined based on the information and the position detected by the user on the touch panel 207B.

  In the modification as well, the remote control process shown in FIGS. 8 to 11 is executed in MFP 100, but steps S13 and S14 in FIG. 9 can be deleted. In step S15, portable information device in step S12. The remote operation is determined from the position in the operation screen included in the remote operation received from 200. Although the remote operation process shown in FIG. 12 is executed in the portable information device, in step S64 of FIG. 12, a process of determining the position in the operation screen from the position in the display portion determined in step S63 is executed. You can do it.

As described above, in the remote control system according to the first embodiment, MFP 100 functions as an image processing apparatus and indicates the ratio of the vertical length to the horizontal length of the display surface of display unit 161. When the 1 aspect ratio is smaller than the second aspect ratio indicating the ratio of the vertical length to the horizontal length of the operation screen, at least a part of the operation screen having the first aspect ratio is determined as the display portion. However, when the first aspect ratio is equal to or higher than the second aspect ratio, the entire operation screen is determined as the display portion. Then, the MFP 100 detects the third aspect ratio indicating the ratio of the vertical length to the horizontal length of the display surface of the display unit 206 included in the portable information device 200, and the first aspect ratio is the second aspect ratio. in the following cases, the third aspect ratio and transmits the image of the display part if the following first aspect ratio to the portable information device 200, the third aspect ratio portable information images OPERA drawing surface if larger than the first aspect ratio Send to the device 200. When portable information device 200 receives the image of the operation screen from MFP 100, mobile information device 200 displays at least a part of the operation screen on display unit 206 in the horizontal size equal to the horizontal size of display unit 206. Therefore, when the third aspect ratio is less than or equal to the first aspect ratio, the MFP 100 transmits the display portion having the first aspect ratio on the display surface having the third aspect ratio to cause the portable information device 200 to display the display portion. When the third aspect ratio is larger than the first aspect ratio, the operation screen is transmitted to cause the portable information device 200 to display the operation screen having the second aspect ratio larger than the first aspect ratio on the display surface having the third aspect ratio. To do. Therefore, since the portable information device 200 displays the operation screen having the second aspect ratio larger than the first aspect ratio on the display surface having the third aspect ratio, the display having the first aspect ratio is displayed on the display surface having the third aspect ratio. The display surface of the display unit 206 of the portable information device 200 can be used more effectively than when displaying a portion.

  Further, the MFP 100 stores the image of the display portion in the VRAM 162 according to the determination of the display portion, and displays the image stored in the VRAM 162 on the display unit 161 in a state where the portable information device 200 is not remotely operated. Then, while not being remotely operated by the portable information device 200, the display portion is moved in the operation screen according to the scroll operation by the user accepted by the operation unit 163, and the image of the display portion after the movement is stored in the VRAM 162. The image of the display portion is displayed on the display unit 161 by. The MFP 100 transmits the image stored in the VRAM 162 to the portable information device 200 in response to the image of the display portion being stored in the VRAM 162 while being remotely operated by the portable information device 200, and the entire operation screen is displayed. Until the image is transmitted to the portable information device 200, the display portion is moved to a portion of the operation screen which is not transmitted to the portable information device 200 in response to the transmission of the image stored in the VRAM 162. By storing the image of the display portion in the VRAM 162, the image of the display portion is transmitted to the portable information device 200. Portable information device 200 generates an image of the operation screen by combining a plurality of images received from MFP 100, and displays the image of the operation screen on display unit 206. Therefore, the image of the operation screen can be displayed on the portable information device 200.

  Further, since MFP 100 transmits partial position information indicating the position on the operation screen of the display portion corresponding to the image to be transmitted to mobile information device 200 together with the image, in mobile information device 200, a plurality of images are displayed. Can be combined on the basis of the relative position of, and an image of the operation screen can be generated from a plurality of images.

  In addition, in response to detecting the position designated by the user by touch panel 165, MFP 100 determines the position in the operation screen based on the position in the display portion corresponding to the detected position, and the mobile information. In the state where the device 200 is remotely operated, the remote operation is performed in response to the remote operation including the partial position information and the position information indicating the position instructed by the user in the image from the portable information device 200. The position in the operation screen is determined based on the display part in the operation screen specified by the included partial position information and the position in the display part specified by the designated position information. Therefore, the remote operation received from the portable information device 200 can be processed in the same manner as when the touch panel 165 detects the position designated by the user on the display surface of the display unit 161.

  In addition, MFP 100 according to the modified example responds to the fact that the position designated by the user is detected by touch panel 165 while not being operated remotely by portable information device 200, and the position in the display portion corresponding to the detected position is displayed. On the basis of the above, the position in the operation screen is determined, and when the mobile information device 200 is remotely operated, the remote operation including the in-screen position information indicating the position in the operation screen is received from the mobile information device 200. Accordingly, the position in the operation screen is determined based on the in-screen position information included in the remote operation. Therefore, the remote operation received from the portable information device 200 can be processed in the same manner as when the touch panel 165 detects the position designated by the user on the display surface of the display unit 161.

<Second Embodiment>
In the remote control system according to the first embodiment, the storage capacity of VRAM 162 included in MFP 100 corresponds to the image displayed on display unit 161. In the remote control system according to the second embodiment, the storage capacity of the VRAM 162 included in the MFP 100 is larger than the storage capacity of the VRAM 162 according to the first embodiment without corresponding to the image displayed on the display unit 161. Therefore, the MFP 100 according to the second embodiment can store all the images on the operation screen in the VRAM 162. The remote control system according to the second embodiment will be described below mainly with respect to differences from the remote control system according to the first embodiment.

  The overall outline of the remote control system according to the second embodiment is the same as the overall outline shown in FIG. The external appearance and hardware configuration of MFP 100 are the same as those shown in FIGS. 2 and 3. The hardware configuration of portable information device 200 is the same as the hardware configuration shown in FIG. Therefore, these descriptions will not be repeated.

  FIG. 13 is a block diagram showing an example of the functions of the CPU included in the MFP according to the second embodiment. Referring to FIG. 13, a difference from the function shown in FIG. 6 is that display control unit 55, storage control unit 59, display portion determination unit 57, position determination unit 63, scroll control unit 69, transmission unit 73, and remote operation. The receiving unit 75 is changed to a display control unit 55A, a storage control unit 59A, a display portion determining unit 57A, a position determining unit 63A, a scroll control unit 69A, a transmitting unit 73A, and a remote operation receiving unit 75A. Since other functions are the same as those shown in FIG. 6, description thereof will not be repeated here.

  The display portion determination unit 57A determines at least a part of the operation screen specified by the screen identification information as a display portion in response to the screen identification information input from the operation screen determination unit 53. Specifically, the display portion determination unit 57A determines that the first aspect ratio indicating the ratio of the length in the vertical direction to the length in the horizontal direction of the display surface of the display unit 161 is in the vertical direction with respect to the horizontal length of the operation screen. When the ratio is equal to or larger than the second aspect ratio indicating the length ratio, the entire operation screen is determined as the display portion. When the first aspect ratio of the display surface of the display unit 161 is smaller than the second aspect ratio of the operation screen, the display portion determination unit 57A determines a part of the operation screen as a display portion for displaying on the display unit 161. The aspect ratio of the display portion is the same as the first aspect ratio of the display surface of the display unit 161. When the screen identification information is input from the operation screen determination unit 53, the display portion determination unit 57A determines the display portion including the uppermost portion of the operation screen. In response to determining the display portion, the display portion determining unit 57A outputs partial position information indicating the position of the display portion in the operation screen to the display control unit 55A, the position determining unit 63A, and the transmitting unit 73A.

  The display portion determination unit 57A includes a scroll control unit 69A. When the first aspect ratio of the display surface of the display unit 161 is smaller than the second aspect ratio of the operation screen, the scroll control unit 69A scrolls the display portion when the direct operation input from the operation determination unit 61 is a scroll operation. The amount of scroll specified by the operation is moved in the direction determined by the scroll operation on the operation screen. When the display part is moved by the scroll control part 69A, the display part determination part 57A provides the display control part 55A, the position determination part 63A and the transmission part with partial position information indicating the position of the display part after the movement in the operation screen. Output to 73A.

  The operation screen may include a fixed part and a variable part. The fixed part is a part corresponding to the header of the operation screen. The display portion determination unit 57A determines the fixed portion of the operation screen as the fixed display portion, determines at least a part of the changed portion of the operation screen as the variable display portion, and combines the fixed display portion and the variable display portion. Determine the display area. Therefore, the vertical length of the variable display portion is a value obtained by subtracting the vertical length of the fixed display portion from the vertical length of the display portion. When the direct operation input from the operation determination unit 61 is a scroll operation, the scroll control unit 69A moves the variable display portion by the scroll amount specified by the scroll operation in the direction determined by the scroll operation on the operation screen.

  Screen identification information is input to the storage control unit 59A from the operation screen determination unit 53. The storage control unit 59 stores the image of the operation screen specified by the screen identification information in the VRAM 162 in response to the input of the screen identification information from the operation screen determination unit 53.

  The display control unit 55A controls the display unit 161 in response to the partial position information being input from the display portion determination unit 57A or the scroll control unit 69A, and causes the display unit 161 to display a portion of the image stored in the VRAM 162. The image of the display portion specified by the position information is displayed.

  When the position on the display surface of the display unit 161 is input from the position detection unit 65, the position determination unit 63A determines whether the position on the operation screen displayed on the display unit 161 is based on the position on the display surface of the display unit 161. Determine the position. The position determination unit 63A receives partial position information indicating the position of the display portion in the operation screen from the display portion determination unit 57. The display unit 161 may display the entire operation screen as a display portion, or may display a part of the operation screen. The position determination unit 63A determines the position in the operation screen by specifying the position in the display portion corresponding to the position in the display surface of the display unit 161 input from the position detection unit 65. The position determination unit 63A outputs the determined position in the operation screen to the operation determination unit 61.

  The transmitter 73A receives the partial position information from the display part determiner 57A or the scroll controller 69A, and receives the third aspect ratio from the aspect ratio detector 71. When the first aspect ratio is equal to or higher than the second aspect ratio, the transmission unit 73A causes the storage control unit 59 to store the VRAM 162 in response to the communication I/F unit 112 establishing a communication path with the portable information device 200. The stored image of the operation screen is transmitted to portable information device 200.

  When the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is larger than the first aspect ratio, the transmission unit 73A causes the communication I/F unit 112 to communicate with the portable information device 200 via a communication path. In response to the establishment of the above, the storage control unit 59 transmits the image of the operation screen stored in the VRAM 162 to the portable information device 200.

  When the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is less than or equal to the first aspect ratio, the transmission unit 73A causes the communication I/F unit 112 to establish a communication path with the portable information device 200. After the establishment, the partial position information is input from the display part determining unit 57A or the scroll control unit 69A, and the partial position information and the partial position of the operation screen image stored in the VRAM 162 by the storage control unit 59 are input. The combination with the image of the display portion specified by the information is transmitted to portable information device 200.

  When the third aspect ratio input from the aspect ratio detector 71 after transmitting the set of the partial position information and the image of the display portion to the portable information device 200 is larger than the first aspect ratio, the transmitter 73A stores the stored information. Of the images of the operation screen stored in the VRAM 162 by the control unit 59, a set of an image of a portion not transmitted to the portable information device 200 and partial position information indicating the position of the portion in the operation screen is set to the portable information device. Send to 200. When there are a plurality of images of the part that have not been transmitted to the portable information device 200, a combination of the image of the part and the partial position information indicating the position of the part on the operation screen is set for each of the plurality of parts. Send to 200.

  Here, the operation of the portable information device 200 according to the second embodiment will be described. FIG. 14 is a block diagram showing an example of functions of the CPU included in the portable information device according to the second embodiment. Referring to FIG. 14, the difference from the functions shown in FIG. 7 is that screen receiving section 263, combining section 255 and remote operation transmitting section 265 respectively include screen receiving section 263A, combining section 255A and remote operation transmitting section 265A. This is a changed point. Since other functions are the same as those shown in FIG. 7, description thereof will not be repeated here.

  The screen receiving unit 263A controls the wireless LAN I/F 208 to output the received image to the combining unit 255A in response to receiving the image from the MFP 100. The screen receiving unit 263A may receive a set of an image and partial position information. When receiving the set of the image and the partial position information, the screen receiving unit 263A outputs the set of the image and the partial position information to the combining unit 255A.

  The combining unit 255A receives the image from the screen receiving unit 263 but does not receive the partial position information in the first case and the second receiving unit in which the image receiving unit 263 inputs one set of the image and the partial position information. And the third case where a plurality of sets of images and partial position information are input from the screen receiving unit 263. In the first case, the combining unit 255A outputs the image input from the screen receiving unit 263 to the screen display unit 257. The screen display unit 257 controls the display unit 206 to display the image in response to the image input from the combining unit 255A. The image received by the screen receiving unit 263 in the first case is an image of the operation screen.

  In the second case, the combining unit 255A outputs the image input from the screen receiving unit 263 to the screen display unit 257 and also outputs one partial position information to the remote operation transmitting unit 265A. The screen display unit 257 controls the display unit 206 to display the image in response to the image input from the combining unit 255A. The image received by the screen receiving unit 263 in the second case is the image of the display portion specified by the partial position information in the operation screen.

  In the third case, the combining unit 255A combines the plurality of images input from the screen receiving unit 263 based on the partial position information of each pair. Since the partial position information indicates the position of the image on the operation screen, the relative positional relationship between the plurality of images can be known. Therefore, the synthesizing unit 255A synthesizes a plurality of images based on their relative positions to generate a synthesized image. The image after being combined by the combining unit 255A is an operation screen image. When the operation screen includes a fixed part and a variable part, the image is divided into the fixed part and the variable part and transmitted. Therefore, by combining a plurality of variable parts with the image of one fixed part, the operation is performed. Generate a screen image. The combining unit 255A outputs the combined image to the screen display unit 257 and also outputs a plurality of partial position information to the remote operation transmission unit 265A. The screen display unit 257 controls the display unit 206 to display the combined image in response to the input of the combined image from the screen receiving unit 263A.

  The remote operation transmission unit 265A receives the area information from the screen display unit 257 and the position on the display surface from the operation reception unit 259. The remote operation transmission unit 265A has a first case in which partial position information is not input from the combining unit 255A, a second case in which one partial position information is input from the combining unit 255A, and a plurality of partial positions from the combining unit 255A. There is a third case where information is input. In the first case or the third case, the remote operation transmission unit 265A regards the image displayed by the screen display unit 257 as the image of the operation screen, and the area information and the position on the display surface input from the operation reception unit 259. The position on the operation screen corresponding to the position on the display surface input from the operation reception unit 259 is determined based on the. In the second case, the remote operation transmission unit 265 determines the position in the operation screen from the partial position information and the position designated by the user on the display surface of the display unit 206. Remote operation transmitting unit 265A controls wireless LAN I/F 208 to transmit a remote operation including a position on the operation screen to MFP 100.

  Returning to FIG. 13, when the communication I/F unit 112 receives a remote operation from the portable information device 200, the remote operation receiving unit 75A outputs the remote operation to the position determining unit 63A. The position determination unit 63A acquires the position in the operation screen included in the remote operation input from the remote operation reception unit 75A, and outputs the acquired position in the operation screen to the operation determination unit 61.

  The display portion determination unit 57A is at least part of the operation screen specified by the screen identification information in response to the screen identification information being input from the operation screen determination unit 53 while being remotely operated by the portable information device 200. To the display part. Specifically, when the first aspect ratio of the display surface of the display unit 161 is equal to or higher than the second aspect ratio of the operation screen, the display portion determination unit 57A determines the entire operation screen as the display portion. When the first aspect ratio of the display surface of the display unit 161 is smaller than the second aspect ratio of the operation screen, the display portion determination unit 57A determines a part of the operation screen as a display portion for displaying on the display unit 161. The aspect ratio of the display portion is the same as the first aspect ratio of the display surface of the display unit 161. When the screen identification information is input from the operation screen determination unit 53, the display portion determination unit 57A determines the display portion including the uppermost portion of the operation screen. In response to determining the display portion, the display portion determining unit 57 outputs partial position information indicating the position of the display portion in the operation screen to the position determining unit 63A and the transmitting unit 73A.

  When the remote control input from the operation determining unit 61 is a scroll operation in the state where the scroll control unit 69A is remotely operated by the portable information device 200, the scroll control unit 69A displays the display portion by the scroll amount specified by the scroll operation. Move in the direction determined by the scroll operation. When the scroll control unit 69 moves the display portion, the display portion determination unit 57A outputs partial position information indicating the position of the moved display portion in the operation screen to the position determination unit 63A and the transmission unit 73A.

  FIG. 15 is a first flowchart showing an example of the flow of remote control processing according to the second embodiment. FIG. 16 is a second flowchart showing an example of the flow of remote control processing according to the second embodiment. Referring to FIGS. 15 and 16, the direct operation execution process of the second embodiment is executed in step S101, and the process proceeds to step S102. Although the details of the straightness operation execution process according to the second embodiment will be described later, it is a process executed according to an operation input by the user to the operation unit 163.

  The processing of steps S102 to S105 is the same as the processing of steps S02 to S05 of FIG. Therefore, the description will not be repeated here. In step S106, it is determined whether the third aspect ratio is larger than the first aspect ratio. If the third aspect ratio is larger than the first aspect ratio, the process proceeds to step S107. If not, the process proceeds to step S108. In step S107, the image of the operation screen is transmitted to portable information device 200, and the process proceeds to step S109. The communication I/F unit 112 is controlled to transmit the image of the operation screen stored in the VRAM 162 to the portable information device 200 using the communication path established in step S103.

  In step S108, the partial position information and the image of the display portion are transmitted to portable information device 200, and the process proceeds to step S109. The communication I/F unit 112 is controlled to show the image of the display portion of the image of the operation screen stored in the VRAM 162 and the position of the display portion in the operation screen by using the communication path established in step S103. The partial position information is transmitted to portable information device 200.

  In step S109, it is determined whether a remote operation has been received. The communication I/F unit 112 determines whether or not a remote operation has been received via the communication path established in step S103. If the remote operation is received, the process proceeds to step S110. If not, the process proceeds to step S119. In step S119, it is determined whether the third aspect ratio has been acquired. If communication I/F unit 112 receives the third aspect ratio from portable information device 200, the process proceeds to step S120. If not, the process returns to step S109. In step S120, it is determined whether the third aspect ratio is larger than the first aspect ratio. If the third aspect ratio is larger than the first aspect ratio, the process proceeds to step S121. If not, the process returns to step S109.

  In step S121, it is determined whether or not there is an untransmitted portion. In the image of the operation screen stored in VRAM 162, it is determined whether or not there is an untransmitted portion that has not been transmitted to portable information device 200. If there is an untransmitted portion, the process proceeds to step S122. If not, the process returns to step S109. In step S122, the set of the image of the untransmitted portion and the partial position information indicating the position of the untransmitted portion in the operation screen is transmitted to the portable information device, and the process proceeds to step S121. For example, when the display portion is in the middle of the operation screen, the upper portion and the lower portion of the display portion are untransmitted portions. In this case, the first untransmitted portion above the display portion and the second untransmitted portion below the display portion are transmitted to portable information device 200.

  In step S110, the position in the operation screen included in the remote operation is acquired, and the process proceeds to step S111. Identify the remote operation from the position on the operation screen. In step S111, the process is branched by remote control. If the remote operation is a setting operation, the process proceeds to step S112, if it is an execution instruction operation, the process proceeds to step S113, if the remote operation is a scroll operation, the process proceeds to step S114, and if the screen transition operation, the process proceeds to step S116. If the remote operation is an end operation, the process proceeds to step S118.

  In step S112, according to the setting operation, the setting value determined by the setting operation is changed, and the process returns to step S109. In step S113, the process determined by the execution instruction operation is executed according to the execution instruction operation, and the process returns to step S109.

  The display portion has been determined before the process proceeds to step S114. In step S114, a new display portion is determined and the process proceeds to step S115. For example, the display portion is determined by moving the display portion in the operation screen in a direction determined by the scroll operation by an amount determined by the scroll operation. In step S115, the partial position information of the display portion and the image of the display portion determined in step S114 are transmitted to portable information device 200, and the process returns to step S109.

  In step S116, the screen transition process shown in FIG. 10 is executed, and the process proceeds to step S117. In step S117, the image of the operation screen determined in step S116 is stored in VRAM 162, and the process returns to step S106. In step S118, the communication path established with portable information device 200 in step S103 is disconnected, and the process returns to step S101.

  FIG. 17 is a flowchart showing an example of the flow of direct operation execution processing according to the second embodiment. Referring to FIG. 17, the point different from the processing shown in FIG. 10 is that steps S34 to S36 are changed to steps S34A to S36A. Other processes are the same as the processes shown in FIG. 10, and therefore description thereof will not be repeated here.

  In step S34A, the image of the operation screen is stored in VRAM 162, and the process proceeds to step S35A. In step S35A, display unit 161 is controlled to display the image of the display portion determined in step S33 among the images of the operation screen stored in VRAM 162, and the process is returned to the remote control process. The display portion is determined before the process proceeds to step S36A. In step S36A, a new display portion is determined and the process proceeds to step S35A. For example, the display portion is determined by moving the display portion in the operation screen in a direction determined by the scroll operation by an amount determined by the scroll operation. In step S35A, when the process proceeds from step S36A, the display unit 161 is controlled to display the image of the display portion determined in step S36A among the images of the operation screen stored in the VRAM 162, and the process is remotely controlled. Return to processing.

  FIG. 18 is a flowchart showing an example of the flow of remote control processing according to the second embodiment. Referring to FIG. 18, the points different from the remote operation processing shown in FIG. 12 are that steps S71 to S77 are executed instead of steps S55 to S60, step S62 is deleted, and step S63. Further, step S64 is changed to step S63A and step S64A. The other processing is the same as the processing shown in FIG. 12, and therefore the description will not be repeated here.

  After transmitting the third aspect ratio to the MFP 100 in step S55, it is determined in step S71 whether the partial position information and the image have been received from the MFP 100. If the wireless LAN I/F 208 receives the partial position information and the image from the MFP 100, the process proceeds to step S72. If not, the process proceeds to step S73. In step S72, the received image is displayed on display unit 206, and the process proceeds to step S61.

  In step S73, it is determined whether the partial position information and the image of the untransmitted portion have been received from MFP 100. If wireless LAN I/F 208 receives the partial position information and the image of the untransmitted portion from MFP 100, the process proceeds to step S74. If not, the process proceeds to step S76. In step S74, the untransmitted image is combined with the image already displayed on the display unit 206 to generate a combined image. The position on the operation screen of the image displayed on the display unit 206 is specified by the partial position information, and the position of the untransmitted part on the operation screen is specified by the partial position information, thereby displaying on the display unit 206. The image and the image of the untransmitted portion are combined. In the next step S75, the composite image is displayed on display unit 206, and the process proceeds to step S61.

  In step S76, it is determined whether the image of the operation screen is received from MFP 100. If the wireless LAN I/F 208 receives the image of the operation screen from the MFP 100, the process proceeds to step S77. If not, the process proceeds to step S61. In step S77, the image of the operation screen is displayed on display unit 206, and the process proceeds to step S61.

  When the instruction is accepted in step S61, the position in the operation screen is determined in step S63A, and the process proceeds to step S64A. When the composite image is generated in step S75, the composite image is the image of the operation screen. When the composite image is displayed on the display unit 206, or when the image of the operation screen is displayed on the display unit 206, the operation screen corresponding to the position on the display surface detected by the touch panel 207B in step S61 is displayed. Determine the position. If the image of the display portion is displayed in step S72, the position in the display portion corresponding to the position in the display surface detected by the touch panel 207B in step S61 is specified, and the operation of the display portion determined by the partial position information is performed. The position on the operation screen is determined based on the position on the screen.

  In the next step S64A, the remote operation is transmitted, and the process returns to step S71. The remote operation is transmitted to the MFP 100 via the communication path established in step S52. The remote operation includes the position on the operation screen determined in step S63A.

  If it is determined in step S67 that the posture has changed, the process returns to step S55, and if not, the process returns to step S71.

  The MFP 100 according to the second embodiment stores the image of the operation screen in the VRAM 162 in response to the determination of the operation screen, and the operation screen stored in the VRAM 162 in the state where it is not remotely operated by the portable information device 200. Of the image of the operation screen displayed on the display unit 161, the display unit is moved in the operation screen according to the scroll operation by the user accepted by the operation unit 163, and the image of the operation screen stored in the VRAM 162 is moved. The image of the subsequent display portion is displayed on the display unit 161. When the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is less than or equal to the first aspect ratio when the portable information device 200 is remotely operated, the image of the operation screen stored in the VRAM 162 is displayed. When the image of the display portion is transmitted to the portable information device 200, and the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is larger than the first aspect ratio, the operation screen stored in the VRAM 162 is displayed. The image is transmitted to portable information device 200. Therefore, when the third aspect ratio is larger than the first aspect ratio, the portable information device 200 displays the operation screen having the second aspect ratio larger than the first aspect ratio on the display surface having the third aspect ratio. The display surface of the display unit 206 of the portable information device 200 can be used more effectively than when the display portion having the first aspect ratio is displayed on the display surface having the third aspect ratio.

  Further, since MFP 100 according to the second embodiment transmits partial position information indicating the position in the operation screen of the display portion corresponding to the image to be transmitted, to portable information device 200 together with the image, portable information device. In 200, the images can be combined based on the relative positions of the images, and an image of the operation screen can be generated from the images.

  Further, MFP 100 according to the second embodiment corresponds to the detected position in response to detection of the position designated by the user by touch panel 165 in the state where remote control is not performed by portable information device 200. In the state where the position in the operation screen is determined based on the position in the display portion and the mobile information device 200 is remotely operated, the remote information including the in-screen position information indicating the position in the operation screen is displayed from the mobile information device 200. In response to receiving the operation, the position in the operation screen is determined based on the in-screen position information included in the remote operation. Therefore, the remote operation received from the portable information device 200 can be processed in the same manner as when the touch panel 165 detects the position designated by the user on the display surface of the display unit 161.

  In the above-described embodiments, the MFP 100 has been described as an example of the screen processing apparatus. However, a remote control method for causing the MFP 100 to execute the remote control processing illustrated in FIGS. It goes without saying that the invention can be understood as a remote control program executed by the CPU 111 for controlling the.

  The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

  1 remote control system, 3 networks, 5 wireless stations, 110 main circuit, 111 CPU, 112 communication I/F unit, 113 ROM, 114 RAM, 115 HDD, 116 facsimile unit, 117 external storage device, 120 automatic document feeder, 130 document reading unit, 140 image forming unit, 150 paper feeding unit, 155 post-processing unit, 160 operation panel, 161 display unit, 162 VRAM, 163 operation unit, 165 touch panel, 165 hard key unit, 167 hard key unit, 200, 200A, 200B portable information device, 201 CPU, 202 camera, 203 flash memory, 204 wireless communication unit, 205 call unit, 206 display unit, 207 operation unit, 207A main key, 207B touch panel, 208 wireless LAN I/F, 210 acceleration sensor , 211 external storage device, 51 operation acceptance unit, 53 operation screen determination unit, 55, 55A display control unit, 57, 57A display portion determination unit, 59, 59A storage control unit, 60 communication control unit, 61 operation determination unit, 63 , 63A position determination unit, 65 position detection unit, 69, 69A scroll control unit, 71 aspect ratio detection unit, 73, 73A transmission unit, 75, 75A remote operation reception unit, 251 portable communication control unit, 253 attitude detection unit, 255, 255A combining unit, 257 screen display unit, 259 operation receiving unit, 261 aspect ratio transmitting unit, 263, 263A screen receiving unit, 265, 265A remote operation transmitting unit.

Claims (11)

A remote control system including an image processing device and a portable information device capable of remotely controlling the image processing device,
The image processing device includes an operation receiving unit that receives an operation by a user,
First display means capable of displaying an image,
The first aspect ratio, which indicates the ratio of the length in the vertical direction to the length in the horizontal direction of the display surface of the first display unit, has a horizontal length of the operation screen at least a part of which is displayed on the first display unit. When it is smaller than the second aspect ratio indicating the ratio of the length in the vertical direction to the height, the portion having the first aspect ratio which is at least a part of the operation screen is determined as the display portion, and the first aspect ratio is A display portion determining means for determining the entire operation screen as a display portion when the aspect ratio is not less than the second aspect ratio;
Display control means for displaying the image of the display portion determined by the display portion determining means on the first display means;
Aspect ratio detecting means for acquiring from the portable information device a third aspect ratio indicating a ratio of a vertical length to a horizontal length of a display surface of the second display means included in the portable information device ,
When the first aspect ratio is smaller than the second aspect ratio and when the third aspect ratio is less than or equal to the first aspect ratio, in order to display the image of the determined display portion on the second display means. When the image of the display portion is transmitted to the portable information device and the third aspect ratio is larger than the first aspect ratio, the portable information device for displaying the image of the operation screen on the second display means. And transmitting means for transmitting the image of the operation screen to
The portable information device, gravity direction detection means for detecting the gravity direction,
Of the sides in the longitudinal direction and the sides in the lateral direction of the display surface of the second display means, one side having a larger angle with the detected gravity direction is the lateral direction, and the third aspect ratio is determined. Means and
When the image of the operation screen is received from the image processing device, at least a part of the operation screen is displayed on the second display unit with a horizontal size equal to the horizontal size of the second display unit. A remote control system comprising: a portable display control means for displaying. The image processing device includes storage control means for storing an image of the display portion in storage means in response to the display portion being determined by the display portion determining means,
Display control means for displaying the image stored in the storage means on the first display means while not being remotely controlled by the portable information device;
When the first aspect ratio is smaller than the second aspect ratio while not being remotely operated by the portable information device, the scroll for moving the display portion in the operation screen according to the operation by the user accepted by the operation accepting unit. Further comprising a control means,
The transmitting means transmits the image stored in the storage means to the portable information device in response to the image of the display portion being stored in the storage means,
A case where the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is less than or equal to the first aspect ratio in a state where the scroll control means is remotely operated by the portable information device ; In the case where the third aspect ratio is larger than the first aspect ratio , the transmitting unit stores the operation screen in the storage unit until the transmitting unit transmits the entire operation screen to the portable information device. In response to the image being transmitted, the display portion is moved to a portion in the operation screen which is not transmitted to the portable information device by the transmitting means,
The remote control system according to claim 1, wherein the portable information device further includes a synthesizing unit that synthesizes a plurality of images received from the image processing device to generate an image of the operation screen. An image processing device remotely operated by a portable information device,
Operation acceptance means for accepting user operations,
First display means capable of displaying an image,
A first aspect ratio indicating a ratio of a vertical length to a horizontal length of the display surface of the first display means, and a second aspect ratio indicating a ratio of a vertical length to a horizontal length of the operation screen. When the ratio is smaller than the ratio, at least a part of the operation screen and the portion having the first aspect ratio is determined as the display part, and when the first aspect ratio is equal to or more than the second aspect ratio, the entire operation screen is displayed. Display portion determining means for determining the display portion,
Display control means for displaying the image of the display portion determined by the display portion determining means on the first display means;
Aspect ratio detecting means for acquiring from the portable information device a third aspect ratio indicating a ratio of a vertical length to a horizontal length of a display surface of the second display means included in the portable information device ,
When the first aspect ratio is smaller than the second aspect ratio and when the third aspect ratio is less than or equal to the first aspect ratio, the image of the determined display portion is displayed on the second display means. When the image of the display portion is transmitted to the portable information device and the third aspect ratio is larger than the first aspect ratio, the portable information device for displaying the image of the operation screen on the second display means. An image processing apparatus, comprising: a transmission unit configured to transmit the image of the operation screen. Storage control means for storing an image of the display portion in the storage means in response to the display portion being determined by the display portion determining means,
Display control means for displaying the image stored in the storage means on the first display means while not being remotely controlled by the portable information device;
When the first aspect ratio is smaller than the second aspect ratio while not being remotely operated by the portable information device, the scroll for moving the display portion in the operation screen according to the operation by the user accepted by the operation accepting unit. And a control means,
The transmitting means transmits the image stored in the storage means to the portable information device in response to the image of the display portion being stored in the storage means,
The scroll control means , when the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is less than or equal to the first aspect ratio in a state of being remotely operated by the portable information device ; In the case where the third aspect ratio is larger than the first aspect ratio , the image stored in the storage unit by the transmission unit until the transmission unit transmits the entire operation screen to the portable information device. The image processing apparatus according to claim 3, wherein the display portion is moved to a portion of the operation screen which is not transmitted to the portable information device in response to the transmission of the.   The image processing apparatus according to claim 4, wherein the transmission unit transmits partial position information indicating a position in the operation screen of a display portion corresponding to an image to be transmitted together with the image. Position detection means for detecting a position designated by the user on the display surface of the first display means,
The display determined by the display portion determining means corresponding to the detected position in response to the position detected by the user being detected by the position detecting means while not being remotely operated by the portable information device. Position determining means for determining a position in the operation screen based on the position in the portion,
After the transmitting means transmits the image of the operation screen and the partial position information, remote operation receiving means for receiving a remote operation including in-screen position information indicating a position in the operation screen from the portable information device, Equipped with
The image processing apparatus according to claim 5 , wherein the position determination means determines a position in the operation screen based on the in- screen position information included in the received remote operation. Position detection means for detecting a position designated by the user on the display surface of the first display means,
The display determined by the display portion determining means corresponding to the detected position in response to the position detected by the user being detected by the position detecting means while not being remotely operated by the portable information device. Position determining means for determining a position in the operation screen based on the position in the portion,
After the transmitting means transmits the image of the operation screen and the partial position information, remote operation receiving means for receiving a remote operation including in-screen position information indicating a position in the operation screen from the portable information device, Equipped with
The image processing apparatus according to claim 5 , wherein the position determining means determines a position in the operation screen based on the in-screen position information included in the received remote operation. Storage control means for storing the image of the operation screen in the storage means,
Display control means for displaying on the first display means an image of the determined display portion of the image of the operation screen stored in the storage means while not being remotely operated by the portable information device. Prepare,
In addition to the case where the first aspect ratio is smaller than the second aspect ratio and the third aspect ratio is larger than the first aspect ratio , the transmitting means may further include: The image processing device according to claim 3 , wherein when the ratio is equal to or higher than the ratio, the image of the operation screen stored in the storage unit is transmitted to the portable information device. Position detection means for detecting a position designated by the user on the display surface of the first display means,
The display determined by the display portion determining means corresponding to the detected position in response to the position detected by the user being detected by the position detecting means while not being remotely operated by the portable information device. Position determining means for determining a position in the operation screen based on the position in the portion,
After the transmission means transmits the operation screen, remote operation reception means for receiving a remote operation including in-screen position information indicating a position in the operation screen from the portable information device,
The position determining means determines a position in the operation screen based on in-screen position information included in the received remote operation in response to the remote operation being received by the remote operation receiving means. The image processing apparatus according to claim 8. A remote control method executed by an image processing device remotely operated by a portable information device, comprising:
The image processing device includes an operation receiving unit that receives an operation by a user,
A first display means capable of displaying an image,
A first aspect ratio indicating a ratio of a vertical length to a horizontal length of the display surface of the first display means, and a second aspect ratio indicating a ratio of a vertical length to a horizontal length of the operation screen. When the ratio is smaller than the ratio, at least a part of the operation screen and the portion having the first aspect ratio is determined as the display part, and when the first aspect ratio is equal to or more than the second aspect ratio, the entire operation screen is displayed. A display part determination step for determining the display part,
A display control step of displaying the image of the display portion determined in the display portion determining step on the first display means;
An aspect ratio detecting step of acquiring from the portable information device a third aspect ratio indicating a ratio of the length in the vertical direction to the length in the horizontal direction of the display surface of the second display means included in the portable information device ;
When the first aspect ratio is smaller than the second aspect ratio and when the third aspect ratio is less than or equal to the first aspect ratio, the image of the determined display portion is displayed on the second display means. When the image of the display portion is transmitted to the portable information device and the third aspect ratio is larger than the first aspect ratio, the portable information device for displaying the image of the operation screen on the second display means. And a step of transmitting the image of the operation screen to the remote control method. A remote control program executed by a computer for controlling an image processing device remotely operated by a portable information device,
The image processing device includes an operation receiving unit that receives an operation by a user,
A first display means capable of displaying an image,
A first aspect ratio indicating a ratio of a vertical length to a horizontal length of the display surface of the first display means, and a second aspect ratio indicating a ratio of a vertical length to a horizontal length of the operation screen. When the ratio is smaller than the ratio, at least a part of the operation screen and the portion having the first aspect ratio is determined as the display part, and when the first aspect ratio is equal to or more than the second aspect ratio, the entire operation screen is displayed. A display part determination step for determining the display part,
A display control step of displaying the image of the display portion determined in the display portion determining step on the first display means;
An aspect ratio detecting step of acquiring from the portable information device a third aspect ratio indicating a ratio of the length in the vertical direction to the length in the horizontal direction of the display surface of the second display means included in the portable information device ;
When the first aspect ratio is smaller than the second aspect ratio and when the third aspect ratio is less than or equal to the first aspect ratio, in order to display the image of the determined display portion on the second display means. When the image of the display portion is transmitted to the portable information device and the third aspect ratio is larger than the first aspect ratio, the portable information device for displaying the image of the operation screen on the second display means. And a transmitting step of transmitting the image of the operation screen to the remote control program.

Images