CN113010130B - Remote large-screen interaction control method based on image stream - Google Patents

Abstract

The invention discloses a long-distance large-screen interactive control method based on image streams: 1) During initialization, the smart phone terminal divides the screen into upper and lower partitions, the upper partition (touch operation area) receives screenshot image streams, and the lower partition ( Motion control area) Control the inertial sensor switch, and create a rectangular cursor on the large screen; 2) Long press the partition to control the movement of the rectangular cursor, and the screenshots of the contents within the rectangular cursor range will be sent to the upper partition for display in the form of image stream; 3) When the rectangular cursor Move to the vicinity of the target and lift off the finger, the rectangular cursor is fixed, and the trigger mode on the smart phone is switched; 4) Touch the target point on the upper partition, and determine the selection according to the relative position of the click; 5) After selecting the target point, long press and lift Drag and drop from the lower partition; double-click the selected point on the upper partition to double-click; press and hold the selected point on the upper partition to right-click. The invention effectively solves the problem of large-screen interaction context retention, and improves accuracy and interaction efficiency.

Description

A remote large-screen interactive control method based on image stream

technical field

The invention relates to a long-distance large-screen interactive control method based on image flow, which belongs to the field of computer application human-computer interaction.

Background technique

In recent years, with the development of science and technology, mobile smart devices have gradually become popular due to their convenience. Therefore, more and more studies consider using mobile devices to replace or integrate original operating devices, giving mobile devices more power to carry out Various interactive operations. Among them, the use of mobile devices to remotely control large screens has become one of the research hotspots in the field of human-computer interaction in recent years, such as using smart phones, tablet computers, etc., because these mobile devices are easy to hold and carry, and integrate touch screens, A variety of sensors such as cameras, inertial sensors, gyroscopes, and magnetometers can sense the surrounding environment and user behaviors, and there is great room for imagination in the interactive behaviors that can be realized.

Due to the high resolution and large size of the large screen, large-screen display devices have been widely used and can be applied to a variety of scientific information visualization and group collaborative work scenarios, so the research on large-screen interaction methods is particularly important . Large screens generally use the PC-side operating system, and use the interactive operation of the mouse to click on the target. However, the large size of the large screen allows it to display more content at one time, and the high-resolution feature enables it to present more parts at the same time. Details, in large-screen interaction scenarios, users often need to observe the panorama from a long distance, and also need to observe the details of the context at close range, which increases the user's interaction range, and the original mouse operation cannot meet the needs of large-screen interaction.

Existing technologies for using smart phones to control large-screen clicking can be divided into three categories, namely, based on smart phone cameras, based on light projection, and based on smart phone touch screen interaction methods. These current technologies have more or less problems, such as the need for users to hold the smartphone for a long time, easy fatigue, lack of flexibility and real-time performance, and are also affected by problems such as occlusion, and the size limit of the small screen of the smartphone It will lead to low accuracy and efficiency. In addition, these technologies lack the follow-up interaction method for the selected target point. After the target on the large screen is selected remotely through the smartphone, it cannot be operated, or it still returns to the original mouse interaction mode. There are big problems in the continuity of the interaction process.

Contents of the invention

The technical solution of the present invention is to overcome the deficiencies of the prior art, and provide a long-distance large-screen interactive control method based on image streams, which divides the smart phone terminal into upper and lower partitions to realize the switching between the inertial sensor control mode and the touch screen control mode. Combining the two input channels of the inertial sensor and the touch screen on the smart phone side, adopting the point-and-click interaction technology of the rectangular cursor screenshot image stream transmission, it solves the problem of retaining the interaction context on the large-screen side, and transfers the point-and-click interaction directly from the big-screen side to the smart phone The small screen on the terminal can click and interact, which improves the accuracy and interaction efficiency. The present invention satisfies the subsequent interactive operation similar to the mouse on the large screen end, and maps the gesture operation on the smart phone end with the interactive operation of clicking on the PC mouse on the large screen end one by one, allowing the user to use the smart phone to perform a one-to-one operation on the selected target point on the large screen. A series of control operations become a complete set of interactive methods for remote control of large-screen targets by smart devices. The present invention supports the re-entry of interactive operation. When the hand moves to an uncomfortable angle, the finger can be lifted away from the motion control area of the smart phone to maintain the state of the rectangular cursor. After changing the posture, press and hold the motion control area again to continue to control the movement of the rectangular cursor. And the real-time transmission of the image stream, so as to ensure the comfort of the user when using it. The present invention is not affected by problems such as distance and occlusion.

The technical solution adopted by the present invention to solve the technical problem is: a long-distance large-screen interactive control method based on image stream, including a large-screen terminal and a smart phone terminal, and using the attitude angle information of the built-in inertial sensor of the smart phone at the smart phone terminal Long-distance control of the rectangular cursor on the large screen, the content screenshot of the rectangular cursor area on the large screen is transmitted to the smart phone as an image stream in real time, and the touch control information is used to confirm the selected target point on the smart phone, and then controlled by the inertial sensor mode or touch screen control mode to achieve multiple interactive operations on the selected target point; the method includes an initialization phase, a cursor control phase, a screen capture phase and an interactive gesture phase.

Specifically include the following steps:

In the first step, in the initialization stage, the large screen terminal and the smart phone terminal run an application program respectively, and establish two-way communication through Bluetooth or Wifi;

In the second step, when initializing the interactive interface of the smartphone terminal, the smartphone terminal divides the screen into upper and lower partitions, the upper partition is used as the area for receiving screenshot image streams (called the touch operation area), and the lower partition is used as the inertial sensor control switch area ( It is called the motion control area), and the application on the smart phone sends the attitude angle information or touch information of the inertial sensor to the large screen (as shown in Figure 2, when the finger on the smart phone long presses the motion control area, the sensor motion data is sent to the large screen. ), and receive the screenshot image stream data of the rectangular cursor on the large screen end to display in the touch operation area (as shown in Figure 2 when the rectangular cursor on the large screen end takes a screenshot, the smart phone receives the screenshot image stream of the cursor on the large screen end);

Step 3: When initializing the interactive interface of the large-screen terminal, a rectangular cursor is generated on the large-screen terminal. The aspect ratio of the rectangular cursor is completely similar to that of the touch operation area on the smartphone terminal. The large-screen terminal sends a rectangular area image stream to the smartphone terminal in real time ( As shown in Figure 2, when the rectangular cursor on the large screen takes a screenshot, send the cursor screenshot image stream to the smartphone), and receive the attitude angle information (that is, the heading angle and pitch angle information) of the inertial sensor on the smartphone and map it to a rectangular cursor on the large screen. The x and y coordinates of the terminal (as shown in Figure 2, when the finger on the smartphone terminal presses the motion control area for a long time, the large screen terminal receives the sensor motion data of the smartphone terminal), and also receives the touch information of the smartphone terminal and maps it to mouse-related operations on the large screen terminal (such as Fig. 2 When the finger on the smart phone operates on the touch operation area, the large screen receives the selected position data or the touch screen event status of the smart phone);

The fourth step is to control the movement of the rectangular cursor on the big screen by long pressing the motion control area on the smart phone; the smart phone sends the attitude angle information collected by the inertial sensor to the big screen, and the big screen controls the rectangular cursor based on the attitude angle information Move on the large screen to frame the target point for rough positioning, and the large screen transmits the content within the rectangular cursor range in real time to the touch operation area of the smartphone for display, entering the cursor control stage;

Step 5: In the cursor control stage, when the rectangular cursor moves to the vicinity of the target point, the finger lifts off the motion control area, the rectangular cursor is fixed, and the smartphone no longer sends the attitude angle information of the inertial sensor. At this time, the smartphone triggers the mode Switch, switch from the inertial sensor control mode to the touch screen control mode, and enter the stage of screenshot selection;

Step 6: In the screenshot selection stage, by touching the target point in the touch operation area of the smartphone terminal to perform precise positioning, the smartphone terminal sends the relative position information on the screen to the large screen terminal to calculate the absolute position of the large screen terminal. Position coordinates, judge the selected target point on the large screen, and enter the stage of interactive gestures;

The seventh step, in the stage of interactive gestures, drag and release the selected point and the rectangular cursor by long pressing and lifting the finger on the smartphone end from the motion control area (as shown in Figure 2, after the target is selected, the finger long press motion on the smartphone end From the control area to the large screen end, the rectangular cursor is fixed to release the data flow of the selected point); by double-clicking the selected point on the screenshot of the touch operation area on the smartphone end, the double-click operation on the large screen end is performed (as shown in Figure 2 after the target is selected on the smartphone end Double-click the data flow of the selected point in the touch operation area); press and hold the selected point on the screenshot of the touch operation area on the smartphone side to perform a right-click operation on the large screen (as shown in Figure 2 after selecting the target, long press the touch point on the smartphone side to the data flow of the point selected in the control operation area).

In the second step, the upper and lower sections of the smart phone terminal are divided into two physically divided screen areas with a fixed area ratio.

Calculate the height of the upper and lower partitions of the smart phone. The specific process is as follows: the height of the motion control area of the lower partition is 1.5 to 2.5 times the width of the thumb to avoid incorrect switching of the mode caused by finger movement. According to the ergonomic size, the width of the thumb is 15mm ~25mm; the height of the upper partition touch operation area is the height of the remaining area of the smart phone screen (Figure 4 shows the height design of the upper partition and the lower partition).

In the third step, the rectangular cursor is a rectangle completely similar to the touch operation area of the smart phone.

高度/>

其中，x'为智能手机端屏幕宽度即触控操作区宽度，y'为智能手机端触控操作区高度，D_res为大屏幕端的分辨率，S_res为智能手机端触摸屏的分辨率。为避免大屏幕端分辨率小于智能手机端或智能手机端分辨率大于大屏幕端的极端情况，限制矩形光标要大于大屏幕端最小边长的物理尺寸1/4倍，小于智能手机端触控操作区的物理尺寸4倍。Calculate the width and height of the rectangular cursor, the specific process is as follows: the width of the rectangular cursor

height />

Among them, x' is the screen width of the smartphone, that is, the width of the touch operation area, y' is the height of the touch operation area of the smartphone, D _res is the resolution of the large screen, and S _res is the resolution of the touch screen of the smartphone. In order to avoid extreme situations where the resolution of the large screen is smaller than that of the smartphone or the resolution of the smartphone is larger than that of the large screen, the restricted rectangular cursor must be larger than 1/4 of the physical size of the minimum side length of the large screen, and smaller than the touch operation of the smartphone 4 times the physical size of the zone.

In said step 4, when the smart phone end detects the state of long pressing of the motion control area, the smart phone end is in the inertial sensor control mode state, and the smart phone end transmits the heading angle and pitch angle information collected by the inertial sensor to the large screen end , to control the movement of the rectangular cursor from a long distance. During the movement, the transmission method on the large screen terminal is to take real-time screenshots of the contents within the frame selection range of the rectangular cursor as pictures, and transmit all the screenshots to the touch screen on the smartphone terminal in the form of image stream data. Control operation area to ensure real-time display of contextual information on the large screen when using a rectangular cursor to roughly locate the target point at a long distance.

In the step 5 and step 6, when the smart phone detects that the motion control area is lifted off, the trigger mode of the smart phone is switched from the inertial sensor control mode to the touch screen control mode, and the smart phone is used to remotely Control the rectangular cursor to roughly locate the target point, and then switch to the touch operation area on the smartphone to directly touch and select the precise target point.

In the step six, the inverse calculation process is as follows:

纵坐标/>

其中，x_mob为触控操作区触摸位置的横坐标，y_mob为触控操作区触摸位置的纵坐标，x'为智能手机端屏幕宽度即触控操作区宽度，y'为智能手机端触控操作区高度，x为矩形坐标的宽度，y为矩形坐标的高度。(1) According to the touch position coordinates (x _mob , y _mob ) of the touch operation area on the smartphone terminal, calculate the relative position abscissa of the rectangular cursor

Ordinate />

Among them, x _mob is the abscissa of the touch position in the touch operation area, y _mob is the ordinate of the touch position in the touch operation area, x' is the screen width of the smartphone terminal, that is, the width of the touch operation area, and y' is the touch control area of the smartphone terminal. Control the height of the operation area, x is the width of the rectangle coordinates, and y is the height of the rectangle coordinates.

(2) Calculate the absolute coordinates of the selected point on the large screen according to the absolute position of the center of the rectangular cursor on the large screen, and judge whether the target point is selected according to the calculation result of the absolute coordinates.

In said step seven, when the target point is selected, the subsequent interactive operation on the selected target point is realized through the interactive gesture on the smart phone end, and the selected target point is simulated on the large screen according to the attitude angle information or touch information collected by the inertial sensor on the smart phone end. Click mouse-related operations, the types of simulated interactive gestures are as follows:

(1) By long pressing and lifting away from the motion control area of the smartphone, the attitude angle information collected by the inertial sensor on the smartphone is sent to the large screen, and the image stream of the rectangular cursor screenshot is displayed in real time in the touch operation area. Select the point and the rectangular cursor to perform the mouse drag operation on the large screen;

(2) Double-click the mouse on the large screen by double-clicking the selected point in the touch operation area of the smart phone;

(3) Right-click the mouse on the large screen by long pressing the selected point in the touch operation area of the smartphone.

The interaction process in the present invention includes three stages, two interaction control modes and three interaction gestures, as shown in FIG. 3 . The three phases are cursor control phase, screenshot selection phase and interactive gesture phase. The interactive control mode adopted includes inertial sensor control mode and touch screen control mode. . According to whether the finger is in contact with the screen and how long the finger is in contact with the screen, the on-screen state of the finger can be divided into three states: "long press", "lift off" and "contact". The switching process is when the finger "long press" the motion control area , the cursor control stage is triggered, which is the inertial sensor control mode at this time; when the finger "lifts off" the motion control area, the screenshot selection stage is triggered, and the inertial sensor control mode is converted to the touch screen control mode at this time; and when the finger "touches "When touching the target point in the operation area, the interactive gesture phase is triggered. After entering the interactive gesture stage, there are three interactive gestures, namely "long press to lift off", "double click" and "long press". When the finger "long press" and "lift off" the motion control area, dragging Move and release the selected point and rectangular cursor operation; when the finger "double-clicks" the touch operation area to select the point, the double-click operation is realized; when the finger "long press" the touch operation area to select the point, the right-click operation is realized.

Compared with prior art solutions, the beneficial effects of the present invention are:

(1) The present invention designs an interactive method for cursor screenshot selection, which divides the screen of the smart phone into upper and lower partitions. The purpose is to realize the conversion of the two operating states. Position the frame selection target, and then transfer the direct click interaction from the large screen to the precise click interaction on the small screen on the upper partition of the smart phone. By directly touching and clicking the target point on the screenshot of the rectangular cursor on the upper partition of the smart phone, the remote The selection of the target point at the end of the large screen is more precise, which improves the speed and accuracy of the click method.

(2) The present invention designs an interactive method of a rectangular cursor screenshot image stream, which sends the screenshot of the content framed by the rectangular cursor to the smart phone in real time in the form of an image stream during rough positioning, fully retaining the high resolution of the large screen terminal Context details to ensure that the user's interaction scene is not interrupted.

(3) The present invention proposes a complete interactive process, utilizes the smart phone terminal to remotely select the target point on the large screen end and designs three kinds of interactive gestures, and utilizes the smart phone terminal to complete subsequent mouse-like operations on the selected target, It not only conforms to the linear behavior of mouse movement, but also ensures the coherence and integrity of long-distance large-screen interactive control.

(4) The present invention uses the built-in inertial sensor of the smart phone to locate the position attribute of the rectangular cursor and drag the selected point and the rectangular cursor to send the attitude angle information to the large screen, thereby controlling the movement of the cursor without being affected by distance or occlusion and other issues. At the same time, it supports the re-entry of interactive operations. When the hand moves to an uncomfortable angle, you can lift your finger away from the motion control area on the smartphone to maintain the state of the rectangular cursor. Change the posture of your hand and press and hold the motion control area again to continue to control the rectangular cursor. Real-time transmission of movement and image streams to ensure user comfort during use.

(5) The interactive system of the present invention is simple to build, easy to obtain materials from smart phones, low in cost, has a certain degree of convenience, and supports multi-person collaborative operations at the same time.

Description of drawings

Fig. 1 is a schematic flow sheet of the method of the present invention;

Fig. 2 is a schematic diagram of data flow;

Fig. 3 is a schematic diagram of interactive state transition;

FIG. 4 is a schematic diagram of partitions on the smartphone side;

FIG. 5 is a schematic diagram of an interactive gesture;

(a) Rectangular cursor screenshot (b) Rectangular cursor control (c) Partition click operation (d) Long press partition gesture-drag operation (e) Double-click selected point gesture-double-click operation (f) Long press selected point gesture - Right click operation;

FIG. 6 is a schematic diagram of an interaction process according to an embodiment of the present invention;

(a) Long press the partition with the finger - cursor screenshot (b) Control the rectangular cursor (c) Lift the finger away - the cursor is fixed (d) Touch the finger - select the target point (e) Long press the finger with the partition - the cursor screenshot drags the selected point Lift your finger away from the lower partition - fix the cursor and release the selected point (f) Double-click the selected point with your finger - double-click the selected point (g) Long press the selected point with your finger - right-click the selected point

Fig. 7 is a system layout diagram of the embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The basic idea of the present invention is to select the target point roughly through the rectangular cursor, and then select the target through the precise positioning of the rectangle cursor screenshot received by the smart phone, and finally provide three kinds of interactive gestures to complete similar large-screen mouse operations . The rough positioning of the screenshot of the rectangular cursor frame selection area preserves the context details of the target point on the large screen as much as possible, and the precise positioning of the rectangular cursor screenshot selection can improve the accuracy of the target point selection on the large screen. The coherence of the subsequent operation of the selected target.

As shown in Figure 1, the specific steps included in the remote large-screen control interaction method based on image streams in the present invention are as follows:

The first step is communication initialization.

The large screen terminal and the smart phone terminal run an application program respectively, and establish two-way communication through Bluetooth or Wifi.

The second step is to initialize the mobile terminal.

The smart phone terminal divides the screen into upper and lower partitions. The upper partition is used as an area for receiving screenshot image streams (called touch operation area), and the lower partition is used as an inertial sensor control switch area (called motion control area). Send the attitude angle information or touch information of the inertial sensor to the large screen, and receive the screenshot image stream data of the rectangular cursor on the large screen and display it in the touch operation area.

The third step is to initialize the large screen.

The large-screen end generates a rectangular cursor whose aspect ratio is completely similar to that of the touch operation area on the smartphone end. The large-screen end sends the image stream of the rectangular area to the smartphone end in real time, and receives the attitude angle information of the inertial sensor on the smartphone end ( That is, heading angle and pitch angle information) are respectively mapped to the x and y coordinates of the rectangular cursor on the large screen, and the touch information received from the smart phone is mapped to mouse-related operations on the large screen.

The fourth step is rough positioning.

By long-pressing the motion control area on the smart phone, using the method of long-pressing the screen with the finger, that is, keeping the finger in contact with the screen ((a) in Figure 6 is the state of long-pressing the partition to start the cursor screenshot), and the inertial sensor on the smart phone The collected attitude angle information is sent to the large screen end, and the large screen end controls the movement of the rectangular cursor on the large screen end according to the attitude angle information, and the large screen end transmits the content within the range of the rectangular cursor to the touch operation area of the smartphone end for display in real time. The steps can control the rectangular cursor to move quickly and long-distance to roughly frame the target point ((b) in Figure 6 is the process of controlling the displacement of the rectangular cursor on the smartphone).

The fifth step is precise positioning.

After the rectangular cursor moves to the frame selection target point in step 4, through the event of lifting the finger in the motion control area, the smart phone no longer sends inertial sensor information, and the rectangular cursor is fixed. At this time, the smart phone triggers mode switching, controlled by the inertial sensor Switch the mode to the touch screen control mode, and enter the screenshot selection stage, and accurately locate the target according to the rectangular cursor screenshot in the touch operation area of the smart phone at the moment, while retaining the context of the target point on the large screen (as shown in (c) in Figure 6 Cursor fixed state for finger lift off).

The sixth step is to select the target.

Through the finger touch operation area event, calculate the absolute position of the large screen end according to the relative position selected by the touch operation area, and judge the target point selected by the user on the large screen end ((d) in Figure 6 is the state of the finger touch selected target point ).

The seventh step is gesture interaction.

After selecting the target point in step six, there are three gesture interaction methods. The first is to drag and drop the selected point and the rectangular cursor by long pressing the motion control area on the smartphone, using the method of long pressing the screen with the finger, that is, keeping the finger in contact with the screen, and at the same time start sending the screenshot image stream of the rectangular cursor to the In the touch operation area, after dragging to the destination, release the selected point and the rectangular cursor through the finger lifting event in the motion control area (as shown in (e) in Figure 6, long press and lift off the lower partition to drag and how to release selected points). The second is to trigger a double-click event of the selected point on the large screen by double-clicking the selected point in the touch operation area on the smart phone (as (f) in Figure 6 is the double-click operation of double-clicking the selected point with a finger). The third is to trigger the right-click event of the selected point on the large screen by long pressing the selected point in the touch operation area on the smart phone, and using the finger to long press the screen, that is, the finger keeps in touch with the screen (as shown in Figure 6 The (g) in (g) is the method of long-pressing the selected point and right-clicking with the finger).

In the first step, the height ratio of the upper and lower partitions of the smart phone terminal is determined in the following manner: the height of the motion control area of the lower partition is 1.5 to 2.5 times the width of the thumb to avoid incorrect mode switching caused by finger movement, according to ergonomics Size The width of the thumb is 15mm to 25mm; the height of the upper partition touch operation area is the height of the remaining area of the smart phone screen (as shown in Figure 4, the height design of the upper partition and the lower partition).

In the third step, the width-to-height ratio of the rectangular cursor is respectively determined by the aspect ratio of the partition touch operation area on the smart phone, that is, the rectangular cursor is completely similar to the touch operation area on the smart phone.

The width value of the rectangular cursor can be determined by the following formula:

The height value of the rectangular cursor can be determined by the following formula:

Among them, x is the width of the rectangular cursor, x' is the screen width of the smart phone, that is, the width of the touch operation area, y is the height of the rectangular cursor, y' is the height of the touch operation area of the smart phone, and D _res is the resolution of the large screen rate, the unit is DPI, and S _res is the resolution of the touch screen of the smart phone, the unit is DPI. In order to avoid extreme situations where the resolution of the large screen is smaller than that of the smartphone or the resolution of the smartphone is larger than that of the large screen, the restricted rectangular cursor must be larger than 1/4 of the physical size of the minimum side length of the large screen, and smaller than the touch operation of the smartphone 4 times the physical size of the zone.

In the step 4, the rough positioning interaction method is to collect the relative position change of the angle of the smartphone terminal through the inertial sensor interaction channel of the smartphone terminal, and map the heading angle and pitch angle information of the smartphone terminal to the center of the rectangular cursor respectively. The x and y coordinates change on the large screen side.

In said step 4, the smart phone terminal is in the state of inertial sensor control mode, and the movement of the rectangular cursor is remotely controlled. The screenshots are sent to the touch operation area of the smart phone in the form of image stream data to ensure that the context information on the large screen is displayed in real time when the target point is roughly positioned with a rectangular cursor at a long distance.

The mode switching in steps 4 and 5 and the selection operation in step 6 are all triggered by the on-screen state and area of the finger, and the on-screen state of the finger is divided into "long press", "lift off" and "contact "Three states, when the finger "long press" the motion control area, the rough positioning stage is triggered, and the smart phone is moved to control the movement of the rectangular cursor, and at the same time, the screenshot of the content selected by the rectangular cursor is sent to the smart phone in real time in the form of an image stream Touch operation area; when the finger "lifts off" the motion control area, the precise positioning stage is triggered; and when the finger "touches" the touch operation area, the large screen side judges the selection based on the relative positioning of the finger touching the touch operation area. Target point, trigger the select target command.

In the step six, the inverse calculation process is as follows:

(1) According to the touch position coordinates (x _mob , y _mob ) of the touch operation area on the smartphone, the abscissa of the relative position in the rectangular cursor can be determined by the following formula:

The ordinate of the relative position in the rectangular cursor can be determined by the following formula:

Among them, x _cur is the abscissa of the relative position of the rectangular cursor, y _cur is the ordinate of the relative position of the rectangular cursor, x _mob is the abscissa of the touch position in the touch operation area, y _mob is the ordinate of the touch position in the touch operation area, x' is the screen width of the smart phone, that is, the width of the touch operation area, y' is the height of the touch operation area of the smart phone, x is the width of the rectangle coordinates, and y is the height of the rectangle coordinates.

(2) Calculate the absolute coordinates of the selected point on the large screen according to the absolute position of the center of the rectangular cursor on the large screen, and judge whether the target point is selected according to the calculation result of the absolute coordinates.

In the step seven, there are three types of simulated interactive gestures, which are triggered by the on-screen state and area of the finger. The finger-on-screen states of the three interactive gestures are divided into "long press to lift off", "double-click" and " Long press "Three states:

(1) When the finger "long presses" and "lifts off" the motion control area, the screenshot image stream of the rectangular cursor is sent to the touch operation area in real time, and according to the attitude angle information collected by the inertial sensor of the smartphone end, the large screen end The selected point and the rectangular cursor implement drag and release operations;

(2) When the finger "double-clicks" the selected point in the touch operation area, double-click the selected point on the large screen;

(3) When the finger "long presses" the selected point in the touch operation area, perform a right-click operation on the selected point on the large screen.

As shown in Figure 7, in order to realize the method of the present invention, implement and use the bigger big screen of physical size, memory is 24G, and operating system is Microsoft Windows 7; Handheld mobile device adopts smart phone, and screen size is 5.5 inches, and screen height The width and width are 150.7mm and 75.3mm respectively, the resolution is 403ppi, the processor is Helio X10 processor octa-core, and the memory is 2GB.

The system layout of the embodiment is shown in Figure 7. The effect of this embodiment is that the user completes the interactive task of controlling the rectangular cursor by using the long press motion of the thumb in the motion control area and the movement of the smart phone. Lift the thumb away from the motion control area to enter the screenshot selection operation stage. Use your finger to click the target point in the touch operation area to trigger the command to select the target and enter the interactive gesture stage. Use the long press motion of the thumb in the motion control area and the movement of the smart phone to drag the selected point and the rectangular cursor on the large screen. After dragging the selected point and the rectangular cursor to the destination, lift off the motion control with the thumb area to release the selected point and the rectangular cursor. Using the touch movement of the finger in the touch operation area, double-click the selected point to realize the double-click operation command on the large screen end, and long press the selected point to realize the right-click operation command on the large screen end.

The interaction process of this embodiment is shown in FIG. 6 .

In the first step, user 1 establishes a connection between the smart phone terminal 1 and the large screen terminal, and starts to interact;

In the second step, in the initial state, long-press the motion control area of the smart phone with a finger, and the rectangular cursor is activated. The user moves the smart phone to send an interactive data packet to the big screen through the inertial sensor channel, and the big screen analyzes the motion and maps the is the displacement of the rectangular cursor, and the touch operation area of the smart phone terminal displays the screenshot image stream of the content selected by the rectangular cursor in real time;

In the third step, when the rectangular cursor moves to the frame to select the target point, the touch screen control mode is activated by lifting the finger away from the motion control area, and the rectangular cursor is fixed at this time;

In the fourth step, the user selects the target by touching the touch operation area of the smart phone with a finger.

Step 5: When the target point on the large screen is in the selected state, the user presses the motion control area for a long time, and the smartphone sends an interactive data packet to the large screen through the inertial sensor channel. After analysis, the large screen maps the motion to the selected point With the displacement of the rectangular cursor, the touch operation area of the smart phone terminal displays the screenshot image stream of the content selected by the rectangular cursor in real time. The user double-clicks the selected point in the touch operation area to control the selected point to perform a double-click operation. The user long-presses the selected point in the touch operation area to control the selected point to perform a right-click operation.

The above is only a specific implementation mode in the present invention, but the scope of protection of the present invention is not limited thereto. Anyone familiar with the technology can understand the conceivable transformation or replacement within the technical scope disclosed in the present invention. All should be covered within the scope of the present invention, therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (7)

1. A remote large-screen interaction control method based on image flow is characterized in that: the method comprises a large screen end and a smart phone end, wherein a rectangular cursor on a large screen of the large screen end is remotely controlled at the smart phone end by utilizing attitude angle information of an inertial sensor built in the smart phone, a screenshot of the rectangular cursor area on the large screen is transmitted to the smart phone end in real time as an image stream, a click target point is confirmed at the smart phone end by utilizing touch control information, and then various interactive operations on the selected target point are realized by an inertial sensor control mode or a touch screen control mode; the method comprises an initialization stage, a cursor control stage, a screenshot clicking stage and an interaction gesture stage, and is specifically realized as follows:

firstly, in an initialization stage, an application program is respectively operated by a smart phone end and a large screen end, and meanwhile, the smart phone end and the large screen end establish two-way communication through Bluetooth or Wifi;

the second step, the smart phone end physically divides the screen into an upper partition and a lower partition, wherein the upper partition is used as a region for receiving the screenshot image stream and is called a touch operation region; the lower partition is used as an inertial sensor control switch area and is called a motion control area, an application program at the smart phone end sends attitude angle information or touch information of the inertial sensor to the large screen end, receives screenshot image stream data of the rectangular cursor at the large screen end and displays the screenshot image stream data in the touch operation area;

thirdly, the application program of the large screen end creates a rectangular cursor, the aspect ratio of the rectangular cursor is completely similar to the touch operation area of the smart phone end, the large screen end sends the regional image stream of the rectangular cursor to the smart phone end in real time and receives attitude angle information of an inertial sensor of the smart phone end, namely course angle and pitch angle information are respectively mapped into x and y coordinates of the rectangular cursor at the large screen end, and touch information of the smart phone end is also received and mapped into mouse related operations of the large screen end;

fourthly, by long-pressing a motion control area of the smart phone end, an application program of the smart phone end sends attitude angle information acquired by the inertial sensor to the large screen end, the rectangular cursor of the large screen end is controlled to move to select a target point for rough positioning, meanwhile, the large screen end transmits the content in the range of the rectangular cursor to a touch operation area of the smart phone end in real time for display, and a cursor control stage is started;

fifthly, in a cursor control stage, when the rectangular cursor moves to the position near a target and is lifted away from the finger, the rectangular cursor at the large screen end is fixed, the smart phone end does not send information of the inertial sensor any more, at the moment, the smart phone end triggers mode switching, the mode is switched from the inertial sensor control mode to the touch screen control mode, and a screenshot clicking stage is entered;

sixthly, in the screenshot clicking stage, accurately positioning a target point to be selected in the screenshot through touch of a touch operation area of the smart phone end, calculating the absolute position of the large screen end in a reverse mode according to the clicked relative position of the touch operation area, judging the target point of the large screen end to be selected, and entering an interactive gesture stage;

seventhly, in the gesture interaction stage, after a target point is selected, the selected target point and a rectangular cursor are dragged on a large screen end by long pressing and lifting away from a motion control area of the smart phone end; double-click operation of the large screen end is realized by double-clicking the selected target point in the touch operation area; and realizing the right click operation of the large screen end by long-pressing the selected target point of the touch operation area.

2. The method of claim 1, wherein: in the second step, the height of the motion control area is 1.5-2.5 times of the width of the thumb, so that the mode is prevented from being switched mistakenly due to the movement of fingers, the width of the thumb is 15mm to 25mm according to the ergonomic size, and the height of the touch control operation area is the height of the remaining area of the screen of the smart phone end.

3. The method of claim 1, wherein: in the third step, the width of the rectangular cursor

Height of

(ii) a Wherein x 'is the width of the screen of the smart phone terminal, namely the width of the touch operation area, y' is the height of the touch operation area of the smart phone terminal, and D _res Resolution at large screen end, S _res The resolution ratio of the touch screen at the smart phone end is defined, namely the rectangular cursor and the touch operation area at the smart phone end are completely similar; in order to avoid the extreme condition that the resolution of the large screen end is smaller than that of the smart phone end or the resolution of the smart phone end is larger than that of the large screen end, the physical size of the rectangular cursor larger than the minimum side length of the large screen end is limited to be 1/4 times, and the physical size of the rectangular cursor is limited to be 4 times smaller than that of a touch operation area of the smart phone end.

4. The method of claim 1, wherein: and in the fourth step, when the smart phone end is in an inertial sensor control mode, remotely controlling the movement of the rectangular cursor, and in the moving process, capturing the content of the rectangular cursor frame selection range into a picture in real time, and transmitting all captured pictures to a touch operation area of the smart phone end in the form of image stream data so as to ensure that the context information of the large screen end is displayed in real time when the target point is roughly positioned by the rectangular cursor remotely.

5. The method of claim 1, wherein: in the fifth step and the sixth step, when the smartphone end detects a state that the motion control area is lifted away, the smartphone end trigger mode is switched to a state that the inertial sensor control mode is switched to a touch screen control mode, the smartphone end is used for remotely controlling the rectangular cursor to roughly position the target point, and then the smartphone end is switched to a touch operation area of the smartphone end to directly touch and click the accurate positioning target point.

6. The method of claim 1, wherein: in the sixth step, the process of inversely calculating the absolute position of the large screen end according to the relative position clicked by the touch operation area is as follows:

(1) According to the touch position coordinates of the touch operation area of the smart phone end (x _mob ,y _mob ) Inverse calculation of relative position abscissa in rectangular cursor

Ordinate of

(ii) a Wherein,x _mob is the abscissa of the touch position of the touch operation area,y _mob the coordinate is a vertical coordinate of a touch position of the touch operation area, x 'is the width of a screen of the smart phone end, namely the width of the touch operation area, y' is the height of the touch operation area of the smart phone end, x is the width of the rectangular cursor, and y is the height of the rectangular cursor;

(2) And calculating the absolute coordinate of the clicked point at the large screen end according to the absolute position of the center of the rectangular cursor at the large screen end, and judging whether to select a target point according to the absolute coordinate calculation result.

7. The method of claim 1, wherein: in the seventh step, after the target point is selected, the subsequent interactive operation on the selected target point is realized through an interactive gesture at the smart phone end, the mouse operation of the large screen end on the selected target point is simulated according to the attitude angle information or the touch information collected by the inertial sensor at the smart phone end, and the types of the interactive gesture are as follows:

(1) Sending attitude angle information acquired by an inertial sensor of the smart phone end to a large screen end by long-pressing and lifting away from a smart phone end motion control area, displaying an image stream of a rectangular cursor screenshot in real time in a touch operation area, and carrying out large screen end mouse dragging operation on the selected target point and the rectangular cursor;

(2) Performing double-click operation on the large-screen-end mouse by double-clicking the selected target point in the touch operation area of the smart phone end;

(3) And carrying out right click operation on the large screen end mouse by long-pressing the selected target point in the touch operation area of the smart phone end.

Images