JP6574295B1 - Information processing system, information processing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing system, a processing method, and a computer program for automating the generation of product molding data. A user terminal and a server device connected to the user terminal via a network are provided, and the first processing unit or the second processing unit includes a moving image or a still image captured by the first imaging unit. It is determined whether the target information satisfies the information amount necessary for generating the three-dimensional data of the member corresponding to the mounting target. As a result of the determination, the target information satisfies the information amount necessary for generating the three-dimensional data. If it is, the 3D data of the member is generated, and the second communication unit transmits the 3D data to the external device. [Selection] Figure 1

Description

  The present invention relates to an information processing system, an information processing method, and a computer program.

  In recent years, the development of a wearable device that is a wearable device worn on an ear has been remarkable. With the development of this hearable device, it is expected that the number of models equipped with sophisticated sensing devices will increase (for example, Non-Patent Document 1).

  Most of these sensing devices are desirably in close contact with the human body, and it is appropriate to adopt a Custom in-Ear Monitor (IEM) earphone shape when developing an earphone-type hearable.

NEC Corporation, “NEC, Prototype of Hearable Device”, [online], May 23, 2017, NEC Corporation Press Release, [Search June 26, 2018], Internet <https: //jpn.nec.com/press/201705/20170523_01.html>

  However, conventionally, IEM earphones have been positioned as expensive products for studio musicians and some high amateurs, and have a high threshold.

  The reason for this is that, first of all, when making a shape that fits the individual's ear, it is necessary to collect an ear shape, and the user goes to a store with a dedicated measuring instrument, so the sales opportunities are limited There is.

  Secondly, the collected ear shape cannot be made into a product shape as it is, conversion to digital data by a 3D scanner, data cleansing work to remove noise portions unnecessary for molding from the converted data, etc. There is a reason that there is a process that requires human intervention and is not suitable for mass production.

  Therefore, an object of the present invention is to provide an information processing system, an information processing method, and a computer program for generating three-dimensional data of a product more easily than in the past.

  An information processing system of the present invention is an information processing system including a user terminal and a server device connected to the user terminal via a network, and the user terminal is a first imaging unit for imaging a mounting target A first communication unit for transmitting and receiving information to and from the first processing unit, the server device includes a second communication unit for transmitting and receiving information to and from the user terminal and an external device, and a second processing unit The first processing unit or the second processing unit includes information necessary for generating the three-dimensional data of the member corresponding to the mounting target, the target information including the moving image or the still image captured by the first imaging unit. If the target information satisfies the information amount necessary for generating the three-dimensional data as a result of the determination, the three-dimensional data of the member is generated, and the second communication unit , 3D data outside And transmitting devices.

  As a result of the determination, the first processing unit or the second processing unit can generate information for requesting additional target information when the target information does not satisfy the amount of information necessary for generating the three-dimensional data.

  The generation of the three-dimensional data can be performed by the first processing unit.

  The generation of the three-dimensional data can be performed by the second processing unit.

  The first processing unit or the second processing unit can generate point cloud data to be mounted from the target information, and can generate mesh data to be mounted from the point cloud data.

  The first processing unit or the second processing unit can extract the outer shape of the member from the mesh data, and generate three-dimensional data corresponding to the mounting target based on the outer shape of the member.

  Based on the depth information of the mesh data, the first processing unit or the second processing unit obtains contour contour curves by slicing the mesh multiple times in the height direction starting from the deepest location of the mesh data. Can be extracted.

  The first processing unit or the second processing unit can smooth the contour curve by curve interpolation and generate surface data passing through the contour line to obtain three-dimensional data.

  The first processing unit or the second processing unit selects one template data from the template data of the members stored in advance in a predetermined storage device based on the target information, and expands the selected template data Alternatively, by reducing the size, it is possible to determine a size at which at least one point is in contact with the mesh data to be attached, and use the template data of the size as three-dimensional data.

  The first processing unit or the second processing unit can perform mask processing on the moving image or the still image captured by the first imaging unit, specify the mounting target, and use it as target information.

  The first processing unit or the second processing unit can generate a guide image or guide sound for imaging the mounting target and output it from the user terminal.

  The processing method of the present invention is a processing method executed in an information processing system including a user terminal and a server device connected to the user terminal via a network, and the user terminal captures a mounting target. The first imaging unit, the first processing unit, and the first communication unit for transmitting and receiving information to and from the server device, and the server device is a second communication unit for transmitting and receiving information to and from the user terminal and the external device And the second processing unit, and the target information including the moving image or the still image captured by the first imaging unit is included in the first processing unit or the second processing unit of the three-dimensional data of the member corresponding to the mounting target Determining whether the amount of information necessary for generation is satisfied and, as a result of the determination, if the target information satisfies the amount of information necessary for generating three-dimensional data, generating the three-dimensional data of the member , The second communication unit, characterized in that it comprises a step of transmitting three-dimensional data to an external device.

  The computer program of the present invention is a computer program executed in an information processing system including a user terminal and a server device connected to the user terminal via a network, and the user terminal is for imaging a mounting target. The first imaging unit, the first processing unit, and the first communication unit for transmitting and receiving information to and from the server device, and the server device includes a second communication unit for transmitting and receiving information to and from the user terminal and the external device A second processing unit, and target information including a moving image or a still image captured by the first imaging unit is generated in the first processing unit or the second processing unit, and three-dimensional data of a member corresponding to the mounting target is generated. If the target information satisfies the information amount necessary for generating the three-dimensional data as a result of the determination, the three-dimensional To realize the function of generating over data, the second communication unit, characterized in that to realize the function of transmitting three-dimensional data to an external device.

  According to the present invention, generation of three-dimensional data of a product can be automated.

It is a conceptual diagram which shows the system configuration | structure of the information processing system of this invention. It is a block diagram which shows the structure of the user terminal which concerns on the information processing system of this invention. It is a block diagram which shows the structure of the server apparatus which concerns on the information processing system of this invention. It is a conceptual diagram which shows the concept of the mounting target and product which concern on the information processing system of this invention. It is a conceptual diagram which shows the concept of the target information which concerns on the information processing system of this invention. It is a conceptual diagram which shows the concept of the contour line which concerns on the information processing system of this invention. It is a flowchart which shows the processing method of this invention. It is a functional block diagram regarding the computer program of this invention.

  Embodiments of an information processing system, a processing method, and a computer program according to the present invention will be described with reference to the drawings.

  First, an embodiment of an information processing system of the present invention will be described with reference to FIG.

  As shown in FIG. 1, the information processing system 100 of the present invention includes a user terminal 200 and a server device 300 connected to the user terminal 200 via a network.

  The user terminal 200 can be an information processing terminal such as a smartphone, a tablet, or a personal computer.

  As shown in FIG. 2, the user terminal 200 includes a first imaging unit 210, a first processing unit 220, and a first communication unit 230.

  The first imaging unit 210 is for imaging a mounting target, and may be a camera unit built in the user terminal 200 or a camera device connected to the user terminal 200 by wire or wirelessly. it can.

  In this embodiment, the wearing target is described as being the user's “auricle”, but is not limited thereto, and for example, a wearable device such as a wrist, neck, head, face, or finger can be worn. It can be a part of a human body.

  The first imaging unit 210 can image the mounting target in the form of a moving image or a still image.

  The first processing unit 220 executes each process. Details of each process will be described later.

  The first communication unit 230 is for transmitting and receiving information to and from the server device 300. This transmission / reception is performed via the network.

  As shown in FIG. 3, the server device 300 includes a second communication unit 310 and a second processing unit 320.

  The second communication unit 310 is for transmitting and receiving information to and from the user terminal 200 and the external device 400. These transmissions / receptions are performed via the network.

  As shown in FIG. 1, the external device 400 can be connected to the server device 300 via a network.

  The second processing unit 320 executes each process. Details of each process will be described later.

  The first processing unit 220 or the second processing unit 320 is configured so that the target information including the moving image or the still image captured by the first imaging unit 210 determines the amount of information necessary for generating the three-dimensional data of the member corresponding to the mounting target. If the target information satisfies the information amount necessary for generating the three-dimensional data as a result of the determination, the three-dimensional data of the member is generated.

  The three-dimensional data is data necessary for manufacturing a member as a three-dimensional shape, and can be data necessary for manufacturing a three-dimensional shape by an external device (for example, a 3D printer or the like) described later.

  As shown in FIG. 4 as an example, the member is integrated with a member to be described later to become a product (earphone 10), and is configured to be able to be in close contact with at least a part of the wearing target (auricle 20).

  The amount of information required for generating the three-dimensional data of this member is about 10 images captured at different angles if it is a still image, and about several seconds captured at a moving angle if it is a movie. It can be. FIG. 5 shows an image of a pinna photograph taken at different angles. Note that FIG. 5 is a photographed still image in which only the region of the auricle is masked by image recognition and machine learning techniques.

  The member can be constituted by a speaker for generating sound, a microphone for a telephone call or command input by the user, and / or a control button for giving an instruction by the user.

  The microphone and the control button may be integrated with the member, or may be a case separate from the member and designed to communicate with the member in a wired or wireless manner. It is desirable that the member has a wireless connection function for connection with a user terminal or a predetermined server device. For the wireless connection function, Bluetooth (registered trademark) or BLE (Bluetooth Low-Energy) technology can be used. Alternatively, the member may be configured to be connected to the user terminal 200 by wire.

  And a member can be further provided with a blood flow sensor for measuring a pulse wave as a sensing member. It is desirable that the blood flow sensor is provided at a part to be crimped to the ear hole or the earlobe of the member or can be crimped and fixed to the user's earlobe with a clip or the like. By doing so, it is possible to accurately measure pulse waves that are not easily disturbed by noise.

  In addition, the member can include a gyro sensor for measuring the orientation of the user's head, a geomagnetic sensor (compass), and an acceleration sensor for measuring the user's movement. These sensing members are preferably fixed in the member body. In this way, it is possible to accurately measure the position and direction of the head of the user wearing the member.

  Alternatively, the sensor is provided on an object such as glasses or a hat that has a fixed positional relationship with the orientation of the user's head, or is fixed with a clip or the like, and is connected to the member, the user terminal 200, or The server apparatus 300 may be provided with sensor information.

  The member may include a proximity sensor for detecting whether the product is worn by the user. Furthermore, you may provide other types of sensors, such as an electroencephalogram sensor and GPS.

  Then, the second communication unit 310 transmits the three-dimensional data to the external device 400.

  An example of the external device 400 is a 3D printer, but it is not particularly limited as long as it is a device capable of modeling three-dimensional data into a three-dimensional shape.

  According to the above configuration, it is possible to automate the generation of the three-dimensional data of the product. Thereby, for example, a wearable device for general users, particularly a wearable device such as an IEM earphone, can be provided at a lower price.

  In addition, when the target information does not satisfy the information amount necessary for generating the three-dimensional data as a result of the determination, the first processing unit 220 or the second processing unit 320 generates information for requesting additional target information. be able to.

  The information that requests additional target information is, for example, information that requests the user to capture an additional still image or moving image, and is displayed as text or an image on the display unit of the user terminal. Alternatively, it can be output as audio.

  According to this configuration, generation of incomplete three-dimensional data can be prevented.

  The generation of the three-dimensional data can be performed by the first processing unit 220. When the first processing unit 220 is for the user terminal, the processing can be stably performed regardless of the communication speed and communication environment between the user terminal and the server device.

  The generation of the three-dimensional data can also be performed by the second processing unit 320. When the second processing unit 320 is a server device, the processing is performed by the server device having a high processing capability regardless of the processing capability of the user terminal, so that the processing can be performed at high speed.

  Next, an example of a method for generating three-dimensional data from target information will be described. The following example is an example, and does not exclude from the scope of the right to generate three-dimensional data by other methods.

  First, the first processing unit 220 or the second processing unit 320 generates point cloud data to be mounted from the target information, and generates mesh data to be mounted from the point cloud data.

  Then, the first processing unit 220 or the second processing unit 320 can extract the outer shape of the member from the mesh data, and generate three-dimensional data corresponding to the mounting target based on the outer shape of the member.

  More specifically, the first processing unit 220 or the second processing unit 320 slices the mesh multiple times in the height direction from the deepest point of the mesh data based on the depth information of the mesh data. The contour of the member can be extracted by obtaining contour lines having contour lines as shown in FIG.

  Then, the first processing unit 220 or the second processing unit 320 can smooth the contour curve by curve interpolation, generate surface data passing through the contour line, and obtain three-dimensional data.

  Alternatively, the first processing unit 220 or the second processing unit 320 selects one template data from template data of members stored in a predetermined storage device in advance based on the target information, and the selected By enlarging or reducing the template data, the size at which at least one point touches the mesh data to be attached can be determined, and the template data of the size can be made into three-dimensional data.

  In addition, the first processing unit 220 or the second processing unit 320 can perform mask processing on the moving image or the still image captured by the first imaging unit 210, specify the mounting target, and use it as target information.

  According to such a configuration, it is possible to generate three-dimensional data with a higher system.

  Moreover, the 1st process part 220 or the 2nd process part 320 can produce | generate the guide image or guide audio | voice for imaging a mounting target, and can output it from a user terminal.

  For example, the guide image can be an object indicating the outer shape of the mounting target. Then, while viewing the object displayed on the display unit of the user terminal 200, the user can take an image so that his / her wearing target fits in this object. At this time, the first imaging unit 210 is preferably an in-camera provided on the same side as the display unit of the user terminal 200.

  At this time, the display unit displays the mounting target that the first imaging unit 210 is about to capture through the in-camera, but the display unit is difficult to see from the user (for example, when the mounting target is an ear or a head). It is appropriate to output a guide voice.

  For example, the guide voice may be a voice such as “Please move the camera to the right” or “Please move the camera away”.

  Next, an embodiment of the processing method of the present invention will be described with reference to FIG.

  The processing method of the present invention is a processing method executed in an information processing system including a user terminal and a server device connected to the user terminal via a network.

  The user terminal includes a first imaging unit for imaging a mounting target, and a first communication unit for transmitting and receiving information to and from the first processing unit and the server device. The details of the first communication unit are as described above. These details are as described in the explanation of the information processing system.

  The server device includes a second communication unit for transmitting / receiving information to / from the user terminal and the external device, and a second processing unit. These details are as described in the explanation of the information processing system.

  As shown in FIG. 7, the processing method of the present invention includes step S510 for generating three-dimensional data and step S520 for transmitting three-dimensional data.

  In step S510 for generating the three-dimensional data, the target information including the moving image or the still image captured by the first imaging unit is sent to the first processing unit or the second processing unit. Whether or not the amount of information necessary for generation is satisfied is determined. As a result of the determination, if the target information satisfies the amount of information necessary for generating three-dimensional data, the three-dimensional data of the member is generated.

  In step S520 for transmitting the three-dimensional data, the second communication unit transmits the three-dimensional data to the external device.

  According to the above configuration, it is possible to automate the generation of the three-dimensional data of the product. Thereby, for example, a wearable device for general users, particularly a wearable device such as an IEM earphone, can be provided at a lower price.

  Finally, an embodiment of the computer program of the present invention will be described with reference to FIG.

  The processing method of the present invention is a computer program executed in an information processing system including a user terminal and a server device connected to the user terminal via a network.

  The user terminal includes a first imaging unit for imaging a mounting target, and a first communication unit for transmitting and receiving information to and from the first processing unit and the server device. These details are as described in the explanation of the information processing system.

  The server device includes a second communication unit for transmitting / receiving information to / from the user terminal and the external device, and a second processing unit. These details are as described in the explanation of the information processing system.

  And the computer program of this invention implement | achieves the function which makes a 1st process part or a 2nd process part produce | generate 3D data, and makes the 2nd communication part transmit the 3D data. These functions are realized by the first circuit 510 and the second circuit 520 shown in FIG. Each of these circuits corresponds to either the first processing unit 220 or the second processing unit 320 described above.

  The function of generating three-dimensional data is that the first processing unit or the second processing unit generates the three-dimensional data of the member corresponding to the mounting target when the target information including the moving image or the still image captured by the first imaging unit If the target information satisfies the information amount necessary for generating the three-dimensional data as a result of the determination, the three-dimensional data of the member is generated.

  The function of transmitting three-dimensional data causes the second communication unit to transmit molding data to an external device.

  According to the above configuration, it is possible to automate the generation of the three-dimensional data of the product. Thereby, for example, a wearable device for general users, particularly a wearable device such as an IEM earphone, can be provided at a lower price.

  Moreover, in order to function as a server apparatus or a terminal apparatus according to the above-described embodiment, an information processing apparatus such as a computer or a mobile phone can be suitably used. Such an information processing apparatus stores a program describing processing contents for realizing each function of the server apparatus or the terminal apparatus according to the embodiment in a storage unit of the information processing apparatus, and the program is executed by the CPU of the information processing apparatus. This can be realized by reading and executing.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  The method described in the embodiment is a program that can be executed by a computer (computer), for example, a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM, DVD, MO, etc.), It can be stored in a recording medium such as a semiconductor memory (ROM, RAM, flash memory, etc.), or transmitted and distributed via a communication medium. Note that the program stored on the medium side includes a setting program that configures in the computer software means (including not only the execution program but also a table and data structure) to be executed by the computer. A computer that implements this apparatus reads a program recorded on a recording medium, constructs software means by a setting program as the case may be, and executes the processing described above by controlling the operation by this software means. The recording medium referred to in this specification is not limited to distribution, but includes a storage medium such as a magnetic disk or a semiconductor memory provided in a computer or a device connected via a network. The storage unit may function as, for example, a main storage device, an auxiliary storage device, or a cache memory.

DESCRIPTION OF SYMBOLS 100 Information processing system 200 User terminal 210 1st imaging part 220 1st process part 230 1st communication part 300 Server apparatus 310 2nd communication part 320 2nd process apparatus 400 External apparatus

Claims (13)

An information processing system comprising a user terminal and a server device connected to the user terminal via a network,
The user terminal is
A first imaging unit for imaging a mounting target;
A first processing unit and a first communication unit for transmitting and receiving information to and from the server device,
The server device
A second communication unit for transmitting and receiving information to and from the user terminal and an external device;
A second processing unit,
The first processing unit or the second processing unit is configured such that the target information including the moving image or the still image captured by the first imaging unit is an amount of information necessary for generating the three-dimensional data of the member corresponding to the mounting target If the target information satisfies the amount of information necessary for generating the three-dimensional data, the three-dimensional data of the member is generated.
The second communication unit transmits the three-dimensional data to the external device ;
The first processing unit or the second processing unit generates a guide voice for imaging the mounting target, and outputs the guide voice from the user terminal.
The guide sound is a sound relating to a position between the wearing target and the user terminal, which changes according to a positional relationship between the user and the user terminal based on a moving image or a still image captured by the first imaging unit. An information processing system.   The first processing unit or the second processing unit generates information for requesting additional target information when the target information does not satisfy an amount of information necessary for generating the three-dimensional data as a result of the determination. The information processing system according to claim 1.   The information processing system according to claim 1, wherein the three-dimensional data is generated by the first processing unit.   The information processing system according to claim 1, wherein the three-dimensional data is generated by the second processing unit. The first processing unit or the second processing unit is
Generating point cloud data of the mounting target from the target information;
The information processing system according to any one of claims 1 to 4, wherein the mesh data to be mounted is generated from the point cloud data.   The first processing unit or the second processing unit extracts an outer shape of the member from the mesh data, and generates three-dimensional data corresponding to the mounting target based on the outer shape of the member. The information processing system according to claim 5.   Based on the depth information of the mesh data, the first processing unit or the second processing unit starts contour contour curves by slicing the mesh a plurality of times in the height direction from the deepest point of the mesh data. The information processing system according to claim 6, wherein the outer shape of the member is extracted by obtaining the information.   The said 1st process part or said 2nd process part makes the said contour curve smooth by curve interpolation, produces | generates the surface data which passes through the said contour line, It is set as the said three-dimensional data, It is characterized by the above-mentioned. Information processing system. The first processing unit or the second processing unit is
Based on the target information, select one template data from the template data of the member stored in a predetermined storage device in advance,
By enlarging or reducing the selected template data, the size at which at least one point touches the mesh data to be mounted is determined,
6. The information processing system according to claim 5, wherein the template data of the size is the three-dimensional data. The first processing unit or the second processing unit is
The masking process is performed on the moving image or the still image captured by the first imaging unit, and the mounting target is specified as the target information. Information processing system. The first processing unit or the second processing unit is
Said mounting object generates a guide image for imaging the information processing system according to claim 1, any one of 10, wherein the output from the user terminal. A processing method executed in an information processing system including a user terminal and a server device connected to the user terminal via a network,
The user terminal is
A first imaging unit for imaging a mounting target;
A first processing unit and a first communication unit for transmitting and receiving information to and from the server device,
The server device
A second communication unit for transmitting and receiving information to and from the user terminal and an external device;
A second processing unit,
The amount of information necessary for the target information including the moving image or the still image captured by the first imaging unit in the first processing unit or the second processing unit to generate the three-dimensional data of the member corresponding to the mounting target If the target information satisfies the information amount necessary for generating the three-dimensional data, the step of generating the three-dimensional data of the member;
Causing the second communication unit to transmit the three-dimensional data to the external device ;
Causing the first processing unit or the second processing unit to generate a guide voice for imaging the mounting target and outputting from the user terminal ;
The guide sound is a sound relating to a position between the wearing target and the user terminal, which changes according to a positional relationship between the user and the user terminal based on a moving image or a still image captured by the first imaging unit. der Ru processing method. A computer program executed in an information processing system including a user terminal and a server device connected to the user terminal via a network,
The user terminal is
A first imaging unit for imaging a mounting target;
A first processing unit and a first communication unit for transmitting and receiving information to and from the server device,
The server device
A second communication unit for transmitting and receiving information to and from the user terminal and an external device;
A second processing unit,
The amount of information necessary for the target information including the moving image or the still image captured by the first imaging unit in the first processing unit or the second processing unit to generate the three-dimensional data of the member corresponding to the mounting target If the target information satisfies the amount of information necessary for generating the three-dimensional data, the function of generating the three-dimensional data of the member is realized. ,
The second communication unit realizes a function of transmitting the three-dimensional data to the external device ,
Causing the first processing unit or the second processing unit to generate a guide voice for imaging the mounting target, and realizing a function of outputting from the user terminal;
The guide sound is a sound relating to a position between the wearing target and the user terminal, which changes according to a positional relationship between the user and the user terminal based on a moving image or a still image captured by the first imaging unit. der Ru computer program.

Images