JP2025067260A - Keyboard musical instrument - Google Patents

Abstract

To allow multiple performers in an ensemble to achieve a sense of unity.SOLUTION: A keyboard musical instrument in one embodiment includes: keys; a sound board; a receiving unit that receives performance data including performance contents for the keyboard musical instrument at other communication bases; a generating unit that generates drive signals according to the performance data; and a sound generator that makes sound based on the drive signals. The performance data includes key position signals indicating key depression amounts on the keyboard musical instrument. The sound generator includes a shaker connected to speakers or the sound board. The drive signal includes a first drive signal for driving the keys to reproduce the key depression amounts according to the key position signal, and a second drive signal for driving the speakers or the shaker based on the key position signal. The generating unit generates the second drive signal so as to pronounce sound at predetermined positions in the middle of the key presses and at a predicted loudness by the time the keys reach the predetermined positions.SELECTED DRAWING: Figure 4

Description

The present invention relates to keyboard instruments.

Technology has been developed that allows multiple communication points where musical instruments are played to be connected via a network, making it possible for ensemble playing to be performed even when the instruments are in different locations. For example, Patent Document 1 discloses technology for reducing the effects of communication delays to achieve a comfortable ensemble playing experience.

JP 2005-195982 A

When playing together at multiple communication points, it is difficult for multiple performers to achieve a sense of unity in many ways compared to playing together in the same place.

One of the aims of this invention is to allow multiple performers playing together to feel a sense of unity.

In one embodiment, the keyboard instrument includes keys, a soundboard, a receiver that receives performance data including performance content for the keyboard instrument at other communication points, a generator that generates a drive signal corresponding to the performance data, and a sound generator that generates sound based on the drive signal. The performance data includes a key position signal that indicates the amount of key depression on the keyboard instrument. The sound generator includes a speaker or a vibrator connected to the soundboard. The drive signal includes a first drive signal for driving the key to reproduce the amount of key depression corresponding to the key position signal, and a second drive signal for driving the speaker or vibrator based on the key position signal. The generator generates the second drive signal so that sound is generated at a predetermined position during key depression and at a predicted volume by the time the key reaches the predetermined position.

The present invention allows multiple performers playing together to feel a sense of unity.

FIG. 1 is a diagram for explaining a communication system configuration in a first embodiment. FIG. 2 is a diagram illustrating the internal configuration of the player piano according to the first embodiment. FIG. 2 is a diagram illustrating a configuration of a control device in the first embodiment. 4 is a diagram illustrating the configuration of an ensemble control function in the first embodiment. FIG. 13A and 13B are diagrams illustrating the positional relationship between a vibrator and a pickup sensor in the second embodiment. 13 is a diagram illustrating a configuration of a drive signal generating unit in a second embodiment. FIG. 13A and 13B are diagrams illustrating the relationship between velocity and delay time in the third embodiment. 13A and 13B are diagrams illustrating the relationship between velocity and correction value in the third embodiment. FIG. 13 is a diagram illustrating the configuration of an environment collection device at a communication base T1 in the fourth embodiment. FIG. 13 is a diagram illustrating a configuration of an environment providing device at a communication base T2 in the fourth embodiment. FIG. 13 is a diagram illustrating a configuration of a control signal generating unit in the fifth embodiment. FIG. 23 is a diagram illustrating a display example of a screen in the sixth embodiment.

One embodiment of the present invention will be described in detail below with reference to the drawings. The embodiments described below are merely examples, and the present invention should not be interpreted as being limited to these embodiments. The configurations described in each embodiment can also be applied to other embodiments. In the drawings referred to in the multiple embodiments described below, identical parts or parts having similar functions are given the same or similar symbols (symbols consisting of a number followed by A, B, etc.), and repeated explanations may be omitted. For clarity of explanation, the drawings may be described diagrammatically with some of the configuration omitted.

First Embodiment
[Communication system]
FIG. 1 is a diagram for explaining the configuration of a communication system in one embodiment. The communication system includes a server 1000 connected to a network NW such as the Internet. The server 1000 includes a control unit such as a CPU, a storage unit, and a communication unit. The control unit provides a service for realizing an ensemble between communication bases by executing a predetermined program. The server 1000 controls communication between a plurality of communication bases connected to the network NW, and executes processing required for the player pianos 1 at each communication base to realize P2P type communication with each other. This processing may be realized by a known method. In FIG. 1, two communication bases T1 and T2 are illustrated as an example, but the number is not limited to this, and more communication bases may exist. In the following description, when the communication bases T1 and T2 are not distinguished from each other, they are simply referred to as communication bases.

In this example, information related to performances at each communication center T1 and communication center T2 is exchanged between them via P2P communication. This communication allows an ensemble to be played between multiple communication centers. An automatic player piano 1 is located at each communication center. In this example, an environment collection device 82 and an environment providing device 88 are connected to the automatic player piano 1.

The environment collection device 82 includes a sensor for collecting information on the surrounding environment of the player piano 1, and outputs a collection signal indicating the measurement result of the sensor. The surrounding environment is, for example, sound, light, vibration, temperature, air flow, etc. When the environment providing device 88 acquires a control signal indicating the surrounding environment, it provides an environment based on the control signal. The environment collection device 82 and the environment providing device 88 may be configured as one unit. The environment providing device 88 may be provided according to the number of other communication bases. For example, if there are three communication bases in addition to the communication base T1, three environment providing devices 88 may be provided at the communication base T1 corresponding to each communication base. At least one of the environment collection device 82 and the environment providing device 88 may be configured to be incorporated in the player piano 1. Specific examples of the environment collection device 82 and the environment providing device 88 will be described later.

The automatic player piano 1 includes a keyboard instrument 10, a control device 20, a sensor 30, and a drive device 40.

[Automatic Piano]
Next, the configuration of the player piano 1 will be described.

Figure 2 is a diagram illustrating the internal configuration of the player piano in the first embodiment. The keyboard instrument 10 of the player piano 1 corresponds to, for example, a grand piano. The keyboard instrument 10 includes a plurality of keys 12. The keyboard instrument 10 includes hammers 14, strings 15, and dampers 18 that are provided corresponding to each key 12. In the player piano 1, the components provided corresponding to each key 12 are shown with a focus on the components provided corresponding to one key 2 shown in Figure 2. Therefore, the components provided corresponding to the other keys 12 are omitted. Some components, such as the damper 18, may not be provided for some keys 12.

The keyboard instrument 10 includes a plurality of pedals 13. The plurality of pedals 13 are, for example, a damper pedal, a shift pedal, and a sostenuto pedal. In the player piano 1, the components provided corresponding to each pedal 13 are shown by focusing on the components provided corresponding to one pedal 13 shown in FIG. 2. Therefore, the components provided corresponding to the other pedals 13 are not shown. The keyboard instrument 10 further includes a fallboard 11, a bridge 16, a soundboard 17, and a straight support 19.

The sensor 30 includes a key sensor 32, a pedal sensor 33, and a hammer sensor 34. The key sensor 32 is provided corresponding to each key 12, and outputs a measurement signal corresponding to the behavior of the key 12 to the control device 20. In this example, the key sensor 32 outputs a measurement signal corresponding to the position (pressing amount) of the key 12 to the control device 20. The position of the key 12 may be measured in continuous amounts (fine resolution), or may be measured by detecting that the key 12 has passed a predetermined position. The positions at which the key 12 is detected may be multiple positions within the pressing range of the key 12 (range from the rest position to the end position).

The hammer sensor 34 is provided corresponding to each hammer 14, and outputs a measurement signal according to the behavior of the hammer 14 to the control device 20. In this example, the hammer sensor 34 measures the position (rotation amount) of the hammer shank immediately before the hammer 14 strikes the string 15, and outputs a measurement signal according to the measurement result to the control device 20. The position of the hammer shank may be measured in a continuous amount (fine resolution), or may be measured by detecting that the hammer shank has passed a predetermined position. The position at which the hammer shank is detected may be multiple positions within a range immediately before the hammer 14 strikes the string 15.

The pedal sensor 33 is provided corresponding to each pedal 13, and outputs a measurement signal corresponding to the behavior of the pedal 13 to the control device 20. In this example, the pedal sensor 33 outputs a measurement signal corresponding to the position (depression amount) of the pedal 13 to the control device 20. The position of the pedal 13 may be detected continuously (fine resolution), or may be detected when the pedal 13 passes a predetermined position. The position at which the pedal 13 is detected may be multiple positions within the depression range of the pedal 13 (range from the rest position to the end position).

The drive device 40 includes a key drive device 42, a pedal drive device 43, a stopper 44, a vibrator 47, and a damper drive device 48. The key drive device 42 is provided corresponding to each key 12, and drives the key 12 to be pressed down by control using a drive signal from the control device 20. This mechanically reproduces the same situation as when the performer presses the key 12. The pedal drive device 43 is provided corresponding to each pedal 13, and drives the pedal 13 to be pressed down by control using a drive signal from the control device 20. This mechanically reproduces the same situation as when the performer presses the pedal 13. The damper drive device 48 is provided corresponding to each damper 18, and drives the damper 18 to be separated from the string 15 by control using a drive signal from the control device 20. The damper drive device 48 may have a configuration that drives all dampers 18 simultaneously.

The stopper 44 is driven by control from the control device 20 so that it is either in a position where it collides with the hammer shank (blocking position) or in a position where it does not collides with the hammer shank (retracted position). When the stopper 44 is in the blocking position, even if the key 12 is pressed, the movement of the hammer shank is restricted and the hammer 14 does not strike the string 15. When the stopper 44 is in the retracted position, the hammer 14 linked to the key 12 strikes the string 15 when the key 12 is pressed. When the string 15 is struck, the keyboard instrument 10 produces a sound.

In this example, the vibrator 47 is supported by a support connected to the straight support 9 so as to contact the surface of the soundboard 17 opposite the portion where the bridge 16 is located. The vibrator 47 vibrates the soundboard 17 under control of a drive signal from the control device 20. When a drive signal including a piano sound is supplied from the control device 20, for example, the vibrator 47 applies vibrations corresponding to the drive signal to the soundboard 17. This causes the piano sound to be emitted from the soundboard 17. Multiple vibrators 47 may be arranged so as to contact the soundboard 17. A speaker that emits sound may be used instead of the vibrator 47 that vibrates the soundboard 17.

The sound produced by the keyboard instrument 10 can be produced either by striking the strings 15 with the hammers 14 or by vibrating the soundboard 17 with the oscillator 47. Therefore, the keyboard instrument 10 can be said to include a sound producing device that produces a string-striking sound by driving the keys 12, and a sound producing device that produces sound from the soundboard 17 by driving the oscillator 47. Driving the keys 12 and the oscillator 47 is achieved by outputting a drive signal to the drive device 40, as described below.

The configuration of the control device 20 will now be described. In this example, the control device 20 is attached to the keyboard instrument 10. The control device 20 does not have to be a device that is attached to the keyboard instrument 10, and may be, for example, a personal computer, a tablet computer, a smartphone, etc.

Figure 3 is a diagram illustrating the configuration of the control device in the first embodiment. The control device 20 has a control unit 21, a storage unit 22, an operation panel 23, a communication unit 24, a sound source unit 25, and an interface 26. Each of these components is connected via a bus 27.

The control unit 21 is an example of a computer equipped with a processor such as a CPU and a storage device such as a RAM. The control unit 21 executes the programs stored in the storage unit 22 using the CPU (processor), and realizes functions for executing various processes in the control device 20. The functions realized in the control device 20 include an ensemble control function described below. This ensemble control function controls each part of the control device 20 and each configuration connected to the interface 26. A sensor 30 and a drive device 40 are connected to the interface 26. In this example, an external device 80 is also connected to the interface 26. The interface 26 transmits drive signals, control signals, etc. generated by the control unit 21 to the target configuration, and receives measurement signals, collection signals, etc. from each target configuration.

The memory unit 22 is a storage device such as a non-volatile memory or a hard disk drive. The memory unit 22 stores the program executed by the control unit 21 and various data required when executing this program.

The operation panel 23 has operation buttons and the like that accept user operations. When a user operation is accepted by this operation button, an operation signal corresponding to the operation is output to the control unit 21. The operation panel 23 may have a display screen. In this case, the operation panel 23 may be a touch panel in which a touch sensor is combined with a display screen.

The communication unit 24 is a communication module that communicates with other devices wirelessly, wired, etc. In this example, the other devices with which the communication unit 24 communicates are the server 1000 or the player piano 1 at another communication base. In this example, performance data indicating the content of the performance on the keyboard instrument 10, environmental data, etc. are communicated between the communication bases.

The sound source unit 25 generates a sound signal under the control of the control unit 21. The sound signal is used as a drive signal (a vibration drive signal described later) for driving the vibrator 47, etc. In this example, the sound signal includes a signal indicating a piano sound. The control unit 21 controls the sound source unit 25 to generate a sound signal indicating a piano sound according to the performance content corresponding to the performance data, for example. The performance data may be data generated based on the measurement signal generated by the sensor 30. The performance data may be, for example, MIDI format data including sound generation control information such as note on, note off, note number, and velocity, or may be information directly indicated by the measurement signal.

The interface 26 is an interface that connects the control device 20 to each external component. As described above, in this example, each component connected to the interface 26 includes the sensor 30, the drive device 40, and the external device 80. The interface 26 outputs a measurement signal output from the sensor 30 to the control unit 21. The interface 26 outputs a drive signal to the drive device 40 for driving each device. The drive signal is generated in the ensemble control function 100, which will be described later. The interface 26 may include a headphone terminal to which a sound signal indicating a piano sound generated in the sound source unit 25 is supplied.

[Ensemble control function]
Next, an ensemble control function that is realized by the control unit 21 executing a program will be described. The configuration that realizes the ensemble control function is not limited to being realized by the execution of a program, and at least a part of the configuration may be realized by hardware. The configuration that realizes the ensemble control function may be realized not by the control unit 20 but by a device connected to the interface 26 (for example, a computer on which this program is installed).

In this example, when the ensemble control function is realized, the control unit 21 controls the stopper 44 to be placed in the blocking position. In this case, when the user inputs a performance operation to the keys 12 and pedals 13, the strings are blocked from being struck by the stopper 44, while a sound signal corresponding to the performance operation (e.g., a piano performance sound) is generated in the sound source unit 25. This sound signal is used by the vibrator 47 to vibrate the soundboard 17, and is emitted as sound. A signal for driving the vibrator 47 is generated in the drive signal generation unit 145, which will be described below.

Figure 4 is a diagram explaining the configuration of the ensemble control function in the first embodiment. The ensemble control function 100 includes a performance data generation unit 131, a performance data transmission unit 133, a performance data reception unit 143, and a drive signal generation unit 145. In this example, the ensemble control function 100 further includes an environmental data generation unit 121, an environmental data transmission unit 123, an environmental data reception unit 183, and a control signal generation unit 185 as functions associated with the ensemble for sharing the surrounding environment of the player piano 1 between communication locations.

The performance data generating unit 131 generates performance data indicating the performance content for the keyboard instrument 10 based on the measurement signal output from the sensor 30. In this example, the performance data includes a measurement signal output from the key sensor 32 (hereinafter referred to as a key position signal) and a measurement signal output from the pedal sensor 33 (hereinafter referred to as a pedal position signal). In this example, the key position signal includes the pitch of the pressed key 12 and the amount of depression of the key 12. If the key sensor 32 is a sensor that measures the amount of depression of the key 12 at four positions, the information on the amount of depression of the key 12 included in the key position signal indicates one of the four positions.

In this example, the pedal position signal includes the type of pedal 13 that has been pressed and the amount of depression of the pedal 13. If the pedal sensor 33 is a sensor that measures the amount of depression of the pedal at three locations, the information on the amount of depression of the pedal 13 indicates one of the three locations. The performance data may further include a measurement signal (hereinafter referred to as a hammer position signal) output from the hammer sensor 34. The hammer position signal includes, for example, the key pitch and the rotational position of the hammer 14.

Let us assume that the performance data generated by the performance data generating unit 131 is data (e.g., in MIDI format) that includes sound generation control information generated based on the measurement results of the key sensor 32 and the pedal sensor 33. In this case, for example, to send a note-on, the depression amount of the key 12 needs to progress to a state that generates a note-on.

On the other hand, the performance data generating unit 131 in this example can sequentially transmit the amount of key 12 pressed while the key 12 is being pressed. This allows the player pianos 1 at other communication locations to recognize that the key 12 has begun to be pressed even before a note-on occurs. For example, when the key 12 begins to be pressed at the player piano 1 at communication location T1, the key 12 at the player piano 1 at communication location T2 can begin to be driven to the recognized amount of pressing even before a note-on occurs. In this way, the key 12 at communication location T2 can also be driven to follow the performance operation on the key 12 at communication location T1 with a short delay time.

The performance data transmission unit 133 transmits the performance data generated by the performance data generation unit 131 to other communication locations.

The performance data receiving unit 143 receives performance data transmitted from other communication locations.

The drive signal generating unit 145 generates a drive signal used in the drive device 40 based on the performance data received by the performance data receiving unit 143. This drive signal includes a signal supplied to the key drive device 42 (key drive signal), a signal supplied to the pedal drive device 43 (pedal drive signal), and a signal supplied to the vibrator 47 (vibration drive signal).

The key drive signal is generated based on the performance data, and more specifically, based on the key position signal included in the performance data. The key drive signal is a signal for controlling the key drive device 42 to drive the key 12 so as to reproduce the amount of depression corresponding to the key position signal. The key 12 to be driven is the key corresponding to the pitch specified by the key position signal. The pedal drive signal is generated based on the performance data, and more specifically, based on the pedal position signal. The pedal drive signal is a signal for controlling the pedal drive device 43 to move the pedal corresponding to the type specified by the pedal position signal to a position corresponding to the amount of depression.

The vibration drive signal is generated based on the performance data, more specifically, it is a signal generated by the sound source unit 25 based on the key position signal and the pedal position signal. When the vibration drive signal causes the vibrator 47 to vibrate the soundboard 17, a sound (in this example, a piano sound) corresponding to the signal generated by the sound source unit 25 spreads around the keyboard instrument 10 via the soundboard 17.

When generating a sound signal in the sound source unit 25, the drive signal generating unit 145 may generate sound control information based on the key position signal and the pedal position signal, and may cause the sound source unit 25 to generate a sound signal based on the sound control information. At this time, the drive signal generating unit 145 may generate the sound control information using a calculation that predicts the timing and velocity of the note-on from the change in the amount of pressing of the key 12 indicated by the key position signal in the performance data. The change in the rotation position of the hammer 14 indicated by the hammer position signal in the performance data may be used for this prediction calculation. The prediction calculation may use a learned model obtained in advance by machine learning, or may use a fitting process that assumes a constant velocity trajectory, a constant acceleration trajectory, etc. from the change in the amount of pressing. This makes it possible to improve the prediction accuracy even when the movements of the key 12 and the hammer 14 are not synchronized. For example, the drive signal generating unit 145 may predict the final string striking speed of the hammer 14 at a predetermined position (e.g., 6.3 mm/10 mm) during the pressing of the key 12, and output a note-on event. By sending note-on events earlier, the overall delay can be reduced.

Let us assume that the key 12 is actuated based on the performance data, and the hammer 14 that operates as a result strikes the string 15. In this case, the timing of sound production is delayed because it takes time from the sound production instruction (e.g. note-on) to actuating the key 12. Therefore, the timing of sound production is affected not only by the communication delay between communication points, but also by the delay when the key 12 is actuated.

On the other hand, according to the drive signal generating unit 145, the key 12 and pedal 13 are driven by the key drive signal and pedal drive signal, but the stopper 44 prevents the hammer 14 from striking the string 15, so no string-striking sound is produced. Instead, the vibration drive signal drives the oscillator 47, causing sound to be produced from the soundboard 17. Producing a sound using the oscillator 47 does not require driving the key 12. Therefore, with regard to the time from a sound production instruction (e.g., note-on) to actual sound production, the time when sound is produced by the oscillator 47 is shorter than the time when sound is produced by striking the string.

At this time, the sound produced by the vibrator 47 and the actuation of the key 12 are controlled separately, so there is a timing discrepancy between them. However, the amount of discrepancy is small, so it has little effect on the user's senses.

The transmitted performance data and sound generation method are not limited to the above combination. For example, sound generation control information may be transmitted as the performance data, rather than the amount of pressing of the key 12. Even in this case, by using sound generation by the vibrator 47 as described above, the time difference in sound generation between communication points can be made shorter than when sound generation by striking the strings is used.

Each drive signal is generated based on the sound generation control information in the performance data. At this time, for the key drive signal, the velocity value in the sound generation control information may be increased so as to be equal to or greater than a predetermined value. When the velocity value is small, i.e., when the pressing speed of the key 12 is slow (when the rotation speed of the hammer 14 is slow), the key drive device 42 slows down the drive speed of the key 12.

At this time, depending on the characteristics of the solenoid, the slower the drive speed is set, the later the key 12 may move than the scheduled timing. In order to compensate for this delay, it is possible to increase the velocity value. If the velocity value is increased, the delay time when the key 12 is driven will be shorter, but the striking sound will be louder. On the other hand, in this example, the hammer 14 is prevented from striking the string by the stopper 44, so no striking sound is produced, and there is no effect on the sound production. Therefore, the velocity value can be increased for keys 12 that do not contribute to the sound production. For example, if the velocity is equal to or less than a predetermined value, the key 12 may be driven by rounding up the velocity to any value greater than or equal to the predetermined value. In this way, the delay may be reduced. For the vibration drive signal, the velocity value is not changed so that the sound production content does not change. At this time, some keys 12 may not be driven in order not to affect the user's performance. The keys 12 that are not driven may be keys with a pitch used in the music piece that is set in advance.

As another example, instead of using the sound produced by the vibrator, the stopper 44 may be controlled to the waiting position to drive the key 12 and produce sound by striking the string. Even in this case, as described above, by transmitting the amount of pressing of the key 12 as performance data, the time difference in sound production between communication centers can be shortened compared to the case where sound production control information is transmitted as performance data. In this case, the sound produced by the user's performance operation (e.g., the sound produced by playing at communication center T1) and the sound produced by playing at another communication center (e.g., communication center T2) both include the sound of the string being struck. In this case, if the velocity is equal to or less than a predetermined value, the velocity may be rounded up to any value greater than or equal to the predetermined value and the key 12 may be driven. In this way, even if the sound of the string being struck becomes slightly louder, priority may be given to reducing the delay. In this case, if the velocity is equal to or less than a predetermined value, the sound corresponding to that velocity may be produced by the vibrator 47 or by a speaker. In this way, priority may be given to reducing the delay even if the sound becomes slightly louder.

At this time, in order to avoid affecting the user's performance, the pedal 13 may not be driven by the pedal driving device 43, while the damper 18 may be driven by the damper driving device 48.

As another example, sound may be generated by the vibrator 47 while controlling the stopper 44 to the waiting position. Sound may be generated by a speaker instead of or together with the vibrator 47. In this case, the drive signal generating unit 145 may not generate a key drive signal or a pedal drive signal so that the key 12 and the pedal 13 do not move. The drive signal generating unit 145 may drive the key 12 by the key drive signal at a speed at which no sound is generated by striking the string even if the key 12 moves, or at a speed at which the volume of the sound generated by striking the string is not noticeable. In other words, a key drive signal with a velocity controlled to a predetermined value or less is generated. In this way, the sound generated by the user's performance operation includes the sound of striking the string, that is, the user can play with raw sound. On the other hand, the sound generated by the performance at another communication site can be a sound generated by the vibrator 47 with little delay. In other words, the movement of the key 12 gives the feeling that a person is actually playing at a remote location. Several examples of combinations of transmitted performance data and sound generation methods have been described, but the method of selection may be set by operating the operation panel 23.

The environmental data generation unit 121 generates environmental data indicative of the surrounding environment based on the collected signal output from the environmental collection device 82. In this example, the surrounding environment includes images and sounds around the device. Therefore, the environmental collection device 82 includes devices for collecting the surrounding environment, namely, a camera (image capture device) for acquiring images and a microphone (sound collection device) for acquiring sounds. In this example, the camera acquires images of an area that includes the player of the keyboard instrument 10.

The information on the image included in the environmental data may be image information showing the image (video) itself, but in this example, it includes motion information obtained by acquiring the player's motion using motion capture technology. The sensor that measures the player's motion is not limited to a camera, but may include an IMU (Inertial Measurement Unit), a pressure sensor, a displacement sensor, etc. The motion information is, for example, information indicated by the coordinates of each part for multiple parts having a predetermined feature extracted from the image. The environmental data may be transmitted in the form of audio data indicating a sound signal. In this case, the motion information can be synchronized with the sound signal included in the audio data by transmitting it as data of a predetermined channel in the audio data. Similarly, the environmental data may be transmitted to another communication base after the data format is converted so that it is transmitted as part of existing data, such as a format indicating sound control information (e.g., MIDI format) or a video data format.

The sounds collected by the environment collection device 82 may include sounds (piano sounds) generated by playing the keyboard instrument 10. The period during which sounds generated by playing the keyboard instrument 10 exist can be identified from key position signals, etc. If the sounds generated by playing are sounds generated by the vibrator 47, the sounds can be identified in the sound source unit 25. Therefore, when generating the environment data, the environment data generation unit 121 may apply signal processing to cancel the components of the sounds generated in the sound source unit 25 from the sounds included in the collected signals.

Even if the sound produced by playing is a string-striking sound, the environmental data generating unit 121 may perform signal processing to cancel the string-striking sound component from the sound contained in the collected signal. The string-striking sound component may be generated in the sound source unit 25 using a key position signal and a pedal drive signal. The environmental data generating unit 121 may generate environmental data for a period in which there is sound produced by playing the keyboard instrument 10, without using the sound contained in the collected signal. In this case, the environmental collection device 82 may recognize the period in which playing is in progress and not collect sound during that period.

The environmental data transmission unit 123 transmits the environmental data generated by the environmental data generation unit 121 to other communication bases.

The environmental data receiving unit 183 receives environmental data transmitted from other communication bases.

The control signal generating unit 185 generates a control signal to be used in the environment providing device 88 based on the environment data received by the environment data receiving unit 183. This control signal is a signal for reproducing the information of the surrounding environment contained in the environment data, and in this example, includes a signal for displaying an image on a display (display device) and a signal for outputting sound from a speaker (sound emitting device). Therefore, the environment providing device 88 includes a display for displaying an image and a speaker for outputting sound. The display may be placed in a position that is easy for the performer to see, such as the fallboard 11 of the keyboard instrument 10 or a music stand. When an image is displayed on the fallboard 11, a projector that projects an image on the fallboard 11 may be used instead of a display. When multiple communication bases are the communication targets, an environment providing device 88 may be provided corresponding to each communication base. In this case, the control signal supplied to the environment providing device 88 is generated based on the environment data received from the communication base corresponding to the environment providing device 88.

The control signal generating unit 185 may use the motion information included in the environmental data to generate an image that resembles the performer, and generate a signal for displaying the generated image on the display. At this time, an image that emphasizes a specific part or motion may be generated. The specific part may be, for example, the performer's eyes, face, fingers, etc. The specific motion may be, for example, eye movement, facial movement, finger movement during performance motion, etc. The control signal generating unit 185 may use the motion information included in the environmental data to generate an image such as a graph that quantifies the performer's motion, and generate a signal for displaying the generated image on the display. The performer can use the displayed information to match their performance.

The control signal generating unit 185 may generate a signal for displaying an image based on the performance data received by the performance data receiving unit 143 on a display. The image based on the performance data may include an image showing the performance content included in the performance data, for example, an image showing the keys or pedals being operated.

In this way, the ensemble control function 100 in the first embodiment reduces the time difference between sounds produced at different communication points, allowing players to feel closer to each other's surroundings. This allows multiple players playing together to feel a sense of unity.

Second Embodiment
The performance content included in the performance data is not limited to the case where the performance operation on the keys 12, etc. is indicated. In the second embodiment, an example will be described in which the performance data includes a signal indicating the vibration of the soundboard 17 to which the string-striking sound caused by the performance is transmitted. In this example, the vibration of the soundboard 17 is measured by a pickup sensor included in the sensor 30.

Figure 5 is a diagram explaining the positional relationship between the vibrator and the pickup sensor in the second embodiment. Figure 5 is a diagram of the keyboard instrument 10 viewed from below. As shown in Figure 5, two vibrators 47 (vibrators 47H, 47L) are provided on the soundboard 17. The vibrators 47H, 47L are connected between the multiple sound bars 17a of the soundboard 17. The vibrator 47H is provided at a position corresponding to the bridge 16H of the two bridges 16 (bridge 16H (long bridge), 16L (short bridge)). The vibrator 47L is provided at a position corresponding to the bridge 16L. The bridge 16H is a bridge that supports the strings 15 on the treble side, and the bridge 16L is a bridge that supports the strings 15 on the bass side. The vibrator 47H is supported by a support portion 97H connected to the straight support 19. The vibrator 47L is supported by a support portion 97L connected to the straight support 19.

The vibrator 47 is not limited to being provided at a position on the soundboard 17 corresponding to the bridge 16, but may be provided at a position away from the bridge 16, or at a position corresponding to the sound bar 17a. If provided at a position corresponding to the sound bar 17a, the vibrator 47 may be provided on the string 15 side of the soundboard 17.

Pickup sensor 37H is attached to soundboard 17 near vibrator 47H, measures vibrations of soundboard 17, and outputs a measurement signal indicating the measurement result. Pickup sensor 37L is attached to soundboard 17 near vibrator 47L, measures vibrations of soundboard 17, and outputs a measurement signal indicating the measurement result. Therefore, the performance data transmitted by performance data transmitting unit 133 to other communication locations, and the performance data received by performance data receiving unit 143 from other communication locations, include measurement signal PU1 from pickup sensor 37H and measurement signal PU2 from pickup sensor 37L.

Figure 6 is a diagram explaining the configuration of the drive signal generating unit in the second embodiment. The drive signal generating unit 145A generates vibration drive signals DS1 and DS2 from the measurement signals PU1 and PU2 contained in the performance data received by the performance data receiving unit 143. The vibration drive signal DS1 is supplied to the vibration exciter 47H. The vibration drive signal DS2 is supplied to the vibration exciter 47L.

The drive signal generating unit 145A includes a crosstalk processing unit 1451, an acoustic providing unit 1453, and an amplifier unit 1455. The crosstalk processing unit 1451 applies a predetermined delay processing and a predetermined filter processing to the measurement signal PU1 and adds it to the measurement signal PU2. The crosstalk processing unit 1451 applies a predetermined delay processing and a predetermined filter processing to the measurement signal PU2 and adds it to the measurement signal PU1. This reduces the crosstalk components contained in each of the measurement signals PU1 and PU2.

The sound applying unit 1453 applies signal processing to the measurement signals PU1 and PU2 to apply sound effects such as delay, compressor, expander, and equalizer. The amplifier unit 1455 amplifies the measurement signals PU1 and PU2 to output the vibration drive signals DS1 and DS2 that are supplied to the vibrators 47H and 47L.

For example, if the size and shape of the soundboard 17, etc., differs between the keyboard instrument 10 at communication point T1 and the keyboard instrument 10 at communication point T2, differences in vibration modes in the soundboard 17 will occur. As a result, the sounds emitted from each keyboard instrument 10 will be different. Parameters in the signal processing in the crosstalk processing unit 1451 and the sound adding unit 1453 are set in accordance with the different configurations of the keyboard instruments 10. This makes it possible to reduce differences in sound production caused by differences in the shapes, etc. of the keyboard instruments 10 at other communication points.

When an ensemble is played between multiple communication bases, the vibrations of the soundboard 17 measured by the pickup sensors 37H, 37L include not only the vibrations caused by the striking sound of the strings, but also the vibrations caused by the vibrators 47H, 47L based on the vibration drive signals DS1, DS2. Therefore, before transmitting the performance data to other communication bases, the performance data transmission unit 133 may apply signal processing to the measurement signals PU1, PU2 contained in the performance data to reduce the components of the vibration drive signals DS1, DS2.

Third Embodiment
As described above, when the velocity value is small, it takes longer for the key drive device 42 to drive the key 12, and the movement of the key 12 is delayed compared to when the velocity value is large. As described above, the velocity value can be increased in a situation where no striking sound is generated, but when a striking sound is generated, increasing the velocity value too much will significantly change the sound produced. In the third embodiment, an example will be described for reducing such changes in sound produced as much as possible.

Figure 7 is a diagram illustrating the relationship between velocity and delay time in the third embodiment. The horizontal axis is the velocity value, which is a value used as a calculation parameter in the drive signal generation unit 145 to generate a key drive signal. The vertical axis corresponds to the delay time when the key drive device 42 drives the key 12 based on the key drive signal. The velocity value is obtained, for example, from information included in the performance data used by the drive signal generation unit 145. More specifically, the velocity value may be obtained by calculation from the key position signal or hammer position signal included in the performance data. If the performance data includes sound generation control information, the velocity value may be obtained from that velocity value. In this example, the velocity has values from "1" to "127".

As shown in FIG. 7, as the velocity decreases, the delay time increases. When the velocity becomes smaller than Vt, the delay time increases rapidly. Therefore, in this example, the drive signal generation unit 145 corrects the delay time to a larger value when the velocity becomes smaller than Vt.

Figure 8 is a diagram explaining the relationship between velocity and correction value in the third embodiment. As shown in Figure 8, when the velocity is a value smaller than Vt, the correction value obtained by increasing the value is used as a calculation parameter in the drive signal generation unit 145. For example, when the velocity changes from "1" to "Vt", the correction value is set to change from "Va" to "Vt". In this way, it is possible to reduce the delay time when the velocity is small. Since the correction value is larger than the value before correction, a louder sound than expected is generated, while the change in the volume of the sound has a smaller impact on the listener than the effect of reducing the delay time. In order to enhance the effect of reducing the delay time, the correction value may be set to "Vb" larger than "Va" when the velocity is "1".

When playing in an ensemble, it is preferable to minimize the delay. On the other hand, if the purpose is not to play in an ensemble but to simply listen to sounds based on performance data, it is preferable that the delay time not change over the entire range of input values, rather than reducing the delay. Therefore, in such a case, the drive signal generating unit 145 may generate a key drive signal that is delayed so as to intentionally delay the timing at which the pressing of the key 12 begins in a range where the velocity value is large (above a predetermined value). In this case, the amount of time by which the timing is delayed may be gradually decreased as the input value decreases from "Vt" toward "1".

Fourth Embodiment
In the fourth embodiment, an example will be described in which different instruments are played at different communication points. Here, communication point T1 is a large hall where an orchestra can perform. Communication point T2 is a small studio such as a soundproof room. In this example, an orchestra performs at communication point T1, and a piano performs at communication point T2. That is, there is no piano player in the orchestra at communication point T1, and there is a piano player at the remote communication point T2.

The sounds of the orchestra playing at communication point T1 are transmitted to communication point T2. The player at communication point T2 plays the player piano 1 while listening to the sounds received from communication point T1. The content of the user's performance is transmitted as performance data from the player piano 1 to the player piano 1 at communication point T2. The player piano 1 at communication point T2 therefore generates sounds to reproduce the performance at communication point T1. In other words, even if the piano player is not present at communication point T1, the piano playing sounds can be heard at the communication point T1 as well as the sounds of the orchestra playing. In the fourth embodiment, the realism of the orchestra performance at communication point T1 can be conveyed to the player of the player piano 1 at communication point T2. The configuration for achieving this is described in detail below.

Figure 9 is a diagram explaining the configuration of the environment collection device at the communication base T1 in the fourth embodiment. At the communication base T1, the conductor's podium CS, the player piano 1, and the chair 50 are installed on the stage ST1 of the hall. The stage ST1 is provided with an environment collection device 82 including vibration measurement plates 821, 822 and a microphone 823. The vibration measurement plate 821 is placed at the location on the stage ST1 where the chair 50 is placed. The vibration measurement plate 821 measures the vibration transmitted through the stage ST1 and outputs a collected signal indicating the measured vibration. The vibration measurement plate 822 is placed at the location on the stage ST1 where the player piano 1 is placed. The vibration measurement plate 822 measures the vibration transmitted through the stage ST1 and outputs a collected signal indicating the measured vibration. In this example, the microphone 823 is placed near the chair 50. The microphone 823 collects the sound that reaches it and outputs a collected signal indicating the sound.

FIG. 10 is a diagram illustrating the configuration of the environment providing device at the communication base T1 in the fourth embodiment. At the communication base T2, the player piano 1 and the chair 50 are installed on the stage ST2 of the studio. The stage ST2 is provided with an environment providing device 88 including vibration generating plates 881, 882 and a speaker 883. The vibration generating plate 881 is placed at the location where the chair 50 is placed on the stage ST2. The vibration generating plate 881 vibrates based on a control signal. The vibration generating plate 882 is placed at the location where the player piano 1 is placed on the stage ST2. The vibration generating plate 882 vibrates based on a control signal. In this example, the speaker 883 is placed near the chair 50 (near the performer). The speaker 883 emits sound based on the control signal. When the control signal used in the vibration generating plates 881, 882 and the speaker 883 is generated, signal processing according to the vibration characteristics, signal processing for changing the transfer characteristics, etc. may be performed.

When the orchestra plays at communication point T1, vibrations accompanying the performance are transmitted via stage ST1 to vibration measuring plates 821 and 822, and the performance sounds are collected by microphone 823. The collected signals output from vibration measuring plates 821 and 822 and microphone 823 are transmitted to communication point T2 as environmental data. When player piano 1 is played at communication point T2, the performance contents are transmitted to communication point T1 as performance data.

As a result, at communication point T1, the player piano 1 is driven to produce sound based on the performance data from communication point T2. That is, the player piano 1 at communication point T1 is driven in accordance with the performance on the player piano 1 at communication point T2. At communication point T2, the speaker 883 produces sound based on the environmental data from communication point T1. This sound is collected by the microphone 823, and corresponds to the sound of the orchestra playing at communication point T1.

Furthermore, vibration generating plate 881 and vibration generating plate 882 are driven to vibrate based on environmental data from communication point T1. The vibrations in vibration generating plate 881 correspond to the vibrations measured by vibration measuring plate 821 at communication point T1. That is, the vibrations transmitted to chair 50 at communication point T1 are also transmitted to chair 50 at communication point T2. The vibrations in vibration generating plate 882 correspond to the vibrations measured by vibration measuring plate 822 at communication point T1. The vibrations in vibration generating plate 882 correspond to the vibrations measured by vibration measuring plate 822 at communication point T1. That is, the vibrations transmitted to player piano 1 at communication point T2 are also transmitted to player piano 1 at communication point T2. Therefore, the performer at communication point T2 can obtain a sense of realism as if he or she were playing at communication point T1.

The vibration measuring plates 821, 822 and microphone 823 at communication point T1 also collect piano sound components as the player piano 1 is driven. Therefore, signal processing is performed to reduce the piano sound components on the path to where the vibration generating plates 881, 882 and speaker 883 at communication point T2 are driven. The piano sound components can be generated from a signal for driving the player piano 1 at communication point T1. Therefore, this signal processing may be performed, for example, by the environmental data generation unit 121 when the environmental data to be transmitted from communication point T1 is generated. In this way, the influence of the performance at communication point T2 can be reduced in the environment provided by the environment providing device 88 at communication point T2.

Fifth Embodiment
In the fifth embodiment, a configuration will be described in which, when the environment providing device 88 displays images of performers at other communication locations, the device also displays its own image (the image of the performer that is transmitted to the other communication location).

Figure 11 is a diagram illustrating the configuration of the control signal generation unit in the fifth embodiment. The control signal generation unit 185B in the fifth embodiment includes a self-image acquisition unit 1851, a remote image acquisition unit 1853, and an image synthesis unit 1855.

The self-image acquisition unit 1851 acquires self-image information about an image including the performer based on a collection signal output from the environment collection device 82. The remote image acquisition unit 1853 acquires remote image information about an image including the performer collected by the environment collection device 82 at another communication point based on the environmental data received by the environmental data receiving unit 183. Both the self-image information and the remote image information are images including the performer and the keyboard portion of the keyboard instrument 10.

The image synthesis unit 1855 generates a synthetic image based on the self-image information and the remote image information. The synthetic image is an image in which an image area of the performer included in the remote image information is extracted and superimposed on the image of the self-image information. At this time, the image synthesis unit 1855 identifies the keyboard part from each of the images in the self-image information and the remote image information, and determines the superimposition position of the image of the performer in the remote image information so that the keyboard parts match each other. For example, the image synthesis unit 1855 superimposes an image obtained by applying a transformation matrix to the remote image information so that the cross-correlation between the keyboard parts is maximized on the image of the self-image information. The image synthesis unit 1855 generates a control signal for displaying the synthetic image and outputs it to the environment providing device 88. The environment providing device 88 may be a display that displays the synthetic image on a display, or a projector that projects at least a part of the image of the remote image information onto the keyboard part. When projecting using a projector, a predetermined transformation matrix according to the position of the keyboard part may be applied to the remote image information.

In a piano duet, where two people play on one piano, the range of notes played by each player is different, and so the players' positions in relation to the keys also differ. Even when a piano duet is split into two communication bases, with each player playing one at a time, the players' positions in relation to the keys are roughly the same as when playing a single piano. Therefore, the composite image that is generated is an image of two players playing one piano.

In this example, the image synthesis unit 1855 determines whether the images of the two performers have come into contact with each other, and if so, modifies the synthesis image so that the area corresponding to the contact can be identified, such as by illuminating the area. At this time, the contact area that is illuminated may be limited to a portion (such as an arm or hand). The performers may be made aware that the images of the two performers have come into contact through something other than the image. For example, when the images of the two performers come into contact with each other, the seat of the chair used by the performer may be vibrated. In this case, the configuration for vibrating the seat of the chair is included in the environment providing device 88, and is controlled by a control signal from the control signal generating unit 185B. In this way, even if the two performers are performing at different communication bases, they can experience a situation in which they are actually performing at the same place.

The remote image acquisition unit 1853 may acquire the above-mentioned action information (remote action information) instead of the remote image information. In this case, the image synthesis unit 1855 may generate an image imitating a performer using the action information, and synthesize it with the image of the self-image information to generate a synthetic image. When generating a synthetic image, the image synthesis unit 1855 may delay the image of the self-image information and then superimpose the image of the remote image information, taking into account a time lag (communication delay, etc.) between the self-image information and the remote image information, or may generate a future predicted image for the delay time from the image of the remote image information or the remote action information, and superimpose it on the image of the self-image information.

Sixth Embodiment
In the sixth embodiment, a case will be described in which two communication bases exist in the same room. In this case, there are two player pianos 1 that can communicate with each other in one room. In such a case, a screen on which images related to the performance content of each of the two player pianos 1 are displayed may be disposed between the two player pianos 1.

Figure 12 is a diagram illustrating an example of a screen display in the sixth embodiment. An automatic player piano 1a corresponding to communication location T1 and an automatic player piano 1b corresponding to communication location T2 are placed in one room. A screen SC onto which an image is projected by a projector PJ is placed between the automatic player piano 1a and the automatic player piano 1b. In this example, an image corresponding to the performance data communicated between the automatic player piano 1a and the automatic player piano 1b is displayed on the screen SC.

The displayed image is an image related to a sound corresponding to the performance content, and in this example is a band-shaped image displayed at a position determined by the pitch and sound generation timing with a length corresponding to the sound duration. In FIG. 12, band-shaped image sba is an image showing a sound corresponding to the performance content of automatic player piano 1a, and band-shaped image sbb is an image showing a sound corresponding to the performance content of automatic player piano 1a. Band-shaped images sba and sbb are displayed to flow according to the direction of communication.

That is, when a key 12 is pressed on the player piano 1a, a band-shaped image sba corresponding to the sound corresponding to that key 12 is displayed on the screen as it moves toward the player piano 1b. Here, when the band-shaped image sba reaches the player piano 1b side, the sound corresponding to that image may be generated on the player piano 1b. In this case, after receiving the performance data corresponding to the band-shaped image sba, the player piano 1b may delay until that timing is reached and then activate the key 12. The relationship between the player piano 1a and the player piano 1b remains the same even if they are swapped.

The projector PJ and screen SC can be considered as an example of an environment providing device 88. In this case, the environment providing device 88 is shared by the two player pianos 1a and 1b.

<Modification>
The present invention is not limited to the above-described embodiment, and includes various other modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. Some modified examples will be described below. The first embodiment will be described as an example of a modified example, but the modified examples can also be applied to other embodiments. A combination of multiple modified examples can also be applied to each embodiment.

(1) The drive signal generating unit 145 may predict sound generation control information such as note-on from changes in the amount of pressing of the key 12 indicated by the key position signal in the performance data, and may also predict sound generation control information using other information. For example, the drive signal generating unit 145 may extract the movement of the fingers toward the key 12 from the image of the performer included in the environmental data, and estimate the movement when the key 12 is pressed from the changes in the finger movement to predict the sound generation control information. The information indicating the finger movement is also information regarding the pressing of the key 12. Therefore, the image of the finger or the finger movement may be acquired by the sensor 30. In this case, the information indicating the finger movement may be transmitted by the performance data.

The sensor 30 may be configured to detect contact with or proximity to the key 12. In this case, the performance data generating unit 131 generates and transmits performance data based on the detection result, thereby making it possible for other communication centers to recognize that the key 12 is about to be pressed before the key 12 actually begins to be pressed. This may improve the prediction accuracy of the sound generation control information.

In these predictions, a trained model may be used that uses machine learning to determine the correlation between movement history information, such as the movement of the keys 12 or the movement of the fingers, and sound generation control information, such as note-on timing and velocity values. The trained model may be generated to correspond to each performer.

Predicting sound production control information in this manner is not limited to applications when controlling an automatic player piano 1 at another communication point, but may also be used for various linkages. For example, it can be applied to a configuration in which a keyboard device and a sound source device are wirelessly connected. For example, if a note-on occurs when a key is pressed on the keyboard device and then the note-on is sent to the sound source device, the timing of sound production will be delayed due to the effects of communication delay. On the other hand, by sending the key movement to the sound source device before the note-on occurs on the keyboard device, the effects of communication delay can be reduced by predictive calculations using that movement.

(2) The key position signal generated in response to the pressing of a key 12 on the player piano 1 may be used to control other keys 12 on the same player piano 1. For example, in response to the pressing of a key 12, it is possible to control a key 12 corresponding to a note one octave higher than the key 12 to be pressed so that it is linked. The linked key 12 is not limited to a note one octave higher, but may be any predetermined note. The predetermined note may be determined relative to the pitch of the pressed key 12, or may be determined absolutely regardless of the pitch. In this case, by using the key position signal instead of the sound generation control information, the time difference between the pressing of the key 12 to be played and the actuation of the linked key 12 can be reduced.

(3) The control device 20 may be configured to record the content of the performance on the player piano 1. The recorded data may be data based on sound generation control information, or data corresponding to signals output from the sensors 30, such as key position signals, hammer position signals, and pedal position signals.

(4) The performance data transmission unit 133 may transmit performance data including information for setting for each key 12 whether or not to drive the key 12 of the player piano 1 at another communication location. Whether or not to drive the key 12 may be set by the player at the transmitting communication location during performance, or may be determined in advance for a specific key or range. At the receiving communication location, the player piano 1 does not drive the key 12 in response to a key position signal relating to a key 12 that is set not to be driven, and instead drives the vibrator 47.

(5) The environment collection device 82 may include a sensor attached to the player, for example, a sensor that measures the player's breathing. The player piano 1 may transmit the measurement results of the player's breathing as environmental data to another communication site, and display information according to changes in breathing on the display of the environment providing device 88 at the other communication site. The player's breathing is closely related to the performance action. For example, the player often takes a deep breath just before starting to play. Therefore, when a deep breath is measured, information indicating this or the time until the performance is thought to start may be displayed on the display. Assuming that the time from taking a deep breath to starting to play differs depending on the player, the time may be set to be different for each player. In predicting this time, a trained model that has been machine-learned to determine the correlation between the timing of a deep breath and the time until the performance starts may be used.

(6) The environment providing device 88 may be a small, mobile device capable of providing various environments, and may be, for example, a humanoid shape imitating some kind of character. For example, the environment providing device 88 may be a humanoid robot whose arms and hands move based on a control signal. The environment providing device 88 may be in a shape that can be attached to the performer (wristwatch-type, shoulder-type, neck-type, etc.). The configuration for providing various environments may be the above-mentioned display and speaker, or may be, for example, a heat source, a cooling source, a fan, etc. for controlling temperature, or may be lighting, a projector, etc. for controlling the brightness, color, pattern, etc. of the room. The environment providing device 88 may include, for example, a structure such as a robot arm for changing the position of a heat source, etc., or may have multiple heat sources arranged so that the position of the heat source is substantially changed by driving any one of them. The heat source may be used, for example, to reproduce the position of the performer at another communication base. The environment collecting device 82 may include a sensor corresponding to the environment providing device 88, and may include, for example, a temperature sensor, an air flow sensor, an illuminance sensor, etc.

(7) The environment providing device 88 may include multiple speakers to localize a sound image or reproduce a specified sound field. In this case, a specified reverb process or filter process such as FIR may be added to the sound signal included in the environmental data. The environment collecting device 82 may collect information for reproducing the sound field characteristics of the room and transmit it to another communication base as environmental data. As a result, the environment providing device 88 at the receiving communication base may reproduce the sound field of the room at the transmitting communication base based on the information included in the environmental data. In this case, the sound field of the room at the transmitting communication base may be reproduced more accurately by including signal processing for canceling the sound field characteristics of the room at the receiving communication base. Such a process for reproducing a sound field may be added to the vibration drive signal.

(8) A common metronome synchronized at each communication point may be realized by sound, light, vibration, etc. When synchronizing with absolute time, for example, time information used in a satellite positioning system such as a GPS signal may be used, or time synchronization technology using NTP (Network Time Protocol) may be used. In this case, the BPM value may be set, and the start timing of the beat may be determined based on the time information. The BPM value may be determined based on a preset piece of music to be played, or may be set by the performer.

Instead of using absolute time as a reference, one of the multiple communication bases may be used as a metronome reference. In this case, the beat position may be analyzed from the performance content at the reference communication base and used as the metronome. When beat positions are detected from the performance content at each of the multiple communication bases, the beat position that is consistent among the largest number of communication bases and can be handled may be used as a metronome at the other communication bases. Predetermined data (data including sound generation control information, sound data, video data, etc.) may be reproduced according to these metronomes. The predetermined data may be obtained by recording a performance. For example, a drum rhythm pattern may be reproduced according to the metronome settings.

When the metronome is realized by vibration, a movable component of the player piano 1 may be vibrated to allow the player to recognize the metronome beat. For example, the drive signal generating unit 145 may generate a drive signal to move the pedal 13 slightly for each metronome beat. If the pedal 13 is a damper pedal, the amount of movement is a small amount that does not cause the damper 18 to separate from the string 15. The component that moves for each metronome beat is not limited to the pedal 13, but may be any of the keys 12. In this case, it is preferable that the key 12 is pressed down an amount that is not enough to strike the string or generate a note-on sound.

(9) The transmitted performance data may include time information when the performance data is transmitted. In this way, the performance data received from multiple communication locations can be adjusted on the time axis according to the time information, thereby correcting for any deviation on the time axis due to communication delays. For example, if the player piano 1 is not played, or if it is played but performance data is not transmitted to other communication locations, even if the performance data from the multiple other communication locations is received at different times due to communication delays, the player piano 1 can be driven as if there was the same amount of delay by shifting the performance data on the time axis so that the time information is aligned.

Using this time information, the delay time of the performance data arriving from each communication base can be recognized. The drive signal generating unit 145 may generate a drive signal such that the velocity value is smaller the longer the delay time. The drive signal generating unit 145 may add reverberation the longer the delay time. In this way, the player piano 1 can realize a sound in which the length of the delay time is an effect of the distance. In other words, a long delay time can give the listener the feeling that the performance is being performed in a distant place. For each communication base, an image visually indicating the length of the delay time may be displayed on the display. For each communication base, an image visually indicating the length of the delay time may be presented using AR (Augmented Reality). For example, the delay time may be converted into a position/distance relationship in the AR space, and an image for each communication base may be presented.

(10) The control device 20 may compare the performance data between the multiple communication centers to calculate the correlation degree, and display the correlation degree on a display. The correlation degree may be calculated using, for example, signal processing or a deep neural network (DNN). At this time, as described above, the correlation degree may be calculated using performance data adjusted so that time information is aligned between the multiple communication centers. The control device 20 may analyze the received performance data to identify chords and beat positions, and display the identified information on a display. At this time, the chord and beat position with the highest likelihood among the performance data at the multiple communication centers may be displayed on the display. Light may be illuminated on the keyboard so that the performer can recognize the keys 12 corresponding to the constituent notes of this chord.

(11) The control device 20 analyzes the chord from the received performance data, and if the likelihood of the chord is higher than a predetermined value, identifies it as the current chord. When the control device 20 generates a sound from the vibrator 47 in response to a performance operation on the key 12, the control device 20 controls the vibrator 47 not to activate the performance operation on the key 12 for a note other than the note corresponding to the chord.

(12) The control device 20 analyzes the beat position from the received performance data, and if the likelihood of the beat position is higher than a predetermined value, identifies it as the current beat position. When the control device 20 generates sound from the vibrator 47 in response to a performance operation on the key 12, if the key 12 is pressed within a predetermined time range until the next predicted beat position, the control device 20 realizes sound generation by the vibrator 47 by delaying the pressing of the key 12 to the predicted beat position. In this way, the performance sound may be aligned with the beat position.

(13) The control device 20 determines the volume from the received performance data, and also determines the volume of the player's own performance on the player piano 1. The volume is determined, for example, by the average velocity value over a specified period of time in the past. The drive signal generator 145 adjusts the volume of the received performance data so that it approaches the volume of the player's own performance, and generates a key drive signal or a vibration drive signal. The volume may be adjusted gradually rather than suddenly. In this way, the volume balance in an ensemble can be adjusted. The volume balance may be set in advance so that one of the volumes is relatively louder.

(14) When the control device 20 generates a sound from the vibrator 47 in response to a performance operation on the key 12, the control device 20 may delay the timing of the sound generated in response to the pressing of the key 12. In this case, the performance data transmitted to the other communication base and the performance data transmitted from the other communication base are not delayed. This allows the performer to play by pressing the key 12 early taking into account the delay time, thereby reducing the impact of communication delays in an ensemble.

(15) In the case where the automatic player piano 1 is not installed at any of the communication locations, and the piano is an acoustic piano in which the sensor 30 and the drive unit 40 are not arranged, the control device 20 does not need to include configuration related to the sensor 30 and the drive unit 40, and may be configured as a desktop personal computer, a tablet computer, etc.

In this case, the control device 20 may convert the performance sounds into performance data and transmit it to another communication location. The performance sounds may be collected by a microphone, and the control device 20 may generate the performance data by analyzing the constituent tones contained in the collected performance sounds and converting them into sound generation control information. This type of processing can also be applied to musical instruments other than the piano.

(16) The environment collection device 82 may have a sensor that detects the opening and closing of the fallboard 11, a sensor that detects the performer sitting on a chair, etc. In this case, the environment providing device 88 may have a display that shows the opening and closing of the fallboard 11 and the performer sitting on a chair. The environment providing device 88 may have a structure that opens and closes the fallboard 11 in response to a control signal. In this case, the fallboards 11 at other communication locations may be linked in response to the opening and closing of the fallboard 11 at a specific communication location.

(17) The keyboard instrument 10 in the player piano 1 is not limited to an acoustic piano such as a grand piano, but may be an electronic keyboard instrument. The electronic keyboard instrument may be a keyboard device having a structure equivalent to the keys 12, or a keyboard device in which the keys 12 have a sheet-like structure. In the case of a keyboard device having a sheet-like structure, it can be placed on the floor and played by stepping on it with the feet, so it can be played even in a situation where the hands cannot be used. In the case of a keyboard device played with the feet, the range of sounds that can be played may be narrow. In such a case, multiple keyboard devices with different ranges preset may be used so that multiple people can play. In the case of a keyboard device having a sheet-like structure, it may be placed on the back of a bedside table. In this case, a rotation mechanism that can switch between the front and back of the side table so that they face upwards may be provided in the support member that supports the side table.

(18) At least some of the functions of the control device 20 may be provided as a plug-in in software that realizes the video conferencing system.

(19) The network NW connecting the communication bases may be a dedicated line realized by an optical cable or the like.

(20) The environment collection device 82 and the environment providing device 88 may include a configuration for detachably attaching them to the player piano 1. The player piano 1 may also include a configuration for attaching the environment collection device 82 and the environment providing device 88. In this case, the environment collection device 82 or the environment providing device 88 may be connected to the interface 26 by being attached to the player piano 1.

The above is an explanation of the modified version.

As described above, according to one embodiment of the present invention, a control device is provided that includes a first transmitting unit that transmits first performance data including the performance content on a keyboard instrument at a first communication location to a second communication location, a first receiving unit that receives second performance data from the second communication location, and a first generating unit that generates a drive signal for producing sound in accordance with the second performance data and outputs the drive signal to a sound producing device at the first communication location, and at least one of the first performance data and the second performance data includes a key position signal that indicates the amount of depression of a key on the keyboard instrument.

The sound generating device may include a vibrator connected to a soundboard of the keyboard instrument. The sound generated in response to the second performance data may be generated by vibration of the vibrator in response to the drive signal.

The sound generating device may include a key of the keyboard instrument, a hammer linked to the key, and a string struck by the hammer. Sound generation according to the second performance data may be generated by driving the key according to the drive signal. The drive signal may be a signal for driving the key so as to reproduce the depression amount according to the key position signal.

The communication system may further include a second transmitting unit that acquires first environmental data corresponding to information on the surrounding environment collected by an environmental collection device at the first communication location and transmits the first environmental data to the second communication location, a second receiving unit that receives the second environmental data from the second communication location, and a second generating unit that generates a control signal for providing the surrounding environment corresponding to the second environmental data and outputs the control signal to an environmental providing device at the first communication location.

1, 1a, 1b: player piano, 10: keyboard instrument, 11: fallboard, 12: key, 13: pedal, 14: hammer, 15: string, 16, 16H, 16L: bridge, 17: soundboard, 17a: sound bar, 18: damper, 19: straight support, 20: control device, 21: control unit, 22: memory unit, 23: operation panel, 24: communication unit, 25: sound source unit, 26: interface, 27: bass, 30: sensor, 32: key sensor, 33: pedal sensor, 34: hammer sensor, 37H, 37L: pickup sensor, 40: drive unit, 42: key drive unit, 43: pedal drive unit, 44: stopper, 47, 47H, 47L: vibrator, 48: damper drive unit, 49H, 49L: Support unit, 50: chair, 82: environment collection device, 88: environment providing device, 100: ensemble control function, 121: environment data generation unit, 123: environment data transmission unit, 131: performance data generation unit, 133: performance data transmission unit, 143: performance data reception unit, 145, 145A: drive signal generation unit, 183: environment data reception unit, 185, 185B: control signal generation unit, 821, 822: vibration measurement plate, 823: microphone, 881, 882: vibration generation plate, 883: speaker, 1000: server, 1451: crosstalk processing unit, 1453: sound imparting unit, 1455: amplifier, 1851: self-image acquisition unit, 1853: remote image acquisition unit, 1855: image synthesis unit

Claims (4)

Keys and
A soundboard and
a receiving unit for receiving performance data including performance content for a keyboard instrument at another communication point;
a generating unit for generating a drive signal in accordance with the performance data;
a sound generating device that generates sound based on the drive signal;
Including,
the performance data includes a key position signal indicating a key depression amount of the keyboard instrument,
The sound generating device includes a speaker or a vibrator connected to the soundboard;
the drive signals include a first drive signal for driving the key so as to reproduce the key depression amount corresponding to the key position signal, and a second drive signal for driving the speaker or the vibrator based on the key position signal;
the generating unit generates the second drive signal so that a sound is generated at a predetermined position during the depression of the key and at a predicted volume by the time the key reaches the predetermined position.
Keyboard instrument. Keys and
A soundboard and
a receiving unit for receiving performance data including performance content for a keyboard instrument at another communication point;
a generating unit for generating a drive signal in accordance with the performance data;
a sound generating device that generates sound based on the drive signal;
Including,
the performance data includes a key position signal indicating a key depression amount of the keyboard instrument,
The sound generating device includes a speaker or a vibrator connected to the soundboard;
the drive signal includes a first drive signal for driving the key so as to reproduce the key depression amount corresponding to the key position signal,
the generation unit calculates a velocity based on the key position signal, and when the velocity is equal to or smaller than a predetermined value, generates the first drive signal using a value rounded up to a value greater than the predetermined value.
Keyboard instrument. Keys and
A soundboard and
a receiving unit for receiving performance data including performance content for a keyboard instrument at another communication point;
a generating unit for generating a drive signal in accordance with the performance data;
a sound generating device that generates sound based on the drive signal;
Including,
the performance data includes a key position signal indicating a key depression amount of the keyboard instrument,
The sound generating device includes a speaker or a vibrator connected to the soundboard;
the drive signals include a first drive signal for driving the key so as to reproduce the key depression amount corresponding to the key position signal, and a second drive signal for driving the speaker or the vibrator based on the key position signal;
the generating unit calculates a velocity based on the key position signal, and generates the second drive signal so as to generate a sound when the velocity is equal to or smaller than a predetermined value.
Keyboard instrument. Keys and
A soundboard and
a receiving unit for receiving performance data including performance content for a keyboard instrument at another communication point;
a generating unit for generating a drive signal in accordance with the performance data;
a sound generating device that generates sound based on the drive signal;
Including,
the performance data includes a key position signal indicating a key depression amount of the keyboard instrument,
The sound generating device includes a speaker or a vibrator connected to the soundboard;
the drive signals include a first drive signal for driving the key so as to reproduce the key depression amount corresponding to the key position signal, and a second drive signal for driving the speaker or the vibrator based on the key position signal;
The generating unit generates the first drive signal with a velocity controlled to a predetermined value or less.
Keyboard instrument.

Images