KR20230045663A - Automotive Pillar Display System - Google Patents

Abstract

Embodiments of the present invention relate to a vehicle pillar display system, and the technical problem to be solved is to display various information related to the vehicle through a display mounted on the interior side of a pillar constituting the body of the vehicle, thereby simply serving as a pillar. It is to provide a vehicle pillar display system that utilizes the pillar in charge for information display. To this end, the present invention provides a first display provided on the interior side of an A-pillar of a first vehicle; a camera provided on an outdoor side surface of the first vehicle to obtain an image of a blind spot caused by the A-pillar; and a processor connected to the first display and controlling an operation of the first display, wherein the processor displays information of a predetermined type on the first display by performing a blind spot mode and a traffic information mode; , When entering the blind spot mode, the processor displays an image of the blind spot on the first display and simultaneously detects an object in the blind spot through image processing of the image of the blind spot, and the detection A risk index of the detected object is calculated, an indicator for guiding the detected object is displayed on the first display, the size, position, shape and color of the indicator are set based on the risk index, and the traffic information is displayed. When entering the mode, the first vehicle establishes a wireless link with the second vehicle and the third vehicle outside to perform sidelink unicast communication, but sidelink unicast communication with the second vehicle and the third vehicle If a failure of the radio link associated with the second vehicle is detected while performing the sidelink medium access control reset associated with the second vehicle in response to the failure of the radio link, and the sidelink medium associated with the third vehicle Access control reset is not performed, but at the same time, critical information in the traffic information is detected through data processing of the traffic information data, a risk index of the detected critical information is calculated, and the detected risk index is guided. Provided is a vehicle pillar display system that displays an indicator on the first display.

Description

Vehicle pillar display system {Automotive Pillar Display System}

An embodiment of the present invention relates to a pillar display system for a vehicle.

Conventional automobiles are equipped with a side view mirror and a room mirror as devices for the driver to observe the surroundings except for the front while driving. In some vehicles, a side view mirror or rear surveillance camera for parking and reversing and a front part covered by the body It is equipped with a convex mirror for viewing. However, the surrounding observation device using a mirror is a structure in which the driver judges the situation by looking at a mirror that reflects a limited range outside the window. In addition, there are various problems such as blind spots due to the inherent properties of mirrors.

On the other hand, the recent around-view monitoring system is of great help in preventing collisions with nearby obstacles in low-speed situations such as parking by providing a top-view image of the 360-degree direction around the vehicle to the user riding in the vehicle. are giving However, there is a limitation in that a user can recognize only information about a surrounding situation within a very limited range through a top-view image provided through the around-view monitoring system.

Meanwhile, a pillar of a vehicle is a vehicle body structure that refers to a pillar supporting a roof of a vehicle, and may be classified into an A-pillar, a B-pillar, a C-pillar, and the like, depending on its location. These pillars generally play only a role of supporting the vehicle body, and inevitably obscure the occupant's view and cause blind spots.

The above-described information disclosed in the background art of the present invention is only for improving the understanding of the background of the present invention, and thus may include information that does not constitute prior art.

The problem to be solved according to the embodiment of the present invention is to display various information related to the vehicle through a display mounted on the interior side of the pillar constituting the body of the vehicle, so that the pillar, which conventionally only serves as a pillar, can be used for information display. It is to provide a vehicle pillar display system that is utilized.

A vehicle pillar display system according to an embodiment of the present invention includes a first display provided on an interior side of an A-pillar of a first vehicle; a camera provided on an outdoor side surface of the first vehicle to obtain an image of a blind spot caused by the A-pillar; and a processor connected to the first display and controlling an operation of the first display, wherein the processor displays information of a predetermined type on the first display by performing a blind spot mode and a traffic information mode; , When entering the blind spot mode, the processor displays an image of the blind spot on the first display and simultaneously detects an object in the blind spot through image processing of the image of the blind spot, and the detection A risk index of the detected object is calculated, an indicator for guiding the detected object is displayed on the first display, the size, position, shape and color of the indicator are set based on the risk index, and the traffic information is displayed. When entering the mode, the first vehicle establishes a wireless link with the second vehicle and the third vehicle outside to perform sidelink unicast communication, but sidelink unicast communication with the second vehicle and the third vehicle If a failure of the radio link associated with the second vehicle is detected while performing the sidelink medium access control reset associated with the second vehicle in response to the failure of the radio link, and the sidelink medium associated with the third vehicle Access control reset is not performed, but at the same time, critical information in the traffic information is detected through data processing of the traffic information data, a risk index of the detected critical information is calculated, and the detected risk index is guided. An indicator is displayed on the first display.

The processor calculates the risk index based on the type, distance, direction, and speed of the detected object and critical information for the first vehicle.

The sidelink medium access control reset associated with the second vehicle causes the first vehicle to stop transmitting and receiving sidelinks associated with the second vehicle.

The sidelink transmission and/or reception includes transmission and reception of a physical sidelink shared channel, a physical sidelink control channel, and a physical sidelink feedback channel.

The sidelink medium access control reset associated with the second vehicle stops sidelink transmission and reception associated with the third vehicle.

An embodiment of the present invention displays various vehicle-related information through a display mounted on the interior side of a pillar constituting a vehicle body, thereby utilizing a pillar, which conventionally simply serves as a pillar, for displaying information. provides

1 shows a block diagram of a vehicle according to one embodiment of the present invention.
FIG. 2 shows an exemplary appearance of the vehicle shown in FIG. 1 .
FIG. 3 shows an example of images generated by a plurality of cameras shown in FIG. 2 .
FIG. 4 shows an example of the vehicle described above with reference to FIG. 1 . For convenience of description, it is assumed that the vehicle is a four-wheeled vehicle.
5 shows a block diagram of a schematic configuration of a display device according to an embodiment of the present invention.
6A and 6B show an example in which a display unit and an outdoor camera included in a display device according to an embodiment of the present invention are mounted on a vehicle.
7 illustrates a flow chart of a process performed by a display device according to one embodiment of the present invention.
8 illustrates a flowchart of a process performed when a display device enters a blind spot mode according to an embodiment of the present invention.
9 is a diagram referenced to explain an operation when a display device according to an embodiment of the present invention enters a blind spot mode.
10 illustrates a flowchart of a process performed when a display device enters a traffic information mode according to an embodiment of the present invention.
11 illustrates a flowchart of a process of processing multi-sidelink communication in a C-V2X module when a display device enters a traffic information mode according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art, and the following examples may be modified in many different forms, and the scope of the present invention is as follows It is not limited to the examples. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the spirit of the invention to those skilled in the art.

In addition, in the following drawings, the thickness or size of each layer is exaggerated for convenience and clarity of explanation, and the same reference numerals refer to the same elements in the drawings. As used herein, the term "and/or" includes any one and all combinations of one or more of the listed items. In addition, the meaning of "connected" in the present specification means not only when member A and member B are directly connected, but also when member A and member B are indirectly connected by interposing member C between member A and member B. do.

Terms used in this specification are used to describe specific embodiments and are not intended to limit the present invention. As used herein, the singular form may include the plural form unless the context clearly indicates otherwise. Also, when used herein, “comprise, include” and/or “comprising, including” refers to a referenced form, number, step, operation, member, element, and/or group thereof. presence, but does not preclude the presence or addition of one or more other shapes, numbers, operations, elements, elements and/or groups.

In this specification, terms such as first and second are used to describe various members, components, regions, layers and/or portions, but these members, components, regions, layers and/or portions are limited by these terms. It is self-evident that These terms are only used to distinguish one element, component, region, layer or section from another region, layer or section. Thus, a first member, component, region, layer or section described in detail below may refer to a second member, component, region, layer or section without departing from the teachings of the present invention.

In addition, the control unit (controller) and/or other related devices or components according to the present invention may be implemented using any suitable hardware, firmware (eg, application specific semiconductor), software, or a suitable combination of software, firmware, and hardware. can For example, various components of a control unit (controller) and/or other related devices or parts according to the present invention may be formed on one integrated circuit chip or on separate integrated circuit chips. In addition, various components of the control unit (controller) may be implemented on a flexible printed circuit film, and may be formed on a tape carrier package, a printed circuit board, or the same substrate as the control unit (controller). In addition, various components of the control unit (controller) may be processes or threads executed in one or more processors in one or more computing devices, which execute computer program instructions to perform various functions mentioned below. and can interact with other components. Computer program instructions are stored in memory that can be executed in a computing device using standard memory vehicles, such as, for example, random access memory. Computer program instructions may also be stored on other non-transitory computer readable media, such as, for example, a CD-ROM, flash drive, or the like. In addition, those skilled in the art related to the present invention will understand that the functions of various computing devices can be combined with each other, integrated into one computing device, or the functions of a particular computing device can be incorporated into one or more other computing devices without departing from the exemplary embodiments of the present invention. It should be recognized that it can be dispersed in the field.

1 shows a block diagram of a vehicle 100 according to one embodiment of the present invention.

The vehicle 100 includes a communication unit 110, an input unit 120, a memory 130, an output unit 140, a vehicle driving unit 150, a sensing unit 160, a control unit 170, an interface unit 180, and a power supply unit. (190) may be included.

The communication unit 110 includes one or more devices that enable wireless communication between the vehicle 100 and external devices (eg, other vehicles, pedestrians (mobile terminals), infrastructure (traffic lights), base stations, external servers, and other devices capable of wireless communication). modules may be included. Also, the communication unit 110 may include one or more modules that connect the vehicle 100 to one or more networks.

The communication unit 110 includes a broadcast reception module 111, a wireless Internet module 112, a short-range communication module 113, a location information module 114, and a Cellular Vehicle-to-Everything (C-V2X) module 115. can do.

The broadcast reception module 111 receives a broadcast signal or broadcast-related information from an external broadcast management server through a broadcast channel. Here, broadcasting includes radio broadcasting or TV broadcasting.

The wireless Internet module 112 refers to a module for wireless Internet access, and may be built into or external to the vehicle 100 . The wireless Internet module 112 is configured to transmit and receive radio signals in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G, 6G, etc. The wireless Internet module 112 transmits and receives data according to at least one wireless Internet technology within a range including Internet technologies not listed above. For example, the wireless Internet module 112 may wirelessly exchange data with an external server. The wireless Internet module 112 may receive weather information and road traffic condition information (eg, Transport Protocol Expert Group (TPEG)) information from an external server.

The short-range communication module 113 is for short-range communication, and includes Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and NFC. Near Field Communication (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and wireless USB (Wireless Universal Serial Bus) technology may be used to support short-distance communication.

The short-range communication module 113 may perform short-range communication between the vehicle 100 and at least one external vehicle by forming wireless area networks. For example, the short-distance communication module 113 may wirelessly exchange data with a mobile terminal of a passenger. The short-distance communication module 113 may receive weather information and road traffic condition information (eg, Transport Protocol Expert Group (TPEG)) from a mobile terminal or an external server. For example, when a user gets on the vehicle 100, the user's mobile terminal and the vehicle 100 may perform pairing with each other automatically or by the user executing an application.

The location information module 114 is a module for acquiring the location of the vehicle 100, and a representative example thereof is a Global Positioning System (GPS) module. For example, if a vehicle utilizes a GPS module, the location of the vehicle may be obtained using a signal transmitted from a GPS satellite.

The C-V2X module 115 allows information such as traffic and road conditions to be shared between vehicles (V2V), pedestrians (V2P), and infrastructure (V2I), which complies with the LTE C-V2X direct communication standard prescribed by 3GPP. can follow

The C-V2X module 115 can perform a short-range direct communication mode capable of recognizing a 360-degree non-line-of-sight (NLOS, non-linear) path, and a built-in line-of-sight (such as a camera, radar, lidar, ultrasonic sensor, etc. to be described below) LOS, linear) sensors, etc. compensate for this. By using the C-V2X module 115 and built-in sensors together, the vehicle can see, hear, and predict potential risk factors while driving even in blind spots at intersections or in adverse weather conditions. In addition, the C-V2X module 115 allows to receive alerts from other C-V2X vehicles, roadside infrastructure (eg, traffic lights), and mobile devices. By integrating short-range C-V2X direct communications with extensive mobile cellular networks to leverage long-distance communications and cloud access, reliability is greatly improved and traffic efficiency is also beneficial.

3GPP Release 14 defines a direct communication mode that standardizes a method for vehicles to communicate with virtually everything, such as other vehicles (V2V), pedestrians (V2P), and infrastructure facilities (V2I) in a short distance without depending on the cellular network. As a result, basic, latency-critical safety messages transmitted to improve road safety and traffic efficiency are delivered using a direct communication mode utilizing the globally agreed upon 5.9 GHz ITS spectrum. And as C-V2X technology continues to evolve, the 3GPP Release 16 NR C-V2X direct mode (or sidelink) specification2 also supports advanced use cases that can enhance autonomous driving technology without using a cellular network.

V2V or V2I technology using sidelinks can be supplemented with vehicle-to-network (V2N) communication through the C-V2X module 115. This requires the help of the mobile network operator to deliver the information over the cellular network. V2N has been used for about 20 years for functions such as telematics, car collision alert, infotainment, cloud services such as maps and other software updates, traffic information and route guidance, and more recently, remote monitoring and control (remote control).

The automotive industry has already seen the benefits of basic safety messages provided by Release 14/15 NR C-V2X sidelinks, and the most common V2V use cases specified in the SAE J2945/1 standard are:

- Forward Collision Warning (FCW)

- Intersection Mobility Assist (IMA)

- Blind Spot Warning/Lane Change Warning (BSW/LCW)

For V2I use cases with roadside units (RSUs), generic traffic signal guidance and timing systems (SpaT) such as:

- Transit Signal Priority (TSP)

- Optimal Speed Advisory (OSA)

- Sharp curve deceleration warning (CSW, Curve Speed Warning)

- PREEMPT (Emergency Vehicle Preempt)

- Pedestrian in Signalized Crosswalk Warning (PED-X)

In 3GPP Release 15, the direct communication mode introduced in Release 14 has evolved, and transmit diversity (cyclic delay diversity) has been added and its functions have been improved. The features added in Release 15 are also fully compatible with Release 14.

The NR C-V2X sidelink shows improved performance, such as higher throughput, lower latency, improved reliability and positioning, all of which can work collectively to improve autonomous driving capabilities. These technology enhancements take advantage of features of 5G NR such as OFDM-based air interfaces with wideband carrier support and scalable sub-carrier spacing, flexible slot-based frameworks, or use of Demodulation Reference Signals (DMRS). This is possible by utilizing methods that make it faster and with advanced channel coding capabilities. In addition, the NR C-V2X sidelink supports sidelink time synchronization, allowing the C-V2X module 115 to operate without GPS coverage.

Moreover, NR C-V2X sidelinks can switch the default mode of operation from broadcast to reliable multicast communication. NR C-V2X sidelink is the first wireless system to introduce distance as a dimension at the physical layer, which helps achieve uniform coverage in a wide range of radio environments, including both line-of-sight and line-of-sight scenarios. Introducing distance as a dimension allows for the formation of 'instant' multicast groups based on distance and application. Multicast groups formed in this way require little overhead for group formation and dismantling.

As advanced use cases evolve, so does the importance of seamless wireless connectivity. Through NR C-V2X sidelinks, vehicles can share their intentions with other vehicles, enabling more sophisticated cooperative driving. NR C-V2X is a cross-sectional intersection call (which improves traffic efficiency by resolving uncertainty that may occur at intersections through intention sharing), cooperative lane change using low-latency communication, improved positioning accuracy, and immediate group formation based on distance. designed to enable

Thanks to the sharing of high-throughput sensor data from cameras, radar, lidar, and ultrasonic sensors, real-time updates of 3D high-definition map information, and the ability to 'see' within the blind spot of a corner or blocked by a vehicle, vehicles can achieve situational awareness and collective The cognitive ability is improved, so you can better understand the road conditions. NR C-V2X Sidelink also enables intent sharing between vehicles, even in densely populated areas thanks to its low latency and high throughput.

The input unit 120 may include a driving control unit 121, a microphone 123, and a user input unit 124.

The driving control means 121 receives a user input for driving the vehicle 100 . The driving control unit 121 may include a steering input unit 121a, a shift input unit 121b, an acceleration input unit 121c, and a brake input unit 121d.

The steering input means 121a receives an input of the driving direction of the vehicle 100 from the user. The steering input unit 121a may include a steering wheel. Depending on the embodiment, the steering input unit 121a may be formed as a touch screen, touch pad, or button.

The shift input unit 121b receives inputs of parking (P), forward (D), neutral (N), and reverse (R) of the vehicle 100 from the user. The shift input unit 121b is preferably formed in a lever shape. Depending on the embodiment, the shift input unit 121b may be formed as a touch screen, touch pad, or button.

The acceleration input unit 121c receives an input for acceleration of the vehicle 100 from a user. The brake input unit 121d receives an input for decelerating the vehicle 100 from the user. The acceleration input means 121c and the brake input means 121d are preferably formed in the form of pedals. Depending on the embodiment, the acceleration input unit 121c or the brake input unit 121d may be formed as a touch screen, touch pad, or button.

The microphone 123 may process an external sound signal into electrical data. The processed data may be utilized in various ways depending on the function being performed in the vehicle 100 . The microphone 123 may convert a user's voice command into electrical data. The converted electrical data may be transmitted to the controller 170 .

Meanwhile, according to embodiments, the microphone 123 may be a component included in the sensing unit 160 rather than a component included in the input unit 120 .

The user input unit 124 is for receiving information from a user. When information is input through the user input unit 124, the control unit 170 may control the operation of the vehicle 100 to correspond to the input information. The user input unit 124 may include a touch input means or a mechanical input means. Depending on the embodiment, the user input unit 124 may be disposed in one area of the steering wheel. In this case, the driver may manipulate the user input unit 124 with his/her fingers while holding the steering wheel.

The input unit 120 may include a plurality of buttons or touch sensors. It is also possible to perform various input operations through a plurality of buttons or touch sensors.

The sensing unit 160 senses a signal related to driving of the vehicle 100 . To this end, the sensing unit 160 includes a collision sensor, a steering sensor, a speed sensor, an inclination sensor, a weight sensor, a heading sensor, a yaw sensor, a gyro sensor, Position module, vehicle forward/backward sensor, battery sensor, fuel sensor, tire sensor, steering sensor by steering wheel rotation, vehicle internal temperature sensor, vehicle internal humidity sensor, infrared sensor, radar 162, lidar ( 163), an ultrasonic sensor 164, and the like.

Accordingly, the sensing unit 160 provides vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward/backward information, battery information, Sensing signals for fuel information, tire information, vehicle lamp information, vehicle internal temperature information, vehicle internal humidity information, steering wheel rotation angle, and the like may be acquired. In addition, the controller 170 accelerates, decelerates, decelerates, and A control signal for direction change, etc. may be generated. Here, the external environment information may be information related to various objects located within a predetermined distance range from the driving vehicle 100 . For example, the external environment information may include information about the number of obstacles located within a distance of 100 m from the vehicle 100, the distance to the obstacle, the size of the obstacle, and the type of the obstacle.

Meanwhile, the sensing unit 160 includes an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), and a throttle position. A sensor (TPS), a TDC sensor, a crank angle sensor (CAS), and the like may be further included.

The sensing unit 160 may include a biometric information sensing unit. The biometric information detection unit detects and obtains the biometric information of the occupant. Biometric information includes fingerprint recognition information, iris-scan information, retina-scan information, hand geo-metry information, facial recognition information, voice recognition ( Voice recognition) information may be included. The biometric information detection unit may include a sensor that senses the biometric information of the occupant. Here, the microphone 123 may operate as a sensor.

The sensing unit 160 may include at least one camera 161 that photographs the exterior of the vehicle 100 . The camera 161 may be referred to as an external camera. For example, the sensing unit 160 may include a plurality of cameras 161 disposed at different locations on the exterior of the vehicle. The camera 161 may include an image sensor and an image processing module. The camera 161 may process still images or moving images obtained by an image sensor (eg, CMOS or CCD). The image processing module may process a still image or video obtained through an image sensor, extract necessary information, and deliver the extracted information to the controller 170 .

The camera 161 may include an image sensor (eg, CMOS or CCD) and an image processing module. Also, the camera 161 may process a still image or moving image obtained by an image sensor. The image processing module may process a still image or video obtained through an image sensor. Also, the camera 161 may obtain an image including at least one of traffic lights, traffic signs, pedestrians, other vehicles, and road surfaces.

The output unit 140 is for outputting information processed by the controller 170 and may include a display unit 141 , a sound output unit 142 and a haptic output unit 143 .

The display unit 141 may display information processed by the controller 170 . For example, the display unit 141 may display vehicle-related information. Here, the vehicle-related information may include vehicle control information for direct control of the vehicle or vehicle driving assistance information for driving guidance to the vehicle driver. In addition, the vehicle-related information may include vehicle state information indicating a current state of the vehicle or vehicle driving information related to driving of the vehicle.

The display unit 141 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. display), a 3D display, and an e-ink display.

The display unit 141 may implement a touch screen by forming a mutual layer structure or integrally with the touch sensor. Such a touch screen may function as a user input unit 124 providing an input interface between the vehicle 100 and the user, and may provide an output interface between the vehicle 100 and the user. In this case, the display unit 141 may include a touch sensor that detects a touch on the display unit 141 so that a control command can be input by a touch method. Using this, when a touch is made to the display unit 141, the touch sensor can sense the touch, and the controller 170 can generate a control command corresponding to the touch based on this. Contents input by the touch method may be letters or numbers, or menu items that can be instructed or designated in various modes.

Meanwhile, the display unit 141 may include a cluster so that the driver can check vehicle condition information or vehicle driving information while driving. Clusters can be located above the dashboard. In this case, the driver can check the information displayed on the cluster while keeping his/her gaze in front of the vehicle.

Meanwhile, according to embodiments, the display unit 141 may be implemented as a Head Up Display (HUD). When the display unit 141 is implemented as a HUD, information may be output through a transparent display provided on the windshield. Alternatively, the display unit 141 may include a projection module to output information through an image projected on the windshield.

The audio output unit 142 converts the electrical signal from the control unit 170 into an audio signal and outputs it. To this end, the sound output unit 142 may include a speaker or the like. The sound output unit 142 may also output a sound corresponding to an operation of the user input unit 124 .

The haptic output unit 143 generates a tactile output. For example, the haptic output unit 143 may vibrate a steering wheel, a safety belt, and a seat so that the user can recognize the output.

The vehicle driving unit 150 may control the operation of various vehicle devices. The vehicle driving unit 150 includes a power source driving unit 151, a steering driving unit 152, a brake driving unit 153, a lamp driving unit 154, an air conditioning driving unit 155, a window driving unit 156, an airbag driving unit 157, a sunroof At least one of the driving unit 158 and the wiper driving unit 159 may be included.

The power source driver 151 may perform electronic control of the power source in the vehicle 100 . The power source driver 151 may include an accelerator that increases the speed of the vehicle 100 and a deceleration device that reduces the speed of the vehicle 100 .

For example, when a fossil fuel-based engine (not shown) is the power source, the power source driver 151 may perform electronic control of the engine. This makes it possible to control the output torque of the engine and the like. When the power source driving unit 151 is an engine, the speed of the vehicle may be limited by limiting engine output torque under the control of the control unit 170 .

As another example, when an electric motor (not shown) is a power source, the power source driver 151 may control the motor. This makes it possible to control the rotational speed and torque of the motor.

The steering driver 152 may include a steering apparatus. Accordingly, the steering driving unit 152 may perform electronic control of a steering apparatus in the vehicle 100 . For example, the steering driver 152 may include a steering torque sensor, a steering angle sensor, and a steering motor, and steering torque applied to a steering wheel by a driver may be detected by the steering torque sensor. The steering driver 152 may control the steering force and steering angle by changing the magnitude and direction of the current applied to the steering motor based on the speed and steering torque of the vehicle 100 . Also, the steering driver 152 may determine whether the driving direction of the vehicle 100 is properly adjusted based on the steering angle information acquired by the steering angle sensor. This makes it possible to change the driving direction of the vehicle. In addition, the steering drive unit 152 reduces the weight of the steering wheel by increasing the steering force of the steering motor when the vehicle 100 travels at a low speed, and reduces the steering force of the steering motor when the vehicle 100 travels at a high speed so as to reduce the weight of the steering wheel. weight can be increased. In addition, when the autonomous driving function of the vehicle 100 is executed, the steering driving unit 152 outputs the sensing unit 160 even in a situation where the driver operates the steering wheel (eg, a situation in which steering torque is not detected). Based on a sensing signal or a control signal provided by the controller 170, the steering motor may be controlled to generate an appropriate steering force.

The brake driver 153 may perform electronic control of a brake apparatus (not shown) in the vehicle 100 . For example, the speed of the vehicle 100 may be reduced by controlling the operation of brakes disposed on wheels. As another example, the driving direction of the vehicle 100 may be adjusted to the left or right by differentiating the operation of the brakes respectively disposed on the left and right wheels.

The lamp driving unit 154 may control turning on/off of at least one or more lamps disposed inside or outside the vehicle. The lamp driver 154 may include a lighting device. In addition, the lamp driver 154 may control the intensity and direction of light output from each lamp included in the lighting device. For example, control of direction indicator lamps, head lamps, and brake lamps may be performed.

The air conditioning driver 155 may perform electronic control of an air cinditioner (not shown) in the vehicle 100 . For example, when the temperature inside the vehicle is high, the air conditioner may be operated to supply cool air to the inside of the vehicle.

The window driver 156 may perform electronic control of a window apparatus in the vehicle 100 . For example, it is possible to control the opening or closing of left and right windows on the side of the vehicle.

The airbag driving unit 157 may perform electronic control of an airbag apparatus in the vehicle 100 . For example, in case of danger, the airbag can be controlled to explode.

The sunroof driving unit 158 may perform electronic control of a sunroof apparatus (not shown) in the vehicle 100 . For example, the opening or closing of the sunroof can be controlled.

The wiper driver 159 may control the wipers 14a and 14b provided in the vehicle 100 . For example, when receiving a user input commanding to drive the wipers through the user input unit 124, the wiper drive unit 159 performs electronic control of the driving frequency and driving speed of the wipers 14a and 14b according to the user input. can be performed. For another example, the wiper driving unit 159 determines the amount or strength of rainwater based on the sensing signal of the rain sensor included in the sensing unit 160, and wipers 14a and 14b without user input. can run automatically.

Meanwhile, the vehicle driving unit 150 may further include a suspension driving unit (not shown). The suspension driving unit may perform electronic control of a suspension apparatus (not shown) in the vehicle 100 . For example, when there is a curve on the road surface, the vibration of the vehicle 100 may be reduced by controlling the suspension device.

The memory 130 is electrically connected to the control unit 170 . The memory 170 may store basic data for the unit, control data for controlling the operation of the unit, and input/output data. In terms of hardware, the memory 190 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, and the like. The memory 130 may store various data for overall operation of the vehicle 100, such as a program for processing or control by the control unit 170.

The interface unit 180 may serve as a passage for various types of external devices connected to the vehicle 100 . For example, the interface unit 180 may have a port connectable to an external device, and communicate with the external device through the port. In this case, the interface unit 180 may exchange data with an external device.

The interface unit 180 may receive turn signal information. Here, the turn signal information may be a turn on signal of a direction indicator lamp for left turn or right turn input by the user. When a left or right direction indicator turn on input is received through the user input unit ( 124 in FIG. 1 ) of the vehicle 100 , the interface unit 180 may receive left or right turn signal information.

The interface unit 180 may receive vehicle speed information, steering wheel rotation angle information, or gear shift information. The interface unit 180 may receive vehicle speed information, steering wheel rotation angle information, or gear shift information sensed through the sensing unit 160 of the vehicle. Alternatively, the interface unit 180 may receive vehicle speed information, steering wheel rotation angle information, or gear shift information from the vehicle control unit 170 . Meanwhile, here, the gear shift information may be information about the state of the shift lever of the vehicle. For example, the gear shift information may be information on which one of the parking (P), reverse (R), neutral (N), driving (D), and first to multi-level gear states the shift lever is in. .

The interface unit 180 may receive a user input received through the user input unit 124 of the vehicle 100 . The interface unit 180 may receive user input from the input unit 120 of the vehicle 100 or through the control unit 170 .

The interface unit 180 may receive information obtained from an external device. For example, when traffic light change information is received from an external server through the communication unit 110 of the vehicle 100, the interface unit 180 may receive the traffic light change information from the control unit 170.

The controller 170 may control overall operations of each unit in the vehicle 100 . The controller 170 may be referred to as an Electronic Control Unit (ECU).

The controller 170 may be configured in terms of hardware, such as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors ), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

The power supply unit 190 may supply power required for operation of each component under the control of the control unit 170 . In particular, the power supply unit 170 may receive power from a battery (not shown) inside the vehicle.

The AVN (Audio Video Navigation) device 200 may exchange data with the controller 170 . The controller 170 may receive navigation information from the AVN device 200 . Here, the navigation information may include set destination information, route information according to the destination, map information related to vehicle driving, or vehicle location information.

Meanwhile, some of the components shown in FIG. 1 may not be essential to implement the vehicle 100 . Accordingly, the vehicle 100 described herein may have more or fewer components than those listed above.

FIG. 2 shows an exemplary appearance of the vehicle 100 shown in FIG. 1 .

Referring to FIG. 2 , four cameras 161a, 161b, 161c, and 161d may be mounted at different locations on the exterior of the vehicle 100. Each of the four cameras 161a, 161b, 161c, and 161d may be the same as the camera 161 described above. The plurality of cameras 161a, 161b, 161c, and 161d may be disposed on the front, left, right, and rear sides of the vehicle 100, respectively. Each of the plurality of cameras 161a, 161b, 161c, and 161d may be included in the camera 161 shown in FIG. 1 .

The front camera 161a may be placed near the windshield, emblem, or radiator grill.

The left camera 161b may be disposed in a case surrounding a left side mirror. Alternatively, the left camera 161b may be disposed outside the case surrounding the left side mirror. Alternatively, the left camera 161b may be disposed in an area outside the left front door, the left rear door, or the left fender.

The right camera 161c may be disposed in a case surrounding a right side mirror. Alternatively, the right camera 161c may be disposed outside the case surrounding the right side mirror. Alternatively, the right camera 161c may be disposed in an area outside a right front door, a right rear door, or a right fender.

Meanwhile, the rear camera 161d may be disposed near a rear license plate or a trunk switch.

Each image captured by the plurality of cameras 161a, 161b, 161c, and 161d is transmitted to the controller 170, and the controller 170 synthesizes the respective images to generate an external image for each direction of the vehicle. . Alternatively, each image captured by the plurality of cameras 161a, 161b, 161c, and 161d is transmitted to the processor 470 of the augmented reality providing device 400 to be described later, and the processor 470 synthesizes the respective images , external images for each direction of the vehicle may be generated.

FIG. 3 shows an example of images generated by the plurality of cameras 161a, 161b, 161c, and 161d shown in FIG. 2 .

Referring to FIG. 3 , the vehicle 100 may generate a synthesized image 300 . The synthesized image 300 includes a first image area 301 corresponding to an external image captured by the front camera 161a, a second image area 302 corresponding to an external image captured by the left camera 161b, A third image area 303 corresponding to an external image captured by the right camera 161c and a fourth image area 304 corresponding to an external image captured by the rear camera 161d may be included. The synthesized image 300 may be named an Around View Monitoring (AVM) image.

When the synthesized image 300 is generated, boundaries 311 , 312 , 313 , and 314 are generated between any two external images included in the synthesized image 300 . The vehicle 100 may be naturally displayed by image blending of the boundary lines 311 , 312 , 313 , and 314 .

In addition, a preset image pointing to the vehicle 100 may be included in the center of the synthesized image 300 . In addition, the synthesized image 300 may be displayed on the augmented reality providing device 400 mounted on the vehicle 100 .

FIG. 4 shows an example of the vehicle 100 described above with reference to FIG. 1 . For convenience of explanation, it is assumed that the vehicle 100 is a four-wheeled vehicle.

Referring to FIG. 4 , the vehicle 100 may include one or more of a radar 162 , a lidar 163 , and an ultrasonic sensor 164 .

The radar 162 may be mounted on one side of the vehicle 100, emit electromagnetic waves toward the periphery of the vehicle 100, and receive electromagnetic waves reflected from various objects existing around the vehicle 100. For example, the radar 162 may obtain information related to the distance, direction, altitude, and the like of the object by measuring the time of the electromagnetic wave reflected and returned by an object.

The lidar 163 may be mounted on one side of the vehicle 100 and emit a laser toward the periphery of the vehicle 100 . The laser emitted by the LIDAR 163 may be scattered or reflected and returned to the vehicle 100, and the LIDAR 163 may determine the return time, intensity, frequency change, and polarization state change based on the laser beam 163. , Information on physical characteristics such as distance, speed, and shape of a target located around the vehicle 100 may be obtained.

The ultrasonic sensor 164 is mounted on one side of the vehicle 100 and generates ultrasonic waves toward the periphery of the vehicle 100 . The ultrasonic wave generated by the ultrasonic sensor 164 has a high frequency (about 20 KHz or more) and a short wavelength. The ultrasonic sensor 164 may be mainly used to recognize an obstacle or the like close to the vehicle 100 .

The radar 162 , lidar 163 , and ultrasonic sensor 164 shown in FIG. 4 may be sensors included in the sensing unit 160 shown in FIG. 1 . In addition, it is obvious to those skilled in the art that the radar 162, the lidar 163, and the ultrasonic sensor 164 may be mounted in a different number at a location different from that shown in FIG. 4, depending on the embodiment.

5 shows a block diagram of a schematic configuration of a display device 500 according to an embodiment of the present invention.

The display device 500 may generate vehicle-related information from an image received from the camera 161 shown in FIG. 1 through computer vision-based signal processing. Here, the vehicle-related information may include vehicle control information for direct control of the vehicle or vehicle driving assistance information for driving guidance to the driver of the vehicle.

Referring to FIG. 5 , the display device 500 includes an input unit 510, a communication unit 520, an interface unit 530, a memory 540, a processor 570, a display unit 580, and a power supply unit 590. can include However, according to embodiments, some of the components shown in FIG. 5 may be omitted or new components may be further included. For example, some components included in the sensing unit 160 shown in FIG. 1 may be included in the display device 500 instead of the vehicle 100 .

The input unit 510 may include a plurality of buttons, switches, or a physical input means for receiving various commands for the display device 500 from a passenger of the vehicle 100 such as a touch screen. It is possible to turn on and operate the power of the display device 500 through a plurality of buttons, switches, or a touch screen. In addition, it is also possible to perform various input operations.

The communication unit 520 is provided with at least one signal receiver, and can exchange data with external devices such as a mobile terminal of a passenger of the vehicle 100, an external server, or another vehicle in a wireless manner. In this case, the signal receiver may be a hardware-based physical signal receiving device. For example, the signal receiver may include an antenna designed to receive signals of at least one frequency band.

The communication unit 520 may wirelessly exchange data with the vehicle driver's mobile terminal. As a wireless data communication method, various data communication methods such as Bluetooth, WiFi Direct, WiFi, APiX, and NFC are possible.

The communication unit 520 may receive weather information and road traffic condition information, eg, Transport Protocol Expert Group (TPEG) information, from an external server. Meanwhile, the display device 500 may transmit the identified real-time information to an external device.

Meanwhile, when a user gets into a vehicle, the user's mobile terminal and the display device 500 may perform pairing with each other automatically or by the user's application execution.

The communication unit 520 may receive traffic light change information from an external server. Here, the external server may be a server located in a traffic control center that controls traffic.

The interface unit 530 may receive vehicle-related data or transmit a signal processed or generated by the processor 570 to the outside. The interface unit 530 may perform data communication with the vehicle 100 and/or the external device 600 by forming a wired/wireless network with the vehicle 100 or an external device within the vehicle 100 . For example, the interface unit 530 may perform data communication with the control unit 170 , the audio video navigation (AVN) device 200 , and the sensing unit 160 . As another example, the interface unit 530 receives navigation information or content data from an external device 600 such as a mobile terminal, a USB memory, an SD card, etc. located inside the vehicle 100, and transmits the navigation information or content data to the processor 570. can be provided to To this end, the interface unit 530 may include at least one physical port for communication with the external device 600 .

Navigation information may include set destination information, route information according to the destination, map information related to vehicle driving, and current location information of the vehicle. Meanwhile, the navigation information may include vehicle location information on the road.

Meanwhile, the interface unit 530 may receive sensor information from the controller 170 or the sensing unit 160 .

Here, the sensor information includes vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward/reverse information, battery information, fuel information, tire information, and vehicle lamp information. , vehicle internal temperature information, vehicle internal humidity information, and object information.

Such sensor information includes heading sensor, yaw sensor, gyro sensor, position module, vehicle forward/backward sensor, wheel sensor, vehicle speed sensor, It can be obtained from a vehicle body tilt sensor, battery sensor, fuel sensor, tire sensor, steering sensor by steering wheel rotation, vehicle internal temperature sensor, vehicle internal humidity sensor, object sensor (e.g., radar, lidar, ultrasonic sensor, etc.) there is. Meanwhile, the position module may include a GPS module for receiving GPS information.

Meanwhile, among sensor information, vehicle direction information, vehicle location information, vehicle angle information, vehicle speed information, and vehicle tilt information related to vehicle driving may be referred to as vehicle driving information.

The interface unit 530 may receive turn signal information. Here, the turn signal information may be a turn on signal of a direction indicator lamp for left turn or right turn input by the user. When a left or right direction indicator turn on input is received through the user input unit ( 124 in FIG. 1 ) of the vehicle 100 , the interface unit 530 may receive left or right turn signal information.

The interface unit 530 may receive vehicle speed information, steering wheel rotation angle information, or gear shift information. The interface unit 530 may receive vehicle speed information, steering wheel rotation angle information, or gear shift information sensed through the sensing unit 160 of the vehicle. Alternatively, the interface unit 530 may receive vehicle speed information, steering wheel rotation angle information, or gear shift information from the vehicle control unit 170 . Meanwhile, here, the gear shift information may be information about the state of the shift lever of the vehicle. For example, the gear shift information may be information on which one of the parking (P), reverse (R), neutral (N), driving (D), and first to multi-level gear states the shift lever is in. .

The interface unit 530 may receive a user input received through the user input unit 124 of the vehicle 100 . The interface unit 530 may receive user input from the input unit 120 of the vehicle 100 or through the control unit 170 .

The interface unit 530 may receive information acquired from an external device. For example, when traffic light change information is received from an external server through the communication unit 110 of the vehicle 100, the interface unit 530 may receive the traffic light change information from the control unit 170.

The memory 540 may store one or more programs and/or instructions executable by the processor 570 . In addition, the memory 540 may store various data that is searched for, manipulated, changed, or stored by the processor 570 . Data stored in the memory 540 may include various types of information for processing or controlling overall operations of the display device 500 .

The memory 540 may store data for checking object/critical information. For example, the memory 540, when a predetermined object/critical information is detected from an image captured by the camera 161/from traffic information obtained from the C-V2X module 115, by a predetermined algorithm, the object / Data to check what the critical information corresponds to can be saved.

The memory 540 may store data about traffic information. For example, the memory 540, when predetermined traffic information is detected from the external image/information of the vehicle 100 acquired through the camera 161/C-V2X module 115, by a predetermined algorithm, Data for confirming what the traffic information corresponds to may be stored.

The memory 540 may store data for determining an area indicated by at least one direction of the driver's line of sight and gesture of the vehicle 100 . For example, when the interior of the vehicle 100 is divided into a plurality of predetermined regions, a set of a plurality of 3D coordinates corresponding to each region may be stored in the memory 540 . That is, each area may be defined with a plurality of 3D coordinates. Also, 3D coordinates defined for one region may be different from 3D coordinates defined for another region. If the direction of the driver's gaze detected by the processor 570 at an arbitrary point in time is expressed as one 3D coordinate, the processor 570 calculates the 3D coordinates corresponding to the direction of the driver's gaze from the memory 540. Only one area associated with coordinates can be determined.

Meanwhile, in terms of hardware, the memory 540 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, and the like.

Camera 550 may be configured to photograph the exterior and/or interior of vehicle 100 . In one embodiment, the camera 550 may include at least one outdoor camera 551 and at least one indoor camera 552, respectively.

The outdoor camera 551 may be provided on an outdoor surface of the vehicle 100 (eg, an outer surface of the A-pillar) to obtain an image of a blind spot formed by the A-pillar of the vehicle 100 . For example, one outdoor camera 551 may be mounted on each of the left and right A-pillars of the vehicle 100 .

The indoor camera 552 is disposed on one side of the interior of the vehicle 100 to generate an indoor image of the vehicle 100 . For example, the indoor camera 552 may be disposed in various locations inside the vehicle 100, such as a dashboard surface, an indoor side surface of a roof, and a rear-view mirror, to photograph occupants of the vehicle 100. In this case, the indoor camera 552 may generate an indoor image of an area including the driver's seat of the vehicle 100 . Also, the indoor camera 552 may generate an indoor image of an area including a driver's seat and a passenger seat of the vehicle 100 . The indoor image generated by the indoor camera 552 may be a 2D image and/or a 3D image. To generate a 3D image, the indoor camera 552 may include at least one of a stereo camera, a depth camera, and a 3D laser scanner. The indoor camera 552 may provide the indoor image generated by the indoor camera 552 to the processor 570 .

The processor 570 may analyze the indoor image provided from the indoor camera 552 to detect various objects or gestures of passengers. For example, the processor 570 may detect the driver's gaze and/or gesture from a portion corresponding to the driver's seat area in the indoor image. As another example, the processor 570 may detect a passenger's gaze and/or gesture from a portion corresponding to an area other than the driver's seat area of the indoor image. Of course, gazes and/or gestures of the driver and passenger may be simultaneously detected.

The processor 570 controls the overall operation of each unit in the display device 500.

The processor 570 may process a front image, a left image, a right image, and/or a rear image of the vehicle provided from the camera 161 .

Separately, the processor 570 may process images/traffic information of blind spots acquired using the camera 550/C-V2X module 115. For example, the processor 570 may perform computer vision-based signal processing on images of blind spots. For example, the processor 570 may perform signal processing on various traffic information acquired by the C-V2X module 115. Accordingly, the processor 570 detects object/threshold information existing in the blind spot from the image/threshold information of the blind spot provided from the camera 550/C-V2X module 115, and the detected object/threshold information. Information can be tracked. In particular, when detecting object/critical information, the processor 570 performs lane detection (LD), vehicle detection (VD), pedestrian detection (PD), and brightspot detection (BD). ), traffic sign recognition (TSR), and road surface detection.

Meanwhile, a traffic sign may mean predetermined information that can be delivered to the driver of the vehicle 100 . Traffic signals can be communicated to drivers through traffic lights, traffic signs, or road surfaces. For example, the traffic signal may be a vehicle or pedestrian Go or Stop signal output from a traffic light. For example, traffic signals may be various designs or texts displayed on traffic signs. For example, traffic signals may be various designs or texts displayed on a road surface.

The processor 570 may detect information from an external image/external traffic information of the vehicle 100 generated by the camera 161/C-V2X module 115.

Traffic information may be information about vehicle driving conditions. For example, the information may be a concept including road information on which the vehicle is traveling, traffic law information, surrounding vehicle information, vehicle or pedestrian traffic light information, construction information, traffic condition information, parking lot information, lane information, and the like.

The processor 570 may detect traffic information from any one of traffic lights, traffic signs, and road surfaces included in the external image/traffic information acquired by the camera 161/C-V2X module 115. For example, the processor 570 may detect a Go or Stop signal of a vehicle or pedestrian from a traffic light included in video/traffic information. For example, the processor 570 may detect various designs or texts from traffic signs included in image/traffic information. For example, the processor 570 may detect various designs or texts from the road included in the image/traffic information.

The processor 570 may check the information by comparing the detected information with the information stored in the memory 540 .

For example, the processor 570 detects a pattern or text indicating a rampway from an object included in the acquired image/traffic information. Here, the object information may be a traffic sign or a road surface. Detect drawings or text. The processor 570 may check the rampway information by comparing the traffic information stored in the memory 540 with the detected design or text.

For example, the processor 570 detects a design or text indicating a vehicle or pedestrian stop from an object included in the acquired image/traffic information. Here, the object may be a traffic sign or a road surface. The processor 570 may check the stop information by comparing the traffic information stored in the memory 540 with the detected design or text. Alternatively, the processor 570 detects a stop line from a road included in the obtained image/traffic information. The processor 570 may check the stop information by comparing the traffic information stored in the memory 540 with the stop line.

For example, the processor 570 may detect whether there is a lane in an object included in the acquired image/traffic information. Here, the object may be a road surface. The processor 570 may check the color of the detected lane. The processor 570 may determine whether the detected lane is a driving lane or a waiting lane.

For example, the processor 570 may detect vehicle Go or Stop information from an object included in the acquired image/traffic information. Here, the object may be a vehicle traffic light. Here, the Go information of the vehicle may be a signal instructing the vehicle to go straight, turn left, or turn right. The vehicle stop information may be a signal instructing the vehicle to stop. Vehicle Go information may be displayed in green, and vehicle Stop information may be displayed in red.

For example, the processor 570 may detect a pedestrian's Go or Stop information from an object included in the obtained image/traffic information. Here, the object may be a pedestrian traffic light. Here, the pedestrian's Go information may be a signal instructing the pedestrian to cross the road at a crosswalk. Pedestrian stop information may be a signal instructing a pedestrian to stop at a crosswalk.

Meanwhile, the processor 570 may control zoom of the cameras 161 and 550 . For example, the processor 570 may control the zoom of the cameras 161 and 550 according to an object detection result. For example, when a traffic sign is detected but content displayed on the traffic sign is not detected, the processor 570 may control the camera 161 to zoom-in. As another example, when a face of a passenger included in indoor images of the vehicle 100 has a size smaller than a threshold value, the processor 570 may control the camera 550 to zoom in.

Meanwhile, the processor 570 transmits weather information and road traffic condition information from all communication capable devices 104 such as other vehicles 101, pedestrians (mobile terminals) 102, and infrastructure 103 through the communication unit 520. , for example, TPEG (Transport Protocol Expert Group) information may be received.

Meanwhile, the processor 570 may determine traffic condition information around the vehicle, determined based on the stereo image, on the display device 500 in real time.

Meanwhile, the processor 570 may receive navigation information and the like from the AVN device 200 through the interface unit 530 .

Meanwhile, the processor 570 may receive sensor information from the control unit 170 or the sensing unit 160 through the interface unit 530 . Here, the sensor information includes vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward/reverse information, battery information, fuel information, tire information, vehicle It may include at least one of lamp information, vehicle internal temperature information, vehicle internal humidity information, and steering wheel rotation information.

Meanwhile, the processor 570 may include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, It may be implemented using at least one of controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

The processor 570 may be controlled by the control unit 170 .

The display unit 580 may include at least one display displaying images processed by the processor 570 . For example, the display unit 580 may include at least one of a cluster, a Head Up Display (HUD), and a transparent display that displays various images related to the operation of the display device 500 or the vehicle 100. .

Specifically, the display unit 580 may include at least a first display 581 . The first display 581 may be provided on the interior side of at least one of A-pillars on both sides of the vehicle 100 .

In one embodiment, the display unit 580 may further include a second display 582 . For example, the second display 582 may be a Head Up Display (HUD) provided on the dashboard or windshield of the vehicle 100 to display an image on the windshield. When the HUD is provided on the dashboard of the vehicle 100 , the HUD may include a projection module that outputs display light for forming an image on the windshield 11 . Alternatively, the HUD may be a transparent display that forms a part of the windshield or is directly attached to the windshield without a separate projection module. When the second display 582 is a transparent display, it has a predetermined transmittance in an off state, and may change transmittance, color, or the like in response to a predetermined electric signal, or display a graphic object corresponding to the electric signal. Accordingly, the user can check various types of information through the second display 582 while looking forward.

The power supply 590 may supply power required for operation of each component under the control of the processor 570 . In particular, the power supply 590 may receive power from a battery inside the vehicle 100 .

6A and 6B show an example in which the display unit 580 and the outdoor camera 551 included in the display device 500 according to an embodiment of the present invention are mounted on the vehicle 100.

First, FIG. 6A shows the arrangement of the outdoor camera 551 for the exterior of the vehicle 100 . As described above, the left A-pillar 10a and the right A-pillar 10b may be provided on the left and right sides of the windshield of the vehicle 100, respectively. In this case, even if the occupant looks forward, a portion of the occupant's visual field is inevitably limited by the left A-pillar 10a and the right A-pillar 10b, respectively. That is, a blind spot by the left A-pillar 10a and a blind spot by the right A-pillar 10b are formed. Hereinafter, the blind spot caused by the left A-pillar 10a will be referred to as a 'left blind spot', and the blind spot caused by the right A-pillar 10b will be referred to as a 'right blind spot'.

In one embodiment, the display device 500 includes an outdoor camera 551a mounted on the outdoor side of the left A-pillar 10a and an outdoor camera 551b mounted on the outdoor side of the right A-pillar 10b. can do.

The outdoor camera 551a captures a blind spot caused by at least the left A-pillar 10a and acquires an image of the left blind spot. In addition, the outdoor camera 551b captures a blind spot caused by at least the left A-pillar 10b and acquires an image of a right blind spot.

6B shows the arrangement of the indoor camera 552, the first display 581, and the second display 582 in the interior of the vehicle 100. Hereinafter, for convenience of explanation, the first display 581 will be referred to as a 'pillar display' and the second display 582 will be referred to as a 'windshield display'.

First, the indoor camera 552 may be mounted at a location inside the vehicle 100 capable of capturing at least a part of the driver's body. An indoor image captured by the indoor camera 552 may be provided to the processor 570 . The processor 570 detects the occupant's body (eg, face, eyes, hands, arms) from the indoor image through image processing on the indoor image, and performs a predetermined operation or function based on the detected body characteristics. can be done For example, the processor 570 may identify a driver's gesture from an indoor image. Next, the processor 570 may execute a specific command related to the identified gesture among commands stored in the memory. As a specific command is executed, a specific function corresponding to the identified gesture may be activated.

One or more pillar displays 581 may be provided on the interior surfaces of the left A-pillar 10a and the right A-pillar 10b, respectively. The pillar display 581a provided on the left A-pillar 10a may display an image of a left blind spot at all times or in a specific situation. In addition, the pillar display 581b provided on the right A-pillar 10b may display an image of a right blind spot at all times or in a specific situation.

In addition to images of left and right blind spots, the pillar display 581 may display various graphic objects related to the vehicle 100, the surrounding environment, or passengers.

Meanwhile, in FIG. 6A , the outdoor cameras 551a and 551b are illustrated as being disposed on the outdoor side of the left A-pillar 10a and the right A-pillar 10b, respectively, but this is merely exemplary. For example, the outdoor cameras 551a and 551b may be disposed at other positions instead of the outdoor sides of the left A-pillar 10a and the right A-pillar 10b, provided that they are positioned to take pictures of the left blind spot and the right blind spot. It should be understood that there is

7 illustrates a flow chart of a process S700 performed by the display device 500 according to an embodiment of the present invention.

Prior to a detailed description of the process S700 shown in FIG. 7, a plurality of steps included in the process S700 are shown sequentially, but may be performed in a different order from that shown in FIG. 7 depending on embodiments. make it clear in advance that For example, one step may be performed in parallel with another step. Also, additional steps may be further included in process S700.

In step S710, the processor 570 may activate the displays 581 provided on the A-pillars 10a and 10b.

In one embodiment, the processor 570 may activate the pillar display 581 based on an input signal corresponding to a command received by the input unit. In this case, the processor 570 can simultaneously activate the left pillar display 581a and the right pillar display 581b or selectively activate either one.

Alternatively, the processor 570 may automatically activate the pillar display 581 when a predetermined condition is satisfied. For example, when the vehicle 100 is located in a specific type of area (eg, a tunnel, a drive-thru, a ramp, a parking lot, or a curve), the pillar display 581 may be automatically activated.

Alternatively, the processor 570 may automatically activate the pillar display 581 based on the user's gesture detected from the indoor image.

In step S720, the processor 570 may enter at least one of a plurality of display modes including a blind spot mode, a traffic information mode, and a route guidance mode. In the present invention,

The blind spot mode may be a mode in which an image of a blind spot acquired by the outdoor camera 551 is displayed on a pillar display.

Also, the traffic information mode may be a mode in which a graphic object corresponding to traffic information data received through the interface unit 530 is displayed on the pillar display 581 . In this case, the traffic information data may include at least one of weather data, multimedia data, other vehicle condition data, pedestrian condition data, and infrastructure condition data.

Also, the route guidance mode may be a mode in which a graphic object corresponding to route guidance data for at least a part of a planned route of the vehicle 100 is displayed on the pillar display.

The processor 570 may enter one of a plurality of display modes or simultaneously enter two or more display modes. For example, the input unit may include a physical switch for selecting a display mode of the pillar display, and the processor 570 may enter a specific display mode selected by the user according to a user's operation of the switch.

Alternatively, the processor 570 may enter at least one of a plurality of display modes based on a sensing signal provided from the vehicle 100 . For example, when the transmission of the vehicle 100 is located in the D range, at least the blind spot mode may be entered. This is to provide the user with an image of a blind spot as the vehicle 100 moves forward when the transmission is in the D range.

In step S730, the processor 570 may display an indicator guiding information of a predetermined type for each of at least one mode entered through step S730. Here, the indicator is a type of graphic object and may have a form of a symbol, number, character, or image.

For example, in the blind spot mode, the processor 570 displays an indicator for guiding an object (eg, a pedestrian, another vehicle 100, a traffic light, a lane) existing in the blind spot along with an image of the blind spot on the pillar display 581. can be displayed on

As another example, in the traffic information mode, the processor 570 may perform other vehicle conditions, pedestrian conditions, infrastructure conditions, weather, music, movies, photos, games, broadcasting, SNS, Various traffic information data related to chatting, shopping, etc. may be received, and a graphic object corresponding to the received traffic information data may be displayed on the pillar display 581 .

As another example, in the route guidance mode, the processor 570 corresponds to at least one of route data to a destination, pre-registered area of interest data (eg, waypoint, tourist destination), and peripheral facility data (eg, gas station, drive-thru). A graphic object may be displayed on the pillar display 581.

8 illustrates a flowchart of a process (S800) performed when the display device 500 enters the blind spot mode according to an embodiment of the present invention.

In step S810, the processor 570 displays the image of the blind spot provided from the outdoor camera 581 on the pillar display 581. In this case, the processor 570 may display an image of a blind spot by selectively activating at least one of the left pillar display 581a and the right pillar display 581b according to the steering angle of the vehicle 100 . For example, when the vehicle 100 is turning left, only the left pillar display 581a is activated to display an image of the left blind spot, and when the vehicle 100 is turning right, only the right pillar display 581b is activated to display the right blind spot. , and when the vehicle 100 is traveling straight, both the left pillar display 581a and the right pillar display 581b may be activated.

In step S820, the processor 570 may detect an object from the blind spot image. In this case, the processor 570 may individually detect the object from the image of the left blind spot and the image of the right blind spot.

When an object is detected from the blind spot image, the processor 570 may calculate at least one of the type, size, shape, position, speed, and direction of the detected object.

In step S830, the processor 570 may calculate a risk index of the detected object. Here, the risk index may be a value numerically expressing the risk of collision between the vehicle 100 and an object. For example, the closer the distance between the vehicle 100 and the object, the larger the size of the object, and the faster the object approaches the vehicle 100, the processor 570 may increase the risk index.

In step S840, the processor 570 may display an indicator for guiding the calculated risk index on the pillar display 581. Specifically, an indicator for guiding a risk index may be displayed in an overlapping manner on a blind spot image on the pillar display 581 . In this case, the indicator may be displayed at a position adjacent to the object indicated by the indicator in the entire area of the blind spot image.

Meanwhile, when displaying an indicator for guiding a risk index, the processor 570 may set at least one of a size, position, shape, and color of the indicator based on the risk index. For example, when a risk index for a specific object within a blind spot increases over time, the processor 570 may increase the size of the indicator in proportion to the risk index. For another example, a first color indicator may be displayed when the risk index for a specific object is less than the threshold value, and a second color indicator may be displayed when the risk index for the same object increases to more than the threshold value.

FIG. 9 is a diagram referenced to describe an operation when the display apparatus 500 according to an embodiment of the present invention enters a blind spot mode.

Referring to (a) of FIG. 9 , the drawing on the left illustrates a state in which the left pillar display 581a is deactivated. In this case, as illustrated, a part of the object 901 in the front is covered by the left A-pillar 10a, and thus, it may be difficult for the user to accurately recognize the object 901.

To solve this problem, the processor 570 may enter the blind spot mode and display the image 910 of the left blind spot on the left pillar display 581a, as shown in (b) of FIG. there is. Accordingly, since a part of the object 901 covered by the left A-pillar 10a is displayed on the left pillar display 581a, the user can easily recognize the object 901 and prepare for danger by moving the vehicle 100 ) can take appropriate measures, such as operating the brake system.

Meanwhile, when the processor 570 enters the blind spot mode, instead of always displaying the image 910 of the blind spot on the left pillar display 581a, only when the risk index of the object present in the blind spot is greater than or equal to the threshold value. , the blind spot image 910 may be displayed on the left pillar display 581a. For example, when the processor 570 enters the blind spot mode, when the risk index of the object 901 within the blind spot is smaller than the threshold value, the processor 570 maintains the left pillar display 581a in an inactive state, and then increases the risk index of the same object. When the value is greater than or equal to the threshold value, the left pillar display 581a may be activated to display the image 910 of the blind spot.

(c) of FIG. 9 illustrates that the indicator 911 for guiding the risk index of the object 901 is displayed on the left pillar display 581a. For example, when the risk index of the object 901 is greater than or equal to a threshold value, the processor 570 warns of a risk of collision between the vehicle 100 and the object 901 and sets a predetermined indicator 911 to the left to dictate the driver's response. It can be displayed on the pillar display 581a. For example, when the risk index is the first value, the processor 570 may display the indicator 911 of the first size on the left pillar display 581a as shown. At this time, the indicator 911 may be displayed in a form overlapping the image 910 of the blind spot.

(d) of FIG. 9 illustrates an indicator 912 displayed when the risk index of the object 901 is higher than that of (c) of FIG. 9 . Compared to the indicator 911 of FIG. 9(c), the indicator 912 may have an enlarged size. Of course, it is also possible to change the color, transparency, flashing cycle, etc. in addition to the size.

Meanwhile, although the display operation of the left pillar display 581a has been described in FIG. 9 as a standard, it goes without saying that the right pillar display 581b may also operate in the same manner.

In addition, although the indicators 911 and 912 are illustrated in (c) and (d) of FIG. 9 as symbols, this is exemplary and may have a combination of numbers, characters, images, etc. other than symbols.

10 illustrates a flowchart of a process performed when a display device enters a traffic information mode according to an embodiment of the present invention.

In step S1010, the processor 570 displays various traffic information provided from the C-V2X module 115 on the pillar display 581. In this case, the processor 570 may display an image related to traffic information by selectively activating at least one of the left pillar display 581a and the right pillar display 581b according to the steering angle of the vehicle 100 . For example, when the vehicle 100 is turning left, only the left pillar display 581a is activated to display an image related to left traffic information, and when the vehicle 100 is turning right, only the right pillar display 581b is activated to display right traffic information. An image related to information is displayed, and both the left pillar display 581a and the right pillar display 581b may be activated when the vehicle 100 is traveling straight.

In step S1020, the processor 570 may detect critical information from traffic information data. In this case, the processor 570 may separately detect critical information from left traffic information data and right traffic information data.

When critical information is detected from traffic information data, the processor 570 may calculate at least one of the type, size, shape, position, speed, and direction of the detected critical information.

In step S1030, the processor 570 may calculate a risk index of the detected critical information. Here, the risk index may be a value numerically expressing the risk of the vehicle 100 colliding with a specific object based on critical information. For example, the closer the distance between the vehicle 100 and the object, the larger the size of the object, and the faster the object approaches the vehicle 100, the processor 570 may increase the risk index.

In step S840, the processor 570 may display an indicator for guiding the calculated risk index on the pillar display 581. Specifically, an indicator for guiding a risk index may be displayed in an overlapping manner on an image related to critical information on the pillar display 581 . In this case, the indicator may be displayed at a position adjacent to the object indicated by the indicator in the entire area of the critical information image.

Meanwhile, when displaying an indicator for guiding a risk index, the processor 570 may set at least one of a size, position, shape, and color of the indicator based on the risk index. For example, when the risk index of a specific object within the critical information increases over time, the processor 570 may increase the size of the indicator in proportion to the risk index. For another example, a first color indicator may be displayed when the risk index for a specific object is less than the threshold value, and a second color indicator may be displayed when the risk index for the same object increases to more than the threshold value.

On the other hand, in the C-V2X module 115, the vehicle 100 may fail in a wireless link with other vehicles 101, pedestrians 102, infrastructure 103, and the like. Here, the radio link failure means that in the RRC_CONNECTED mode during wireless communication, the vehicle 100 monitors the radio link in the active BWP based on the reference signal (SSB/CSI-RS) and signal quality threshold set in the network (Radio Link Monitoring). The SSB-based RLM is based on the SSB associated with the initial DL BWP and can be described only for the DL BWP including the initial DL BWP and the SSB associated with the initial DL BWP, and for other DL BWPs, the RLM This can be performed only based on the CSI-RS. When one of the following criteria is satisfied, the vehicle 100 must declare a radio link failure (RLF).

- Timer expiration started after indication of radio problem at physical layer (if radio problem recovers before timer expires, UE stops timer); or

- random access procedure failure; or

- RLC failure.

After the RLF is declared, the vehicle 100:

- stay in RRC_CONNECTED mode;

- select an appropriate cell and then initiate RRC re-configuration;

- If no suitable cell is found within a certain period of time after RLF is declared, RRC_IDLE is entered.

In 3GPP TS 23.287, identifiers for NR V2X communication are introduced as follows:

When V2X communication is carried over a PC5 unicast link, the following principles apply:

- The granularity of the PC5 unicast link is equal to the pair of application layer IDs of both vehicles. Therefore, if the V2X service is associated with the same application layer ID pair, one PC5 unicast link supports one or more V2X services (eg, PSID or ITS-AID). For example, vehicle 1 has one PC5 unicast link with a peer vehicle identified by application layer ID 2 and another PC5 unicast link with a peer vehicle identified by application layer ID 4 .

After successful PC5 unicast link establishment, the first vehicle and the second vehicle use the same layer-2 ID pair for subsequent PC5-S signaling message exchange and V2X service data transmission. When transmitting a message through the established PC5 link, the V2X layer of the transmitting vehicle determines whether the message is a PC5-S signaling message (i.e., direct communication request/acceptance, link identifier update request/response, connection disconnection request/response) or service data transmission. It indicates to the AS layer whether or not While the V2X layer of the receiving vehicle processes the message if it is a PC5-S signaling message, the V2X layer of the receiving vehicle delivers the message to the upper layer if the message is an application data message.

Unicast mode supports a per-flow QoS model. During unicast link establishment, each vehicle assigns itself a PC5 link identifier and associates the PC5 link identifier with the unicast link profile for the established unicast link. The PC5 link identifier is a unique value within the vehicle. The unicast link profile identified by the PC5 link identifier includes the service type(s) (e.g. PSID or ITS-AID), the application layer ID and layer-2 ID of the first vehicle, the application layer ID of the second vehicle and Layer-2 ID and a set of PC5 QoS flow identifier(s) (PFI(s)). Each PFI is associated with a QoS parameter (ie PQI and optionally a range). The PC5 Link Identifier and PFI(s) are unchanged values for established unicast links regardless of changes in Application Layer ID and Layer-2 ID. The vehicle uses PFI to indicate the PC5 QoS flow to the AS layer, so that the AS layer identifies the corresponding PC5 QoS flow even if the source and/or destination layer-2 IDs have changed, for example due to privacy support. do. A vehicle uses a PC5 link identifier to indicate a PC5 unicast link to a V2X application layer, so even if there is more than one unicast link associated with one service type, the V2X application layer identifies the corresponding PC5 unicast link (e.g. For example, a vehicle establishes multiple unicast links to multiple vehicles for the same service type).

At the 3GPP RAN2#106 meeting, radio link failure monitoring for both transmitting and receiving vehicles for sidelink (SL) unicast communication was agreed, but detailed procedures related to NR SL RLF declaration are still unclear, and RLF procedure in Uu It may not be suitable to apply directly.

On the other hand, according to the current specifications for radio link monitoring and radio link failure declaration for NR Uu links as discussed in 3GPP TS 38.331, if the vehicle detects a radio link failure with the network, the vehicle can perform other actions. and can jump to RRC_IDLE or initiate a connection reset procedure with the network. Some actions may not be applicable to SL RLF since the vehicle may not have connectivity issues with the network and therefore does not need to leave RRC_CONNECTED mode for Uu. In addition, when the first vehicle declares an SL RLF with the second vehicle, the cell reselection (or vehicle reselection) associated with the connection re-establishment is performed because the first vehicle does not affect the recovery of the connection with the second vehicle. You may not need to do it.

11 illustrates a flowchart of a process of processing multi-sidelink communication in the C-V2X module 115 when the display device enters the traffic information mode according to an embodiment of the present invention. Here, the flow is from the perspective of the first vehicle.

In step 1110, the first vehicle performs or establishes sidelink unicast communication with the second vehicle. In step 1120, the first vehicle performs or establishes sidelink unicast communication with the third vehicle. In step 1130, the first vehicle detects a radio link failure associated with the second vehicle. In step 1140, the first vehicle performs a sidelink medium access control reset associated with the second vehicle in response to the radio link failure and does not perform a sidelink medium access control reset associated with the third vehicle.

In one embodiment, the sidelink medium access control reset associated with the second vehicle may include or consist of the first vehicle stopping all sidelink transmissions and/or receptions associated with the second vehicle. . Preferably or alternatively, resetting the sidelink medium access control associated with the second vehicle may not include stopping all sidelink transmissions and/or receptions associated with the third vehicle. A sidelink transmission or reception may include or consist of a physical sidelink shared channel, a physical sidelink control channel, and/or a physical sidelink feedback channel transmission or reception.

In one embodiment, the sidelink medium access control reset associated with the second vehicle may include or consist of the first vehicle considering one or more timers associated with the second vehicle to expire. Additionally or alternatively, the sidelink medium access control reset associated with the second vehicle may include or consist of the first vehicle not taking into account one or more timers associated with the third vehicle to expire. there is. The sidelink medium access control reset associated with the second vehicle may also include or consist of canceling all pending Scheduling Requests (SRs) and/or Buffer Status Reports (BSRs) associated with the second vehicle. Alternatively, the sidelink medium access control reset associated with the second vehicle may not include canceling all pending scheduling requests (SRs) and/or buffer status reports (BSRs) associated with the third vehicle.

In one embodiment, the sidelink medium access control reset associated with the second vehicle may include or consist of flushing all soft buffers associated with the sidelink unicast communication with the second vehicle. Preferably or alternatively, the sidelink medium access control reset associated with the second vehicle may not include flushing all soft buffers associated with the sidelink unicast communication with the third vehicle.

In one embodiment, sidelink medium access control reset may include releasing sidelink unicast resources associated with the second vehicle scheduled by the network or reserved by the first vehicle. In one embodiment, a first vehicle may maintain one sidelink medium access control entity for sidelink unicast communication with a second vehicle and for sidelink unicast communication with a third vehicle.

In this way, in one embodiment of the present invention, while the first vehicle performs a sidelink unicast operation with the second and third vehicles, for example, when a radio link failure associated with the second vehicle is detected, the second vehicle By performing only the sidelink wireless medium access control reset associated with the third vehicle and not performing the sidelink wireless medium access control reset associated with the third vehicle, the overhead is reduced in the C-V2X operation of the first vehicle, thereby saving through the pillar display. It is possible to provide delayed high-speed traffic information.

What has been described above is only one embodiment for implementing the vehicle pillar display system according to the present invention, and the present invention is not limited to the above-described embodiment, and the subject matter of the present invention as claimed in the claims below Anyone with ordinary knowledge in the field to which the present invention pertains without departing from the technical spirit of the present invention to the extent that various changes can be made.

Claims (5)

a first display provided on an interior side of an A-pillar of a first vehicle; a camera provided on an outdoor side surface of the first vehicle to obtain an image of a blind spot caused by the A-pillar; and a processor connected to the first display and controlling an operation of the first display, wherein the processor displays information of a predetermined type on the first display by performing a blind spot mode and a traffic information mode; , the processor,
Upon entering the blind spot mode, an image of the blind spot is displayed on the first display, and an object within the blind spot is detected through image processing on the image of the blind spot, and a risk index of the detected object is detected. Calculate, display an indicator for guiding the detected object on the first display, set the size, position, shape and color of the indicator based on the risk index,
When entering the traffic information mode, the first vehicle establishes a wireless link with the second vehicle and the third vehicle outside to perform sidelink unicast communication, but the sidelink with the second vehicle and the third vehicle If a failure of the radio link associated with the second vehicle is detected while performing unicast communication, a sidelink medium access control reset associated with the second vehicle is performed in response to the failure of the radio link and associated with the third vehicle. Sidelink medium access control reset is not performed, but at the same time, critical information in the traffic information is detected through data processing on the data of the traffic information, a risk index of the detected critical information is calculated, and the detected risk index is calculated. A vehicle pillar display system for displaying an indicator for guiding on the first display. According to claim 1,
The processor calculates the risk index based on the type, distance, direction, and speed of the detected object and critical information for the first vehicle. According to claim 1,
The sidelink medium access control reset associated with the second vehicle causes the first vehicle to stop transmitting and receiving sidelinks associated with the second vehicle. According to claim 3,
The sidelink transmission and / or reception includes a physical sidelink shared channel, a physical sidelink control channel, and a physical sidelink feedback channel transmission and reception. According to claim 1,
Wherein the sidelink medium access control reset associated with the second vehicle stops sidelink transmission and reception associated with the third vehicle.

Images