End-to-End Simulation of 5G mmWave Networks

This work has been submitted to IEEE Communication Surveys and Tutorials for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. 1 End-to-End Simulation of 5G mmWave Networks arXiv:1705.02882v3 [cs.NI] 5 Feb 2018 Marco Mezzavilla, Member, IEEE, Menglei Zhang, Michele Polese, Student Member, IEEE, Russell Ford, Sourjya Dutta, Student Member, IEEE, Sundeep Rangan, Fellow, IEEE, Michele Zorzi, Fellow, IEEE Abstract—Due to its potential for multi-gigabit and low latency wireless links, millimeter wave (mmWave) technology is expected to play a central role in 5th generation (5G) cellular systems. While there has been considerable progress in understanding the mmWave physical layer, innovations will be required at all layers of the protocol stack, in both the access and the core network. Discrete-event network simulation is essential for end-to-end, cross-layer research and development. This paper provides a tutorial on a recently developed full-stack mmWave module integrated into the widely used open-source ns–3 simulator. The module includes a number of detailed statistical channel models as well as the ability to incorporate real measurements or raytracing data. The Physical (PHY) and Medium Access Control (MAC) layers are modular and highly customizable, making it easy to integrate algorithms or compare Orthogonal Frequency Division Multiplexing (OFDM) numerologies, for example. The module is interfaced with the core network of the ns–3 Long Term Evolution (LTE) module for full-stack simulations of end-to-end connectivity, and advanced architectural features, such as dualconnectivity, are also available. To facilitate the understanding of the module, and verify its correct functioning, we provide several examples that show the performance of the custom mmWave stack as well as custom congestion control algorithms designed specifically for efficient utilization of the mmWave channel. Index Terms—mmWave, 5G, Cellular, Channel, Propagation, PHY, MAC, multi-connectivity, handover, simulation, ns–3 I. I NTRODUCTION Millimeter Wave (mmWave) communications are emerging as a central technology in 5G cellular wireless systems due to their potential to achieve the massive throughputs required by future networks [1]–[5]. In particular, mmWave has become a key focus of the 3rd Generation Partnership Project (3GPP) New Radio (NR) effort currently under development [6]. Due to the unique propagation characteristics of mmWave signals and the need to transmit in beams with much greater directionality than previously used in cellular systems, much of the recent work in mmWave communications has focused on channel modeling, beamforming and other physical layer procedures. However, the design of End-to-End (E2E) cellular systems that can fully exploit the high-throughput, low-latency capabilities of mmWave links will require innovations not only at the physical layer, but also across all layers of the communication protocol stack. For mmWave systems, E2E design and analysis are at a much earlier stage of research [7]–[9]. Marco Mezzavilla, Menglei Zhang, Russell Ford, Sourjya Dutta and Sundeep Rangan are with NYU WIRELESS, NYU Tandon School of Engineering, Brooklyn, NY, USA (email: {mezzavilla, russell.ford, menglei, sdutta, srangan}@nyu.edu). Michele Polese and Michele Zorzi are with the Department of Information Engineering, University of Padova, Padova, Italy (email: {polesemi, zorzi}@dei.unipd.it) Discrete-event network simulators are fundamental and widely used tools for developing new protocols and analyzing complex networks. Importantly, most network simulators enable full-stack simulation, meaning that they model all layers of the protocol stack as well as applications running over the network. This full-stack capability will play a critical role in the development of 5G mmWave systems. The unique characteristics of the underlying mmWave channel have wide ranging effects throughout the protocol stack. For example, the use of highly directional beams increases the complexity of a number of basic MAC-layer procedures such as synchronization, control signaling, cell search and initial access, which in turn affect delay and robustness [7]. MmWave signals are also highly susceptible to blockage [1], [10]– [12], which results in high variability of the channel quality. This erratic behavior complicates the design of rate adaptation algorithms and signaling procedures, requiring advanced solutions for multi-connectivity, fast handover and connection re-establishment [13]–[16]. New transport layer mechanisms may also be required in order to utilize the large capacity, when available, and to react promptly to rapid fading to avoid congestion [9], [17]–[19]. The need for ultra-low latency applications [1], [20], [21] may require solutions based on edge computing and distributed architectures that will determine a considerable departure from current cellular core network designs. To better capture these design challenges, this work presents a comprehensive tutorial on the open-source mmWave simulation tool developed by New York University and the University of Padova for LTE-like 5G mmWave cellular networks, which can be used to evaluate cross-layer and end-to-end performance. This mmWave simulation tool is developed as a new module within the widely used ns–3 network simulator [22]. ns–3 is an open-source platform, that currently implements a wide range of protocols in C++, making it useful for crosslayer design and analysis. The new mmWave module presented here is based on the architecture and design patterns of the LTE LENA module [23], [24] and implements all the necessary Service Access Points (SAPs) needed to leverage the robust suite of LTE/Evolved Packet Core (EPC) protocols provided by LENA. The code (publicly available at GitHub [25], along with examples and test configurations) is highly modular and customizable to facilitate researchers to design and test novel 5G protocols. The ns–3 mmWave module was first presented in [26], [27]. The 3GPP channel model implementation is introduced in [28], and the dual connectivity functionality is described in [14], [29]. This paper extends these works by presenting the ns–3 mmWave module from a single and organic point of view, and is intended as a tutorial for any researcher that plans to use the simulator. In addition to its comprehensive description 2 and discussion, we provide in Sec. X a brief guide on how to set up a simulation, followed by a number of representative examples. The rest of the paper is organized as follows. In Section II, we provide some background on mmWave cellular communications and highlight some key problems at the higher protocol layers to motivate the need for a robust full-stack simulator. We also describe the main challenges related to the design of a mmWave cellular networks simulator. Then, in Section III, we introduce ns-3, the network simulator on which our mmWave module is developed, and in Section IV we present the overall architecture of the mmWave module. We then take a closer look at each component, starting with the suite of Multiple Input, Multiple Output (MIMO) channel models in Section V. In addition to an implementation of the latest 3GPP “above 6 GHz” model [30], several custom channel models are also provided. Section VI discusses the features of the OFDMbased PHY layer, which has a customizable frame structure for evaluating different numerologies and parameters. In Section VII, we provide a MAC-layer discussion that includes our proposed flexible/variable Transmission Time Interval (TTI) Time Division Multiple Access (TDMA) MAC scheme, which is supported by several scheduler implementations. Section VIII presents the enhancements that we introduced to the LTE Radio Link Control (RLC) layer. The dualconnectivity architecture is reported in Sec. IX. In Section X, we show how the module can be used for cross-layer evaluation of multi-user cellular networks through a number of representative examples, and provide pointers to a large set of general results that have been obtained so far with this module. The integration of native Linux Transmission Control Protocol (TCP) implementations, performed through the ns– 3 Direct Code Execution (DCE) framework, is discussed in Section XI. In Section XII, we provide details on our future plans for the simulator and suggest possible research topics in which it could be used. Finally, we conclude this tutorial paper in Section XIII. II. M ILLIMETER WAVE C ELLULAR BACKGROUND Millimeter wave communications is an advanced PHY layer technology, which has recently come to the forefront of research interest and may be able to rise to the challenge of providing high-rate mobile broadband services, in addition to offering opportunities for reducing over-the-air latency for 5G New Radio. MmWave makes use of the radio frequency spectrum roughly between 30 and 300 GHz, even though the research challenges extend also to lower frequencies (i.e., above 6 GHz) which are considered for 3GPP NR. Systems that can operate in these bands are attractive because of the large quantities of available spectrum at these higher frequency ranges and the spatial degrees of freedom afforded by very high-dimensional antenna arrays, which are possible thanks to the smaller size of antenna elements at higher frequencies. Most current commercial wireless systems operate below 6 GHz, where lower frequencies allow for long-range propagation and low penetration loss (i.e., attenuation by walls and other obstacles), which makes them well-suited for radio communications. As a result, the sub-6 GHz spectrum has become heavily congested and individual bands are generally not available in contiguous chunks wider than 200 MHz. However, large swaths of spectrum are available at the higher mmWave frequencies, which offer the possibility of very wide bandwidths of over 1 GHz, in some cases. Although the mmWave bands are already used by a variety of commercial applications, such as satellite and pointto-point backhaul communications, until recently they were considered impractical for mobile access networks due to the poor isotropic propagation and the vulnerability to shadowing at these higher frequencies. However, it has now been shown that the limitations of the mmWave channel can be overcome with the help of high-gain, directional antennas so that this vast region of spectrum can now be exploited to provide an order of magnitude or more increase in throughput for mobile devices [3], [31]. Directional smart antennas are the major technology enabler that will make it possible for mmWave devices to overcome the poor propagation effects and unlock this high-frequency spectrum. The theoretical free space path loss (as governed by Friis’ Equation) is proportional to the square of the frequency, resulting in the magnitude of received power for a mmWave signal being over 30 dB (1000x) less than conventional cellular systems at equivalent distances between transmitter and receiver [32]. Multi-element antenna arrays and MIMO beamforming techniques offer a means of compensating for this high attenuation. With millimeter waves, the antenna size and spacing shrinks to be on the order of millimeters, making it possible to pack hundreds of elements onto a small cell base station and dozens onto a handheld device. Smaller antenna size also allows for multiple arrays to be integrated onto mobile devices to provide diversity and maintain connectivity even if the signal from one array is blocked (for instance, by the user’s hand) [3]. It is clear that mmWave will be highly disruptive in the wireless space thanks to the prospect of massive bandwidth and high-gain antennas. Nevertheless, before mmWave technology can be effectively realized in 5G networks, there are numerous challenges to be addressed, not only at the physical layer, but also at higher layers of the radio stack, namely: • Adaptive beamforming and beam tracking: The requirement of directionality introduces new challenges for supporting mobility in mmWave networks. The transmitter and receiver must continually track the channel as the mobile user moves in order to align their antenna arrays to achieve the maximum directional gain. MmWave signals are also known to be particularly susceptible to shadowing and can be completely blocked by many materials such as brick, tinted glass and even the human body [33], [34]. Fortunately, recent field measurements have demonstrated that reflected power can be sufficient for Non Line of Sight (NLOS) communications to be possible. A blocked link may therefore be able to recover by steering the beam from the primary Line of Sight (LOS) path to an alternate NLOS path. The UE and BS 3 • • must then jointly initiate a procedure to search for and select another path to reestablish the link. Directional synchronization and broadcast channels: Directionality also complicates the design of many control channels and procedures. The cell discovery and initial access procedures, where the UE must search for nearby base stations to which it can attach, will require an innovative approach to be handled efficiently. Traditional cells periodically broadcast out synchronization signals (known as the Primary Synchronization Signal (PSS) in LTE systems) omnidirectionally, which are received by all devices within the cell’s coverage range and used to initially connect to the cell. If a 5G mmWave evolved Node Base (eNB) were to broadcast the PSS in an omnidirectional antenna pattern, the signal would not benefit from the directional gain and might not have adequate range to be detected by many UEs. Therefore, the eNB and User Equipment (UE) must perform an angular search in order for users to detect the PSS and hone in on the optimal Transmitter (TX)/Receiver (RX) beamforming angles [35]. A similar problem also arises for other control signals, such as the Downlink Control Information (DCI) assignments, which indicate the resources assigned to each user for Downlink (DL)/Uplink (UL) transmission in the subframe or slot. Issues for the MAC, Network and Transport Layers: The rapid channel dynamics and vulnerability of mmWave links to shadowing will require frequent, near instantaneous handovers between neighboring 5G or 4G cells. Dual-connectivity, where mobiles are continuously connected to both the 5G and legacy 4G network, may therefore be essential to recover from an abrupt failure of the primary 5G link [14], [29]. Additionally, at the transport layer, the congestion control and avoidance mechanisms provided by TCP must be able to quickly adapt to sudden fluctuations in capacity to maximally utilize the link bandwidth while avoiding overwhelming the network by sending too many packets, resulting in congestion and affecting other flows in the network. Current versions of TCP may not be optimized for mmWave channel dynamics [9], [18], so new algorithms may be called for to provide high rates for E2E sessions [17], [36], [37]. Potentials and Challenges of System-level Simulations of mmWave Networks An End-to-End network simulator for mmWave cellular networks is an invaluable tool that can help address these challenges by allowing the evaluation of the impact of the channel and of the PHY layer technology on the whole protocol stack. However, given the characteristics of mmWave communications described in the previous paragraphs, in order to have accurate results it is of paramount importance to model in detail the behavior of the different elements that interact in a cellular system. In the following paragraphs we will introduce and discuss some of the most important elements that need to be considered when designing a mmWave cellular systems simulation, and show how they depend on one another: • • • The channel model is the fundamental component of every wireless simulation. Given the harsh propagation conditions at mmWaves, the channel is one the main elements that affect the end-to-end network performance. Firstly, it has to account for the different LOS and NLOS states for the propagation loss and the fading [30]. Moreover, beamforming should be applied on top of the channel to accurately model directional transmissions, which have an impact on the link budget, the interference and the control procedures. Finally, the Doppler effect is particularly relevant at mmWave frequencies, especially with high mobility [3]. An important consideration related to the channel model is the trade off between the accuracy and the computational complexity: very accurate models that require the computation of the complete channel matrix are usually also computationally intensive [28]. The users’ mobility and the network deployment have an important impact on the communication performance, intertwined with that of the channel model. Given the small range of the mmWave cells, the deployment will be dense and will require frequent access point updates, which should be simulated for a realistic performance assessment [38]. Moreover, mobility affects the performance of beam tracking algorithms [15]. Therefore, when simulating a mmWave network it is important to use realistic deployments and mobility models. The level of detail when modeling the protocol stack of the mmWave links and of the end devices is another important parameter for network simulations. A simplified model of the protocol stack can be accurate enough for studies that do not involve complex interplay between different layers, but cannot capture the behaviors that emerge from the interaction of the different layer, and therefore could not generate realistic results for end-toend performance evaluations. For example, at mmWave frequencies, it has been shown that the channel behavior has an impact on the TCP performance [9], [18], therefore a model of the TCP/IP stack is needed when analyzing the data rate that an application can reach in an end-toend mmWave network. To the best of our knowledge, there are no open source simulators capable of thoroughly modeling the mmWave channel along with the cellular network protocol stack as well as other protocols (e.g., the TCP/IP stack), realistic scenarios and mobility. There exists an ns–3-based simulator for IEEE 802.11ad in the 60 GHz band [39], [40], which however cannot be used to simulate cellular and 3GPP-like scenarios. Other papers [41]–[44] report results from system level simulations, with custom (often MATLAB-based and not publicly available) simulators which are not able to capture the complexity of the whole stack with a very high level of detail. This is what motivated us to develop an open source cellular mmWave module for the ns–3 simulator, which we will describe in the following sections. 4 Figure 1: Class diagram of the end-to-end mmWave module. III. NS –3 The ns–3 discrete-event network simulator [22], [45] is a very powerful tool available to communication and networking researchers for developing new protocols and analyzing complex systems. It is the successor to ns–2, a tried and tested tool that has been in use by the networking community for over a decade in the design and validation of network protocols. ns–3 is open source, and can be downloaded from the website of the project1 . An active community of researchers from both industry and academia has enriched the basic core of the simulator with several modules, and ns–3 can be now used to simulate a wide variety of wireless and wired networks, protocols and algorithms. There is a complete documentation2 on the models in the ns–3 website, in terms of both the design of the models and what a user can do with the models. Moreover, a complete tutorial on how to install ns–3, set up ns–3 scenarios and topologies, handle the collection of statistics and log useful messages is provided in the documentation3 . The tutorial is a good starting point for a researcher who approaches ns–3 for the first time. The ns–3 simulator is organized into multiple folders. The src folder provides a collection of C++ classes, which implement a wide range of modular simulation models and network protocols. The different modules can be aggregated and instantiated to build diverse simulated network scenarios, making ns-3 especially useful for cross-layer design and analysis. The modularity and use of object-oriented design patterns also allows for new algorithms to easily be incorporated into the network stack and experimented with. Each module is itself organized into multiple subfolders, which contain the documentation and the source code of the model itself, the helpers, the examples and the tests. The helpers associated with each model have a very important role. They are classes which hide to the final user the complexity involved in setting up a complete scenario, for example by automatically assigning IP addresses, or connecting the different classes of a protocol stack. The build folder contains the binaries of the simulator. Finally, the scratch folder is a special folder in which scripts with examples and scenarios can be built onthe-fly. Besides the core module, which provides the basic structure of the simulator, there are modules for networking protocols (e.g., the TCP/IP stack protocols [46]), wireless protocols (LTE [23], Wi-Fi [47], WiMAX [48]), routing algorithms [49], mobility, embedding obstacles and buildings in the simulation scenarios, and data collection. All the modules are listed in the model library4 . In the following sections, we will describe in detail the mmWave module for ns–3, following the same approach which is used for the other ns–3 modules. We will first describe the model in terms of implementation of the different components of a mmWave cellular network and protocol stack, and then the examples and scenarios that can be simulated with it and how they can be set up. IV. MM WAVE M ODULE OVERVIEW The ns–3 mmWave module is designed to perform end-to-end simulations of 3GPP-style cellular networks. The architecture 1 http://www.nsnam.org 2 https://www.nsnam.org/documentation/ 3 https://www.nsnam.org/docs/tutorial/html/ 4 https://www.nsnam.org/docs/release/3.27/models/html/index.html for ns–3 version 3.27 5 ns3::MmWave MmWaveEnbNetDevice MmWaveNetDevice +int m_cellId +int m_earfcn MmWaveUeNetDevice +Int m_mtu +int m_imsi +int m_earfcn +Receive(Packet): void +DoSend(Packet): bool +DoSend(Packet): bool LteEnbRrc LteUeRrc +ConfigureCell(...): void +SynchronizeToStrongestCell(): void +SendMeasurementReport(...): void LtePdcp LTE +int m_rnti +int m_lcid +DoTransmitPdcpSdu(Packet): void +DoReceivePdcpSdu(Packet): void LteRlc +int m_rnti +int m_lcid LteRlcSm +DoTransmitPdcpPdu(Packet): void +DoReceivePdu(Packet): void +DoNotifyTxOpportunity(...): void LteRlcTm LteRlcUm LteRlcAm MmWaveUeMac MmWaveEnbMac +DlHarqInfo Harq +DlCqiInfo cqi MmWaveMac +MacPduInfo +DoTransmitPdu(params): void +DoRxPdu(Packet): void MMWaveAmc +int m_rnti +UlHarqInfo_t Harq +DoTransmitPdu(params): void +DoRxPdu(Packet): void MmWavePhy MmWaveEnbPhy MmWaveUePhy +double m_txPower +GetMcsFromCqi(cqi): mcs +StartSubFrame(): void +EndSubFrame(): void +GenerateDlCqiReport(SINR): void +SubframeIndication(...): void MmWaveHarqPhy MmWaveMacScheduler +GetHarqProcessInfoDl(...): harq +GetHarqProcessInfoUl(...): harq +DoSchedDlTriggerReq(...): bool +DoSchedUlTriggerReq(...): bool MmWaveSpectrumPhy MmWaveMiErrorModel +StartTxDataFrame(...): bool +StartTxCtrlFrame(...): bool +StartRx(...): void +GetTbDecodificationStats(...): TbStats_t PropagationLossModel +DoCalcRxPower(...): double MultiModelSpectrumchannel +StartTx(params): void +StartRx(params): void MmWave3gppPropagationLossModel +string m_channelConditions +string m_scenario +double m_shadowing MmWaveInterference +AddSinrChunkProcessor(...): void SpectrumPropagationLossModel +DoCalcRxPowerSpectralDensity(...): Ptr<SpectrumValue> MmWave3gppChannelModel +Params3gpp_map m_channelMap +CalBeamformingGain(...): Ptr<SpectrumValue> Figure 2: Unified Modeling Language (UML) class diagram for the end-to-end mmWave module. 6 builds upon the ns–3 LTE module (LENA) [23], [24]. It leverages the detailed implementation of LTE/EPC protocols, and implements custom PHY and MAC layers. Additionally, it is possible to connect the module to a patched version of Direct Code Execution [50], a tool that allows the Linux stack TCP/IP implementation to run as the TCP/IP stack of ns–3 nodes, as well as to execute POSIX socket-based applications (i.e., wget, iPerf, etc). Figure 1 depicts the high-level composition of the MmWaveEnbNetDevice and MmWaveUeNetDevice classes, which represent the mmWave eNB5 and UE radio stacks, respectively, along with a perspective on the end-toend structure of the simulator. A more detailed UML class diagram is provided in Figure 2. The ns–3 mmWave module also includes a McUeNetDevice, which is a NetDevice with a dual stack (LTE and mmWave), i.e., a device capable of connecting to both technologies. More details will be given in Section IX. The MmWaveEnbMac and MmWaveUeMac MAC layer classes implement the LTE module Service Access Point (SAP) provider and user interfaces, which enable the interoperation with the LTE RLC layer. Support for RLC Transparent Mode (TM), Saturation Mode (SM), Unacknowledged Mode (UM), Acknowledged Mode (AM) is built into the MAC and scheduler classes (i.e., MmWaveMacScheduler and derived classes). The MAC scheduler also implements a SAP for configuration at the LTE Radio Resource Control (RRC) layer (LteEnbRrc). Hence, every component required to establish Evolved Packet Core (EPC) connectivity is available. The MmWavePhy classes handle directional transmission and reception of the DL and UL data and control channels based on control messages from the MAC layer. Similar to the LTE module, each PHY instance communicates over the channel (i.e., SpectrumChannel) via an instance of the MmWaveSpectrumPhy class, which is shared for both the DL and the UL (since our design of the mmWave PHY layer is based on Time Division Duplexing (TDD), as detailed in Section VI-A). Instances of MmWaveSpectrumPhy encapsulate all PHY-layer models: interference calculation (MmWaveInterference), Signal to Interference plus Noise Ratio (SINR) calculation (MmWaveSinrChunkProcessor), the Mutual Information (MI)-based error model (MmWaveMiErrorModel), which computes the packet error probability, as well as the Hybrid Automatic Repeat reQuest (HARQ) PHY-layer entity (MmWaveHarqPhy) to perform soft combining. Since the structure, high-level functions and naming scheme of each class closely follow the LTE LENA module, the reader is also referred to the LENA project documentation for more information [51]. V. C HANNEL AND MIMO M ODELING A. Channel Models The ns–3 mmWave module allows the user to choose among different channel models, which provide a trade-off between 5 Recently, 3GPP has proposed the term next generation Node Base (gNB) for the 5G NR base station. computational complexity, flexibility and accuracy of the results. The most flexible and detailed channel model is the one described in detail in [28], which is based on the official 3GPP channel model for the 6-100 GHz frequency band [30]. It accounts also for spatial consistency of mobility-based simulations and provides a random blockage model, as well as the modeling of outdoor to indoor communications. The second model is based on traces from measurements or thirdparty ray-tracing software. This makes the channel model detailed and realistic, but constrains the simulation to limited measurements/ray-tracing routes. The third is the statistical channel model introduced in [26] and based on MATLAB traces, which makes the computation less demanding, but is available only for the 28 and 73 GHz frequencies. In the following paragraphs we will provide architectural details of all the available channel models. 1) 3GPP Statistical Channel Model: The 3GPP model for the 6-100 GHz band, described in [30], is applicable for bandwidths up to 10% of the carrier frequency and accounts for mobility. It provides several optional features that can be plugged into the basic model, in order to simulate, for example, spatial consistency (i.e., the radio environment conditions of close-by users are correlated) and random blockage. The model defines different scenarios, which describe different possible cellular network deployments: urban (with macrocells and microcells), rural and indoor. Pathloss: The pathloss of the propagation channel is implemented in the MmWave3gppPropagationLossModel class. The model provides a statistical LOS/NLOS condition characterization, as well as pathloss computation considering outdoor to indoor penetration loss, as described in [30, Sec. 7.4]. The MmWave3gppBuildingPropagationLossModel class, instead, determines the LOS condition according to the reciprocal position of the UE and the eNB and to the presence of buildings or obstacles in the scenario. These classes also optionally apply an additional shadowing component to the pathloss. For a moving UE, the shadowing is correlated in space. Given the distance ∆d2D > 0 on the horizontal plane from the last position in which the shadowing was computed, the exponential correlation parameter is computed as R(∆d2D ) = e−∆d2D /dcor , where dcor is the correlation distance. In our implementation, pathloss and shadowing (if enabled) are updated at every transmission. Figure 3 shows the pathloss in dB for the 3D distance from the smallest value supported in each scenario to 103 m for outdoor and 102 m for indoor. Small-scale fading: The small-scale fading model is implemented in the MmWave3gppChannel class, and follows the step by step approach of [30, Sec. 7.5]. Small-scale fading is the bottleneck of this channel model implementation, since it is very detailed and computationally demanding. The fading is generated following the 3D statistical spatial approach originally proposed in [52]. The channel is described by a channel matrix H(t, f ), where t is the time and f is the frequency, of size U × S, where U and S are the number of antennas at the receiver and the transmitter. Each entry depends on N ≤ 20 different multipath components, called 7 Figure 3: Typical realization of the 3GPP pathloss model [30] using spatial consistency for 3 outdoor scenarios (RMa stands for Rural Macro, UMa for Urban Macro and UMi for Urban Micro) and 2 indoor scenarios (InH stands for Indoor, either in the office or in a shopping mall). The channel can be LOS, NLOS with or without the optional equations for the propagation loss and outdoor to indoor (O2I). clusters, which have different delays and received powers, according to an exponential power delay profile. A cluster is itself a combination of M = 20 rays, each with a slightly different arrival and departure angle in the vertical and horizontal planes. The MmWave3gppChannel class has a method that generates the channel matrix, and stores the coefficient for each transmit element s, receive element u and cluster n in a data structure, that can be accessed by other methods in order to update the channel matrix or compute the beamforming gain. We introduced some assumptions with respect to the 3GPP model, in order to decrease the computational overhead introduced by the high level of detail of the channel. For example, we consider only antennas with vertical polarization, and the speed-dependent Doppler effect is not computed for each ray, but only for the central angle of each cluster. Further details on this implementation are given in [28]. Spatial consistency: The basic channel model described in the previous paragraphs can be used for drop-based simulations with limited mobility, i.e., for UEs that move in an area in which the channel is very correlated and the fading parameters do not change. However, for simulations in which mobility is an important factor, the spatial consistency of the channel throughout the path on which the UE moves can be simulated by enabling this option in the MmWave3gppChannel class. In the current implementation, we support spatial consistency with Procedure A of [30, Sec 7.6.3.2] for both LOS and NLOS communications. It is possible to set the period of update tP ER , and every tP ER the cluster delays, powers and departure and arrival angles are updated with a transformation that accounts for the speed of the UE and for the distance traveled on the horizontal plane. Blockage: This optional feature can be used to model the attenuation in certain clusters, according to their angle of arrival. The attenuation can be caused by the human body that holds the UE, or by external elements such as for example cars, other human bodies, trees. The blockage model is implemented in the MmWave3gppChannel class and can be optionally activated. In our implementation we consider blockage model A, which only distinguishes between selfblocking and non-self-blocking, and is generic and computationally efficient [30]. In particular, this model randomly generates K + 1 blocking regions, one for self-blocking, with different parameters according to the orientation of the UE (i.e., portrait or landscape mode), and K for non-self-blocking. The attenuation is of 30 dB for self-blocking, and dependent on the scenario and on horizontal and vertical arrival angles for non-self-blocking. Moreover, the blocking of a certain cluster is correlated in both space and time, according to the UE mobility, the blocker speed and the simulation scenario. Notice that, if both the blockage and the spatial consistency options are used, then the update of the channel with both features is synchronized, i.e., the cluster blockage is updated before the channel coefficients are recomputed with the spatial consistency procedure. 2) Ray-tracing or Measurement Trace Model: MmWaveChannelRaytracing uses software-generated or measurement traces to model the channel in ns–3, for pathloss and fading. The trace samples need to contain the number of paths and the propagation loss, delay, angle of arrival and angle of departure for each path. The following trace files have been tested in our channel and are available in mmwave/model/Raytracing/. Ray-tracing: Any ray-tracing software (e.g., WinProp [53]) can be used to generate the channel information for a specific route. This means that the simulation scenario must be chosen a priori, and cannot be random since it has to be given as input to the ray-tracing software. An example of ray-tracing route6 is shown in Figure 4b. QuaDRiGa: The Quasi Deterministic Radio Channel Generator model [55], supports consistent user mobility and massive MIMO at several frequencies (10, 28, 43, 60, 82 GHz). It also adds some time evolution characterization on top of the statistical channel to capture user mobility, which makes it suitable for system level simulations. 3) NYU Statistical Model: This channel model is based on the approach described in [31] and implemented in our previous work [26]. A MATLAB implementation of the same channel model is also available in [56]– [58]. It provides two pathloss models, which differ in how they capture the LOS/NLOS condition. The first, MmWavePropagationLossModel, is based on a statistical characterization of the LOS state, while the second, BuildingsObstaclePropagationLossModel, lever6 The ray tracing data was provided by the Communication Systems and Networks Group, University of Bristol, UK [9], [54]. 8 40 Blocked SINR (dB) SINR (dB) 40 20 Long-term cov matrix (update) Long-term cov matrix (update) + blockage Long-term cov matrix (without update) 0 Simulated channel Soft LOS/NLOS transition 40 Beam search (update) Beam search (update) + blockage Beam search (without update) 0 SINR (dB) 20 Blocked 0 20 40 60 Time (s) LOS 50 0 LOS NLOS -50 80 (a) 3GPP statistical channel model 100 SINR (dB) 40 SINR (dB) 0 -20 -20 -20 20 -100 0 50 100 NLOS OUTAGE 150 200 20 0 Simulated channel Blockage overlaid 250 Position along route (m) -20 0 (b) Ray-tracing Trace Model 2 4 6 Time (s) 8 10 (c) NYU Statistical Model Figure 4: Example of average SINR plots for the three channel models. ages the ns–3 buildings module in order to decide whether there is an obstacle between the UE and the eNB or not. In particular, it is possible to deploy – deterministically or randomly – objects of different sizes to mimic humans, cars, and buildings. A virtual line is drawn between the transmitter and the receiver: If this line intersects any object, the state is NLOS, otherwise it is LOS. In both classes, once the channel state is selected, the propagation loss is computed as in [31]. Channel configuration: Since the channel matrices and optimal beamforming vectors do not depend on the distance between the UE and the eNB, they are pre-generated in MATLAB to reduce the computational overhead in ns–3. At the beginning of each simulation we load 100 instances of the spatial signature matrices, along with the beamforming vectors. Moreover, in order to simulate realistic channels with large-scale fading, the channel matrices are updated periodically and independently (block fading). Currently, no results are available for modeling how the large-scale statistics of the mmWave channel change over time for a mobile user, thus it should be noted that the accuracy of this method is not verified at this time. The matrix update can take place at some fixed intervals, specified by the LongTermUpdatePeriod attribute of the MmWaveBeamforming class. The small-scale fading, instead, is calculated at every transmission, where we obtain the speed of the user directly from the mobility model. The remaining parameters that depend on the environment are assumed to be constant over the entire simulation time. Semi-empirical feature: Finally, as shown in Fig. 4c, the soft transition between LOS/NLOS conditions can be modeled in a “semi-empirical” fashion, meaning that we overlay the statistical channel with blockage measurements performed in our lab [59]: Waving a hand in front of the receiver (hand blockage), walking between the transmitter and the receiver (human blockage), and placing a metal plate between the transmitter and the receiver to emulate an obstacle, like a car or a building. B. Beamforming Gain For the long-term statistical channel model, the beamforming vectors are directly loaded from MATLAB generated files. For the other channel models, two methods are implemented to compute beamforming vectors, i.e., the long-term covariance matrix method and the beam search method. Currently, the only available beamforming architecture for data transmission is analog, meaning that devices can transmit or receive in only one direction at a time. As part of our future work, we plan to integrate hybrid and digital transceiver designs. In the long-term covariance matrix method, we assume that the transmitter estimates the spatial correlation matrix Qtx = E[H† (t, f )H(t, f )], where the expectation is taken over the frequencies and some interval of time. An analogous operation is done for the receiver. In practice, the TX and RX would estimate the spatial covariance matrix from reference or synchronization signals and beam scanning. Estimation of this covariance matrix is discussed in [60]. We do not, however, model the covariance estimation directly; instead we simply assume that the TX and RX know the correct long-term channel with some configurable delay. Beamforming vectors can then be computed from the maximal eigenvectors of the covariance matrices [61]. A computationally simple procedure is to use the power method [62]. The algorithm selects a random initial beamforming vector and iteratively multiplies it with the spatial correlation matrix Qtx , normalizing the results at each iteration. Finally, the output will converge to the correct eigenvector. The computation for the receiver is done in the same way, starting from Qrx = E[H(t, f )H† (t, f )]. In the beam search method, we assume that the TX and the RX scan a discrete number of beams from a pre-designed codebook [63]. Codebook design is discussed in detail in [64]. The beamforming vector is selected as the one with the highest power, possibly with some time-averaging. C. Interference MmWave systems may be interference- or power-limited. Albeit potentially less significant for directional mmWave signals, which are generally assumed to be power-limited, there are still some cases where interference is non-negligible [65]. We report in Figure 5 some representative results obtained in [66], where, by plotting the Interference to Noise Ratio (INR), we show that the majority of the links are still interference- 9 1 0.8 0.7 0.7 Empirical CDF Empirical CDF 28 GHz 73 GHz 0.9 0.8 0.6 0.5 INTERFERENCE LIMITED 0.4 0.3 NOISE LIMITED 0.6 0.5 0.4 0.3 0.2 0.2 0.1 0 -80 be decoded or not. Because each TB can be composed of multiple Code Blocks (CBs), whose size depends on the channel capacity, the BLER can be formulated as follows: !# " γi − bCSIZE ,M CS 1 , (3) 1 − erf √ CBLER,i (γi ) = 2 2cCSIZE ,M CS 1 NOISE LIMITED 0.9 -60 -40 -20 0 20 40 INTERFERENCE LIMITED 0.1 28 GHz 73 GHz 0 -80 60 -60 -40 INR [dB] -20 0 20 40 60 INR [dB] (a) Empirical CDF of the INR for (b) Empirical CDF of the INR for λU E = 300 UEs/km2 and λBS = 30 λU E = 1200 UEs/km2 and BSs/km2 . λBS = 120 BSs/km2 . Figure 5: INR trends at different user and base station density levels [66]. limited for some dense topologies. Additionally, although intra-cell interference (i.e., from devices of the same cell) can be neglected in TDMA or Frequency Division Multiple Access (FDMA) operation, it does need to be explicitly calculated in the case of Spatial Division Multiple Access (SDMA)/MultiUser MIMO, where users are multiplexed in the spatial dimension but operate in the same time-frequency resources. Therefore, we propose an interference computation scheme that takes into account the beamforming vectors associated with each link. As an example, we compute the SINR between nodes eN B 1 and U E 1 in the presence of an interferer, eN B 2 . To do so, we first need to obtain the beamforming gains associated with both the desired and interfering signals, i.e., † G11 = |wrx H(t, f )11 wtx11 |2 , 11 (1) † G21 = |wrx H(t, f )21 wtx22 |2 . 11 The SINR is then computed as: SIN R11 = PT x,11 P L11 G11 PT x,22 P L21 G21 + BW × N0 , (2) where PT x,ii is the transmit power of eN B i , P Lij is the pathloss between eN B i and U E j , and BW × N0 is the thermal noise. D. Error Model The mmWave module exploits the error model introduced in the ns-3 LTE LENA project, which follows a link abstraction technique for simulating Transport Block (TB) errors in the downlink of an LTE system. In a nutshell, the model described in [67] defines an accurate and lightweight procedure for the computation of the residual errors after PHY layer processing. This is achieved by exploiting: • Mutual Information-based multi-carrier compression metrics to derive a unique SINR value of the channel, known as effective SINR, which is represented as γi in Eq. 3, and • Link-Level performance curves obtained with a MATLAB bit-level LTE PHY simulator [68], which have been used to match a mathematical approximation of the Block Error Rate (BLER), as reported in Eq. (3). The ultimate goal is to let the receiver derive the error probability of each TB to determine whether the packet can where γi corresponds to the mean mutual information per coded bit of code block i, as explained earlier, and bCSIZE ,M CS and cCSIZE ,M CS represent the mean and standard deviation of the Gaussian cumulative distribution, respectively, which have been obtained from the link level performance curves mentioned above. Finally, the TB block error rate is given by: TBLER = 1 − C Y i=1 (1 − CBLER,i (γi )). (4) E. Examples An example of SINR plots for the three channel models is presented in Figure 4. An example related to the setup of the channel model can be found in the examples folder of the mmWave module, in the file mmwave-3gpp-channel-example.cc. Figure 4a shows an example of a rural scenario with an eNB at coordinates (0, 0) m and at the height of 35 m, and a UE in position (100, 0) m, at the height of 1.5 m and moving at 1 m/s along the y axis, maintaining LOS connectivity. The channel is updated consistently every 100 ms. The top figure shows the SINR when the Beamforming (BF) vector is updated with the long-term covariance matrix method, while in the bottom one it is updated with the beam search method. Notice that the current implementation of the beam search method uses a fixed elevation angle of 90 degrees and sweeps only the horizontal plane. Therefore, the beam search method cannot align with the LOS cluster and the power is reduced by 20 dB. Moreover, after enabling the blockage model, the SINR achieved by the long-term covariance matrix method dropped by 20 dB when the LOS cluster was blocked. However, the beam search method experienced less blockage impact, as it did not align with the LOS cluster. In the other case, without update, the BF vector is computed at t = 0 s but never updated, and this causes the SINR to drop as the UE moves. Comparing the blue and black curves, it is possible to observe that for the first 20 s the performance with and without BF update is similar, because of the consistency of the channel and of the low mobility of the UE, but after t = 20 s the SINR without update degrades by nearly 30 dB. The last observation is that the long-term covariance matrix method finds the optimal BF vector whenever the channel is changed, therefore the SINR is very stable. On the other hand, the beam search method shows an SINR drop after 20 s even with update, because when the UE moves both the UE and the eNB are unable to optimally adapt the BF vector and just select one of the available sectors. Figure 4b plots the average SINR of a ray tracing channel indicating both LOS intervals and NLOS channel states. The ray tracing data contains 5000 samples along a 500 meter route. The SINR has a sudden change when the channel state 10 In this section, we discuss the key features of the mmWave PHY layer. Specifically, we have implemented a TDD frame and subframe structure, which has similarities to TDD-LTE, but allows for more flexible allocation and placement of control and data channels within the subframe and is suitable for the variable TTI MAC scheme described in Section VII. Moreover, we implemented an error model and HARQ model based on those in LENA, but compatible with our custom mmWave PHY and numerology (for instance, they support larger TB and codeword sizes as well as multi-process stopand-wait HARQ for both DL and UL). A. Frame Structure It is widely contended that 5G mmWave systems will target Time Division Duplex (TDD) operation because it offers improved utilization of wider bandwidths and the opportunity to take advantage of channel reciprocity for channel estimation [3], [69]–[72]. In addition, shorter symbol periods and/or slot lengths have been proposed in order to reduce radio link latency [73]–[75]. The ns–3 mmWave module therefore implements a TDD frame structure which is designed to be configurable and supports short slots in the hope that it will be useful for evaluating different potential designs and numerologies. These parameters, shown in Table I, are accessible through the attributes of the common MmwavePhyMacCommon class, which stores all user-defined configuration parameters used by the PHY and MAC classes. Examples related to the setup of the PHY layer parameters can be found in the mmwave-tdma.cc and mmwave-epc-tdma.cc files. The frame and subframe structures share some similarities with LTE in that each frame is subdivided into a number of subframes of fixed length [76]. However, in this case, the user is allowed to specify the subframe length in multiples of OFDM symbols7 . Within each subframe, a variable number of 7 Though many waveforms are being considered for 5G systems, OFDM is still viewed as a possible candidate. In [71], [77], Verizon and the consortium led by Korea Telecom propose a frame structure and OFDM numerology. However, this is still under debate in 3GPP [78]. We naturally chose to adopt OFDM, at least initially, for the mmWave module, which allows us to leverage the existing PHY models derived for OFDM from the LTE LENA module. As soon as the 3GPP NR will be standardized, the protocol stack in our module can be adapted to the updated parameters. Subframe 10 1 Frame, 1 ms DL DL/UL CTRL DATA Sym 23 Sym 24 DL/UL UL DATA CTRL 1 Subframe, 100 µs 1 Sub-band Sym 2 (48 sub carriers = 13.89 MHz 1 Subframe = 24 OFDM symbols Sym 1 1 OFDM Symbol Total Bandwidth VI. P HYSICAL L AYER 1 Frame = Subframe Subframe 2 10 Subframes 1 (72 Sub-bands = 1 GHz) switches. We note that the SINR curve within LOS is relatively stable, whereas more random variations are introduced for NLOS. Finally, Figure 4c shows the average SINR trace generated with the NYU channel model [31] in two cases, a walking user blocked by a building (top) or by other humans (bottom). The main difference is that, with buildings, the link capacity drops rapidly and the blocking interval lasts seconds; on the other hand, with humans, the channel deteriorates slowly and the blockage lasts only for a short interval. From the top figure, we can observe that with soft LOS/NLOS transition enabled, the SINR curve changes less suddenly when the channel condition switches. In the bottom graph, three human blockage events, at 1, 4 and 7 seconds, are added on top of the statistical channel. 4.16 µs Figure 6: Proposed mmWave frame structure. symbols can be assigned by the MAC scheduler and designated for either control or data channel transmission. The MAC entity therefore has full control over multiplexing of physical channels within the subframe, as discussed in Section VII. Furthermore, each variable-length time-domain data slot can be allocated by the scheduler to different users for either the uplink or the downlink. Figure 6 shows an example of frame structure with the numerology taken from our proposed design in [74]. Each frame of length 1 ms is split in time into 10 subframes, each of duration 100 µs, representing 24 symbols of approximately 4.16 µs. In this particular scheme, the downlink and uplink control channels are always fixed in the first and last symbol of the subframe, respectively. A switching guard period of one symbol period is introduced each time the direction changes from UL to DL. In the frequency domain, the entire bandwidth of 1 GHz is divided into 72 subbands of width 13.89 MHz, each composed of 48 subcarriers. It is possible to assign UE data to each of these subbands, as is done with Orthogonal Frequency-Division Multiple Access (OFDMA) in LTE, however only TDMA operation is currently supported for reasons we shall explain shortly. B. PHY Transmission and Reception The MmWaveEnbPhy and MmWaveUePhy classes model the physical layer for the mmWave eNB and the UE, respectively, and encapsulate similar functionalities as the LtePhy classes from the LTE module. Broadly, these objects (i) handle the transmission and reception of physical control and data channels (analogous to the Physical Downlink Control Channel (PDCCH)/Physical Uplink Control Channel (PUCCH) and Physical Downlink Shared Channel (PDSCH)/Physical Uplink Shared Channel (PUSCH) channels of LTE), (ii) simulate the start and the end of frames, subframes and slots, and (iii) deliver received and successfully decoded data and control packets to the MAC layer. In the MmWaveEnbPhy and MmWaveUePhy classes, calls to StartSubFrame() and EndSubFrame() are scheduled at fixed periods, based on the user-specified subframe length, to mark the start and end of each subframe. The timing of variable-TTI slots, controlled by scheduling the StartSlot() and EndSlot() methods, is dynamically 11 Parameter Name Default Value Description SubframePerFrame SubframeLength SymbolsPerSubframe SymbolLength NumSubbands SubbandWidth SubcarriersPerSubband CenterFreq NumRefScPerSymbol NumDlCtrlSymbols NumUlCtrlSymbols GuardPeriod MacPhyDataLatency PhyMacDataLatency NumHarqProcesses 10 100 24 4.16 72 13.89 48 [6-100] 864 (25% total) 1 1 4.16 2 2 20 Number of subframes in one frame Length of one subframe in µs Number of OFDM symbols per slot Length of one OFDM symbol in µs Number of subbands Width of one subband in MHz Number of subcarriers in each subband Possible carrier frequencies in GHz∗ Reference subcarriers per symbol Downlink control symbols per subframe Uplink control symbols per subframe Guard period for UL-to-DL mode switching in µs Subframes between MAC scheduling request and scheduled subframe Subframes between TB reception at PHY and delivery to MAC Number of HARQ processes for both DL and UL Table I: Parameters for configuring the mmWave PHY. ∗ The NYU channel model [31] supports only 28 and 73 GHz. configured by the MAC via the MAC-PHY SAP method SetSfAllocInfo(), which enqueues an SfAllocInfo allocation element for some future subframe index specified by the MAC. A subframe indication to the MAC layer triggers the scheduler at the beginning of each subframe to allocate a future subframe. For the UE PHY, SfAllocInfo objects are populated after reception of DCI messages. At the beginning of each subframe, the current subframe allocation scheme is dequeued, which contains a variable number of SlotAllocInfo objects. These, in turn, specify contiguous ranges of OFDM symbol indices occupied by a given slot, along with the designation as either DL or UL and control (CTRL) or data (DATA). The data packets and the control messages generated by the MAC are mapped to a specific subframe and slot index in the packet burst map and control message map, respectively. Presently, in our custom subframe design, certain control messages which must be decoded by all UEs, such as the DCIs, are always transmitted in fixed PDCCH/PUCCH symbols at the first and last symbol of the subframe, but this static mapping can be easily changed by the user8 . Other UE-specific control and data packets are recalled at the beginning of each allocated TDMA data slot and are transmitted to the intended device. To initiate transmission of a data slot, the eNB PHY first calls AntennaArrayModel::ChangeBeamformingVector() to update the transmit and receive beamforming vectors for both the eNB and the UE. In the case of control slots, no beamforming update is applied since we currently assume an “ideal” control channel. For both DL and UL transmissions, either the MmWaveSpectrumPhy method StartTxDataFrame() or StartTxCtrlFrame() is called to transmit a data or control slot, respectively. The functions of MmWaveSpectrumPhy, which is similar to the corresponding LENA class, are as follows. After the reception 8 As in [73], [74], we assume either FDMA or SDMA-based multiple access in the control regions. However, we do not currently model these modulation schemes nor the specific control channel resource mapping explicitly. We intend for this capability to be available in later versions, which will enable more accurate simulation of the control overhead. of data packets, the PHY layer calculates the SINR of the received signal in each subband, taking into account the path loss, MIMO beamforming gains and frequency-selective fading. This triggers the generation of Channel Quality Information (CQI) reports, which are fed back to the base station in either UL data or control slots. The error model instance is also called to probabilistically compute whether a packet should be dropped by the receiver based on the SINR and, in the case of an HARQ retransmission, any soft bits that have been accumulated in the PHY HARQ entity (see Section VII-B). Uncorrupted packets are then received by the MmWavePhy instance, which forwards them up to the MAC layer SAP. VII. MAC L AYER TDMA is widely assumed to be the de-facto scheme for mmWave access because of the dependence on analog beamforming, where the transmitter and receiver align their antenna arrays to maximize the gain in a specific direction (rather than with a wide angular spread or omni-directionally, as in conventional systems). Many early designs and prototypes have been TDMA-based [69], [70], [72], with others incorporating SDMA for the control channel only [73]. SDMA or FDMA schemes (as in LTE) are possible with digital beamforming, which would allow the base station to transmit or receive in multiple directions at the same time. Furthermore, one of the foremost considerations driving innovation for the 5G MAC layer is latency. Specifically, the Key Performance Indicator of 1 ms over-the-air latency has been proposed as one of the core 5G requirements by such standards bodies as the International Telecommunication Union [79], as well as by recent pre-standardization studies such as those carried out under the METIS 2020 project [80]. However, a well-known drawback of TDMA is that fixed slot lengths or TTIs can result in poor resource utilization and latency, which can become particularly severe in scenarios where many intermittent, small packets must be transmitted to/received from many devices [74]. Based on these considerations, variable TTI-based TDMA frame structures and MAC schemes have been proposed in 12 A. Adaptive Modulation and Coding The role of the Adaptive Modulation and Coding (AMC) mechanism is to adapt the modulation scheme and the coding applied on top to the channel quality, measured using CQIs. In the simulator, this translates into (i) mapping the CQI into the Modulation and Coding Scheme (MCS), using the error model implemented in the MmWaveMiErrorModel and described in Sec. V-D, and (ii) computing the available TB size for a subframe given the MCS. This information is then used by the scheduler to perform radio resource management. The AMC is implemented in the MmWaveAmc class, which uses most of the code of the corresponding LENA module class. Some minor modifications and additional methods were necessary to accommodate the dynamic TDMA MAC scheme and frame structure. For instance, the GetTbSizeFromMcsSymbols() and GetNumSymbolsFromTbsMcs() methods are used by the scheduler to compute the TB size from the number of symbols for a given MCS value, and vice versa. Also the CreateCqiFeedbackWbTdma() method is added to generate wideband CQI reports for variable-TTI slots. Figure 7 shows the results of the test case provided in mmwave-amc-test.cc. This simulation serves to demonstrate the performance of the AMC and CQI feedback mechanisms for a single user in the uplink (although a multi-user scenario could easily be configured as well). The default PHY/MAC parameters in Table I are used along with the default scheduler and default parameters for the statistical path loss, fading and beamforming models (i.e., MmWavePropagationLossModel and MmWaveBeamforming). We compute the rate versus the average SINR over a period of 12 seconds (long enough for the small-scale fading to average out). The average PHY-layer rate is computed as the average sum of the sizes of successfully-decoded TBs per second. Every 12 seconds we artificially increase the path loss while keeping the UE position fixed. As the SINR decreases, the MAC will select a lower MCS level to encode the data. The test is performed for the Additive White Gaussian Noise (AGWN) case (i.e., no fading) as well as for small-scale 3500 AWGN fast fading 3000 2500 Rate (Mbps) [20], [73]–[75], [81]. This approach allows for slot sizes that can vary according to the length of the packet or TB to be transmitted and are well-suited for diverse traffic since they allow bursty or intermittent traffic with small packets as well as high-throughput data like streaming and file transfers to be scheduled efficiently. The MAC layer implementation can be found in the MmWaveEnbMac and MmWaveUeMac classes, whose main role is the coordination of procedures such as scheduling and retransmission. Moreover, they interact with the RLC layer to receive periodic reports on the buffer occupancy, i.e., the Buffer Status Reports (BSRs), and with the physical layer classes for the transmission and reception of packets. To carry out their functionalities, the MAC classes interact with several other classes, that we will describe in the following paragraphs. 2000 1500 1000 500 0 4 123 10 5 56 78 9 0 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 5 10 SINR (dB) 15 20 25 30 Figure 7: Rate and MCS vs. SINR for a single user under AGWN and fastfading mmWave channels. fading. Although the UE position relative to the base station is constant, we can generate time-varying multi-path fading through the MmWaveBeamforming class by setting a fixed speed of 1.5 m/s to artificially generate Doppler, which is a standard technique for such an analysis. Also, we assume that the long-term channel parameters do not change for the duration of the simulation. Figure 7 therefore shows the data rate that it is possible to achieve with a certain SINR and with a certain modulation and coding scheme. If this plot is compared to the one generated from a similar test in Figure 3.1 of the LENA documentation [51], we notice that the AGWN curve from the mmWave test is shifted by approximately 5 dB to the left, indicating that the LENA version is transitioning to a lower MCS at a much higher SINR. This is because the LENA test is using the more conservative average SINR-based CQI mapping, which targets a much smaller TB error probability. In our test, we use the Mutual Information-Based Effective SINR scheme described in Sec. V-D with a target maximum TB error of 10% in order to maximize the rate for a given SINR [67]. B. Hybrid ARQ Retransmission Full support for HARQ with soft combining is now included in the mmWave module. HARQ is a technique introduced in [82] and extensively used in LTE networks [20], which enables fast retransmissions with incremental redundancy in order to increase the probability of successful decoding and the efficiency of the transmissions. In LTE, the HARQ mechanism is based on multiple stop and wait retransmission processes, and a maximum of 8 simultaneous HARQ processes can be active at any given time [83]. The HARQ retransmissions have priority with respect to new transmissions, thus the available resources are given first to HARQ processes and then to the data queued in the RLC buffers. Despite being fundamental in protecting from the losses of packets due to rapid variations in the channel quality, the HARQ mechanism introduces additional latency [18], [20], therefore the optimization of its performance is necessary to enable the target of sub-1-ms latency for ultra-low-latency communications. The MmWaveHarqPhy class along with the functionalities within the different scheduler classes are based heavily on 13 the LENA module code. The scheduler at the eNB uses the information provided by HARQ feedback messages to assign new resources to the HARQ processes that require retransmissions. Each transport block is granted a maximum number of transmission attempts, which is set to 3, as in LTE. However, some novelties are introduced in MmWaveHarqPhy in order to account for the more challenging channel conditions of the mmWave scenario. First, multiple HARQ processes per user can be created not only for the downlink but also for the uplink. Second, the number of processes per user is not fixed to 8, but can be configured through the NumHarqProcesses attribute in MmWavePhyMacCommon. This makes it possible to control (and, if needed, increase) the number of the simultaneous stop and wait retransmission processes and optimize the bandwidth utilization. Third, additional modifications were needed to support larger codeword sizes in both the MmWaveHarqPhy and MmWaveMiErrorModel classes. Finally, the integration with the flexible TTI physical layer allows a reduction in the latency of the retransmissions, as discussed in [20]. C. Schedulers We now present the implementations of four scheduler classes for the variable TTI scheme. These differ significantly from the OFDMA-based schedulers available in ns–3 LENA [51] as, instead of allocating Resource Blocks/Resource Block Groups of frequency-domain resources, these TDMA-based schedulers allocate time-domain symbols within a periodic subframe to different users in the DL or UL direction. Before scheduling new data, Buffer Status Report and CQI messages are first processed. The MCS is computed by the AMC model for each user based on the CQIs for the DL or SINR measurements for the UL data channel. The MCS and the buffer length of each user are used to compute the minimum number of symbols required to schedule the data in the user’s RLC buffers. This procedure for estimating the optimal MCS and determining the minimum number of symbols is common to each of the schedulers described in the following. Round Robin (RR) Scheduler: The MmWaveFlexTtiMacScheduler class is the default scheduler for the mmWave module. It supports the variable TTI scheme previously described in Section VI and assigns OFDM symbols to user flows in Round-Robin order. Upon being triggered by a subframe indication, any HARQ retransmissions are automatically scheduled using the available OFDM symbols. While the slot allocated for a retransmission does not need to start at the same symbol index as the previous transmission of the same TB, it does need the same number of contiguous symbols and MCS, since an adaptive HARQ scheme (where the re-transmission can be scheduled with a different MCS) has not yet been implemented. To assign symbols to users, the total number of users with active flows is first calculated. Then the total available data symbols in the subframe are divided evenly among users. If a user requires fewer symbols to transmit its entire buffer, the remaining symbols (i.e., the difference between the available and required slot length) are distributed among the other active users. One also has the option to set a fixed number of symbols per slot by enabling the fixed TTI mode. Although the same general subframe structure is maintained, slots will then be allocated in some multiple of SymPerSlot symbols. Setting the SymPerSlot attribute of the scheduler class to the number of slots per subframe, for instance, will result in only one UE being scheduled per subframe, which would be highly inefficient in a multi-user cell. Proportional Fair (PF) Scheduler: Proportional Fair is another well-known discipline, and is provided by the MmWaveFlexTtiPfMacScheduler class. The PF scheduler attempts to prioritize traffic for high-SINR users while maintaining some measure of fairness by ensuring that low-SINR, cell-edge users are also scheduled [84]. Earliest Deadline First (EDF) Scheduler: The MmWaveFlexTtiEdfMacScheduler class implements an Earliest Deadline First policy, which is a priority queue-based policy that weighs flows by their relative deadlines for packet delivery. The deadlines are initially set according to the delay budget of the QoS Class Indicator (QCI) configured by the RRC layer [85], [86]. The deadline of the Head-of-Line (HOL) packets of each RLC buffer is then compared, and that with the earliest deadline is scheduled first. Any remaining symbols in the subframe are allocated to the packet with the next smallest relative deadline and so forth until all Nsym symbols are assigned. The EDF scheduler is the only deliberately delaysensitive scheme included in the mmWave module and can be useful for evaluating the latency performance of mmWave links, as in the simulations presented in Sec. X. Maximum Rate (MR) Scheduler: The Maximum Rate policy realized in the MmWaveFlexTtiMrMacScheduler class schedules only the users with the highest SINRs to maximize cell throughput. Initially, UEs are sorted based on their optimal MCS values. Symbols are distributed in roundrobin fashion among UEs at the highest MCS level until the minimum number of symbols required to transmit the entire buffers of these users has been assigned. This is then repeated for UEs at the second highest level, and so forth until all symbols of the subframe are allocated. The MR scheduler potentially suffers from extremely poor fairness when there are both high- and low-rate users, and some users may not be scheduled at all, thus making it impractical for any real-world multi-user system. However, it may still be useful for testing system capacity and performance. VIII. RLC L AYER The RLC layer is inherited directly from the LTE module described in [23], and therefore all the LTE RLC entities are included. Moreover, the RLC AM retransmission entity is modified to be compatible with the mmWave PHY and MAC layers, and Active Queue Management (AQM) for the RLC buffers is introduced as a new optional feature. 14 A. Modified RLC AM Retransmission Reordering and retransmission play an important role in RLC AM. Due to the shortened mmWave frame structure, the timers of the RLC entity should also be reduced accordingly, e.g., the PollRetransmitTimer is changed to 2 ms from 20 ms. Moreover, the original LTE module does not perform resegmentation for retransmissions, and the RLC segment waits in the retransmission buffer until the transmission opportunity advertised by the lower layers is big enough. This becomes problematic when the transmission is operated over an intermittent channel, as a sudden channel capacity drop would halt the retransmission entirely. Therefore, we added to the RLC AM layer implementation the capability of performing segmentation also for the retransmission process, in order to support an intermittent mmWave channel. The re-segmentation process deployed in our RLC AM class works as follows: If the number of bytes that can be transmitted in the next opportunity is smaller than the bytes of the segment that should be retransmitted, then the segment will be split into smaller subsegments with a re-segment flag set to be true. The RLC layer at the receiver side will check the flags of the subsegments, and wait until the final one if the flag is set to be true. Finally, the subsegments are assembled to construct the original segment and forwarded to the upper Packet Data Convergence Protocol (PDCP) layer if all subsegments are received correctly. Otherwise, all subsegments are discarded and another retransmission is triggered. B. Active Queue Management Active Queue Management techniques are used in the buffers of routers, middleboxes and base stations in order to improve the performance of TCP and avoid the manual tuning of the buffer size. Different strategies have been defined in the literature [87], [88], and several of them are implemented in ns–3 [89]–[91]. AQM strategies allow the network to avoid congestion at the buffers, because they react early to the increase in the buffer occupancy by dropping some packets before the buffer is full. With respect to the default Drop-tail approach, in which no packet is dropped until the buffer is full, AQM techniques make TCP aware of possible congestion earlier, avoiding the latency increase which is typical of the bufferbloat phenomenon [92]. Some early AQM, such as Random Early Detection (RED) [93], [94], were widely studied in the literature, but failed to find market traction because of the intrinsic complexity of their tuning parameters. Recently, a simpler AQM technique, namely Controlled Delay (CoDel) [95], was proposed to replace RED queues. CoDel adapts to dynamic link rates without parameter configuration, and is able to discriminate “good” and “bad” queues: good queues can quickly empty the buffer, whereas “bad” queues persistently buffer packets. CoDel works by monitoring the minimum queue delay in every 100 ms interval, and only drops packets when the minimum queue delay is more than 5 ms. In the RLC layer of the LTE module, the default queue management is Drop-tail. In the mmWave module, the RLC layer can use either the default Drop-tail approach or more sophisticated AQM techniques, that can be enabled by setting the EnableAQM attribute to true. The default AQM is the CoDel scheme, however it is possible to use any of the queues available in ns–3 by modifying the queue attribute in the LteRlcAm class. The evaluation of the AQM scheme is further discussed in Section X-D. IX. D UAL C ONNECTIVITY E XTENSION The ns–3 mmWave module is also capable of performing simulations with dual-stack UEs connected both to an LTE eNB and to a mmWave eNB. This feature, partially described in [29], was introduced because mmWave 5G networks will likely use multi-connectivity and inter-networking with legacy Radio Access Technologies (RATs) in order to increase the robustness with respect to mobility and channel dynamics [14]–[16], [96]–[98]. The source code can be found in the new-handover branch of the ns–3 mmWave module repository. The Dual Connectivity (DC) implementation of this simulation module assumes that the core networks of LTE and of mmWave will be integrated, as in one of the options described in [99]. Therefore the LTE and the mmWave eNBs share the same backhaul network, i.e., they are connected to each other with X2 links and to the Mobility Management Entity (MME)/Packet Gateway (PGW) nodes with the S1 interface. As to the Radio Access Network (RAN), the DC solution of this module is an extension of 3GPP’s LTE DC proposal [100]. In particular, a single bearer per DC flow is established, with a connection from the core network to the LTE eNB, where the flow is split and forwarded either to the local stack or to the remote mmWave stack. We chose the PDCP layer as the integration layer, since it allows a non-colocated deployment of the eNBs and a clean-slate approach in the design of the PHY, MAC and RLC layers [101]. A basic diagram for a DC UE device, an LTE eNB and a mmWave eNB is shown in Figure 8. The core of the DC implementation is the McUeNetDevice class, which is a subclass of the ns–3 NetDevice and provides an interface between the ns–3 TCP/IP stack and the custom lower layers. The McUeNetDevice holds pointers to the custom lower layer stack classes, and has a Send method that forwards packets to the TCP/IP stack. This method is linked to a callback on the DoRecvData of the EpcUeNas class, which as specified by the 3GPP standard acts as a connection between the LTE-like protocol stack and the TCP/IP stack. The McUeNetDevice describes a dual connected UE with a single EpcUeNas, but with a dual stack for the lower layers, i.e., there are separate LTE and mmWave PHY and MAC layers. Moreover, there is an instance of the RRC layer for both links. This grants a larger flexibility, because the functionalities and the implementation of the two layers may differ. Besides, the LTE RRC manages both the LTE connection and the control plane features related to DC, while the mmWave RRC handles only the mmWave link. The usage of a secondary RRC, dedicated to the mmWave link, avoids latency in control commands (i.e., the mmWave eNB does not have to encode and transmit the control Packet Data Unit (PDU)s to the master 15 McUeNetDevice EpcUeNas LteUeRrc mmWaveUeRrc McUePdcp LteRlc mmWaveRlc LteUeMac LteUePhy mmWaveUeMac mmWaveUePhy LteSpectrumPhy mmWaveSpectrumPhy Channel classes MultiModelSpectrumChannel MultiModelSpectrumChannel and mmWaveBeamforming LteSpectrumPhy mmWaveSpectrumPhy LteEnbPhy LteEnbMac mmWaveEnbPhy mmWaveEnbMac LteRlc mmWaveRlc face nter X2 i LteEnbNetDeviceMcEnbPdcp LteEnbRrc MmWaveEnbNetDevice mmWaveEnbRrc EpcEnbApplication S1 interface Core Network Figure 8: Block diagram of a dual-connected device, an LTE eNB and a mmWave eNB [29]. LTE eNB). The EpcUeNas layer has an interface to both RRC entities to exchange information between them. The LTE RRC manages also the data plane for the DC devices. In particular, for each bearer, a dual connected PDCP layer is initialized and stored in the LTE RRC. The classes describing the DC PDCP layer are McEnbPdcp and McUePdcp, respectively at the eNB side and at the UE side. They both extend the LtePdcp class with a second interface to the RLC. However, while McUePdcp simply has to communicate with a local RLC in the UE, the implementation of McEnbPdcp requires new interfaces to the class describing the X2 links between eNBs (i.e., EpcX2). In particular, in downlink the eNB PDCP has to send packets to the X2 link and the mmWave RLC layer has to receive them, and vice versa in uplink. The DC module can be used to simulate different dual connected modes, i.e., it can support both Fast Switching (FS) and throughput-oriented dual connectivity, according to which RRC and X2 procedures and primitives are implemented. With FS, the UE is in the RRC CONNECTED state with respect to both eNBs, but only transmits data to one of the two. With the other option, the UE can transmit data simultaneously on both RATs, and different flow control algorithms can be plugged in and tested. As to the physical layer, the two stacks rely on the mmWave and LTE channel models. Notice that since the two systems operate at different frequencies, modeling the interference between the two RATs is not needed. Each of the two channel models can therefore be configured independently. In order to use an McUeNetDevice as a mobile User Equipment in the simulation, the helper class of the mmWave module was extended with several features, such as (i) the initialization of the objects related to the LTE channel; (ii) the installation and configuration of the LTE eNBs, so that they can be connected to the LTE stack of the McUeNetDevice; and (iii) the methods to set up a McUeNetDevice and link its layers as shown in Fig 8. An example on how to set up a dual-connectivity based simulation is provided in the file mc-twoenbs.cc. RRC Layer for Dual Connectivity and Mobility. The RRC layer implementation of the original LTE ns–3 module was extended in order to account for DC-related control procedures. In particular, the multi-connectivity uplink-based measurement framework described in [15] was added with changes to the MmWaveEnbPhy, EpcX2 and LteEnbRrc classes. The MmWaveEnbPhy instance simulates the reception of uplink reference signals (which are accounted for as overhead in the data bearers resource allocation), computes the SINR for each UE in the scenario9 , and sends this information to the LTE eNB on the X2 link. This also allows the simulation of a delay in the reporting, since the control packets with the SINR values must be transmitted on an ns–3 PointToPointLink, which adds a certain latency and has a certain bitrate. Thanks to this framework, the LTE eNB is able to act as a coordinator for the surrounding mmWave eNBs, and learns which is the best association (in terms of SINR) between UEs and mmWave secondary eNBs. This enables automatic cell selection for mmWave eNBs at the beginning of a simulation, and the control of mobility-related operations. The DC module is indeed capable of simulating FS procedures between mmWave and LTE links and Secondary Cell Handover (SCH) (i.e., handovers between mmWave eNBs that do not involve the MME in the core network) initiated by the central controller in the LTE eNB. It is also possible to use the DC module to simulate X2-based RAT handovers between the LTE and mmWave eNBs, i.e., to use standalone UEs based on McUeNetDevice that can perform handovers from the LTE to the mmWave eNBs, and vice versa. Different handover (either inter-RAT or SCH) algorithms can be tested, by implementing them in the LteEnbRrc class. In order to make the handover simulation more compliant with the 3GPP specifications, the lossless handover option implemented for ns–3 in [102] was adapted to the DC module in order to forward the RLC buffer content to the target RAT/eNB RLC layer for both the SCH and the FS. Moreover, in order to model the additional latency given by the interaction with the MME for inter-RAT handovers for standalone UEs, the link between the eNBs and the MME is modeled in this module as a PointToPointLink, while in the original ns–3 LTE module it is an ideal connection. 9 The framework assumes that the optimal beam is always chosen, so the actual directional scan procedure described in [15] is not simulated 16 X. U SE C ASES 1 0.8 Empirical Probability In this section, we illustrate various examples of scenarios10 that can be simulated to show the utility of the module for the analysis of novel mmWave protocols and for testing higherlayer network protocols, such as TCP, over 5G mmWave networks. After each particular use case example, we also provide to the interested readed some references to recent papers that report additional results obtained using the ns–3 mmWave simulation module. 0.6 0.4 mr pf rr edf 0.2 0 0.1 A. Simulation Setup Walk-through B. Multi-User Scheduling Simulation In this experiment, the throughput and latency of users of a mmWave cell with 1 GHz of bandwidth are simulated for 10 The simulations in this section are all configured with the basic PHY and MAC parameters in Table I, with other notable parameters given in the sequel. 5 10 50 100 (a) Empirical CDF of rate 1 rr pf edf mr 0.8 Empirical Probability In order to proficiently use the mmWave ns–3 module, a basic knowledge of ns–3 is required. We therefore advise the interested users to first study the extensive documentation referenced in Sec. III. Moreover, we provide some basic ns–3 scripts in the examples folder of the mmWave module, that can be a basis for the design of any simulation script that uses the mmWave module. In the following paragraphs, we will describe the basic structure of a typical example in simple steps. The first step is to configure all the attributes needed in a simulation. A complete list of attributes related to the mmWave module can be found in the mmWaveAttributesList file in the module repository. The second step involves the setup of the MmWaveHelper object, which provides methods to create the entities involved in the simulation (e.g., the channel-related objects and the MmWavePhyMacCommon object), to install the mmWave stack over ns–3 nodes (for both UEs and eNBs), to perform the initial attachment of a UE to the closest eNB and to enable or disable the generation of simulation traces. Moreover, if the scenario of interest is an end-to-end scenario, the core network and the internet must be set up as well. The first is created by the MmWavePointToPointEpcHelper, which also provides a pointer to the Packet Gateway (PGW) node. This is then usually connected to a remote host, and the internet stack (i.e., the TCP/IP protocol suite) is added to the UEs and to the remote host. In the third step, the positions and velocities of the eNBs and UEs are specified using one or more MobilityHelper objects and different mobility models. Moreover, buildings and obstacles can be added to the scenario using the ns–3 buildings module and the Buildings and BuildingsHelper objects. The fourth step requires the setup of applications in the UEs and in the remote host (if an end-to-end scenario is considered), in order to simulate downlink and uplink traffic. ns–3 provides a wide range of different applications, and helpers that take care of their setup. They can run on either UDP or TCP sockets, and several TCP congestion control versions are available. Finally, the simulation can be run using the Simulator object of ns–3, and traces are generated. 0.5 1 Mean PHY Rate (Mbps) 0.6 1 ms 0.4 0.2 0 10-1 100 101 102 103 Mean IP-Layer Latency (ms) (b) Empirical CDF of latency Figure 9: Distributions of PHY-layer throughput and IP-layer latency for 70 UEs, 10 Mbps/UE arrival rate variable TTI and each of the scheduling policies described in Section VII.11 We shall see how the choice of the scheduler has a significant impact on the subframe utilization and latency of the multi-user cell. In these scenarios, UEs have similar distances from the eNB but are assigned the constant speed of 25 m/s (typical of vehicular users), which results in a lower achievable rate, on average, as well as increased packet errors compared to walking users due to the more rapid variation in the channel. The simulation is again run over 10 drops for each of two scenarios and using default parameters from Table I. In the first scenario, 70 UEs are simulated with each UE generating IP-layer traffic at an average arrival rate of 10 Mbps. In the second scenario, 7 UEs are simulated with a 100 Mbps arrival rate per UE. These specific combinations of users and rates are deliberately chosen because they illustrate the cut-off point at which the system is no longer able to service most users at the requested rate, leading to backlogged queues and increased latency. That is, we wish to analyze the performance at the knee in the curve of the delay taken as a function of the system utilization. In the variable TTI system, this bottleneck effect has the following potential causes: (i) the number of users that must be serviced exceeds the number of available slots (ultimately limited by the number of time-domain symbols), independently of the total throughput requested by the users; (ii) the number of users that are connected to eNB is smaller than the number of available slots, but the total throughput 11 The multi-user scheduling experiment can be reproduced by running the the mmwave-epc-tdma.cc example simulation [25]. 17 PHY-layer throughput [Mbit/s] Policy Cell Mean UE Mean 5% Worst UE Max UE 70 UE/ 10 Mbps RR PF MR EDF 1815.92 1494.61 2273.18 925.80 25.94 21.35 32.47 13.23 1.11 3.26 0.00 7.31 48.80 43.94 151.36 31.02 7 UE/ 100 Mbps RR PF MR EDF 715.26 758.32 766.26 647.98 102.18 108.33 109.47 92.57 49.18 52.32 47.26 63.89 134.28 158.16 158.22 121.76 Table II: DL PHY throughput for RR, PF, MR and EDF scheduling policies. IP-layer latency [ms] Policy Mean UE Mean 5% Worst UE Max UE 70 UE/10 Mbps RR PF MR EDF 7.47 2.83 0.65 1.63 69.35 34.48 1.89 7.91 118.62 106.54 3.07 30.65 7 UE/100 Mbps RR PF MR EDF 0.67 0.55 0.56 0.69 2.01 0.68 0.78 1.41 2.37 0.77 1.09 1.44 Table III: IP-layer latency for RR, PF, MR and EDF scheduling policies. 0.8 Empirical Probability Fairness Utilization 70 UE/10 Mbps RR PF MR EDF 0.71 0.76 0.28 0.96 0.53 0.73 0.39 0.87 7 UE/100 Mbps RR PF MR EDF 0.95 0.90 0.91 0.99 0.74 0.77 0.76 0.84 Table IV: Fairness index and utilization (received IP-layer rate/allocated PHY rate) for RR, PF, MR and EDF scheduling policies. worst-case) user PHY rates and IP-to-IP layer latencies are also provided in Tables II and III along with the utilization and Jain’s Fairness Index in Table IV. 1 0.6 0.4 mr pf rr edf 0.2 0 50 100 200 Mean PHY Rate (Mbps) (a) Empirical CDF of rate 1 0.8 Empirical Probability Policy 0.6 1 ms 0.4 rr pf edf mr 0.2 0 0.5 1 1.5 2 2.5 Mean IP-Layer Latency (ms) (b) Empirical CDF of latency Figure 10: Distributions of PHY-layer throughput and IP-layer latency for 7 UEs, 100 Mbps/UE arrival rate they request exceeds the available resources in the given time period; or (iii) a combination of the previous cases. These effects are demonstrated in Figures 9 and 10 for the 70 UE/10 Mbps and 7 UE/100 Mbps arrival rate scenarios, respectively. The mean, maximum and cell-edge (i.e., 5% For the 70 UE case, Figure 9a shows the distribution of the mean rate experienced by each UE over the simulation duration. It can be seen that the MR and RR policies exhibit the greatest disparity between users scheduled with high and low rates. It can also be observed that the PHY rate significantly exceeds the 10 Mbps arrival rate for some users, which leads to the poor utilization for these two policies, as shown in Table IV. The reason why the utilization (defined as the ratio of the received IP-layer rate to the allocated PHY-layer rate for each terminal) suffers in these cases is that the UEs with higher achievable rates are heavily favored by the MR and RR schedulers. As these users are typically scheduled at a higher MCS level, even a single 4.16 µs-long time-domain symbol has the capacity to transmit kilobytes of data, which cannot be fully taken advantage of given the low 10 Mbps arrival rate. Insufficient data is buffered at the MAC layer to utilize the full slot and useless padding bits must be added. This effect is felt less by users under the PF and EDF policies, which are inherently more fair and allow more resources to be scheduled for lower-MCS users. The ensuing effect of these trends on latency is shown in Figure 9b. Here latency is measured as the time between the arrival time of packets at the PDCP layer of the eNB stack and the time they are delivered to the IP layer at the UE. Naturally, the MR scheduler offers the best delay performance because only 40% of the users, i.e., those with the highest rates, are ever scheduled (unscheduled users with zero rate are not included in the figure). The RR policy offers the highest Avg. IP-layer latency (ms) 101 1 ms 100 10 -1 0 20 40 60 80 100 Deadline Miss Ratio Deadline Miss Ratio Avg. IP-layer latency (ms) 18 100 10 -1 fixed (66 us sf) fixed (100 us sf) var (66 us sf) var (100 us sf) 10-2 10-3 0 0 20 40 60 80 100 Number of UEs 101 1 ms 100 10 -1 0 2 4 6 8 10 100 10-1 fixed (66 us sf) fixed (100 us sf) var (66 us sf) var (100 us sf) 10-2 10-3 0 0 2 4 6 8 10 Number of UEs (a) 10 Mbps per UE arrival rate (100-bytes packets) (b) 100 Mbps per UE arrival rate (1200-bytes packets) Figure 11: Latency and Deadline Miss Ratio as a function of the downlink IP-layer arrival rate for fixed and variable TTI radio frame structures. Description Value Subframe length in µs OFDM symbols per slot HARQ processes (DL and UL) Number of UEs Number of UEs Traffic model Traffic model 100/66.67 24/16 20 DL/20 UL Case 1: {10, 20, 30, 40, 50, 60, 70, 80, 100} Case 2: {1, 2, 3, 4, 5, 6, 7, 8, 10} Case 1: Poisson, λ = 12.5K pck/s, 100 B packets Case 2: Poisson, λ = 83K pck/s, 1200 B packets Table V: Additional parameters for variable and fixed TTI latency experiment. worst-case delays but is able to achieve mean latencies of less than 1 ms for more than 60% of the users. Of all the policies besides MR, Earliest-Deadline First offers the best worst-case delays, as it attempts to balance the delay of all users by scheduling them exclusively based on their relative deadlines (not taking into account achievable rates). The EDF scheduler is able to achieve a mean UE latency of 1.6 ms, which, as we will see from the experiment in the next section, drops below 1 ms for 60 or fewer users (with the same arrival rate). Finally, it can be observed from Figure 10b that, despite having the same total packet arrival rate of 700 Mbps as in the 70 UE case, latencies are much lower overall in the 7 UE/100 Mbps per UE case. This can be clearly explained by the higher utilization factor in Table IV. In this scenario, the number of available slots for scheduling different users is no longer the bottleneck. Though we still see that a significant number of users are scheduled at rates that exceed their 100 Mbps arrival rates, the utilization is notably better than the 10 Mbps case for all scheduling policies. Thus, the channel capacity itself is better utilized, allowing most users to be scheduled at the requested rate, thereby avoiding additional queue wait time and delay. Additional results on the impact of the scheduling on throughput and latency can be found in [103]. C. Latency Evaluation for Variable and Fixed TTI Schemes The results and the discussion introduced in this section are taken from our previous article [20], where the interested reader will find a more comprehensive treatment of techniques to achieve low latency in mmWave 5G cellular systems. While the qualitative benefits of variable TTI over fixed TTI may seem self-evident, in this section we quantify the performance gains for a multi-user TDMA mmWave system with 1 GHz of bandwidth. We also demonstrate that, with the low-latency scheduling loop enabled by the proposed frame structure, LTE-style Hybrid ARQ can still be employed for enhanced link-layer reliability without excessively exceeding the delay constraints. We model the subframe formats shown in Figure 6 for two subframe periods: the default 100 µs subframe, equivalent to 24 OFDM symbols, and a 66.67 µs subframe, equivalent to 16 OFDM symbols. The symbol length of 4.16 µs is based on the numerology in [104]. Each subframe has one fixed DL-CTRL and one UL-CTRL symbol, with the remaining symbols used for DL or UL data slots. For fixed TTI mode, the entire subframe is allocated to a single user, whereas for variable TTI mode, the scheduler may allocate any number of data symbols within the subframe to match the throughput required by each user. We also note that UEs are again modeled as moving at 25 m/s, typical of vehicular speeds, which causes fast channel variation and frequent packet errors from small-scale fading (it is observed that between 0.5% and 3% of the transport blocks are lost and require retransmission). We consider a simple traffic model with Poisson arrivals, where each UE sends an average of 12.5K packets per second (100-bytes packets resulting in an average rate of 10 Mbps), as well as a separate, higher-throughput case where each UE sends an average of 83K packets per second (1200-bytes packets resulting in an average rate of 100 Mbps). Scheduling is performed based on the EDF policy where the scheduler attempts to deliver each IP packet within 1 ms from its arrival Packets-in-flight RLC queue (Packets) RTT (ms) 40 20 0 -20 40k Drop-tail CoDel CoDel packet drop Drop-tail(Dynamic RW) 20k 0 20k 10k 0 200 150 100 40 Rate (Gb/s) at the PDCP layer and packets are assigned a priority based on how close they are to the deadline. Priority is therefore always given to HARQ retransmissions. We simulate the performance for between 10 and 100 UEs for a 10 Mbps (per UE) arrival rate and between 1 and 10 UEs for the 100 Mbps case, equivalent to a total IP-layer arrival rate of between 100 and 1000 Mbps in both cases. In Fig. 11, the downlink radio link latency is averaged among the best 95% of the users (i.e., the 5% of UEs experiencing the highest latency are not considered). The Deadline Miss Ratio (DMR), which represents the fraction of packets delivered after the 1 ms deadline, is also given for the top 95th percentile UEs. We see that, for a 10 Mbps arrival rate (Figure 11a), variable TTI is able to achieve sub-ms average latency and a DMR of about 10% with over 60 users (corresponding to a 600 Mbps total packet arrival rate) and consistently outperforms fixed TTI. Fixed TTI, despite the relatively short subframe compared to LTE, exceeds 1 ms average latency and has a DMR of over 60% even for the 20 UE case and of more than 90% for 40 or more users. This result shows that variable TTI will be essential for reliable, low-latency service, particularly when considering use cases with many lower-rate devices, such as Machine-Type Devices (MTDs) [4]. For the higher-load (100 Mbps arrival rate per UE) case in Figure 11b, we expect the deviation between the variable and fixed TTI schemes to be less pronounced, as the bottleneck is now the multi-user channel capacity and not the minimum slot size. However, we do find a reduction in radio link latency of around 500 µs, or 30%, for the variable TTI scheme in some cases. We also find that, for a smaller number of users, the shorter 66.67 µs subframe offers some improvement over the longer 100 µs subframe thanks to the decreased turnaround time. In particular, the DMR is consistently less for the 100 Mbps/UE case for both variable and fixed TTI. However, this trend reverses with more users due to the lower ratio of control to data symbols in the 100 µs subframe case. We note that the control overhead could be somewhat mitigated by multiplexing data in the DL-CTRL region. We also note that, in real-world implementations, there may be some additional delay related to beam tracking (i.e., for computing and applying the optimal TX/RX beamforming vectors), although the performance limitations of adaptive beamforming transceivers and channel tracking techniques in future implementations are still unknown. We assume that this delay can be neglected in our analysis because data is constantly being transmitted to each UE and channel state feedback is being transmitted by the UEs to the eNB in each subframe period (which is well within the coherence time observed in many studies), thus ensuring that the channel state information is always up-to-date at the eNB. The performance of a mmWave cellular network with respect to the end-to-end or the RAN latency has been studied in several papers with the ns-3 mmWave module. In [20], [103] the authors discuss architectural and protocol solution for low latency networks based on mmWave, while [9], [18], [37], [105] propose and evaluate latency-reduction techniques for transport protocols and video streaming. SINR (dB) 19 4 2 0 0 1 2 3 4 5 Time (s) 6 7 8 9 10 Figure 12: TCP performance of a single UE with human blockages D. TCP Performance over mmWave Another typical use case involves the simulation of end-toend networks, in order to assess the performance of higherlayer protocols on top of mmWave links. An example of these kinds of simulations is shown in Figure 12, where we evaluated the TCP performance with an eNB that uses Drop-tail or CoDel queue management at the RLC layer, and a mobile UE is experiencing blockages from other humans. The sender opens a File Transfer Protocol (FTP) connection and sends a large file to the UE. The congestion control is TCP Cubic, with delayed acknowledgment disabled. The maximum queue length is 50K packets. The core network latency is 40 ms. The UE is walking at 1 m/s, 300 meters away from the base station, while maintaining LOS connectivity, and experiencing 3 human blocking events. Drop-tail: Since the RLC queue size is large enough and all packets lost in the wireless link are recovered by means of lower layer retransmissions (RLC AM and MAC HARQ), the sender is unaware of the packet loss, thus keeping a large congestion window that results in high throughput, but also high buffer occupancy and consequent high delay. CoDel: CoDel has the ability to actively drop packets when it detects high buffering delay. The CoDel packet drop events are also labeled in Figure 12. At 0.5 s, as the RLC queue is building up, the first packet is dropped, which informs the sender to reduce the congestion window. At 0.8 s, a human blockage deteriorates the wireless link capacity and causes the RLC queue to grow, thus triggering one more packet drop. Similarly, at 4 s and 6.7 s, two more packets are dropped. As a consequence, the congestion window will decrease and all the packets buffered in the RLC will be delivered. Nonetheless, after the wireless link has recovered from a human blockage, the congestion window ramps up to link capacity very slowly. Dynamic Receive Window (RW): Our previous results in [17] showed that a user may mitigate the delay by sending TCP acknowledgments containing Dynamic RW. The optimal RW is fed back to the sender, and the sender takes the minimum of the receiver window and the congestion window as the sending 20 80 70 LTE + mmWave eNB mmWave eNB 60 Y [m] 50 40 30 20 10 UE 0 UE path at speed v 40 60 80 100 X [m] 120 140 160 Figure 13: Random realization of the simulation scenario. The grey rectangles are randomly deployed non-overlapping obstacles. 3 500 2 250 CellId Throughput [Mbits] 1 8 9 10 11 12 0 (a) PDCP throughput with Fast Switching. 3 500 2 250 CellId Throughput [Mbits] 1 8 9 10 11 12 0 its LOS/NLOS position and distance from the eNBs. These scenarios can be used to evaluate the performance of different mobility management and multi connectivity schemes. An example can be found in the mc-twoenbs.cc file. The purpose of this example is to compare a system with DC and fast switching among Radio Access Technologies (RATs) and a stand-alone architecture, with the UE connected to a single RAT (either LTE or mmWave) at any given time. The benefit of the first architecture is evident when the UE connected to the mmWave RAT experiences an outage: thanks to dual connectivity, it can immediately recover the communication using the LTE link, while the stand-alone UE must sense the link failure and then perform a handover to the LTE eNB. This takes more time and leads to a time interval in which the throughput that can be achieved at the PDCP layer is zero. This is shown in Figure 14. The green line represents the current cell to which the UE is attached, and this information can be retrieved from the CellIdStats.txt and CellIdStatsHandover.txt files. The purple line, instead, is the instantaneous PDCP throughput, sampled from the DlPdcpStats.txt trace. Besides this specific example, more general results on dual connectivity for LTE and mmWave can be found in [14], [29], where smart handover strategies are proposed and extensively evaluated with UDP as the transport protocol, and in [38], [106], which instead use TCP. (b) PDCP throughput with Hard Handover. Figure 14: PDCP throughput with multiple RATs and eNBs. eNBs with CellId 2 and 3 are mmWave eNBs, while CellId 1 stands for the LTE eNB co-located with mmWave eNB 2. window. As a result, the sending rate is precisely regulated so that the delay is reduced without rate degradation. Additional results on the performance of TCP on mmWave cellular networks obtained with ns-3 mmWave can be found in the literature. In [9], [17], [18], [106], [107] the authors investigate the performance of different congestion control algorithms, considering in particular the throughput-latency trade off. Proxy architectures that improve the throughput [108] and both throughput and latency [37] have been proposed, as well as uplink-based cross layer congestion control algorithms [36]. The performance of TCP with different mobility management schemes is studied in [38]. Finally, [18], [19] use ns-3 mmWave to analyze the performance of Multipath TCP [109] in mmWave and LTE cellular networks. E. LTE-aided Multi Connectivity Finally, a fourth use case regards the study of mobility management schemes for mmWave radio access networks. The example described in the following paragraphs uses the Dual Connectivity extension introduced in Section IX. Thanks to the McUeNetDevice, it is possible to simulate scenarios similar to that in Figure 13, with an LTE eNB and multiple mmWave eNBs under its coverage, either co-located, or connected via the X2 interface. The UE can move with different patterns, according to the mobility models available in ns–3, and while moving experiences different channel conditions according to XI. I NTEGRATION WITH DCE AND E XAMPLES DCE was introduced in [50] as a powerful tool that combines the flexibility of a network simulator such as ns–3 with the robustness of the TCP/IP stack of the Linux kernel and the authenticity of real applications. There are several benefits in using this tool. First, the Linux kernel implements protocols which are not yet available for ns–3, or which are in an early development phase and present some limitations. An example is Multipath TCP (MPTCP), the multipath extension of TCP which makes it possible to transmit data on multiple subflows (i.e., a mobile user could simultaneously transmit on a Wi-Fi subflow and a cellular subflow) [109]. At the time of writing, it was implemented for ns–3 by different projects [110], [111], but none of them is completely compliant with the MPTCP specification, and they are not integrated in the main ns–3 release and validated. With DCE, instead, it is possible to use the MPTCP code developed and tested by the same MPTCP protocol designers [112]. Second, the Linux kernel TCP/IP stack is the most widely used in real production environments and datacenters, besides being the basis for the Android mobile operating system. Therefore, it is a very well tested codebase, with very few bugs. Moreover, its usage in network simulations provides a higher level of realism. Finally, with DCE it is possible to use real POSIX socket-based applications. For example, the well known iPerf tool [113] can be used to measure the maximum achievable datarate in the network. It is also possible to simulate a website, with an http daemon in the server and wget as a client. Besides, standard ns–3 applications (OnOffApplication, BulkSendApplication) can be used with the Linux TCP/IP stack thanks to DCE Cradle. 21 30 20 0 UE path at speed v Y [m] 10 LTE + mmWave eNB -10 -20 UE -30 0 20 40 60 X [m] 80 100 Throughput [Gbit/s] Figure 15: Random realization of the simulation scenario. The grey rectangle represents a building. iPerf BW LTE throughput mmWave throughput 3 2 1 0 5 10 15 Figure 16 shows the output of a simulation, with the TCP throughput for two different congestion control algorithms for MPTCP, together with the per-subflow Radio Access Network throughput. In particular, Figure 16a shows the performance of BALIA [115], which is a coupled congestion control algorithm, i.e., it tries to adapt the congestion window of each MPTCP subflow according to the congestion experienced on all links. In Figure 16b, instead, the congestion controls on the LTE and on the mmWave subflows are uncoupled, i.e., each subflow is independent, and TCP Cubic is used. The first observation is that the LTE subflow has, as expected, a much smaller throughput compared to the mmWave subflow, and thus the total throughput measured with iPerf is similar to that of the mmWave connection. The second is that the uncoupled solution manages to reach a more stable throughput in NLOS conditions, compared to the coupled solution, as was observed in [18], [19], showing that the current coupled congestion control algorithms are not well suited for a deployment over these kinds of links. 20 Time [s] XII. P OTENTIAL USES AND FUTURE EXTENSIONS Throughput [Gbit/s] (a) Balanced Link Adaptation (BALIA) iPerf BW LTE throughput mmWave throughput 3 2 1 0 5 10 15 20 Time [s] (b) Uncoupled Cubic Figure 16: Throughput with MPTCP. In order to integrate DCE with the ns–3 mmWave module, it is necessary to patch the KernelFdSocketFactory class so that it recognizes the MmWaveUeNetDevice. The patch can be found in the utils folder of the ns–3 mmWave module repository. Then, replace the standard ns–3 folder with our mmWave module. Notice that, if MPTCP is used as the transport protocol, the DC extension must be used with the patch provided in the utils folder. MPTCP on mmWave links: The latest Linux kernel implementation of MPTCP compatible with DCE can be found in the net-next-nuse library of the LibOS project [114]. The standard DCE distribution already provides MPTCP examples, which can be promptly extended in order to account for mmWave and LTE subflows, as long as they operate on links with different carrier frequencies (i.e., it is possible to simulate an MPTCP connection on a 2.1 GHz LTE link and a 28 GHz mmWave link). It is possible to simulate different state of the art congestion control algorithms for MPTCP, either coupled or uncoupled, as shown in [18], [19]. An example is in the file dce-example-mptcp-mmwave, which creates the scenario shown in Figure 15. The application used is iPerf, and the mobile device creates two uplink subflows to a remote server, the first on mmWave, and the second on LTE. The UE moves along the y-axis and switches from a LOS to a NLOS condition, and then returns to LOS. Given the full-stack nature of the simulation framework introduced in this paper, the 5G mmWave research community can leverage this tool to bring and test innovation at every layer. Each module can be easily extended while maintaining full backward compatibility. The fundamental components are in the form of functions, classes and modules, which can be implemented to design novel algorithms, procedures, and, more in general, architectures. For example, the scheduling and allocation strategies proposed in [116]–[118] can be readily integrated and tested in our framework with some simple tweaks. Similarly, due to the importance of coping with mobility and frequent handovers, innovative approaches like the one proposed in [119], which exploits caching, can take advantage of the modular structure of the ns–3 framework to test flexible and reprogrammable logics. Additionally, as previously mentioned, several papers already fully exploit the capabilities of this simulator to capture the performance of TCP in mmWave networks, and propose some novel approaches to mitigate the limiting effects of congestion control procedures with intermittent multi-Gbps mmWave links [9], [17], [37], [38], [108]. As part of our future work, we aim at expanding the code to include additional components such as: • 3GPP-inspired signaling/beamtracking procedures to better accommodate novel techniques like those proposed in [118], [120]; • multi connectivity based on Carrier Aggregation [121]; • novel applications such as virtual & augmented reality, to ultimately test key 5G metrics as done in [122], where the authors leverage our mmWave module to run a performance analysis of traditional video delivery over mmWaves, and in [105], where ns-3 mmWave is used to assess the performance of network coding and multi connectivity for reliable video streaming over mmWave; • multi-hop architectures for both the access and the backhaul links, as presented in [123]; 22 vehicular channel and traffic models to test and capture the end-to-end performance of mmWave communications for high-mobility scenarios [124]–[126]; • public safety scenarios, including aerial communications and robotics, where the propagation environments and the performance requirements differ from those of traditional cellular networks, as detailed in [127]. Moreover, we plan to address the challenge of scalability. 5G networks will likely comprise a large number of nodes, with high mobility, and thus channel states must be updated frequently. In addition, the use of low-latency applications requires that packet timelines must be scheduled at very fast interaction times. To do so, we will explore the following two design options: • Low-rank channel modeling: We will develop computationally simple low-rank models that approximate the end-to-end phased array system well. • Migration to cluster computing: Further scaling will be achieved by investigating the deployment of the simulator onto open-source large cluster platforms such as Amazon Web Services (AWS) [128]. Finally, we plan to officially merge our mmWave module into the main ns–3 release, in order to regularly maintain its compatibility with every new update. • XIII. C ONCLUSIONS In this tutorial paper, we have presented the current status of the ns–3 framework for simulation of mmWave cellular systems. The code, which is publicly available at GitHub [25], is highly modular and customizable to allow researchers to test novel 5G protocols. We have shown some performance trends based on the mmWave channel models available. A detailed explanation of our configurable physical and MAC layers is provided, along with a corroborating set of simulation results for varying configurations. Implementations of advanced 5G architectural features, such as dual connectivity, are also available, and we have reported different representative results. We have also shown that the module can be interfaced with the higher-layer protocols and core network models from the ns–3 LTE module to enable full-stack simulations of end-toend connectivity, along with the simulation of real applications through the implementation of direct code execution. The module is demonstrated through several example simulations showing the performance of our custom mmWave stack as well as custom congestion control algorithms, specifically designed for efficient utilization of the mmWave channel. R EFERENCES [1] F. Khan and Z. Pi, “An introduction to millimeter-wave mobile broadband systems,” IEEE Commun. Mag., vol. 49, no. 6, pp. 101 – 107, Jun. 2011. [2] T. S. Rappaport, S. Sun, R. Mayzus, H. Zhao, Y. Azar, K. Wang, G. N. Wong, J. K. Schulz, M. Samimi, and F. Gutierrez, “Millimeter Wave Mobile Communications for 5G Cellular: It Will Work!” IEEE Access, vol. 1, pp. 335–349, May 2013. [3] S. Rangan, T. S. Rappaport, and E. Erkip, “Millimeter-wave cellular wireless networks: Potentials and challenges,” Proc. IEEE, vol. 102, no. 3, pp. 366–385, Mar. 2014. [4] F. Boccardi, R. W. Heath Jr, A. Lozano, T. L. Marzetta, and P. Popovski, “Five disruptive technology directions for 5G,” IEEE Commun. Mag., vol. 52, no. 2, pp. 74–80, Feb. 2014. [5] T. S. Rappaport, R. W. Heath Jr., R. C. Daniels, and J. N. Murdock, Millimeter Wave Wireless Communications. Pearson Education, 2014. [6] 3GPP, “TR 38.913, Study on Scenarios and Requirements for Next Generation Access Technologies, V14.1.0,” 2017. [7] H. Shokri-Ghadikolaei, C. Fischione, G. Fodor, P. Popovski, and M. Zorzi, “Millimeter wave cellular networks: A MAC layer perspective,” IEEE Trans. Comm., vol. 63, no. 10, pp. 3437–3458, Oct. 2015. [8] Y. Niu, Y. Li, D. Jin, L. Su, and A. V. Vasilakos, “A survey of millimeter wave communications (mmWave) for 5G: opportunities and challenges,” Wireless Networks, vol. 21, no. 8, pp. 2657–2676, Nov. 2015. [9] M. Zhang, M. Mezzavilla, R. Ford, S. Rangan, S. Panwar, E. Mellios, D. Kong, A. Nix, and M. Zorzi, “Transport layer performance in 5G mmWave cellular,” in IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), April 2016, pp. 730–735. [10] K. Allen et al., Building penetration loss measurements at 900 MHz, 11.4 GHz, and 28.8 MHz, ser. NTIA report – 94-306. Boulder, CO: U.S. Dept. of Commerce, National Telecommunications and Information Administration, 1994. [11] S. Singh, F. Ziliotto, U. Madhow, E. M. Belding, and M. J. W. Rodwell, “Millimeter Wave WPAN: Cross-Layer Modeling and Multi-Hop Architecture,” in 26th IEEE International Conference on Computer Communications (INFOCOM), May 2007, pp. 2336–2340. [12] J. S. Lu, D. Steinbach, P. Cabrol, and P. Pietraski, “Modeling human blockers in millimeter wave radio links,” ZTE Communications, vol. 10, no. 4, pp. 23–28, Dec. 2012. [13] A. Ghosh, T. A. Thomas, M. C. Cudak, R. Ratasuk, P. Moorut, F. W. Vook, T. S. Rappaport, G. MacCartney, S. Sun, and S. Nie, “Millimeter wave enhanced local area systems: A high data rate approach for future wireless networks,” IEEE J. Sel. Areas Commun., vol. 32, no. 6, pp. 1152–1163, June 2014. [14] M. Polese, M. Giordani, M. Mezzavilla, S. Rangan, and M. Zorzi, “Improved Handover Through Dual Connectivity in 5G mmWave Mobile Networks,” IEEE Journal on Selected Areas in Communications, vol. 35, no. 9, pp. 2069–2084, Sept 2017. [15] M. Giordani, M. Mezzavilla, S. Rangan, M. Zorzi, “An Efficient Uplink Multi-Connectivity Scheme for 5G mmWave Control Plane Applications,” Submitted to IEEE Transaction on Wireless Communications (TWC), 2017. [Online]. Available: https://arxiv.org/abs/1610.04836 [16] F. B. Tesema, A. Awada, I. Viering, M. Simsek, and G. P. Fettweis, “Mobility modeling and performance evaluation of multi-connectivity in 5G intra-frequency networks,” in IEEE Globecom Workshops (GC Wkshps), Dec. 2015. [17] M. Zhang, M. Mezzavilla, J. Zhu, S. Rangan, and S. Panwar, “The bufferbloat problem over intermittent multi-Gbps mmwave links,” arXiv:1611.02117 [cs.NI]., Nov. 2016. [18] M. Polese, R. Jana, and M. Zorzi, “TCP in 5G mmWave Networks: Link Level Retransmissions and MP-TCP,” in IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, 2017. [19] M. Polese, R. Jana, and M. Zorzi, “TCP and MP-TCP in 5G mmWave Networks,” IEEE Internet Computing, vol. 21, no. 5, pp. 12–19, Sep. 2017. [20] R. Ford, M. Zhang, M. Mezzavilla, S. Dutta, S. Rangan, and M. Zorzi, “Achieving Ultra-Low Latency in 5G Millimeter Wave Cellular Networks,” IEEE Communications Magazine, vol. 55, no. 3, pp. 196–203, March 2017. [21] R. Ford, A. Sridharan, R. Margolies, R. Jana, and S. Rangan, “Provisioning Low Latency, Resilient Mobile Edge Clouds for 5G,” in IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, 2017. [22] T. R. Henderson, M. Lacage, G. F. Riley, C. Dowell, and J. Kopena, “Network simulations with the ns-3 simulator,” SIGCOMM demonstration, vol. 14, no. 14, p. 527, 2008. [23] N. Baldo, M. Miozzo, M. Requena-Esteso, and J. Nin-Guerrero, “An open source product-oriented lte network simulator based on ns-3,” in Proceedings of the 14th ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, 2011, pp. 293–298. [Online]. Available: http://doi.acm.org/10.1145/ 2068897.2068948 [24] “LTE-EPC Network Simulator,” Available at http://iptechwiki.cttc.es/ LTE-EPC_Network_Simulator_(LENA), Feb. 2012. 23 [25] NYU WIRELESS, University of Padova, “ns-3 module for simulating mmwave-based cellular systems,” Available at https://github.com/nyuwireless-unipd/ns3-mmwave. [26] M. Mezzavilla, S. Dutta, M. Zhang, M. R. Akdeniz, and S. Rangan, “5G mmWave Module for the ns-3 Network Simulator,” in Proceedings of the 18th ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, 2015, pp. 283–290. [Online]. Available: http://doi.acm.org/10.1145/2811587.2811619 [27] R. Ford, M. Zhang, S. Dutta, M. Mezzavilla, S. Rangan, and M. Zorzi, “A Framework for End-to-End Evaluation of 5G mmWave Cellular Networks in ns-3,” in Proceedings of the Workshop on ns-3. New York, NY, USA: ACM, 2016, pp. 85–92. [Online]. Available: http://doi.acm.org/10.1145/2915371.2915380 [28] M. Zhang, M. Polese, M. Mezzavilla, S. Rangan, and M. Zorzi, “ns-3 Implementation of the 3GPP MIMO Channel Model for Frequency Spectrum Above 6 GHz,” in Proceedings of the Workshop on ns-3. New York, NY, USA: ACM, 2017, pp. 71–78. [Online]. Available: http://doi.acm.org/10.1145/3067665.3067678 [29] M. Polese, M. Mezzavilla, and M. Zorzi, “Performance Comparison of Dual Connectivity and Hard Handover for LTE-5G Tight Integration,” in Proceedings of the 9th EAI International Conference on Simulation Tools and Techniques, ser. SIMUTOOLS’16, 2016, pp. 118–123. [Online]. Available: http://dl.acm.org/citation.cfm?id= 3021426.3021445 [30] 3GPP, “TR 38.900, Study on channel model for frequency spectrum above 6 GHz, V14.2.0,” 2017. [31] M. Akdeniz, Y. Liu, M. Samimi, S. Sun, S. Rangan, T. Rappaport, and E. Erkip, “Millimeter wave channel modeling and cellular capacity evaluation,” IEEE J. Sel. Areas Commun., vol. 32, no. 6, pp. 1164– 1179, June 2014. [32] T. S. Rappaport, Wireless Communications: Principles and Practice, 2nd ed. Upper Saddle River, NJ: Prentice Hall, 2002. [33] H. Zhao, R. Mayzus, S. Sun, M. Samimi, J. K. Schulz, Y. Azar, K. Wang, G. N. Wong, F. Gutierrez, and T. S. Rappaport, “28 GHz millimeter wave cellular communication measurements for reflection and penetration loss in and around buildings in New York City,” in Proc. IEEE ICC, 2013. [34] G. R. MacCartney, S. Deng, S. Sun, and T. S. Rappaport, “MillimeterWave Human Blockage at 73 GHz with a Simple Double Knife-Edge Diffraction Model and Extension for Directional Antennas,” in Proc. IEEE 81st Vehicular Technology Conference, Sep. 2016. [35] C. N. Barati, S. A. Hosseini, S. Rangan, P. Liu, T. Korakis, S. S. Panwar, and T. S. Rappaport, “Directional cell discovery in millimeter wave cellular networks,” IEEE Transactions on Wireless Communications, vol. 14, no. 12, pp. 6664–6678, Dec. 2015. [36] T. Azzino, M. Drago, M. Polese, A. Zanella, and M. Zorzi, “X-TCP: a cross layer approach for TCP uplink flows in mmwave networks,” in 16th Annual Mediterranean Ad Hoc Networking Workshop (Med-HocNet), June 2017. [37] M. Polese, M. Zhang, M. Mezzavilla, J. Zhu, S. Rangan, S. Panwar, and M. Zorzi, “milliProxy: a TCP Proxy Architecture for 5G mmWave Cellular Systems,” in Asilomar Conference on Signals, Systems and Computers, Oct. 2017. [38] M. Polese, M. Mezzavilla, S. Rangan, and M. Zorzi, “Mobility Management for TCP in mmWave Networks,” in Proceedings of the 1st ACM Workshop on Millimeter-Wave Networks and Sensing Systems, co-located with ACM MobiCom’17, ser. mmNets ’17, 2017, pp. 11– 16. [39] H. Assasa and J. Widmer, “Implementation and Evaluation of a WLAN IEEE 802.11 ad Model in ns-3,” in Proceedings of the Workshop on ns-3. ACM, 2016, pp. 57–64. [40] H. Assasa and J. Widmer, “Extending the IEEE 802.11 ad Model: Scheduled Access, Spatial Reuse, Clustering, and Relaying,” in Proceedings of the Workshop on ns-3. ACM, 2017, pp. 39–46. [41] T. Kim, J. Park, J.-Y. Seol, S. Jeong, J. Cho, and W. Roh, “Tens of Gbps support with mmWave beamforming systems for next generation communications,” in IEEE Global Communications Conference (GLOBECOM), Dec. 2013, pp. 3685–3690. [42] K. Zheng, L. Zhao, J. Mei, M. Dohler, W. Xiang, and Y. Peng, “10 Gb/s HetSNets with Millimeter-Wave Communications: Access and Networking - Challenges and Protocols,” IEEE Communications Magazine, vol. 53, no. 1, pp. 222–231, Jan. 2015. [43] C. Dehos, J. L. Gonzalez, A. D. Domenico, D. Ktnas, and L. Dussopt, “Millimeter-wave access and backhauling: the solution to the exponential data traffic increase in 5G mobile communications systems?” IEEE Communications Magazine, vol. 52, no. 9, pp. 88–95, Sep. 2014. [44] R. Taori and A. Sridharan, “Point-to-multipoint in-band mmwave backhaul for 5G networks,” IEEE Communications Magazine, vol. 53, no. 1, pp. 195–201, Jan. 2015. [45] T. R. Henderson, S. Roy, S. Floyd, and G. F. Riley, “ns-3 project goals,” in Proceeding of the 2006 workshop on ns-2: the IP network simulator. ACM, 2006, p. 13. [46] M. Casoni, C. A. Grazia, M. Klapez, and N. Patriciello, “Implementation and validation of TCP options and congestion control algorithms for ns-3,” in Proceedings of the 2015 Workshop on ns-3. ACM, 2015, pp. 112–119. [47] G. Pei and T. Henderson, “Validation of ns-3 802.11 b PHY model,” Technical Report, 2009. [Online]. Available: https://www.nsnam.org/ ∼pei/80211b.pdf [48] J. Farooq and T. Turletti, “An IEEE 802.16 WiMAX Module for the ns-3 Simulator,” in Proceedings of the 2nd International Conference on Simulation Tools and Techniques, 2009, pp. 8:1–8:11. [Online]. Available: http://dx.doi.org/10.4108/ICST.SIMUTOOLS2009.5644 [49] H. Narra, Y. Cheng, E. K. Çetinkaya, J. P. Rohrer, and J. P. G. Sterbenz, “Destination-sequenced Distance Vector (DSDV) Routing Protocol Implementation in ns-3,” in Proceedings of the 4th International ICST Conference on Simulation Tools and Techniques, 2011, pp. 439–446. [Online]. Available: http://dl.acm.org/citation.cfm? id=2151054.2151132 [50] H. Tazaki, F. Uarbani, E. Mancini, M. Lacage, D. Camara, T. Turletti, and W. Dabbous, “Direct Code Execution: Revisiting Library OS Architecture for Reproducible Network Experiments,” in Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies, ser. CoNEXT ’13. New York, NY, USA: ACM, 2013, pp. 217–228. [Online]. Available: http://doi.acm.org/10.1145/2535372.2535374 [51] Centre Tecnologic de Telecomunicacions de Catalunya (CTTC), “The LENA ns-3 LTE Module Documentation,” Available at http://iptechwiki.cttc.es/ LTE-EPC_Network_Simulator_(LENA), Jan 2014. [52] 3GPP, “TR 25.996, Spatial channel model for multiple input multiple output (MIMO) simulations, V6.1.0,” 2003. [53] “Winprop software,” Available at http://www.awecommunications.com/Products/. [54] R. N. Almesaeed, A. S. Ameen, E. Mellios, A. Doufexi, and A. Nix, “3D Channel Models: Principles, Characteristics, and System Implications,” IEEE Communications Magazine, vol. 55, no. 4, pp. 152–159, April 2017. [55] S. Jaeckel, L. Raschkowski, K. Borner, and L. Thiele, “QuaDRiGa: A 3-D Multi-Cell Channel Model With Time Evolution for Enabling Virtual Field Trials,” IEEE Transactions on Antennas and Propagation, vol. 62, no. 6, pp. 3242–3256, Mar. 2014. [56] S. Sun, G. R. MacCartney, Jr., and T. S. Rappaport, “A novel millimeter-wave channel simulator and applications for 5G wireless communications,” in IEEE International Conference on Communications (ICC), May 2017, pp. 1–7. [57] M. K. Samimi and T. S. Rappaport, “3-D millimeter-wave statistical channel model for 5G wireless system design,” IEEE Transactions on Microwave Theory and Techniques, vol. 64, no. 7, pp. 2207–2225, July 2016. [58] NYU Wireless, “NYUSIM: The Open Source 5G Channel Model Simulator software,” 2016. [Online]. Available: http://wireless. engineering.nyu.edu/5g-millimeter-wave-channel-modeling-software/ [59] M. Giordani, M. Mezzavilla, A. Dhananjay, S. Rangan, and M. Zorzi, “Channel Dynamics and SNR Tracking in Millimeter Wave Cellular Systems,” in 22th European Wireless Conference, May 2016, pp. 1–8. [60] P. A. Eliasi, S. Rangan, and T. S. Rappaport, “Low-rank spatial channel estimation for millimeter wave cellular systems,” CoRR, vol. abs/1410.4831, 2014. [Online]. Available: http://arxiv.org/abs/1410. 4831 [61] D. J. Love and R. W. Heath, “Equal gain transmission in multiple-input multiple-output wireless systems,” IEEE Transactions on Communications, vol. 51, no. 7, pp. 1102–1110, July 2003. [62] J. H. Wilkinson, The Algebraic Eigenvalue Problem. Clarendon Press Oxford, 1965, vol. 87. [63] J. Wang, “Beam codebook based beamforming protocol for multi-Gbps millimeter-wave WPAN systems,” IEEE Journal on Selected Areas in Communications, vol. 27, no. 8, Sep. 2009. [64] R. W. Heath, N. Gonzalez-Prelcic, S. Rangan, W. Roh, and A. M. Sayeed, “An Overview of Signal Processing Techniques for Millimeter Wave MIMO Systems,” IEEE Journal of Selected Topics in Signal Processing, vol. 10, no. 3, pp. 436–453, Apr. 2016. 24 [65] M. Rebato, J. Park, P. Popovski, E. D. Carvalho, and M. Zorzi, [89] P. Imputato and S. Avallone, “Design and implementation of the traffic “Stochastic Geometric Coverage Analysis in mmWave Cellular Netcontrol module in ns-3,” in Proceedings of the Workshop on ns-3. works with a Realistic Channel Model,” in GLOBECOM 2017 - 2017 ACM, 2016, pp. 1–8. IEEE Global Communications Conference, Dec 2017. [90] P. Imputato and S. Avallone, “Traffic differentiation and multiqueue [66] M. Rebato, M. Mezzavilla, S. Rangan, F. Boccardi, and M. Zorzi, networking in ns-3,” in Proceedings of the Workshop on ns-3. ACM, “Understanding Noise and Interference Regimes in 5G Millimeter2017, pp. 79–86. Wave Cellular Networks,” in 22th European Wireless Conference, 2016, [91] A. Deepak, K. Shravya, and M. P. Tahiliani, “Design and implementapp. 1–5. tion of aqm evaluation suite for ns-3,” in Proceedings of the Workshop [67] M. Mezzavilla, M. Miozzo, M. Rossi, N. Baldo, and M. Zorzi, “A on ns-3. ACM, 2017, pp. 87–94. Lightweight and Accurate Link Abstraction Model for the Simulation [92] J. Gettys and K. Nichols, “Bufferbloat: Dark buffers in the internet,” of LTE Networks in ns-3,” in Proceedings of the 15th ACM InternaACM Queue, vol. 9, no. 11, pp. 40:40–40:54, Nov 2011. tional Conference on Modeling, Analysis and Simulation of Wireless [93] S. Floyd and V. Jacobson, “Random early detection gateways for and Mobile Systems, ser. MSWiM ’12, 2012, pp. 55–60. congestion avoidance,” IEEE/ACM Transactions on networking, vol. 1, [68] “The Vienna LTE Simulators,” Available at no. 4, pp. 397–413, Aug. 1993. https://www.nt.tuwien.ac.at/research/mobile-communications/vccs/vienna-lte-a-simulators/lte-advanced-link-level-si [94] T. J. Ott, T. Lakshman, and L. H. Wong, “SRED: stabilized RED,” [69] Z. Pi and F. Khan, “System design and network architecture for in INFOCOM’99. Eighteenth Annual Joint Conference of the IEEE a millimeter-wave mobile broadband MMB system,” in Proc. IEEE Computer and Communications Societies, vol. 3. IEEE, 1999, pp. Sarnoff Symposium, May 2011. 1346–1355. [70] A. Ghosh, T. A. Thomasa, M. C. Cudak, R. Ratasuk, P. Moorut, [95] K. Nichols and V. Jacobson, “Controlling queue delay,” CommunicaF. W. Vook, T. S. Rappaport, G. R. MacCartney, S. Sun, and S. Nie, tions of the ACM, vol. 55, no. 7, pp. 42–50, July 2012. “Millimeter-wave enhanced local area systems: A high-data-rate ap[96] S. Chandrashekar, A. Maeder, C. Sartori, T. Höhne, B. Vejlgaard, and proach for future wireless networks,” IEEE J. Sel. Areas in Comm., D. Chandramouli, “5G multi-RAT multi-connectivity architecture,” in vol. 32, no. 6, pp. 1152–1163, June 2014. IEEE International Conference on Communications Workshops (ICC), [71] Korea Telecom (KT) 5G-SIG, “5G.201, KT 5th Generation Radio May 2016, pp. 180–186. Access; Physical Layer; General description (5G pre-specifications),” [97] J. G. Rois, B. Lorenzo, F. J. Gonzalez-Castano, and J. C. Burguillo, 2016. “Heterogeneous millimeter-wave/micro-wave architecture for 5G wire[72] M. Cudak, T. Kovarik, T. A. Thomas, A. Ghosh, Y. Kishiyama, less access and backhauling,” in European Conference on Networks and T. Nakamura, “Experimental mmWave 5G cellular system,” in and Communications (EuCNC), June 2016, pp. 179–184. Globecom Workshops (GC Wkshps), 2014. IEEE, 2014, pp. 377–381. [98] A. Osseiran, F. Boccardi, V. Braun, K. Kusume, P. Marsch, M. Mater[73] T. Levanen, J. Pirskanen, and M. Valkama, “Radio interface design for nia, O. Queseth, M. Schellmann, H. Schotten, H. Taoka et al., “Sceultra-low latency millimeter-wave communications in 5G era,” in Proc. narios for 5G mobile and wireless communications: the vision of the IEEE Globecom Workshops (Gc Wkshps), Dec. 2014, pp. 1420–1426. METIS project,” IEEE Commun. Mag., vol. 52, no. 5, pp. 26–35, May [74] S. Dutta, M. Mezzavilla, R. Ford, M. Zhang, S. Rangan, and M. Zorzi, 2014. “Frame structure design and analysis for millimeter wave cellular [99] AT&T, “Migration and Interworking Aspects - SA WG2 Temporary systems,” IEEE Transactions on Wireless Communications, vol. 16, Document S2-163348,” 2016. [Online]. Available: http://www.3gpp. no. 3, pp. 1508–1522, Mar. 2017. org/ftp/tsg sa/WG2 Arch/TSGS2 116 Vienna/Docs/S2-163348.zip [75] S. Dutta, M. Mezzavilla, R. Ford, M. Zhang, S. Rangan, and M. Zorzi, [100] 3GPP, “TR 36.842, study on small cell enhancements for E-UTRA and “MAC layer frame design for millimeter wave cellular system,” in E-UTRAN, v12.0.0,” 2013. European Conference on Networks and Communications (EuCNC), [101] I. D. Silva, G. Mildh, J. Rune, P. Wallentin, J. Vikberg, P. SchliwaJune 2016, pp. 117–121. Bertling, and R. Fan, “Tight Integration of New 5G Air Interface and [76] D. Astély, E. Dahlman, A. Furuskär, Y. Jading, M. Lindström, and LTE to Fulfill 5G Requirements,” in IEEE 81st Vehicular Technology S. Parkvall, “LTE: the evolution of mobile broadband,” IEEE CommuConference (VTC Spring), May 2015. nications magazine, vol. 47, no. 4, May 2009. [102] B. Nguyen, A. Banerjee, V. Gopalakrishnan, S. Kasera, S. Lee, [77] Verizon, “5G TF; Air Interface Working Group; Verizon 5th A. Shaikh, and J. Van der Merwe, “Towards Understanding TCP Generation Radio Access; Physical channels and modulation (Release Performance on LTE/EPC Mobile Networks,” in Proceedings of 1),” 2016. [Online]. Available: http://www.5gtf.net/V5G 211 v1p7.pdf the 4th Workshop on All Things Cellular: Operations, Applications, [78] 3GPP, “TR 38.802, Study on New Radio Access Technology - Physical and Challenges, ser. AllThingsCellular ’14. New York, NY, USA: Layer Aspects, V14.0.0,” 2017. ACM, 2014, pp. 41–46. [Online]. Available: http://doi.acm.org/10. [79] ITU-R WP5D, Working Document Toward Preliminary Draft New 1145/2627585.2627594 Recommendation ITU-R M.[IMT.VISION], “Framework and overall objectives of the future development of IMT for 2020 and beyond,” [103] R. Ford, “Low Latency Fifth-Generation Cellular Networks,” Ph.D. dissertation, Polytechnic Institute of New York University, 2017. 2014. [Online]. Available: https://www.itu.int/dms pubrec/itu-r/rec/m/ [104] Z. Pi and F. Khan, “A millimeter-wave massive MIMO system for next R-REC-M.2083-0-201509-I!!PDF-E.pdf generation mobile broadband,” in Proc. 46th Asilomar Conference on [80] P. Popovski, V. Brau, H.-P. Mayer, P. Fertl, Z. Ren, D. GonzalesSignals, Systems and Computers (ASILOMAR), Nov. 2012, pp. 693– Serrano, E. G. Ström, T. Svensson, H. Taoka, P. Agyapong et al., “EU 698. FP7 INFSO-ICT-317669 METIS, D1. 1 scenarios, requirements and [105] M. Drago, T. Azzino, M. Polese, C. Stefanovic, and M. Zorzi, KPIs for 5G mobile and wireless system,” 2013. “Reliable Video Streaming over mmWave with Multi Connectivity and [81] P. Kela, M. Costa, J. Salmi, K. Leppanen, J. Turkka, T. Hiltunen, and Network Coding,” in 2018 International Conference on Computing, M. Hronec, “A novel radio frame structure for 5G dense outdoor radio Networking and Communications (ICNC), March 2018. [Online]. access networks,” in Proc. IEEE 81st Vehicular Technology Conference Available: https://arxiv.org/abs/1711.06154 (VTC Spring), May 2015, pp. 1–6. [82] S. Choi and K. G. Shin, “A class of adaptive hybrid ARQ schemes for [106] P. Jimenez Mateo, “Analysis of TCP Performance in 5G mm-wave Mobile Networks,” Master Thesis, Universidad Carlos III de Madrid, wireless links,” IEEE Transactions on Vehicular Technology, vol. 50, Spain, 2017. no. 3, pp. 777–790, May 2001. [83] 3GPP, “TS 36.321, Medium Access Control (MAC) protocol specifi- [107] A. Kassler and M. Pieskä, “TCP Performance over 5G mmWave LinksTradeoff between Capacity and Latency,” in The 13th IEEE Internacation, V14.0.0,” 2016. tional Conference on Wireless and Mobile Computing, Networking and [84] S. Sesia, M. Baker, and I. Toufik, LTE-the UMTS long term evolution: Communications, 9.11 th October 2017 Rome Italy, 2017, pp. 385–394. from theory to practice. John Wiley & Sons, 2011. [108] M. Kim, S.-W. Ko, and S.-L. Kim, “Enhancing TCP End-to-End [85] E. Dahlman, S. Parkvall, J. Sköld, and P. Beming, 4G LTE/LTEPerformance in Millimeter-Wave Communications,” arXiv preprint Advanced for Mobile Broadband. Oxford, UK: Academic Press, 201. arXiv:1709.00717, 2017. [86] 3GPP, “TR 23.203, Technical Specification Group Services and System [109] A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, “Architectural Aspects; Policy and charging control architecture, V14.2.0,” 2017. Guidelines for Multipath TCP Development,” RFC 6182, 2011. [87] G. Abbas, Z. Halim, and Z. H. Abbas, “Fairness-driven queue management: A survey and taxonomy,” IEEE Communications Surveys [110] B. Chihani and C. Denis, “A Multipath TCP model for ns-3 simulator,” in Workshop on ns-3 held in conjunction with SIMUTools 2011, 2011. Tutorials, vol. 18, no. 1, pp. 324–367, Jan. 2016. [88] F. Baker and G. Fairhurst, “IETF recommendations regarding Active [111] M. Coudron and S. Secci, “An implementation of multipath TCP in Queue Management,” RFC 7567, 2015. ns-3,” Computer Networks, vol. 116, pp. 1–11, 2017. 25 [112] C. Paasch, S. Barre, and al., “Multipath TCP in the Linux Kernel,” available at http://www.multipath-tcp.org. [113] “Iperf 2.0,” available at https://iperf.fr. [114] H. Tazaki, R. Nakamura, and Y. Sekiya, “Library operating system with mainline linux network stack,” in Proceedings of Netdev 0.1, 2015. [115] Q. Peng, A. Walid, J. Hwang, and S. H. Low, “Multipath TCP: Analysis, Design, and Implementation,” IEEE/ACM Transactions on Networking, vol. 24, no. 1, pp. 596–609, Feb 2016. [116] O. Semiari, W. Saad, and M. Bennis, “Joint millimeter wave and microwave resources allocation in cellular networks with dual-mode base stations,” IEEE Transactions on Wireless Communications, vol. 16, no. 7, pp. 4802–4816, July 2017. [117] O. Semiari, W. Saad, and M. Bennis, “Downlink cell association and load balancing for joint millimeter wave-microwave cellular networks,” in IEEE Global Communications Conference (GLOBECOM), Dec. 2016, pp. 1–6. [118] M. E. Rasekh, Z. Marzi, Y. Zhu, U. Madhow, and H. Zheng, “Noncoherent mmwave path tracking,” in Proceedings of the 18th International Workshop on Mobile Computing Systems and Applications. New York, NY, USA: ACM, 2017, pp. 13–18. [Online]. Available: http://doi.acm.org/10.1145/3032970.3032974 [119] O. Semiari, W. Saad, M. Bennis, and B. Maham, “Caching Meets Millimeter Wave Communications for Enhanced Mobility Management in 5G Networks,” IEEE Transactions on Wireless Communications, vol. PP, no. 99, pp. 1–1, 2017. [120] J. Palacios, D. D. Donno, and J. Widmer, “Tracking mm-Wave channel dynamics: Fast beam training strategies under mobility,” in 2017 IEEE Conference on Computer Communications (INFOCOM), May 2017, pp. 1–9. [121] Z. Khan, H. Ahmadi, E. Hossain, M. Coupechoux, L. A. DaSilva, and J. J. Lehtomäki, “Carrier aggregation/channel bonding in next generation cellular networks: Methods and challenges,” IEEE Network, vol. 28, no. 6, pp. 34–40, Nov.-Dec. 2014. [122] Y. Hou, Z. Wen’an, L. Song, and M. Gao, “A QoE Estimation Model for Video Streaming over 5G Millimeter Wave Network,” in International Conference on Broadband and Wireless Computing, Communication and Applications. Springer, 2016, pp. 93–104. [123] Á. Drozdy, J. Kapanen, and J. Manner, “User level performance analysis of multi-hop in-band backhaul for 5G,” Wireless Networks, Apr 2017. [Online]. Available: https://doi.org/10.1007/s11276-017-1513-2 [124] A. Tassi, M. Egan, R. J. Piechocki, and A. Nix, “Modeling and design of millimeter-wave networks for highway vehicular communication,” IEEE Transactions on Vehicular Technology, vol. 66, no. 12, pp. 10 676–10 691, Dec. 2017. [125] I. Mavromatis, A. Tassi, R. J. Piechocki, and A. R. Nix, “MmWave System for Future ITS: A MAC-layer Approach for V2X Beam Steering,” to appear in Proc. IEEE VTC-Fall, 2017. [Online]. Available: http://arxiv.org/abs/1705.08684 [126] I. Mavromatis, A. Tassi, R. J. Piechocki, and A. R. Nix, “Beam alignment for millimetre wave links with motion prediction of autonomous vehicles,” Proc. of IET Colloquium on Antennas, Propagation & RF Technology for Transport and Autonomous Platforms, 2017. [127] M. Mezzavilla, M. Polese, A. Zanella, A. Dhananjay, S. Rangan, C. Kessler, T. T. Rappaport, and M. Zorzi, “Public safety communications above 6 ghz: Challenges and opportunities,” IEEE Access, vol. PP, no. 99, pp. 1–1, Nov. 2017. [128] “AWS Storage Services Overview,” Available at https://goo.gl/h2yJje.