Modulation, Pre-Equalization and Pulse Shaping

for
PCM Voiceband Channels

Nader Sheikholeslami Alagha

Department of Electrical and Computer Engineering

McGill University
Montreal, Canada

December 2001

A thesis submitted to the Faculty of Graduate Studies and Research in partial fulfillment of the
requirements for the degree of Doctor of Philosophy.

c 2001 Nader Sheikholeslami Alagha
 i

Abstract

For the past few decades, analog voiceband modems have been used extensively to carry digi-
tal information over the Public Switched Telephone Network (PSTN). Conventional voiceband
modems treat the PSTN as an analog communication channel. However, today’s PSTN is mostly
a digital network except for the basic telephone services provided via analog subscriber lines.
The conventional model of analog voiceband channels is no longer adequate to characterize the
physical connection between terminals with direct digital access to the network and voiceband
modems connected to analog subscriber lines. Such a connection requires a different model in
each direction. There are now international modem standards which support rates of up to 56
kbits/s for the down-stream channel.
This dissertation examines the more challenging part of the voiceband communication channel,
i.e., the up-stream direction connecting an analog subscriber to the digital network. The major
source of distortion on the up-stream channel is quantization error caused by analog-to-digital
conversion performed as part of the encoding to Pulse-Code Modulation (PCM) process. A
bandpass filter prior to the PCM encoder restricts the bandwidth while the sampling rate of the
PCM encoder is predetermined by the network. Signalling in the presence of such constraints
lead to theoretical problems as well as practical concerns in modem design.
Communication models that characterize PCM voiceband channels are developed. We inves-
tigate modulation design and related issues including index mapping, constellation design and
constellation probability assignment to match the pre-determined structure of the detector at the
receiver, i.e., the PCM encoder at the central office.
We develop a framework for transmitter structures that can avoid or reduce Inter-Symbol
Interference (ISI) at the receiver in order to sidestep the limited bandwidth of the up-stream
channels and the fixed sampling rate of the up-stream channel. Techniques employed include
linear filtering, spectral shaping and precoding to reduce the ISI, while limiting the average
transmitted signal power. A filterbank structure for pre-equalizing channels with spectral nulls is
also described.
A new method for pulse shaping design is proposed. The new pulse shaping filters provide a
compatible design that can be used for the up-stream PCM channel as well as to the cascade of
the up-stream and the down-stream channels.
Compared to conventional modem design, the proposed modulation and pre-equalization tech-
niques together allow for an increase of the data transmission rates in the up-stream direction of
up to 50%.
 ii

Résumé

Depuis quelques dizaines d’années, les modems téléphoniques analogiques servent dans une large
mesure à acheminer des informations numériques sur le réseau téléphonique public commuté
(RTPC). Les modems téléphoniques conventionnels traitent le RTPC comme une voie de com-
munication analogique. Or, le RTPC actuel est surtout un réseau numérique, hormis les services
téléphoniques de base fournis par le truchement des lignes d’abonnés analogiques. Le modèle
traditionnel de voie téléphonique analogique ne suffit plus à caractériser la connexion matérielle
entre les terminaux à accès numérique direct au réseau et les modems téléphoniques branchés
aux lignes d’abonnés analogiques. Une telle connexion nécessite un modèle différent dans chaque
direction. Il existe maintenant des recommandations internationales de modem fondées sur les
caractéristiques de la voie aval acheminant des débits pouvant atteindre 56 kbit/s.
Cet essai porte sur la portion la plus délicate de la voie de communication téléphonique, soit la
direction amont reliant l’abonné analogique au réseau numérique. La source majeure de distorsion
de la voie amont est l’erreur de quantification causée par la conversion analogique-numérique
réalisée au cours du codage en modulation par impulsions codées (MIC). Un filtre passe-bande
précédant le codeur MIC limite la largeur de bande de la voie alors que le taux d’échantillonnage
du codeur MIC est prédéterminé par le réseau. La signalisation en présence de telles limites pose
de nombreux problèmes théoriques et des difficultés pratiques dans la conception du modem.
Nous élaborons des modèles de communication pour caractériser les voies téléphoniques MIC.
La conception de la modulation et les enjeux associés, y compris la cartographie des indices, la
conception de la constellation et l’affectation de la probabilité de constellation, sont étudiées afin
de correspondre à la structure prédéterminée du détecteur au récepteur, c’est-à-dire le codeur
MIC au bureau central.
Pour contourner la largeur de bande limitée et le taux d’échantillonnage fixe de la voie amont,
nous créons un cadre de conception de l’émetteur qui évite ou réduit le brouillage intersymbole au
récepteur. Parmi les techniques utilisées, mentionnons la mise en forme spectrale et le précodage
afin de réduire le brouillage intersymbole tout en maintenant une limite sur la puissance moyenne
du signal émis. Nous proposons aussi une structure de banc de filtres afin de précorriger les voies
avec une phase non minimum ou des extinctions spectrales.
Nous proposons aussi une nouvelle méthode de conception de la mise en forme des impulsions.
Les nouveaux filtres de mise en forme des impulsions ont un design compatible qui peut être
appliqué à la voie MIC amont ainsi qu’à la cascade de voies amont et aval.
Comparées à la conception du modem conventionnel, les techniques de modulation et de
précorrection proposées permettent ensemble d’accroître jusqu’à 50% les débits de transmission
de données en amont.
 iii

Acknowledgments

I would like to express my sincere gratitude to my supervisor, Professor Peter Kabal for his patient
and friendly support during the course of my Ph.D. studies. His contributions to this thesis and
my education are highly appreciated.
The financial support of this research was mainly provided by Professor Kabal via a grant
from the Natural Sciences and Engineering Research Council of Canada. I would like to thank
Professor Kabal and recognize NSERC for this support.
My special thanks go to the members of the Telecommunications and Signal Processing Lab-
oratory including the present graduate students and others who have graduated from this lab in
the past few years. I have many good memories from the TSP Lab and have had many inspiring
conversations with other students. Professor Kabal has provided the leadership and set the tone
for such a friendly environment.
I would like to thank Sylviane Duval for her editorial comments regarding this dissertation.
I am greatly thankful to Paul Mercier and Martin Côté, my colleagues at EMS Technologies
Canada, for the French translation of the thesis abstract. My special thanks go to Marin Côté
for his generous support of my studies since the beginning of my work at EMS.
I am indebted to my parents for encouraging me to pursue my studies. I am thankful to my
father for being such a supportive friend in all stages of my life. I am grateful to my mother for
being such a caring person at home and an excellent teacher at school.
My journey towards graduate studies was inspired by many individuals: those scholars who
care so much for education and educating people to create a better society. Among them, my
special gratitudes go to Dr. Jalil Mostashari. Just remembering his words has helped me overcome
many difficulties in this journey. I would like to thank him for being such an inspiring figure in
my life.
Last but not least, I am greatly thankful to my dear wife, Firouzeh, for her unconditional
support of my studies in the past two years. I greatly appreciate her patience, care and encour-
agements.
 iv

Contents

1 Introduction 1
1.1 Data transmission over voiceband channels . . . . . . . . . . . . . . . . . . . . . . 1
1.2 PSTN: A hybrid communication network . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Our research objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5 Thesis organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 The PCM Voiceband Channel Characteristics 7

2.1 Subscriber access to the PSTN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Subscriber loop interface circuit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Subscriber loop characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3.1 Hybrid Circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.2 Characteristics of the filters in the line circuit . . . . . . . . . . . . . . . . . 11
2.3.3 PCM encoding and decoding . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.4 Non-uniform Quantizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.5 Multiplexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Power Constraints of the PCM channel . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5 Capacity of PCM voiceband channels . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.6 PCM Modems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.6.1 Digital network imperfections . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.7 The down-stream PCM channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.8 The up-stream PCM channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.8.1 Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.8.2 Signal mapping and constellation design . . . . . . . . . . . . . . . . . . . . 26
2.8.3 Channel Pre-equalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.8.4 Echo cancellation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.8.5 Channel Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Contents v

2.9 End-to-end PCM voiceband channels . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.9.1 Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.9.2 Channel estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.9.3 Channel Equalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3 Modulation Design for the Up-Stream PCM Channel 28

3.1 Modulation scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2 Optimal signal levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2.1 Performance criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2.2 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2.3 Performance results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2.4 How to choose detector thresholds . . . . . . . . . . . . . . . . . . . . . . . 41
3.2.5 Bit-to-symbol mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.3 Non-uniform probability distributions for constellation points . . . . . . . . . . . . 51
3.3.1 Maximizing the entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3.2 Examples of maximum entropy distributions for a PAM signal . . . . . . . 52
3.3.3 Channel capacity for non-uniform PAM modulation . . . . . . . . . . . . . 53
3.3.4 The Huffman algorithm for bit-to-symbol mapping . . . . . . . . . . . . . . 57
3.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4 Transmitter Design for the Up-stream PCM Channel 64

4.1 Optimal transmitter filter design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.1.1 Peak distortion criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.1.2 Mean-Square Error (MSE) criterion . . . . . . . . . . . . . . . . . . . . . . 68
4.2 Optimal transmitting filter design for the up-stream PCM channels . . . . . . . . . 71
4.2.1 Observations and remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.3 Spectrum shaping and filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3.1 Methods of Spectrum Shaping . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.3.2 Spectrum shaping by inserting redundant symbols . . . . . . . . . . . . . . 83
4.4 Precoding: combined filtering and spectral shaping . . . . . . . . . . . . . . . . . 92
4.4.1 Tomlinson-Harashima Precoding . . . . . . . . . . . . . . . . . . . . . . . . 92
4.4.2 Precoding design for the up-stream PCM channels . . . . . . . . . . . . . . 96
4.4.3 Linear block-by-block pre-filtering . . . . . . . . . . . . . . . . . . . . . . . 97
4.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Contents vi

5 Filterbank Structures for ISI Channel Pre-Equalization 104

5.1 Signal design based on non-uniform sampling . . . . . . . . . . . . . . . . . . . . . 104
5.1.1 Pre-equalizer design for the up-stream PCM channel . . . . . . . . . . . . . 108
5.2 Non-maximally decimated filterbanks . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.2.1 IIR implementation of the FIR filters . . . . . . . . . . . . . . . . . . . . . 113
5.2.2 Average transmitted power . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.3 Performance Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.4 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

6 A New Family of Pulse Shaping Filters 119

6.1 Pulse shaping filters for voiceband PCM channels . . . . . . . . . . . . . . . . . . . 119
6.2 Nyquist filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6.2.1 Nyquist Filters with compensated phase . . . . . . . . . . . . . . . . . . . . 122
6.3 Extension of Nyquist’s first criterion . . . . . . . . . . . . . . . . . . . . . . . . . . 123
6.3.1 Square-root Nyquist filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
6.3.2 Phase compensated square-root raised-cosine filter . . . . . . . . . . . . . . 124
6.3.3 Special Case: Full excess bandwidth . . . . . . . . . . . . . . . . . . . . . . 126
6.3.4 Eye pattern diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.3.5 SNR degradation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.4 Generalized Raised-Cosine Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
6.4.1 Nyquist filter design using phase compensation . . . . . . . . . . . . . . . . 132
6.5 Design examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
6.5.1 Nyquist filters with smoother spectra . . . . . . . . . . . . . . . . . . . . . 134
6.5.2 Nyquist filters with reduced ISI due to truncation . . . . . . . . . . . . . . 136
6.6 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

7 Concluding Remarks 141

7.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

A Optimal Linear Transmitter Structure 146

B Optimal MMSE Transmitting Filters 148

B.1 Transmitting filter with infinite length impulse response . . . . . . . . . . . . . . . 148
B.2 Finite Impulse Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

C Inner boundaries of the eye pattern for the square-root filter 152
Contents vii

D The impulse response of Hsqrt(f ) 154

References 155
 viii

List of Figures

1.1 Schematic graph network access via PSTN . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1 Subscriber loop access via line interface circuits at the central office. . . . . . . . . . . . 8
2.2 The T-equivalent circuit for a unit length segment of a subscriber line . . . . . . . . . . . 9
2.3 The magnitude response of subscriber lines with a 26-AWG cable. . . . . . . . . . . . . 10
2.4 Magnitude response of a loaded line . vs. an unload line . . . . . . . . . . . . . . . . . . 10
2.5 Measurement results of the magnitude response of several subscriber lines. . . . . . . . . 12
2.6 A circuit model of a hybrid coupler . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.7 Example of the frequency response of a send filter. . . . . . . . . . . . . . . . . . . . . 14
2.8 Frequency response of the overall send-receive filter is compared with Recommendation
G.712 requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.9 Empirical results as provided in Recommendation V.56. . . . . . . . . . . . . . . . . . . 16
2.10 Bit allocation in a µ-Law codeword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.11 A channel bank multiplexing 24 voice channels to a DS1 stream. . . . . . . . . . . . . . 20
2.12 Three types of PCM channels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.13 The Up-stream PCM Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.1 A schematic model for the Up-stream PCM channel. . . . . . . . . . . . . . . . . . . . . 28

3.2 In the up-stream PCM channel the ADC at the central office is interpreted as a detector . 30
3.3 Placement of the signal points relative to the thresholds . . . . . . . . . . . . . . . . . . 32
3.4 Two examples of conditional probabilities of the received signal. . . . . . . . . . . . . . . 35
3.5 Performance results of 32-PAM modulation designed for the up-stream PCM channels. . . 38
3.6 Performance results for non-uniform 64-PAM modulation designed for the up-stream PCM
channels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.7 Performance results for the non-uniform 128-PAM designed for the up-stream PCM channels. 40
3.8 Performance results for non-uniform PAM modulation with an average of −12 dBm0 signal
power. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.9 The average transmitted bit-rate as a function of symbols per frame on a PCM channel. . 43
List of Figures ix

3.10 The average Hamming distance between indices of adjacent points is shown as a function
of K for a 58-PAM modulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.11 Bit-to-symbol mapping based on hybrid natural-Gray encoding. . . . . . . . . . . . . . 47
3.12 Compare the average Hamming distance generated by the natural index mapping with that
generated by the Hybrid Gray-Natural index mapping . . . . . . . . . . . . . . . . . . . 49
3.13 Probability of bit error for a 58-PAM modulation over a PCM up-stream channel. . . . . 50
3.14 Probability distribution of the PAM symbols. The average signal power is −10 dBm0 . . . 54
3.15 Probability distribution of the PAM symbols. The average signal power is −12 dBm0. . . 54
3.16 Probability of symbol error for PAM constellations with a maximum entropy distribution
and uniform probability distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.17 Transmitted bit rates of PAM modulation over a PAM channel. . . . . . . . . . . . . . 55
3.18 Channel capacity of a Gaussian channel with a continuous input signal and a PAM-
modulated input signal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.19 Probability distributions of a constellation point with Huffman bit assignments compared
to the maximum entropy distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.20 Transmitted bit-rates as a function of SNR for three different probability distributions:
maximum entropy, the Huffman algorithm and uniform distributions. . . . . . . . . . . . 62

4.1 The general structure of an optimal transmitter filter. . . . . . . . . . . . . . . . . . . . 66

4.2 A discrete-time model of the up-stream PCM channel. . . . . . . . . . . . . . . . . . . 67
4.3 The magnitude response of the channel equivalent filter as discussed in Example 1. . . . . 72
4.4 The impulse response of the channel discussed in Example 1. . . . . . . . . . . . . . . . 72
4.5 Pole-zero plot of a channel filter of Example 1. . . . . . . . . . . . . . . . . . . . . . . 73
4.6 Signal-to-interference ratio as a function of the average transmitted power . . . . . . . . 73
4.7 Probability of symbol error for several PAM signals as a function of signal-to-interference
ratio. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.8 Probability of symbol error as a function of SNR. . . . . . . . . . . . . . . . . . . . . . 75
4.9 Channel characteristics in Example 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.10 Pole-zero plot of a channel filter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.11 Signal-to-ISI ratio as a function of the average transmitted power in Example 2. Compared
to Example 1, a significant increase in the average transmitted power is required to obtain
the same SIR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.12 Probability of symbol error for several PAM signals as a function of signal-to-interference
ratio. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.13 Probability of symbol error as a function of SNR. . . . . . . . . . . . . . . . . . . . . . 79
List of Figures x

4.14 Spectrum shaping cascaded with an optimal filtering to control MMSE while maintaining
the limit on the average transmitted power. . . . . . . . . . . . . . . . . . . . . . . . . 80
4.15 Spectrum shaping by inserting redundant symbols into a block of data symbols. . . . . . . 84
4.16 Power spectrum including spectral null . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.17 Transmitter design for a channel with spectral null at DC. . . . . . . . . . . . . . . . . . 88
4.18 Spectrum shaping reduces the average transmitted power. . . . . . . . . . . . . . . . . 90
4.19 Average transmitted power as a function of the block size for a 90-PAM signal. . . . . . . 91
4.20 Data transmission rate as a function of transmitted power . . . . . . . . . . . . . . . . 91
4.21 Tomlinson-Harashima Precoder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.22 A TH-precoder can be viewed as combined filtering and spectrum shaping. . . . . . . . . 94
4.23 A precoding followed by linear filter at the transmitter. . . . . . . . . . . . . . . . . . . 97
4.24 A block-by-block inverse filtering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.25 Redundancy insertion in a block of symbols to control the transmitted power. . . . . . . . 100
4.26 The average signal power as a function of the block size. . . . . . . . . . . . . . . . . . . 101
4.27 Comparing the power gain of two inverse filters. . . . . . . . . . . . . . . . . . . . . . . 102

5.1 Signal design for a bandlimited channel with non-uniform symbol timing distribution. . . 106
5.2 Signal design for a channel with W = 3500 Hz bandwidth and sampling rate of 1/Ts =8000
Hz. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.3 A non-maximally decimated structure as a channel pre-equalizer. . . . . . . . . . . . . . 109
5.4 A set of FIR filters can partially invert the channel filter. . . . . . . . . . . . . . . . . . 110
5.5 Power gain as a function of the effective symbol rate for H(z) = 1 − z −1 . . . . . . . . . 116
−2
5.6 Power gain as a function of the effective symbol rate for H(z) = 1 − z . . . . . . . . . 117
−1
5.7 Power gain as a function of the effective symbol rate for H(z) = (1+z )(1−ej0.8π z −1 )(1−
e−j0.8π z −1 ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

6.1 Pulse shaping filter design for PCM channels. . . . . . . . . . . . . . . . . . . . . . . . 121

6.2 Piece-wise linear phase compensator for square-root raised-cosine filter. . . . . . . . . . . 125
6.3 Normalized impulse responses of the square-root raised-cosine filter with α=1, (dashed
line). The phase compensated filter has a delayed impulse response (solid line). . . . . . . 127
6.4 Binary eye patterns for a raised-cosine filter and a square-root raised-cosine filter. . 129
6.5 Eye patterns (4 level PAM) for a raised-cosine filter and a square-root raised-cosine
filter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.6 Binary eye patterns for a raised-cosine filter and a phase compensated square-root
raised-cosine filter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.7 Polynomials Pn (x) to generate the generalized raised-cosine filter. . . . . . . . . . . . . . 136
List of Figures xi

6.8 Impulse responses of generalized raised-cosine filters generated by P1 (x) and P3 (x) are
compared. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.9 Inner boundaries of the eye diagram for several generalized raised-cosine filters with α = 1. 138
 xii

List of Tables

3.1 Huffman algorithm for bit-to-signal mapping. The bit sequence associated with each con-
stellation is presented. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

5.1 Comparison of decimation rates and the average filterbank power gain. . . . . . . . 115

6.1 Comparison of several generalized raised-cosine filters with α = 1. . . . . . . . . . . 135

6.2 Nyquist filter design to reduce ISI due to truncation. . . . . . . . . . . . . . . . . . 140
 xiii

Glossary
First appears on page
ADC Analog to Digital Converter 2
ADSL Asymmetric Digital Subscriber Line 1
AWGN Additive White Gaussian Noise 21
BER Bit Error Rate 45
CODEC Encoder and Decoder 17
CRTC Canadian Radio and Telecommunication Commission 19
DAC Digital-to-Analog Converter 11
DLC Digital Loop Carrier 8
FCC Federal Communication Commission 19
HGN Hybrid Gray-Natural (index mapping) 46
ICI Inter-Channel Interference 108
ISDN Integrated Service Digital Network 1
ISI Inter-Symbol Interference 22
ISP Internet Service Provider 2
(M)MSE (Minimum) Mean Square Error 67
PSTN Public Switched Telephone Network 1
PAM Pulse Amplitude Modulation 29
PSD Power Spectrum Density 68
SER Symbol Error Rate 37
SIR Signal to Interference (ISI) Ratio 71
SNR Signal to Noise Ratio 21
SQNR Signal to Quantization Noise Ratio 17
TH-precoder Tomlinson-Harashima precoder 92
 1

Chapter 1

Introduction

1.1 Data transmission over voiceband channels

For the past few decades, data transmission over the Public Switched Telephone Network (PSTN)
has been an active area of scientific research as well as product development. Although the PSTN
was originally designed for voice communications, its worldwide reach and relatively low service
cost have made it convenient to also use the PSTN for data communication.
Conventional telephone service provides a limited bandwidth to each user. The bandwidth
of a telephone channel is limited to frequencies ranging approximately from 200 Hz to 3600 Hz.
Apart from limited bandwidth, a telephone channel has other effects on the transmitted voiceband
signal such as non-linear phase and amplitude distortion, crosstalk from the signals present on
the adjacent wires, and signal echo. Several recommendations have been developed (or are being
developed) by international standards bodies to characterize the impairments of telephone circuits
and to provide guidelines for voiceband modem design.
Over the past few years, it has been repeatedly predicted that network access via telephone
lines would be replaced by new services based on emerging technologies. Despite all these predic-
tions, voiceband modems are still used by the majority of home computer users and small business
owners for data communications and network access. There is no doubt that the demand for more
reliable and efficient information access will bring changes to this situation. Several competing
technologies are available to provide the solution to the so-called “last mile problem”, that is, how
to connect customers premises to the broadband network in a more flexible and affordable way.
Services such as Integrated System Digital Network (ISDN) and Asymmetric Digital Subscriber
Line (ADSL) are two examples of digital data transmission over the twisted-pair subscriber lines.
Compared to voiceband modems, these technologies require investment by the telephone compa-
nies to install special equipment at the central offices. Depending on the distance between user
1 Introduction 2

premises and the serving central office, some users may not be able to use such services.
Other technologies for network access such as coaxial cable, wireless access, direct satellite
access and fiber optic lines are available. Technologies developed for each of these media have
shown merits for particular applications. However, there seems to be no universal technology for
all locations and all applications.
Among all access technologies, the “plain old telephone service” provides the widest access
coverage in the world. A voiceband modem connected to any phone line can reach millions of
other modems connected to the PSTN. Therefore, it is desirable to investigate new techniques to
reach the maximum achievable data transmission rates over voiceband channels.

1.2 PSTN: A hybrid communication network

In most parts of the world, the public switched telephone network has evolved towards a digital
communication network. The interconnections among central offices are based on fully digital
transmission via electrical cable, radio links, and increasingly, fiber optic cables. There are many
traffic sources such as Internet Service Providers (ISP) and medium to large size businesses which
have a direct digital connection to the PSTN. The only remaining analog links are the analog
subscriber loops providing subscribers with basic telephone services. Figure 1.1 shows a schematic
diagram of the telephone network. As shown in this figure, subscribers such as D1 or D2 use
technologies such as ISDN or even radio links for direct digital access to the PSTN, whereas
other telephone users are connected to the network via an analog subscriber loop. Since the
rest of the network carries only digital data, the analog signal received must be converted to a
stream of binary digits. The conversion scheme used at the central office is known as Pulse Code
Modulation (PCM) encoding. The analog signal is passed through a band-limited filter followed
by a sampler and subsequent Analog-to-Digital Converter (ADC). A non-uniform encoding rule
(µ-law or A-Law) is used to map each sample to an 8-bit codeword. The output of the PCM
encoder is a stream of bits constructed from codewords. To reconstruct the analog signal, the bit
stream is passed through a PCM decoder. The PCM encoding and decoding processes add signal
distortion. Although the distortion caused by the PCM encoding process is tolerable for voice
signals, it has a significant impact on the performance of voiceband modems. In fact, quantization
distortion is the predominant source of data transmission errors over voiceband PCM channels.
Until the early 1990s, it was widely believed that voiceband modems had reached their maxi-
mum theoretical limits in terms of data transmission rate. The theoretical limit was based on a
linearized model of the voiceband channels. In this model, a voiceband channel was essentially
viewed as an analog medium where the quantization distortion caused by the PCM encoding and
decoding process was treated as additive noise.
1 Introduction 3

U2 U1

Analog Subscriber Loop Analog Subscriber Loop

Central Office Digital Switching Central Office

C2 Network C1

Analog Subscriber Loop

Digital Link
Digital Link U3
(T1, E1, xDSL, ISDN, ...)

D1 D2

Fig. 1.1 Schematic graph network access via PSTN

However, the conventional model of voiceband channels is not adequate to describe the different
network access scenarios over the PSTN. Analog subscriber loops can be part of three different
types of connections. One type is a connection from a digital source (D1 in Fig. 1.1) to an analog
subscriber. A signal flowing in this direction is not distorted by quantization error since it does not
pass through any PCM encoder. We refer to the channel model associated with this connection as
the down-stream PCM channel. The second type of connection begins with an analog subscriber
and ends with a digital user. We call the corresponding channel the up-stream PCM channel.
This channel includes a PCM encoder with an analog input signal and a digital bit stream at the
output. The third type of connection starts from an analog subscriber, passes through a PCM
encoder and decoder and ends with another analog subscriber line. We refer to this channel as
an end-to-end PCM voiceband channel. Communication models describing these three types of
channels are different.
In a down-stream PCM channel, where the transmitted signal is not distorted by quantization
error at the central office, the maximum achievable information rate is higher than the 33 kbits/s
provided by state of the art analog voiceband modems. There are now international modem
standards for down-stream PCM channels that can support rates of up to 56 kbits/s.
For the up-stream PCM channel, the presence of analog-to-digital conversions as part of the
PCM encoder is a major source of signal distortion. Furthermore, the bandwidth of the up-stream
channel is limited by a bandpass filter ahead of the PCM encoder. The bandpass filter, followed by
1 Introduction 4

the PCM encoder, acts as the receiver front-end for the up-stream channel. The characteristics of
this receiver, including the sampling rate, sample timing and detection process are pre-determined
by the network.
An end-to-end PCM voiceband channel consists of two channels; an up-stream PCM channel
followed by a down-stream PCM channel. Compared to the up-stream and the down-stream
channels, achieving a higher transmission rate over an end-to-end PCM channel is more difficult
since the end points do not have direct access to the digital data stream.

1.3 Related work

Theoretical studies as well as industrial R&D led to a significant increase in achievable data-rates
on voiceband channels. Data transmission rates increased from 300 bits/s in the mid 1950s to
rates of up to 33.6 kbits/s in early 1990s. This advancement was mainly due to improved channel
modelling and the invention of sophisticated techniques for mitigating the channel impairments1 .
Many of these techniques were examined by standards bodies to develop recommendations and
guidelines for voiceband modems. The Series V Recommendations from ITU-T2 have global ac-
ceptance in modem development. Recommendation V.34 [2] was the last recommendation for
voiceband modem design that considered the conventional view of additive noise for the quanti-
zation error [3].
Although the development of the V.34 Standard seemed to be the end of the voiceband mo-
dem era, an interesting observation regarding the PCM voiceband channels created a new wave
of research and development on voiceband modem design. Instead of ignoring the underlying
structure of signal conversions in the PCM encoders and decoders, this structure can be employed
in order to reduce the distortion caused by these conversions.
To obtain a higher data rate over a PCM voiceband channel, the quantization distortion
causing symbol detection errors should be avoided. Voiceband modems designed to achieve this
goal are known as PCM modems. The idea of PCM voiceband modems was first investigated in [4].
This work was followed by [5] which establishes a method for equalizer design for PCM modems.
The error correction method to combat echo was described in [6]. Many of these published articles
have companion patents [7, 8, 9, 10, 11]. A report on the actual system design based on the idea
of PCM modems is described in [12].
The first recommendation on PCM modems, known as Recommendation V.90, was completed
in 1998 [13]. Recommendation V.90 uses a spectrum shaping known as convolutional spectrum
shaping [14] to reduce the power density of the transmitted signal at certain frequencies.
1
A comprehensive review of earlier work on voiceband modems is presented in [1].
2
International Telecommunication Union–Telecommunication Standardization Sector.
1 Introduction 5

Recommendation V.90 provides asymmetrical data rates in two directions. During the de-
velopment of Recommendation V.90, the support of higher data-rates in the up-stream PCM
channel was proposed [15, 16]. Due to technical uncertainties on several issues such as channel
equalization, channel coding, signal mapping and constellation design, the ITU-T committee did
not reach an agreement on a new standard for the up-stream PCM channels [17]. As a result, the
conventional voiceband transmission techniques based on Recommendation V.34 were adopted
for the up-stream direction.
There is ongoing work by an ITU-T study group to develop new recommendation to achieve
higher transmission rates over the up-stream PCM channels3 . For data transmission over the
up-stream PCM channels, a technique based on convolutional spectrum shaping [14] [13] and a
linear filtering is proposed [18].

1.4 Our research objectives

The main objective of our research is to identify methods of obtaining higher rates of reliable data
transmission over voiceband PCM channels. We particularly target the up-stream PCM channel,
where the pre-determined structure of the receiver front-end creates several theoretical, as well as
practical, problems in the modem design.
We first need to identify alternative models of data transmission over a PCM channel. We
characterize sources of impairment in the PCM channel and develop strategies to avoid or minimize
the impact of distortions on the data transmission performance.
There are several constraints imposed by the telephone network on the up-stream PCM channel
receiver as well as the transmitted signal. We investigate methods of modulation design which are
compliant with network requirements and match the pre-determined structure of the receiver. We
investigate design techniques in constellation design, index mapping and constellation probability
assignment that can improve the system performance under a transmitted power constraint.
Due to limited bandwidth and fixed sampling rate, the up-stream PCM channel creates inter-
symbol interference (ISI) at the receiver. In order to eliminate or reduce the impact of ISI on the
system performance, we investigate methods of compensating for the channel at the transmitter.
Since there is no direct access to the receiver front-end of the up-stream channel, the channel
compensation must be performed entirely at the transmitter. We examine the existing techniques,
such as those proposed for the V.92 Standard and investigate alternative methods to improve the
system performance by reducing the impact of the ISI at the receiver.
An appropriate design of pulse shaping filters prevents ISI due to signal modulation. We
3
The new recommendation from ITU-T is called V.92 with a feature for higher bit-rates in the up-stream
directions.
1 Introduction 6

investigate the design of pulse shaping filters that can be used in two different scenarios: in an
up-stream channel where the receiver front-end is fixed, and end-to-end PCM channels where the
transmitter and receiver filters are a pair of matched filters.

1.5 Thesis organization

In Chapter 2, an overview of data transmission over the PCM voiceband channels is presented. We
examine the characteristics of different components of PCM channels. The conventional model of
data transmission over voiceband channel is compared to the hybrid model of connection between
analog subscribers and the digital network.
Chapter 3 describes the choice of a modulation scheme that is compatible with the predeter-
mined structure of the receiver in the up-stream PCM channel. We investigate several problems
related to the constellation design, index mapping and non-equally-probable signalling for up-
stream PCM channels. The modulation performance, with an average power constraint in the
presence of additive Gaussian noise and residual inter-symbol interference, is analyzed.
Chapter 4 describes the linear distortion due to the channel and receiver filters. We describe
linear transmitter filter design procedure which minimizes certain optimization criteria such as
the peak distortion error or the mean-square error. A unified approach is employed to define
several transmitter structures. These transmitter structures include combinations of optimal
linear filtering and spectral shaping. We also describe precoding techniques to compensate for
the PCM channel in the transmitter while maintaining an average transmitted power constraint.
In Chapter 5, a non-maximally decimated filterbank structure is introduced to compensate
for the channel at the transmitter. The channel model can be a non-minimum phase filter and
its frequency response may contain spectral nulls. The merits and limitations of such a filterbank
structure for channel pre-equalization are investigated.
Chapter 6 describes new methods of designing pulse shaping Nyquist filters. We introduce
a general relationship between the phase and amplitude response of bandlimited Nyquist filters.
Using this relationship, we show that a bandlimited zero-phase Nyquist filter can always be split
into two cascaded matched filters, such that each one is also a Nyquist filter. We explain how the
new pulse shaping filters can be used for the up-stream PCM channel as well as the end-to-end
PCM channels. We also introduce a new family of Nyquist filters. The standard raised-cosine
spectra can be considered as a special case of this family. Compared to raised-cosine filters, the
generalized raised-cosine filters provide more flexibility and control in the filter design.
In Chapter 7, concluding remarks, contributions and future research directions are discussed.
 7

Chapter 2

The PCM Voiceband Channel

Characteristics

This chapter describes a more detailed communication model of PCM voiceband channels. Three
different connection scenarios are considered: the down-stream direction, the up-stream direction
and end-to-end connections. We describe different constituents of a voiceband telephone network
that contribute to the overall characteristics of the PCM channels.

2.1 Subscriber access to the PSTN

In a telephone network, the physical connection to customer premises is provided by a pair of

twisted copper wires known as a subscriber loop or a subscriber line. A subscriber loop is used for
signalling operations such as dialing, ringing and the off-hook indicator as well as for bi-directional
voiceband signal transmission [19]. As shown in Fig. 2.1, at both ends of a subscriber loop, a
hybrid circuit separates the receiving and transmitting signal paths by converting a two-wire link
to a four-wire connection.

2.2 Subscriber loop interface circuit

The interface between the analog subscriber loop and a digital central office switching system is
provided by a line interface circuit. A line circuit provides several functionalities such as power
supply, over-voltage protection, ringing for incoming calls, monitoring the line status (supervision),
coding/decoding, hybrid and testing [19]. Figure 2.1 shows the components of the line circuit that
directly affect a voiceband transmitted signal. The actual locations of the line interface circuits
depend on the telephone network and the associated loop plant. In the majority of existing
2 The PCM Voiceband Channel Characteristics 8

Sample
A/D & Receiver
Hold

Bandpass
Filter
Signalling Hybrid Hybrid

Lowpass Subscriber
Filter Loop
Zero- Transmitter
D/A order
Hold
Terminal Equipment

Line Circuit

Fig. 2.1 Subscriber loop access via line interface circuits at the central office.

telephone networks, the line interface circuit cards are installed at the central office. In some
cases, line circuits are installed as part of a digital loop carrier (DLC) in a remote switching
terminal[20]. The data transmission between a DLC and the central office is carried over twisted-
pair or fiber-optic connections. By employing DLCs, the actual length of each analog subscriber
loop is shortened and many signal transmission impairments over long analog subscriber loops
are mitigated.

2.3 Subscriber loop characteristics

A subscriber loop cable has electrical characteristics that introduce changes to the carried signal.
The theory of transmission line can be used to quantify signal distortions. As with any other
transmission line, the electrical characteristics of a twisted pair of copper wires can be described
by a set of parameters known as the line primary constants [19]. The primary constants consist of
series inductance L, shunt capacitance C, shunt conductance G and series resistance R. The pri-
mary constants are all defined per unit length of cable. Figure 2.2 shows a lumped circuit element
model of a segment of transmission line. The actual requirements of twisted copper lines used in
subscriber loops are described in several standards and guideline documents. Nominal values for
these parameters can be found in [19], [21] and [22]. The transfer function of a transmission line
is obtained by solving a set of partial differential equations describing the current and voltage at
each point of the line as a function of time and distance [23]. A set of secondary parameters known
as propagation constants and characteristic impedance are computed to simplify the voltage and
2 The PCM Voiceband Channel Characteristics 9

R/2 L/2 L/2 R/2

G C

Fig. 2.2 The T-equivalent circuit for a unit length segment of a subscriber line

current expressions:


γ = α + jβ = (R + jωL)(G + jωC)

R + jωL (2.1)
Z0 =
G + jωC

For a given load impedance ZL , the transfer function and the input impedance of the line are
determined as follows [21]:
ZL
T (ω) = (2.2)
ZL cosh(γd) + Z0 sinh(γd)

ZL + Z0 tanh(γd)
Zin (ω) = Z0 (2.3)
Z0 + ZL tanh(γd)
where d is the length of the line. If the line consists of several segments with different electrical
characteristics, the input impedance of each part is the load of the preceding segment, and the
overall transfer function is the product of the transfer functions of cascaded segments. A more
detailed analysis and examples including unterminated lines (bridged-tap sections) or lines with
extra series inductance (loading coils) can be found in [21] and [23].
In order to show the filtering effect of the subscriber lines, we consider a set of lines with
the same cable characteristics composed of wires with 26 American-Wire-Gauge (AWG)1 . The
line termination consists of a 600 Ω resistance. Figure 2.3 shows the frequency responses of
different cable lengths. It is evident that the signal attenuation becomes more significant at
higher frequencies for longer cables.
For subscriber lines longer than 5.5 km (18 kft), the signal attenuation is significant even
at frequencies as low as 1 kHz, causing unacceptable distortion to the transmitted voice signal.
A compensation for the line characteristics can be achieved by inserting several series inductors
along the line.
A set of appropriately chosen series inductors (loading coils) can be used to compensate for
1
AWG is a measure of wire diameter. Typical gauges are 19, 22, 24 and 26. The wire diameter decreases by
increasing AWG [20].
2 The PCM Voiceband Channel Characteristics 10

−5

−10
|H(f)| (dB)

−15

−20
0.5 km
−25 2 km
4 km
6 km
−30
0 2000 4000 6000 8000
Frequency (Hz)

Fig. 2.3 The magnitude response of subscriber lines with a 26-AWG cable. The
magnitude response is shown for different cable lengths. The terminal load is set to
a 600 Ω resistance.

−5

−10
|H(f)| (dB)

−15

−20

−25
Unloaded
Loaded
−30
0 2000 4000 6000 8000
Frequency (Hz)

Fig. 2.4 Magnitude response of a 5.5 km (18 kft) long unloaded subscriber line with
a 26-AWG cable is compared with that of a loaded line with the same characteristics
and the same length. The loaded line contains two 88 mH loading coils which are
inserted at 1.83 km (6 kft) from each end.
2 The PCM Voiceband Channel Characteristics 11

the signal attenuation in the lower frequency range (100–2500 Hz). The disadvantage of this
compensation is that the loading coils create a lowpass frequency response where signals with
frequency components higher than the cutoff frequency will be significantly attenuated. This
attenuation reduces the channel bandwidth and the maximum data transmission rate. Figure 2.4
shows an example of a compensated line. In this example, the total length of the line is 5.5 km
(18 kft) and there are two 88 mH loading coils inserted 1.83 km (6 kft) from each end.
For simulation and testing purposes, there have been studies that identify the transmission
impairment conditions encountered on the telephone networks in most countries. For example,
Recommendation V.56 series from the ITU-T provides network models to be used in the evaluation
and comparison of voiceband modems [24]. Figure 2.5 shows the measurement results of the
frequency response for several subscriber lines, as given in [24]. In each case, a resistive load of
600 Ω is used.

2.3.1 Hybrid Circuits

In a public telephone network, access to the central office for a majority of users is provided via a
pair of twisted copper wires. Since the send and receive signals share the same pair of wires, this
connection is called two-wire transmission. Four-wire transmission is where separate pairs of wires
are used for transmitting and receiving signals. At the central office as well as the user premises,
a circuit known as a hybrid coupler (or simply “hybrid”) is used as an interface between two-
wire and four-wire connections. A hybrid circuit can be implemented in several ways. In many
telephone sets, the hybrid circuit is implemented using transformers with line impedance matching
to achieve a maximum energy transfer to the line and a minimum leakage from the transmitter to
the receiver. Figure 2.6 shows the functionality of a hybrid using active components. Ideally, the
hybrid circuit should prevent any energy leakage from the local transmitter to the local receiver. In
practice, the line impedance varies from line to line. It is infeasible to choose one impedance that
matches all subscriber lines. We can only choose a compromised impedance value that provides
a reasonable match for most lines. As shown in Fig. 2.6, the signal from the transmitter to the
subscriber line attenuates in the impedance voltage divider.

2.3.2 Characteristics of the filters in the line circuit

Among other tasks, a line circuit performs filtering on the received and transmitted voiceband
signals. As shown in Fig. 2.1, there are two filters used in a line circuit: one prior to the ADC
circuit, and one following the Digital-to-Analog Converter (DAC).
The bandpass anti-aliasing filter prior to the sampler is referred to as a send filter. The send
filter limits the bandwidth of the input signal to avoid aliasing in the sampled signal. This filter
2 The PCM Voiceband Channel Characteristics 12

−4
|H(f)| (dB)

−8

LL−1
LL−3
LL−4
−12
0 1000 2000 3000 4000
Frequency (Hz)

(a) Magnitude response of several unloaded subscriber lines.

The wire diameter for all these cases is 0.41 mm. EIA LL-1
is 0.61 km long, EIA LL-3 is 2.13 km long and EIA LL-4 is
3.66 km long.

−4

−8
|H(f)| (dB)

−12

−16
LL−6
LL−7
−20
0 1000 2000 3000 4000
Frequency (Hz)

(b) Magnitude response of two loaded lines. The loading coils

are 88 mH. The line shown as EIA LL-6 includes four loading
coils and the line shown as LL-7 includes five loading coils.

Fig. 2.5 Measurement results of the magnitude response of several subscriber lines
as reported in [24].
2 The PCM Voiceband Channel Characteristics 13

Transmitter +
-

Zt Zt

+
Receiver
- Z eq
Subscriber Line

Z ref

Fig. 2.6 A circuit model of a hybrid coupler

also attenuates the low frequency interferences from power lines.

The lowpass reconstructing filter following the DAC circuit is referred to as a receive filter.
This filter is used to suppress out-of-band frequency components in the reconstructed signal.
The send and receive filters should meet certain requirements in terms of their magnitude
response and group delay. These requirements are specified in standard references such as ITU-T
Recommendation G.712 [25]. Recommendation G.712 specifies a range for the attenuation and
group delay for the send and receive filters (digital to analog channel), as well as the overall
frequency response of the cascade of both filters (analog-to-analog channel). Figures 2.7 shows
a design example of a send filter (prior to the ADC circuit) that satisfies the requirements of
Recommendation G.712. The filter consists of a 5th -order elliptic lowpass filter cascaded with a
bi-quad highpass filter. The same lowpass elliptic filter is used as the receive filter (following the
DAC). Figure 2.8 shows the overall response of the send and receive filters.
Some design specifications for send and receive filters can be found in [20]. Some empirical
measurements of the frequency response of these filters are reported in [24]. For example, Fig. 2.9
presents some of the measurement results for several subscriber lines, as reported in [24].

2.3.3 PCM encoding and decoding

In a line circuit, the signal received from an analog subscriber loop is converted into a stream of
binary digits. The encoding scheme used for digital telephone networks is Pulse Code Modulation
(PCM). Conceptually, a PCM encoder performs three operations:
2 The PCM Voiceband Channel Characteristics 14

Magnitude Response (dB)

−1

−3

−5
0 1000 2000 3000 4000
Frequency (Hz)

(a) Magnitude response of a send filter (solid line) is compared

with the bounds provided by Recommendation G.712 (dashed
lines).

2
Group Delay (mSec)

0
0 1000 2000 3000 4000
Frequency (Hz)

(b) Group delay of a send filter (solid line) is compared with

the bound provided by Recommendation G.712 (dashed line).

Fig. 2.7 Example of the frequency response of a send filter. The filter consists of a
5th -order lowpass elliptic filter cascaded with a highpass bi-quad filter.
2 The PCM Voiceband Channel Characteristics 15

1 0
Magnitude Response (dB)

−10
−1.5

−20
−3

−4.5 −30
0 1000 2000 3000 4000 3000 3500 4000 4500
Frequency (Hz) Frequency (Hz)

(a) Overall magnitude response of the cascade send and receive filter is compared with the bounds
of Recommendation G.712.

4
Group Delay (mSec)

0
0 1000 2000 3000 4000
Frequency (Hz)

(b) Overall group delay of the cascade of send and receive filter
is compared with the requirements given in Recommendation
G.712.

Fig. 2.8 Frequency response of the overall send-receive filter is compared with Rec-
ommendation G.712 requirements.
2 The PCM Voiceband Channel Characteristics 16

−5
Magnitude Response (dB)

−10

−15

−20

−25

−30 Send Filter

Receive Filter
−35

−40
0 500 1000 1500 2000 2500 3000 3500 4000
Frequency (Hz)

(a) Measured magnitude responses of send and receive filters.

3000
Send Filter
Receive Filter
2500
Group Delay (µ Sec)

2000

1500

1000

500

0
0 500 1000 1500 2000 2500 3000 3500 4000
Frequency (Hz)

(b) Measured group delays of send and receive filters.

Fig. 2.9 Empirical results as provided in Recommendation V.56.

2 The PCM Voiceband Channel Characteristics 17

1. sampling the input waveform,

2. quantizing the samples, and

3. representing each quantizer level with a binary index.

The continuous-time analog signal at the input of the PCM encoder is sampled at 8000 sam-
ples/sec. Each sample is represented by an 8-bit codeword. A practical device for implementing
this process is a sample-and-hold circuit followed by an Analog-to-Digital Converter (ADC) cir-
cuit [26, p. 187]. The ADC maps each sample into a binary codeword representing a quantized
amplitude closest to the input.
A PCM decoder restores the analog waveform corresponding to the PCM encoded bit stream.
The input bit stream to a PCM decoder is parsed into 8-bit codewords. As part of PCM decoder,
a DAC circuit converts each 8-bit codeword into an amplitude. The discrete-time reconstructed
amplitudes are converted to a continuous-time waveform. An example of a physical device which
performs such conversions is zero-order-hold followed by a lowpass filter [26, p. 187].

2.3.4 Non-uniform Quantizer

The tandem operation of a PCM enCOder and a DECoder (CODEC), in terms of representing
a discrete-time sample by a quantized amplitude, can be viewed conceptually as scalar quantiza-
tion2 . A codec is characterized by two finite sets of amplitudes: decision boundaries and output
levels. A CODEC approximates each input sample falling between two adjacent decision bound-
aries with an amplitude taken from the set of output levels. The approximation error manifests
itself as signal distortion in the reconstructed waveform.
For a given bit resolution (i.e., the number of quantization output levels), the goal of a CODEC
design is to specify the decision boundaries and the output levels of a quantizer that minimize
certain error criteria. For voice telephony, a CODEC should provide a robust quantization per-
formance over a wide range of input signals and signal levels. The performance of a quantizer can
be measured in terms of Signal-to-Quantization Noise Ratio (SQNR)3 .
There are requirements imposed on a CODEC performance in terms of lower bounds (masks)
for the SQNR with respect to the input signal power. Recommendation G.712 defines several
masks for SQNR for different input test signals [25]. These requirements cannot be satisfied by
an 8-bit uniform quantizer (equal spacing between decision boundaries).
By choosing smaller quantizer steps for small signals and large steps for large signals, we
can obtain a more robust quantizer. Non-uniform spacing between quantization levels can be
2
Here, we assume that the digital network is transparent from the viewpoint of signal transmission.
3
Note that a robust quantizer does not necessarily provide the maximum SQNR for a given number of bits but
it maintains a level of performance for a larger dynamic range of the input signal.
2 The PCM Voiceband Channel Characteristics 18

achieved by using a non-linear compression characteristic F (·) prior to a uniform quantizer, then
expand the quantized levels using the inverse function F −1 (·). The compressor characteristic F (·)
is referred to as (compressing and expanding) companding law.
Two companding laws specified in Recommendation G.711 [27] are the µ-Law and the A-Law4 .
The compression characteristic function for the µ-Law quantizers can be expressed as:

log(1 + µ|x|/xmax )
F (x) = xmax sgn(x) (2.4)
log(1 + µ)

where sgn(x) is the sign of x. The inverse characteristic equation is:

1
F −1 (x̂) = xmax [(1 + µ)|x̂|/xmax − 1] sgn(x̂) (2.5)
µ

The actual characteristics of a µ-law (and also an A-Law) CODEC are completely specified in
Recommendation G.711. In this recommendation, the companding laws are approximated as
piece-wise linear functions. For µ = 255, the µ-Law companding function is approximated by 15
linear segments (In fact, there are 16 segments but the two segments close to the center have the
same slope). Each segment specifies 16 output levels.
As shown in Fig. 2.10, each 8-bit codeword at the output of encoder consists of three parts.
The most significant bit in each codeword represents the sign bit. The next three bits give the
segment index and the last four bits identify one of 16 possible output levels in each segment.

(MSB) (LSB)

b7 b6 b5 b4 b3 b2 b1 b0

Sign bit Segment # Step within the segment

Fig. 2.10 Bit allocation in a µ-Law codeword.

For the µ-Law quantizer, the positive decision levels X(1) to X(128) as well as the output
levels Y (0+ ) to Y (127) are specified as:

X(16i + j + 1) = 2(i+1) (j + 17) − 33 for 0 ≤ i ≤ 7 , 0 ≤ j ≤ 15 (2.6)

Y (16i + j) = 2(i+1) (j + 16.5) − 33 for 0 ≤ i ≤ 7 , 0 ≤ j ≤ 15. (2.7)
4
Here, we only give the relationship for µ-Law. A-Law companding and its approximation is described in [22,
pp. 621-627].
2 The PCM Voiceband Channel Characteristics 19

Note that the µ-Law quantizer has a mid-tread characteristic, i.e., zero is one of the allowed
output levels. Since there are 256 possible output levels in total and the quantizer is symmetrical;
two output levels are assigned to the same value:

Y (0+ ) = Y (0− ) = 0

Since the quantizer is symmetrical, negative decision levels are specified as:

X(−i) = −X(i) for 1 ≤ i ≤ 128

Y (−i) = −Y (i) for 0 ≤ i ≤ 127. (2.8)

The outer levels X(−128) and X(128) are virtual decision boundaries corresponding to the am-
plitude of a 3.17 dBm0 sinusoid, that is, a sine-wave with a normalized amplitude X(128) = 8159
will have a power of 3.17 dBm05 .

2.3.5 Multiplexing

The output of a PCM encoder is a 64 kbits/s DS0 bit stream corresponding to each voice channel
[20]. The bit rate DS0 is the lowest rate in the hierarchy of data multiplexing in the switched
digital network. For example, in Fig. 2.11, a channel bank comprising 24 voice channels is depicted.
The outputs of these channels are encoded at 64 kbits/s DS0 rates and multiplexed into a 1.54
Mbits/s DS1 stream6 [20].

2.4 Power Constraints of the PCM channel

Each country has certain regulations and guidelines for the electrical characteristics of terminal
equipment connected to a subscriber loop7 . The regulations are imposed to prevent any damage to
the network. Any equipment for loop terminal use must pass several certification tests in order to
be connected to the network. One particular requirement concerns the signal power transmitted
over the subscriber loop. The average power applied to a loop in the voice frequency band (200
Hz – 4000 Hz) should be limited to −13 dBm under all conditions when averaged over a period
of 3 seconds8 [19]. The average power limitation reduces the maximum data transmission rate.
5
dBm0 is a power measure with respect to a (virtual) reference point, known as zero Transmission Level Point
(TLP) [22, p.23].
6
The digital multiplexing rates given here are based on North American Standards.
7
For example, the Federal Communication Commission (FCC) in the United States, and the Canadian Radio
and Telecommunication Commission (CRTC) in Canada govern these regulations.
8
There are ongoing discussions to relax the power constraint
2 The PCM Voiceband Channel Characteristics 20

Sample
A/D &
Hold
Bandpass
Filter
Subscriber Loop
Signalling Hybrid

Lowpass
Filter
Zero-
D/A order
Hold

DS1
MUX CODEC

Subscriber Loop
CODEC

Subscriber Loop
CODEC

Channel Bank

Fig. 2.11 A channel bank multiplexing 24 voice channels to a DS1 stream.

2 The PCM Voiceband Channel Characteristics 21

2.5 Capacity of PCM voiceband channels

A conventional design of voiceband modems targets data transmission between two terminals
connected to analog subscriber lines. In this design, the voiceband channel is treated as an analog
communication medium, where the underlying sources of channel impairment are modeled as
additive distortions. For an end-to-end PCM voiceband channel, the main source of distortion
is signal quantization in the analog-to-digital conversion. The quantization distortion can be
approximated as Additive White Gaussian Noise (AWGN) under certain conditions [4, 28]:

• the transmitted signal contains independent, identically-distributed random symbols,

• the symbol timing is independent of the sampling clock at the PCM encoder, and

• the number of quantization levels is large.

Under the AWGN assumption of the quantization noise, the voiceband channel capacity can be
computed using Shannon’s classical results [29]:

C = W log2 (1 + SNR) bits/sec (2.9)

where W is the bandwidth and SNR is the Signal-to-Noise power Ratio. For a typical PCM
voiceband channel, W is in the 3–3.5 kHz range. In a PCM channel, the dominant source of
distortion is the quantization error. The SQNR for an 8-bit non-uniform A-law or µ-law quantizer
is in the 33–39 dB range. For the given nominal values of the SQNR and the channel bandwidth,
the capacity of a voiceband channel computed from Eq. (2.9) is in the 33 to 45 kbits/s range. Based
on the AWGN model of distortions in voiceband channels, there are several recommendations and
guidelines developed by international standards bodies such as the ITU-T to design voiceband
modems. For example, Recommendation V.34 provides data transmission rates up to 33.6 kbits/s
on ordinary telephone lines [2].
In the absence of quantization error, the maximum information rate on a PCM voiceband
channel is limited by the channel bandwidth and the PCM codeword length9 . The Nyquist theory
indicates that the maximum symbol rate for data transmission with no inter-symbol interference
is limited to
D = 2W symbols/sec. (2.10)

Since each symbol at the output of PCM encoder is represented by eight bits, the maximum
channel bit rate is 16W bits/sec. For a typical PCM channel, the bandwidth is limited to 3–3.5
9
Here, we ignore any other sources of noise. In Chapter 3, we revisit the maximum achievable rate of an
up-stream PCM channel.
2 The PCM Voiceband Channel Characteristics 22

kHz. Hence the channel bit rate can theoretically reach 48–56 kbits/sec. Compared to achievable
rates using V.34 modems, there is a potential increase of data transmission rate on the PCM
voiceband channels by 50%–70%.

2.6 PCM Modems

The idea of PCM modems was first introduced in [4]. Assuming an ideal bandlimited filter model
for the overall PCM voiceband channel, [4] specifies a set of pulse shaping filters to ensure zero
Inter-Symbol Interference (ISI) in a subset of the sampling instants (say 6 samples out of every
8), prior to ADC at the central office10 . Such signalling is feasible only if the transmitter modem
is synchronized to the network clock at the central office. If the transmitted symbols are selected
from the set of PCM CODEC output levels, there will be no quantization distortion added to
the subset of samples. The PCM decoder (located at the central office serving the destination
end user) does not add any significant distortion to the reconstructed signal, except for a band-
limiting filter at the receiver. In [4], it is suggested to use a maximum likelihood sequence detection
algorithm (e.g., Viterbi Algorithm) to combat ISI at the receiver.
Although the idea of using PCM modems for end-to-end PCM channels is appealing in theory,
it is challenging to implement in practice. The connection between two end-users as shown in
Fig. 2.12(c) consists of two separate links, an up-stream PCM channel and a down-stream PCM
channel with different characteristics. Without side information from within the digital switched
telephone network, it is difficult, if not impossible, to solve problems such as synchronization,
echo cancellation and channel equalization for the end-to-end PCM channel.
As discussed in Section 1.2, there are alternative scenarios for network connection over a PCM
voiceband channel. Figure 2.12 shows three types of connection over a PCM voiceband channel.
In the next sections, we describe the issues related to PCM modem design for each channel.

2.6.1 Digital network imperfections

Before we describe different connections over a PCM channel, we should note that some voice-
band channels are not qualified to be considered as PCM channels. The model we consider for a
voiceband PCM channel does not include any signal distortion within the digital network. It is
implicitly assumed that the digital network provides a 64 kbits/s data transmission rate between a
pair of PCM encoders and decoders. However, there are some possible sources of distortion in the
digital network. For example, in the North American standard of digital multiplexing hierarchy,
the least significant bit of each codeword of a fraction of voice channels is reserved (robbed) for
10
This method is described in Chapter 5 in more detail.
2 The PCM Voiceband Channel Characteristics 23

Noise + Echo
Digital PSTN

Digital PCM
Transmitter Channel Receiver
Interface Decoder
64 kbit/s 64 kbit/s

Digital Subscriber Analog Subscriber

Line Line

(a) The down-stream PCM Channel.

Noise + Echo
Digital PSTN

PCM Digital
Transmitter Channel Receiver
Encoder Interface
64 kbit/s 64 kbit/s 64 kbit/s

Analog Subscriber Digital Subscriber

Line Line

(b) The up-stream PCM Channel.

Noise + Echo
Noise + Echo
Digital PSTN

PCM PCM
Transmitter Channel Channel Receiver
Encoder Decoder
64 kbit/s 64 kbit/s

Analog Subscriber Analog Subscriber

Line Line

(c) The end-to-end PCM Channel.

Fig. 2.12 A PCM modem is used to transmit/receive data over one of the following
channels: (a) a down-stream PCM channel, (b) an up-stream PCM channel, (c) an
end-to-end PCM channel.
2 The PCM Voiceband Channel Characteristics 24

signaling and supervisory information [20]. Robbed-bit signalling reduces the maximum transmis-
sion rate over the PCM channels. A PCM modem should detect the use of robbed-bit signaling
on a PCM channel and adjust the transmission rate accordingly11 .
Digital data conversion is another example of signal distortion in a digital network. The digital
data conversion can be due to

1. µ-law to A-law conversion or vice versa,

2. conversion to other encoding standards such as ADPCM and back to PCM, or

3. conversion to an analog signal and back to digital (multiple encoding in one channel).

If a voiceband channel contains any of these data conversions, our underlying assumption about
quantization error model will no longer be valid, hence, a PCM modem cannot be used for that
channel.

2.7 The down-stream PCM channel

Figure 2.12(a) shows a down-stream PCM channel. While the transmitter modem is connected
to the PSTN via a digital subscriber line (e.g., T1 link or ISDN), the receiver is connected to an
analog subscriber line.
There is no quantization error due to analog-to-digital conversion. The ultimate information
rate over this channel is 8 bits/symbol × 8000 symbols/sec = 64 kbits/sec. However, due to the
non-uniform distribution of the symbol levels as well as the limited bandwidth of the reconstruction
filter at the central office, the maximum achievable rate is around 56 kbits/s. Recommendation
V.90 [13] provides guidelines for designing PCM modems in the down-stream direction to achieve
rates of up to 56 kbits/s12 .

2.8 The up-stream PCM channel

Figure 2.12(b) shows an up-stream PCM channel connecting an analog subscriber to the digital
network. There are many applications that require data transmission over an up-stream PCM
channel. These applications include: uploading files, sending Email with attachments and Internet
video-conferencing.
11
Note that the Robbed-bit signalling is no longer a common signalling method. Instead, signalling scheme known
as Common channel signalling is employed[20].
12
This number is limited to 53 kbits/s in North America, due to regulatory power constraints which are expected
to change in the near future.
2 The PCM Voiceband Channel Characteristics 25

Recommendation V.90 uses a conventional analog data transmission (based on the Recommen-
dation V.34) in the up-stream direction [13] which does not exploit the maximum transmission
rate of the up-stream channel13 .
n(t) Echo Central Office Encoder

x(t) y(nT)
C(t) t=nT

Channel Additive Anti-Aliasing Sampler Quantizer

Noise Filter

Fig. 2.13 The Up-stream PCM Channel

Figure 2.13 shows components of an up-stream PCM channel. An analog signal x(t) generated
by the transmitting modem passes through the subscriber loop and reaches the central office where
a PCM encoder samples the received signal and converts the sample into binary codewords. Each
codeword corresponds to a quantized level y(nT ).
Compared to the down-stream PCM channel, modem design for the up-stream PCM channel
is a more challenging task, primarily due to the quantization distortion at the PCM encoder.
Unlike an analog modem, a PCM modem in the up-stream direction treats the PCM encoder at
the central office as the receiver front-end that includes a bandpass filter, a sampler and a signal
level detector (the quantizer). The choice of receiver analog filter, sampling rate, sample timing
and symbol detector are constrained by the network hardware. Some of technical challenges
associated with the up-stream PCM modem are:

2.8.1 Synchronization

In an up-stream channel, the PCM encoder acts as the receiver front-end. Since the sampling time
is controlled by the telephone network, the transmitter should be synchronized to the sampling
clock of the line circuit. This arrangement is unconventional since the transmitter has no direct
access to the sampling clock at the receiver.
The symbol timing recovery in the down-stream direction can be used as a reference clock for
the transmitter. The down-stream symbol timing is useful to the up-stream transmitter only if
the encoder and decoder at the central office share the same timing clock14 (see Fig. 2.11).
13
There is ongoing work on a new Recommendation called V.92 that provides guidelines for designing PCM
modems in the up-stream direction.
14
In the development of the V.92 recommendation, the same assumption has been considered.
2 The PCM Voiceband Channel Characteristics 26

2.8.2 Signal mapping and constellation design

In the up-stream PCM channel, the symbol detector at the receiver is a scalar quantizer. As we will
describe in Chapter 3, the appropriate choice of modulation scheme is a baseband Pulse Amplitude
Modulation (PAM). Issues related to the optimal constellation design and bit-to-symbol mapping
should be investigated.

2.8.3 Channel Pre-equalization

The anti-aliasing filter at the central office, along with the subscriber loop filtering effect limit the
frequency band of the up-stream channel to the 300–3600 Hz range. While the channel bandwidth
is restricted to about 3300 Hz, the sampling frequency at the central office is fixed at 8000 Hz.
As a result, the received samples are distorted by Inter-symbol Interference (ISI). Since there is
no access to the receiver front-end, the compensation for the channel linear distortion must be
implemented entirely at the transmitter.

2.8.4 Echo cancellation

As shown in Fig. 2.1, the PCM up-stream and down-stream links share the same pair of wires.
Ideally, a hybrid circuit should provide isolation between the two directions of transmissions. In
practice however, there is an echo path from the down-stream link to the up-stream channel.
Although, there are conventional methods for echo cancellation in analog voiceband modems,
echo cancellation in up-stream PCM modem requires further consideration, due to the non-linear
characteristics of PCM quantizer.

2.8.5 Channel Coding

Since there are always residual ISI and error induced by echo, it is important to analyze the
system performance to characterize the potential error correcting coding gain on the up-stream
PCM channel. According to the result of such an analysis, we should decide on the use and the
type of coding over the up-stream channels.

2.9 End-to-end PCM voiceband channels

An end-to-end PCM channel is shown in Fig. 2.12(c). This channel consists of a cascade of an
up-stream and a down-stream channel. From the end users point of view, there is no information
available in the channel output regarding the PCM CODEC, the underlying sampling and the
CODEC filters individually. It is difficult to acquire this information from the end points. Some
challenging problems on and end-to-end PCM channels can be identified as:
2 The PCM Voiceband Channel Characteristics 27

2.9.1 Synchronization

Since there is no access to the signal at the central office for the transmitter modem nor the
receiver modem, it is not straightforward to synchronize the modems to the sampling clock of the
ADC at the central office. Without such synchronization, the underlying structure of the PCM
channel is not visible to the end users.

2.9.2 Channel estimation

Since there is no access to the digital stream of bits at the central office, the up-stream and the
down-stream channel filters involved in an end-to-end channel cannot be estimated separately.

2.9.3 Channel Equalization

Without any information from within the network, channel compensation can only be performed
based on partial knowledge about the up and down-stream channels composing the end-to-end
channel.
Problems described above become much simpler if we assume that there are some assistance
available from within the digital network to obtain the channel characteristics and the network
timing information. For example, an end-to-end PCM connection between two analog end-users
can be established via a third party who has direct access to the digital network and provides
both end-users the channel characteristics and the symbol timing. In fact, in this approach, the
end-to-end PCM channel is decoupled into an up-stream channel and a down-stream channel.
 28

Chapter 3

Modulation Design for the

Up-Stream PCM Channel

Following the general review of PCM voiceband channels presented in Chapter 2, this chapter
focuses on data transmission issues related to the up-stream PCM channel and investigates the
choice of modulation scheme suitable for the up-stream PCM channels. Considering the transmit
power limitation over the voiceband channel, we discuss a constellation design to minimize the
probability of error, and maximize the bit-rate at a given signal-to-noise ratio.

3.1 Modulation scheme

The choice of modulation scheme for the up-stream PCM channel is mainly influenced by the
structure of the front-end receiver, i.e., the PCM encoder at the central office. Figure 3.1 shows
a model of the up-stream PCM channel. The PCM encoder converts the received signal into
a 64 kbits/s DS0 digital data stream. The DS0 rate is clearly the upper bound of the data

%LW QW %LW
6WUHDP 7UDQVPLWWHU 6XEVFULEHU/RRS &HQWUDO2IILFH 6WUHDP

VQδ W − Q7V [W UW 'HWHFWRU
%LWWR6\PERO
0DSSLQJ  ,QYHUVH
KW W KF W KU W 0DSSLQJ
6KDSLQJ Q7V
7UDQVPLWWHU &KDQQHO $QWLDOLDVLQJ 6DPSOH µ/DZ
)LOWHU )LOWHU )LOWHU +ROG $/DZ

7V 7V

Fig. 3.1 A schematic model for the Up-stream PCM channel.

3 Modulation Design for the Up-Stream PCM Channel 29

transmission rate over a PCM voiceband channel. Each 8-bit codeword at the ADC output
should represent a distinct transmitted symbol to achieve a DS0 data rate over this channel.
The ADC operates as a threshold detector, or in communications terms, as a one-dimensional
slicer. Figure 3.2 shows schematically the relationship between the ADC decision boundaries and
the received signal at the ADC input. As shown in this figure, different transmitted symbols are
expected to generate different signal levels at the ADC input. Each signal level corresponds to a
constellation point in a one-dimensional signal space. The choice of modulation scheme for the
up-stream PCM channel should be compliant with such a detector.
The natural choice of modulation for the up-stream channel is baseband Pulse Amplitude
Modulation1 (PAM). A PAM transmitted signal can be represented as:
∞

x(t) = ai ht (t − iTs ) (3.1)
i=−∞

where ht (t) is the impulse response of the transmitting filter2 and ai ’s are the transmitted symbols
taking values from a one-dimensional PAM symbol alphabet. In a PCM channel, the symbol rate
is 8000 samples per second. We assume that the transmitting modem is synchronized with the
sampling clock at the central office. The up-stream transmitter can achieve such synchronization
by extracting the symbol timing information from the received signal in the down-stream direction.
The correct symbol timing of the transmitter can be determined during a training phase and be
adjusted prior to data transmission3 .
A PCM encoder has an 8-bit output word-length corresponding to a maximum of 256 input
decision intervals. However, in the presence of noise and other distortions, a subset of symbol
levels is chosen in order to increase the minimum distance between consecutive signal levels. At
the output of the anti-aliasing filter in the PCM encoder, the received signal can be expressed as:

r(t) = x(t) ⊗ hc (t) ⊗ hr (t) + n(t) ⊗ hr (t)

∞
 (3.2)
= ai h(t − iTs ) + η(t)
i=−∞

where ‘⊗’ denotes the linear convolution and h(t) is the overall impulse response of three cascaded
filters: the transmitting filter, channel filter and receiving filter. Unless otherwise stated, we
assume that these filters are linear and time-invariant over the time period of data transmission.
1
Note that a multi-dimensional modulated signal, such as Quadrature Amplitude Modulation, cannot be detected
by a single slicer.
2
For the optimum detection of the PAM signal, the transmitter and (channel-) receiver filters should be matched
to one another. The design of transmitter filter will be discussed in the next chapters.
3
We assume the PCM encoder and decoder share a common network clock [20].
3 Modulation Design for the Up-Stream PCM Channel 30

$'&'HFLVLRQ 'HWHFWLRQ &RQVWHOODWLRQ

&RGHZRUGV 3 U _ TL
%RXQGDULHV 7KUHVKROGV 3RLQWV
[ = 
G%P

[ = 


τ L +
[  = 


[  =  TL + 3 U _ TL +


τL
[  = 
TL 3 U _ TL
 τ L −
[ = 


[ = 


[ = 
 τ
[ = 

T
[ =  3 U _ T
 [ = 

 [ =  τ = 
[ = 

Fig. 3.2 In order to avoid quantization error in the up-stream PCM channel, the
ADC at the central office is employed as a slicer. One or more codewords represent a
transmitted symbol.
3 Modulation Design for the Up-Stream PCM Channel 31

The impulse response of the overall filter is determined by convolving the impulse responses of
three cascaded filters. The additive noise component η(t), represents the thermal noise as well
as other additive distortions, such as cross-talk from the adjacent wire lines and echo (should it
not be mitigated by an echo canceller). The discrete-time signal at the input of the ADC can be
represented as:
∞

r(nTs + t0 ) = ai h(nTs − iTs + t0 ) + η(nTs + t0 ) (3.3)
i=−∞

To simplify the signal presentation, we use a discrete time notation with a normalized symbol
rate:
∞

r[n] = ai h[n − i] + η[n] (3.4)
i=−∞
∞

= an−n0 h[n0 ] + ai h[n − i] + η[n] (3.5)
i=−∞
i=n0

Equation (3.5) identifies two sources of distortion affecting the symbol detection result; additive
noise and Inter-Symbol Interference (ISI) caused by the memory of the overall impulse response
h[n]. In Eq. (3.5), we consider the time offset n0 to compensate for the delay of the overall channel
filter h[n]. The value of n0 identifies the optimal sampling time of the pulse shape h[n] with respect
to a reference time. In the absence of noise and ISI, each sample r[n] represents a transmitted
symbol an−n0 scaled by the filter gain h[n0 ]. In the presence of noise and ISI, the signal levels
corresponding to different symbol alphabets should be chosen such that the probability of symbol
error is minimized. We will discuss the optimal selection of signal alphabets for a given set of
detector thresholds in the next section.

3.2 Optimal signal levels

For a given set of detector threshold levels, chosen from the ADC decision boundaries, we wish
to determine the optimal PAM symbol alphabet that minimizes the probability of symbol error.
Consider a set of 2N one-dimensional signal levels that is positioned symmetrically about zero.
In the absence of noise and ISI, the desired signal levels at the input of the ADC are denoted as
{q−N , q−(N −1) , ..., q−1 , q1 , ..., q(N −1) , qN }. Note that the qi ’s represent the symbol alphabet at the
input of ADC at the receiver.
Let us define a set of detector thresholds,

{−∞, −τ(N −1) , . . . , τ0 = 0, . . . , τ(N −1) , ∞},

3 Modulation Design for the Up-Stream PCM Channel 32

in ascending order that identify 2N decision intervals. A sample that falls between τi−1 and τi
is detected as symbol qi . The two outmost intervals are semi-infinite. We consider two virtual
detector thresholds ±τN for these two regions. The use of these virtual thresholds will be described
below. Figure 3.3 shows the placement of the signal levels together with the detector thresholds.
Except for ±τN , all other threshold levels are chosen from the set of ADC decision levels4 . If
more than one codeword at the ADC output represents a transmitted symbol (i.e. 2N ≤ 256),
only a subset of ADC decision levels will be used as detector thresholds.

T−L T T T1

τ τ τ τ1
T−L = − TL
6\PPHWULF$OSKDEHW
TL 6LJQDOSRLQWV
τL 7KUHVKROGV

Fig. 3.3 Placement of the signal points relative to the thresholds

Each constellation point qi corresponds to a transmitted PAM symbol. We denote P (qi ) as

the probability of transmitting qi . In a conventional data transmission system, each point in
the constellation is equally likely to be transmitted. This corresponds to a uniform probability
distribution for the constellation points. In our discussion, we allow for non-uniform probability
distributions. However, we assume that the distribution of probabilities of constellation points is
symmetrical:
P (qi ) = P (q−i ) for i = 1, 2, . . . , N.

Due to the symmetry of the constellation and the distribution of probabilities of the signal points,
the transmitted signal has a zero-mean:


N
P (qi )qi = 0 (3.6)
i=−N
i=0

As shown in Eq. (3.5), the discrete signal at the input of the detector can be written as a sum
of the desired signal, noise and the ISI. We consider a zero-mean Gaussian distribution for the
noise. Since the ISI component is a weighted sum of transmitted levels, it also has a symmetrical
distribution5 . As a random variable, the ISI in general does not have a continuous Probability
4
Note that the decision intervals corresponding to the outermost symbols are semi-infinite are only virtual
boundaries of these intervals.
5
We assume transmitted symbols are independent identical distributed random variables. Since transmitted sym-
3 Modulation Design for the Up-Stream PCM Channel 33

Density Function (PDF) [30, p. 60]. However, the sum of noise and ISI is a random variable with
a continuous PDF. We denote w[n] as the resulting additive distortion caused by noise and ISI at
each sampling instance:
r[n] = h[n0 ] an−n0 + w[n] (3.7)

Under the following conditions (that we assume all hold), w[n] has a symmetric PDF:

• The noise component has a symmetric distribution.

• The noise and ISI components are independent.

• The PAM signal constellation is symmetrical.

• The probability distribution of the constellation points is symmetric, P (qi ) = P (q−i ).

3.2.1 Performance criterion

A basic performance measure for any digital modulation scheme is the probability of error in the
presence of noise and other distortions. We use the probability of symbol error as the performance
criterion by which to choose the optimal signal levels. In the absence of noise and ISI, the received
signal takes a value from a PAM symbol alphabet {qi | i = ±1, . . . , ±N }. The probability of symbol
error can be written in terms of conditional error probabilities:

−1
 
N
Pe = P (error|qi )P (qi ) + P (error|qi )P (qi ) (3.8)
i=−N i=1

Due to the symmetry of constellation and symbol probability distributions, the probability of
symbol error can be written as:


N
Pe = 2 P (error|qi )P (qi ). (3.9)
i=1

For a fixed set of symbol probabilities P (qi ) and detector thresholds τi ’s, we select an optimal set
of signal levels qi ’s that minimize the probability of symbol error.
For the inner signal points, each conditional probability P (error|qi ) is computed as:

P (error|qi ) = P (r[n] < τi−1 |qi ) + P (r[n] > τi |qi ) for i = 1, 2, . . . , (N − 1)

 τi −qi (3.10)
=1− fW (w) dw
τi−1 −qi

bols have symmetrical probability distributions, the weighted sum of these random variables has also symmetrical
probability distribution.
3 Modulation Design for the Up-Stream PCM Channel 34

where fW (w) is the PDF of the random variable w[n]. From Eq. (3.10), it is evident that:
 τi −qi
min P (error|qi ) = 1 − max fW (w) dw τi−1 ≤ qi ≤ τi
qi qi τi−1 −qi (3.11)
for i = 1, ..., N − 1

In general, the value of qi that minimizes the probability of symbol error depends on the
PDF of w[n]. However, there are cases where the choice of qi is only dependent on the values
of decision boundaries (detector thresholds). For example, Fig. 3.4(a) shows a conditional PDF,
P (r[n] < τi−1 |qi ) = fW (w − qi ), that is monotonically decreasing on each side of its mean. For
this distribution, the optimal choice of the signal point qi is the midpoint between the two closest
detector thresholds6 :

τi + τi−1
qi = for i = 1, 2, ..., N − 1 (3.12)
2
Note that, in some cases, the choice of qi as the average of two detector thresholds may not
be optimal. For example, Fig. 3.4(b) shows an extreme case where the optimal value of qi is
either of the two decision boundaries. The probability of error in this case is larger than one
half. Such degenerate cases can be caused by a large ISI or a relatively short distance between
adjacent detector thresholds. We can trade off the number transmitted bits per symbol for a
larger minimum distance between the adjacent decision boundaries. We will use pre-equalization
techniques to avoid or to reduce the ISI. These methods will be described in the next chapters.
The computation of the probability of symbol error for the outermost signal points qN is
slightly different. Given that qN is transmitted, the probability of symbol error is computed as:

P (error|qN ) = P (r < τ(N −1) |qN )

 τ(N−1) −qN (3.13)
= fW (w) dw
−∞
6
Assuming that fW (w) is a continuous function, we can verify the optimal choice of qi by taking the derivative of
the integral with respect to qi . The value of qi that maximizes the integral in Eq.(3.11) should satisfy the following
equation:  τi −qi
∂
fW (w) dw = 0
∂qi τi−1 −qi
By taking the derivative of the integral, we have:

−fW (τi − qi ) + fW (τi−1 − qi ) = 0

Since fW is symmetrical and also monotonic on each side of the center, we find that τi − qi = qi − τi−1 . Solving
this equation for qi , we determine the optimal value of qi as given in Eq.(3.12). By taking the second derivative
of the integral in Eq.(3.11) and evaluating the result at qi , we can verify that this point indeed corresponds to a
maximum for the integral, or a minimum for the probability of symbol error.
3 Modulation Design for the Up-Stream PCM Channel 35

3 U _ TL 3 U _ TL

TL

τ L − TL τL τ L − τL

τ L − + τ L T L = τ L −
D TL = E


Fig. 3.4 Two cases of conditional probability distribution functions of the received
signal, given that qi is transmitted. In case (a) the PDF has a single peak. To
maximize the integral of the right-hand side of Eq. (3.11), the signal point qi should
fall midway between detector thresholds τi−1 and τi . In case (b), the optimal selection
of the signal level is not unique. In this particular example, the probability of symbol
error for qi is larger than 1/2. Such an extreme case is caused by severe ISI and/or a
short distance between detector thresholds.

From Eq. (3.13), we note that if the value of qN increases to infinity, P (error|qN ) tends to zero.
However, there is a constraint on the average transmitted signal power:


N
2 P (qi ) qi2 ≤ Pave (3.14)
i=1

where Pave is the average transmitted power in a linear scale. The power constraint Eq. (3.14)
leads to the following expression for the outermost signal point:

 −1
 Pave N
 P (qi ) qi2
 2 −
 i=1
qN = . (3.15)
P (qN )

Note that we assume the detector thresholds and transmitted power are chosen such that a solution
exists. We define a virtual decision boundary:

τN = 2qN − τ(N −1)

so that Eq. (3.12) also holds for the outermost signal points.
3 Modulation Design for the Up-Stream PCM Channel 36

3.2.2 Remarks

• For a given set of detection thresholds, the optimal PAM symbols fall midway between
each pair of adjacent thresholds if the PDF of the additive distortion is symmetrical and
monotonic on each side of its mean. In more general cases, the PDF of additive distortion
is required to identify the optimal signal levels.

• The minimum probability of symbol error is computed as:

N
−1  δi  ∞
2
min Pe = 1 − 2 P (qi ) fW (w) dw + P (qN ) fW (w) dw (3.16)
qi −
δi
−
δN
i=1 2 2

where δi = τi − τi−1 is the distance between two successive decision boundaries. From
Eq. (3.16), it is clear that increasing the distance between decision boundaries will reduce
the probability of error.

• The actual symbol error probability depends on the PDF of w[n]. As an example, consider a
Gaussian distribution for w[n] with zero mean and a standard deviation of σ. The minimum
probability of symbol error is simplified to:

 N
−1
δi δN
min Pe = 2 2 P (qi )Q( ) + P (qN )Q( ) (3.17)
qi 2σ 2σ
i=1

where Q(.) is the area under the tail of the Gaussian PDF:
 ∞
1 −u2
Q(x) = √ e 2 du (3.18)
2π x

Note that, at a high signal-to-noise ratio operating point, the probability of error is deter-
mined by the minimum distance between the adjacent decision boundaries:

δmin
P (error) ≈ KQ( ) (3.19)
2σ

where K is proportional to the number of signal levels with the minimum distance between
their decision boundaries.

• The up-stream PCM channel imposes a power limit on the transmitted signal. We can
take into account the energy cost of each constellation point by choosing a non-uniform
probability distribution for the constellation points. Examples of non-uniform probability
distribution for constellation points will be discussed later in this chapter.
3 Modulation Design for the Up-Stream PCM Channel 37

• The minimum probability of error is a function of the distances between detector thresholds.
In the up-stream PCM channel, the threshold levels are a subset of the PCM encoder decision
boundaries. These boundaries are determined based on a non-linear companding rule (i.e.
A-Law or µ-Law) with non-uniform spacing. Although companding rules are efficient for
voice communication, the choice of non-uniform spacing between threshold levels is not
optimal for data communication over the PCM channel. It can be shown that for equally
probable PAM constellation points with a constraint on average transmitted power, equal
spacing between signal constellation points minimizes the probability of symbol error at a
high signal-to-noise ratio [31].

3.2.3 Performance results

In this section, we examine the performance of PAM modulation with non-uniform symbol spacing
over the upstream PCM channel. We model the channel distortion w[n] as an additive Gaussian
component7 .
As a baseline, we use conventional M-ary PAM modulation with equally spaced and equally
likely constellation points. The probability of symbol error for the conventional PAM modulation
over an AWGN channel is computed as [32]:

2(M − 1) 3 Pave
PM (error) = Q (3.20)
M (M 2 − 1)σ 2

where σ 2 and Pave are the noise and signal power in linear scale. The average signal power is:

1  2
M
Pave = qi (3.21)
M
i=1

The average power limit for a transmitted signal over a voiceband channel is around −12 dBm0. As
described in Section 2.3.4, the outermost decision boundary of a µ-Law PCM encoder corresponds
to the maximum amplitude of a sine-wave with 3.17 dBm0 power. The signal power in the dBm0
scale is computed as:
2
PdBm0 = 3.17 + 10 log10 Pave . (3.22)
(8159)2
From Eq. (3.20), it is clear that the Symbol Error Rate (SER) for uniform PAM modulation
is a function of the signal-to-noise power ratio Pave /σ 2 , and not the absolute value of the signal
or noise power.
7
The additive Gaussian random variable can be either a model for the noise or an approximate model for the
combined noise and ISI.
3 Modulation Design for the Up-Stream PCM Channel 38

In our examples, we consider PAM constellations with 32, 64 and 128 signal points. These
constellations are designed for an up-stream PCM channel with a µ-Law encoder at the receiver.
Depending on the signal power, different subsets of the encoder decision levels are selected as
detector thresholds. The criterion for selecting the detection thresholds is the maximization of
the minimum distance between adjacent boundaries while maintaining average power constraint.
As discussed in the previous section, the PAM constellation points are chosen at the mid-point of
each decision interval.
Figure 3.5 shows symbol error probabilities in terms of the signal-to-noise power ratio and the
average signal power for non-uniform 32-PAM modulations. As a benchmark, the probability of
symbol error of a uniform 32-PAM is also depicted. For a given constellation size, the probability
of symbol error for a uniform PAM is only a function of the signal-to-noise power ratio. Figure 3.5
shows that, for a non-uniform PAM (designed for the up-stream PCM channel), the probability
of symbol error depends on both signal and noise power. Note that for a non-uniform PAM
modulation over a PCM channel, an increase in average signal power does not necessarily cause
an increase in the minimum distance between constellation points (or more precisely, detector
thresholds). This point is further explained in the example described below.

−1
10
µ Law, Pave=−8 dBm0
µ Law, P =−10 dBm0
−2 ave
10 µ Law, P =−12 dBm0
Probabilty of Symbol Error

ave
Uniform Spacing
−3
10

−4
10

−5
10

−6
10
35 36 37 38 39 40 41 42 43
2
Pave/σ (dB)

Fig. 3.5 Performance results of 32-PAM modulation designed for the up-stream
PCM channels. The performance results are compared to those of conventional 32-
PAM signals. The distortion is modeled as additive white Gaussian noise. Compared
to the conventional 32-PAM, the non-uniform spacing between constellation points
causes performance degradations.
3 Modulation Design for the Up-Stream PCM Channel 39

As shown in Eq. (3.19), at a high signal-to-noise ratio, the probability of symbol error is
determined by the minimum distance between adjacent detector thresholds. Since the detector
thresholds are a subset of predetermined values (i.e. the PCM encoder decision boundaries), an
increase in average signal power does not always change the minimum distance between selected
detector thresholds. For example, consider the performance results shown in Fig. 3.5. The
minimum distance between adjacent detector thresholds are 64, 128 and again 128 unit counts
while the average signal powers are −12 dBm0, −10 dBm0 and −8 dBm0 respectively8 . For the
same noise power level, the performance of the two PAM constellations with -10 dBm0 and -8
dBm0 average signal power levels are the same. Results shown in Fig. 3.5 confirm this, since
the difference between the two performance curves at a high SNR is 2 dB which accounts for the
difference in signal powers. In other words, in this example, increasing the average signal power
from −10 dBm0 to −8 dBm0 does not improve performance.
Let us now compare the performance curve of a signal with a −12 dBm0 average power to
that of signal with −10 dBm0 average power. At −12 dBm0 average signal power, the minimum
distance between adjacent detector thresholds is 64 unit counts, or half of the minimum distance
of the PAM signal with −10 dBm0 average signal power. From Eq. (3.19), for the same probability
of symbol error, the noise power for the signal with the average power of −12 dBm0 is four times
(or 6 dB) larger than that of the signal with −10 dBm0 signal power9 . The difference in signal
power and noise power together account for around 4 dB difference in the SNR between the two
performance curves. Results shown in Fig. 3.5 confirm this difference.
Performance results for the 64-PAM and the 128-PAM are shown in Fig. 3.6 and Fig. 3.7
respectively. The results for the 64-PAM show the same trend as those for the 32-PAM. For the
128-PAM, the minimum distance between adjacent detector thresholds is the same for all three
average signal power levels. Therefore, increasing the average signal power does not improve
performance. Note that the performance gap between uniform and non-uniform modulation for
the 128-PAM is larger than that for the 64-PAM and the 32-PAM. For a larger constellation size,
the spacing between constellation points becomes more non-uniform and the minimum distance
between adjacent detector thresholds becomes relatively smaller.
Figure 3.8 compares the performance results for different constellation sizes with a fixed av-
erage signal power of −12 dBm0. At a high SNR, the cost of maintaining the same performance
when we add one more bit per symbol (doubling the constellation size) is around 6 dB.
8
The outermost decision boundary of the PCM encoder corresponds to 8159 unit counts.
9 δ1
If K1 Q( 2σ 1
) ≈ K2 Q( 2σ
δ2
2
) and δ2 = 2δ1 then σ22 = 4σ12 where the impact of K is insignificant.
3 Modulation Design for the Up-Stream PCM Channel 40

−1
10
µ Law, Pave=−8 dBm0
µ Law, P =−10 dBm0
−2 ave
Probabilty of Symbol Error 10 µ Law, P =−12 dBm0
ave
Uniform Spacing
−3
10

−4
10

−5
10

−6
10
40 42 44 46 48 50
2
Pave/σ (dB)

Fig. 3.6 Performance results for non-uniform 64-PAM modulation designed for the
up-stream PCM channels.

−1
10
µ Law, Pave=−8 dBm0
µ Law, P =−10 dBm0
−2
ave
10 µ Law, P =−12 dBm0
Probabilty of Symbol Error

ave
Uniform Spacing
−3
10

−4
10

−5
10

−6
10
48 50 52 54 56 58
2
Pave/σ (dB)

Fig. 3.7 Performance results for the non-uniform 128-PAM designed for the up-
stream PCM channels.
3 Modulation Design for the Up-Stream PCM Channel 41

−1
10
32−PAM
64−PAM
10
−2 128−PAM

Probabilty of Symbol Error

−3
10

−4
10

−5
10

−6
10
40 45 50 55
Pave(−12 dBm0)/σ2 (dB)

Fig. 3.8 Performance results for non-uniform PAM modulation with an average of
−12 dBm0 signal power.

3.2.4 How to choose detector thresholds

As stated previously, detector thresholds are selected from the set of PCM encoder decision
boundaries. There are several conflicting conditions effecting the selection of detector thresholds.
For a given constraint on the average signal power, we wish to achieve a high transmission bit-rate
while keeping a low probability of symbol error. We consider two different approaches in selecting
the detector thresholds.

Selecting thresholds based on a given minimum distance

The minimum distance between adjacent decision thresholds determines the modulation perfor-
mance (expressed in terms of the symbol error rate). For a given signal and noise power, we
can set a value for the minimum distance between the detector thresholds, say Dmin , to obtain
a desired performance. For a given value of Dmin , the selection of the detector thresholds can
be accomplished by applying a pruning procedure to the PCM encoder decision boundaries. The
procedure of selecting detector thresholds τi can be described as follows:

1. Set the origin as a detector threshold τ0 = 0.

2. Select the smallest PCM decision boundary that is greater than Dmin as the first positive
threshold τ1 . For symmetry, select τ−i = −τi as the first negative threshold.
3 Modulation Design for the Up-Stream PCM Channel 42

3. At each stage, consider τi such that τi − τi−1 ≥ Dmin . Continue until the last PCM decision
boundary is reached.

4. Compute the constellation points qi using Eq. (3.12). For a given distribution of symbol
probabilities P (qi ), compute the average signal power. Keep a subset of constellation points,
so that the average signal power constraint given in Eq. (3.14), is satisfied10 .

In this procedure, the number of PAM symbol levels is determined based on Dmin and the
distribution of the symbol probabilities.

Selecting thresholds based on a given number of symbol levels

In this approach, a fixed number of detector threshold levels is selected so that the average
signal power is satisfied and the probability of symbol error is minimized. The minimum distance
between adjacent detector thresholds is used as a measure of the modulation performance. For
a PAM system with a large constellation size, an exhaustive search to determine the optimal set
128!
of thresholds levels is impractical. For a 2N -PAM symmetrical constellation, there are N !(128−N )!
ways of selecting the thresholds.
We use a suboptimal approach in selecting thresholds. Starting from a set of equally-spaced
constellation points, we adjust the levels so that the average power constraint is satisfied. A brief
description of the selection algorithm is given below:

1. Design a 2N -PAM constellation with equally-spaced points with average signal power below
a pre-set level Pmax .

2. Replace each positive threshold τi found in the previous stage with the nearest PCM decision
boundary that is greater or equal to τi .

3. It is possible that more than one threshold is mapped to a decision boundary. Select an
appropriate number of the outmost unused decision boundaries that are unused to maintain
2N distinct thresholds.

4. Compute the constellation point and the average signal power. If the average signal power
is larger than the constraint, reduce Pmax and start from step 1.
10
As we will see in the next sections, an appropriate distribution of symbol probabilities can allow for a larger
constellation size while satisfying the average power constraint.
3 Modulation Design for the Up-Stream PCM Channel 43

3.2.5 Bit-to-symbol mapping

In this section, we describe methods of mapping the binary data into symbol levels. For a 2N -
PAM modulation, the maximum number of bits transmitted per symbol is log2 (2N ) bits. If
the constellation size 2N is not an integer power of 2, the bit-to-symbol mapping on a symbol-
by-symbol basis cannot reach the maximum bit rate of log2 (2N ) bits/symbol. In this case, a
bit-to-symbol mapping performed on a frame-by-frame basis can increase the average number of
bits transmitted per symbol. For a PCM channel, the overall increase of bit-rate due to frame-
by-frame bit assignment, as opposed to symbol-based bit mapping, can be up to (but less than)
8 kbits/sec. As an example, Fig. 3.9 shows the bit-rate of a 58-PAM modulation as a function of
the number of symbols per frame (K). The maximum number of bits per symbol in this case is
equal to log2 (58). If we choose 6 or 7 symbols per frame, the average number of bits per symbol is
close to this maximum. Note that the average number of bits per symbol does not monotonically
increase when we increase the number of symbols per frame11 .

48
8* log2(58) kbits/s
46
Bit Rate (kbits/sec)

44

42

40

38

36

34

2 4 6 8 10 12 14
Number of symbols per frame (K)

Fig. 3.9 The average transmitted bit-rate as a function of symbols per frame on
a PCM channel. The number of levels per symbol (constellation size) is 58. The
bit-rate shown in this figure is based on an 8000 symbol/sec transmission.

The mapping from a string of input bits to a frame of symbols can be viewed as an index
assignment problem, where each point in a K-dimensional constellation should be uniquely spec-
ified by an Nb -bit binary index. There are two important issues concerning the index assignment
11
However, one can show that for a 2N -PAM signal, there exists a value for K such that the average number of
bits transmitted per symbol is as close as desired to log 2 (2N ).
3 Modulation Design for the Up-Stream PCM Channel 44

strategy:

1. due to a large multi-dimensional constellation size ((2N )K ), the bit-to-symbol mapping

should be such that it can be implemented algorithmically rather than in a look-up-table.

2. the bit-to-symbol mapping should be such that an error event due to the channel imperfec-
tions causes a small number of bit errors. In other words, we would like two constellation
points with a small Euclidean distance to be assigned indices which differ in as few bit po-
sitions as possible (i.e., indices of adjacent points should have a small Hamming distance).

In the rest of this section, we first describe a simple index mapping algorithm that has been
used in The V.90 Standard. We refer to this method as natural index mapping. Since the
natural index mapping algorithm does not provide the minimum Hamming distance between
adjacent points, we investigate other methods of index mapping. We propose a new index mapping
algorithm that can be viewed as a generalization of Gray encoding [32, p. 175] in multi-dimensional
space.

Natural Index mapping

A simple method for index assignment is by the natural indexing of constellation points. Suppose
the PAM symbol alphabet contains 2N levels, each frame consists of K symbols and the frame
index is an Nb -bit binary sequence. For a unique index assignment, the following inequality should
hold:
2Nb ≤ (2N )K .

Each index represents an Nb -digit number in base-2, while each constellation point in a K-
dimensional frame is naturally represented by a K-digit number in base-(2N ). A natural index
assignment would be to convert the representation of the number from base-2 to base-(2N ).

(D(K−1) . . . D1 D0 )(2N ) = (b(Nb −1) . . . b1 b0 )2 (3.23)

At the receiver, the reverse conversion from base-(2N ) to base-2 is used to recover the binary
sequence. The natural index assignment can be implemented algorithmically using modulo arith-
metics. The V.90 Standard uses this algorithm for bit-to-symbol mapping [13] described as
“modulus encoding”12 .
A drawback of the natural index mapping is that the average Hamming distance of the indices
of the adjacent points can be relatively high13 . In an extreme case, a pair of constellation points
12
A similar method of bit-to-symbol mapping has been proposed for recommendation V.92.
13
In our discussion, we consider two constellation points in a K-dimensional space to be adjacent if their coordinate
positions are the same except for one coordinate that their values differ with a minimum amount.
3 Modulation Design for the Up-Stream PCM Channel 45

with a small Euclidean distance might correspond to two indices with a maximum Hamming
distance (bits in all positions in the two indices are different). When we use the natural index
mapping, the average Hamming distance between adjacent constellation points increases when
the number of symbols per frame increases. As an example, we consider a 58-PAM modulation
and investigate the average Hamming distance of indices of the adjacent points as a function of
K. If we assume that channel signal-to-noise ratio is relatively high, a symbol in error is most
likely detected as one of its neighbouring points. We also assume that the probability of more
than one symbol error in each frame is negligible. Based on the above assumptions, the bit error
rate (BER) can be approximated as a function of the probability of symbol error:

K
Pb ≈ dH Ps (3.24)
Nb

where K is the number of symbols per frame, Nb is the number of bits transmitted per frame and
dH is the average Hamming distance between the indices of the adjacent points.
Figure 3.10 shows the average Hamming distance of indices of adjacent points as a function
of K. For a natural index mapping, the Hamming distance between adjacent points increases
when the number of symbols per frame (K) increases. A more efficient index mapping strategy
can reduce the probability of bit error by reducing the Hamming distance between the indices of
adjacent points.

10
Average Hamming distance

0
1 2 3 4 5 6 7
Number of symbols per frame (K)

Fig. 3.10 The average Hamming distance between indices of adjacent points is
shown as a function of K for a 58-PAM modulation
3 Modulation Design for the Up-Stream PCM Channel 46

A Hybrid Gray-Natural (HGN) index mapping

The mapping of information bits to a one-dimensional PAM signal can be performed by a Gray
encoding [32, p. 175], in which the indices of adjacent symbols differ by only one bit (a Hamming
distance of one). However, the conventional Gray encoding is not directly applicable to a frame
of symbols when the number of bits assigned to each symbol is not integer.
Here, we introduce a Hybrid Gray-Natural (HGN) index mapping that partitions the constel-
lation points into two regions: one region where indices are assigned by using Gray encoding,
and a second region where indices are obtained by a mapping similar to natural index mapping.
Compared to the natural index mapping, the HGN index mapping reduces the Hamming distance
between indices of adjacent points. The HGN index mapping can be implemented algorithmically,
without using a large look-up table.
We describe the HGN mapping by a simple example. Consider a frame of 2 symbols (K = 2)
with a 6-PAM (2N -PAM) constellation per symbol. The number of bits transmitted per frame is
Nb = 5. Each 2-dimensional constellation point in a frame can be specified by a number in base-6
as (D1 D0 )6 where D1 and D0 take integer values between 0 and 2N − 1 = 5. Figure 3.11 shows
the 2-dimensional constellation points in each frame (region(A) and region(C)).
Each 5-bit index is parsed into two segments (corresponding to two symbols). The first three
bits of each index are mapped into an 8-level Gray encoded symbol G0 . The last two bits of each
index are used to specify a 4-level Gray encoded symbol G1 .
If G0 is less than 2N = 6, the combination of (G1 G0 ) can be used directly to represent a
constellation point in the frame:
(D1 D0 )6 = (G1 G0 )

This constellation is in region (A) (see Fig. 3.11).

If G0 is greater or equal to (2N = 6), the combination of (G1 G0 ) is not a legitimate constel-
lation index (i.e., a point in region (B)). We need to apply a second mapping to project points
in region (B) into points in region (C). Points in region (B) and (C) are separately labeled. A
point from region (B) is mapped to a point in region (C) which has the same integer label14 .
14
In order to specify the mapping, we define new coordinates for region (B) and region (C). The new coordinates
are constructed by a simple shift of axis. For region (B) we define:

E0 = G0 − 5
(3.25)
E1 = G1

where 0 ≤ E0 ≤ 1 and 0 ≤ E1 ≤ 3. We present each point in region (C) with a new coordinate (F1 F0 ) where
0 ≤ F0 ≤ 5 and 0 ≤ F1 ≤ 1. In order to specify F1 and F0 , we use the following equation:

6F1 + F0 = E1 + 4E0 (3.26)

3 Modulation Design for the Up-Stream PCM Channel 47

E1 (0-3)

4 5 6 7
G0(0-7)

E0(0-1)

00100 01100 11100 10100

0 1 2 3
B
00101 01101 11101 10101

00111 01111
A 11111 10111 01100
C
4
D0(0-5)

00110 01110 11110 10110 00100

00010 01010 11010 10010 10101

F0(0-5)
2

00011 01011 11011 10011 11101

1 7

00001 01001 11001 10001 01101 10100

0 6

00000 01000 11000 10000 00101 11101

F1 (0-1)

G1 (0-3)
D1 (0-5)

Fig. 3.11 Bit-to-symbol mapping based on hybrid natural-Gray encoding.

 3 Modulation Design for the Up-Stream PCM Channel 48

Since region (C) contains more points than region (B), some points in region (C) are not used.
Figure 3.11 shows the index of each constellation point. For this particular example, there are
four unused constellation points in region (C).
As shown in the above example, the HGN index mapping creates a Gray encoded region (A)
where the indices of adjacent points have a Hamming distance of one. For points in region (C),
the Hamming distance between adjacent points can be larger than one.
In general, the proposed index mapping can be summarized in the following steps:

Step 1 For a given constellation 2N -PAM and a frame size K, the number of bits per each frame
index is computed as: Nb = K log2 (2N ) . We parse each Nb index bits into K segments.
The number of bits assigned to each segment is Nbi which takes an integer value equal to
log2 (2N ) or log2 (2N ) while:

Nb = Nb1 + Nb2 + · · · + NbK .

We choose the first M segments to have log2 (2N ) bits per segment and the last K − M
segments to have log2 (2N ) bits per segment15 . For example, for a 58-PAM with 6 symbols
per frame, the number of bits per each frame index is 35. Each frame index is parsed into
6 segments where the first segment contains 5 bits and the next 5 segments contain 6 bits
per segment.

Step 2 We use a Gray encoding to map K segments into K indices. If these K indices are all less
than 2N , they represent a valid point in the constellation space (region (A) in Fig. 3.11).
If one or more indices are greater or equal to 2N , we will use a second index mapping, as
described in the next step.

Step 3 If the indices generated by Gray encoding do not represent a valid constellation point (region
(B) in Fig. 3.11 ), we use a natural index mapping to convert the Gray encoded indices into
indices of a valid constellation point (region (C)). We label each point in region (B) and
region (C) by an integer number. The labeling is performed by natural indexing of points
in each region. A point from region (B) is mapped into a point in region (C) that has the
same integer label.

At the receiver, a constellation point that belongs to the Gray-mapping region (region (A))
15
It is easy to show that the number of segments with log2 (2N ) bits is:

M = K − K log 2 (2N ) + Klog2 (2N )

3 Modulation Design for the Up-Stream PCM Channel 49

can be easily decoded to a sequence of bits. For points in region (C), a reverse mapping of Step
3 should be applied.
We use the hybrid mapping for a 58-PAM signal. Figure 3.12 shows the average Hamming
distance of the indices of adjacent point as a function of K. Compared to the previous results
presented in Fig. 3.10, the avenge Hamming distance between adjacent points is significantly
reduced. We investigated the impact of the index mapping on the probability of bit error. As
an example, we applied the natural index mapping and the HGN index mapping to a 58-PAM
signal. Figure 3.13 shows that the HGN index mapping outperforms the natural index mapping.
As a lower bound of the probability of error, we assume there exists an index mapping that can
maintain a Hamming distance of one between the indices of the adjacent points. The minimum
probability of bit error can be expressed as:

K
min Pb ≈ Ps (3.27)
Nb

where K is the the number of symbols per frame, Nb is the number of bits transmitted per frame
and Ps is the probability of symbol error. This lower bound is also presented in Fig. 3.13. The
question of the existence of a better index mapping with reasonable computation complexity and
storage size to recover the gap between the probability of the bit error and its lower bound is a
subject for further study.

10
Hybrid
Natural
8
Average Hamming distance

0
1 2 3 4 5 6 7
Number of symbols per frame (K)

Fig. 3.12 Compare the average Hamming distance generated by the natural in-
dex mapping with that generated by the Hybrid Gray-Natural index mapping. The
number of levels per symbol is 58.
3 Modulation Design for the Up-Stream PCM Channel 50

−2
10

−3
10
Probability of Bit Error

−4
10

−5
10
Natural index mapping
Hybrid index mapping
A lower bound
−6
10
40 41 42 43 44
2
Pave(−12 dBm0)/σ (dB)

Fig. 3.13 Probability of bit error for a 58-PAM modulation over a PCM up-stream
channel. The probability of bit error based on natural index mapping and hybrid
natural-Gray index mapping is compared with a lower bound. The lower bound is
based on this assumption that the Hamming distance between the indices of the
adjacent points is equal to one.

Related work on index assignment

In quantization theory, problems similar to what we described here as index mapping have been
investigated. In a vector quantization of an analog source, a proper index assignment to codevec-
tors can reduce the distortion caused by an error in the bits representing each index. The index
error is caused by a noisy transmission channel.
Simulated annealing [33], pseudo Gray codes via binary switching [34] and Hadamard Trans-
form [35] [36] are several techniques suggested for the index assignment.
Despite the fact that there are similarities between the index assignment problem and the
bit-to-symbol mapping, there are subtle but significant differences between the two problems:

• The constellation points specified in frame of symbols are structured over a grid. Although
signal levels may not be equally spaced, the spacing is the same for different symbols. On
the other hand, the codevectors specified by a vector quantizer do not follow a specific
structure. Therefore, the only way to specify the distance between two codevectors is
Euclidean distance.
3 Modulation Design for the Up-Stream PCM Channel 51

• In general, in a vector quantization problem, the codevectors are stored in memory (unless
structured VQ is used). In that case the index assignment can be seen as re-ordering the
codebooks in the memory. In a bit-to-symbol mapping we would like to avoid any extensive
use of memory or a look-up table for index mapping.

Although we do not rule out any potential use of the above techniques in the index mapping
problem (or the possible use of HGN in the index assignment problem), we defer establishing
these links to future studies.

3.3 Non-uniform probability distributions for constellation points

In our performance analysis of PAM modulation, we assumed equally probable constellation

points. Here, we discuss a potential performance improvement that can be achieved by using a
non-uniform probability distribution for the PAM symbols. For a given PAM constellation size, the
average number of bits per symbol is maximized if the constellation points are equally probable
(i.e. maximum entropy per symbol). However, constellation design based on equally-probable
symbols does not take into account the energy cost of the outer constellation points.
Choosing constellation points with a smaller energy more often than points with a higher
energy reduces the average transmitted power. As a result, we can choose a larger constellation
size while maintaining the same average signal power. We will show that the overall impact of
increasing the constellation size with unequally-probable constellation points is an increase in the
average number of bits per symbol for a given average signal power.
In this section, we first describe the optimal probability distribution that maximizes the av-
erage number of bits per symbol, given a constraint on the average signal power. We apply the
probability distribution to our previous examples of PAM modulation over the PCM channel to
quantify the increase in the bit rate. As a baseline, we compute the upper bound of average
transmitted bits per symbol for a PAM modulation over an AWGN channel. Finally, we describe
the use of Huffman coding to map the information bits to a set of PAM symbols with non-uniform
probability distribution.

3.3.1 Maximizing the entropy

Considering the constraint on the average signal power, we would like to find the probability
distribution of a set of PAM symbols that maximizes the average transmitted bit-rate. From
the information theory point of view, the information content of a random signal is measured by
3 Modulation Design for the Up-Stream PCM Channel 52

entropy. The entropy H(X) of a discrete random variable X is defined as [37]:


H(X) = − PX (x) log2 (PX (x)) bits
x∈X

For a PAM signal with 2N constellation points, the entropy of transmitted symbols is computed
as:

N
H =− P (qi ) log2 P (qi ) (3.28)
i=−N
i=0

where P (qi ) is the probability of symbol qi . The entropy H(P ) is maximized if all symbols are
equally probable [37]:
max H(P ) = log2 (2N )
P

For a PAM signal with power constraint, the maximum entropy principle [37, p. 267] leads to
the following probability distribution, known as Maxwell-Boltzmann distribution, that maximizes
the number of bits received per symbol:

exp(−λqi2 )
P (qi ) =  (3.29)
exp (−λqi2 )
i

where λ is a positive parameter chosen such that the average signal power constraint is satisfied:

qi2 exp(−λqi2 )
i
 ≤ Pave (3.30)
exp (−λqi2 )
i

The left hand-side is a monotone decreasing function of λ. If the inequality does not hold for
λ = 0, we will solve Eq. (3.30) for λ under equality condition.

3.3.2 Examples of maximum entropy distributions for a PAM signal

Considering PAM modulation design for a PCM channel, we apply the maximum entropy dis-
tribution discussed above to the PAM constellation points. From Eq. (3.19), the probability of
symbol error is a function of the minimum distance between the adjacent detector thresholds.
By controlling the minimum distance, we can maintain a desired probability of symbol error at a
given signal-to-noise ratio.
As an example, we consider a PCM channel with a µ-Law compander. We choose a PAM
constellation with 156 points and a minimum distance of 32 unit counts between detector thresh-
3 Modulation Design for the Up-Stream PCM Channel 53

olds. Note that the maximum decision level is normalized at 8159 unit counts. Figures 3.14 and
3.15 show the maximum entropy probability distributions with average power constraints of −10
dBm0 and −12 dBm0 respectively. As shown in these figures, reduction of the average signal
power reduces the probability of selecting the outer signal points. Since the probability of select-
ing the outer points is significantly smaller than that of the inner points, we can eliminate a few
outmost points from the constellation without any significant change in the bit rate. The average
numbers of bits per symbol are 6.82 and 6.64 bits/symbol for average signal power of −10 dBm0
and −12 dBm0 respectively.
To quantify the impact of maximum entropy distribution on the bit rate, we consider a PAM
constellation with equally-probable symbols and an average signal power of −12 dBm0. We
maintain the same minimum distance between detector thresholds as that given in the above
example. Figure 3.16 confirms that for the SNR range in which we are interested, the SER
performance of the two schemes is similar.
The constellation size of the PAM modulation with uniform probability distributions is 90
which accounts for log2 (90) = 6.49 bits/symbol. Since the symbol rate for the up-stream PAM
channel is 8000 symbols/sec, the bit-rate of the PAM modulation with uniform probability distri-
bution is 1.22 kbits/sec lower than the bit-rate of a PAM modulation with the maximum entropy
probability distribution.
Figure 3.17 shows the bit-rate versus the SNR for an average signal power of −12 dBm0.
Results are shown for the uniform probability distribution of constellation points as well as the
maximum entropy distribution. As shown in this figure, the maximum entropy probability dis-
tribution can increase the bit rate by up to 1.8 kbits/sec. This can account for up to 1.5 dB
performance improvement. In fact, it can be shown that maximum performance improvement
due to optimal probability distribution for constellation points is πe/6 or 1.53 dB [38] [39]. Note
that at higher transmission bit-rates, the performance difference between the uniform probability
distribution and the maximum entropy distribution becomes smaller. For PAM modulation on a
PCM channel, the maximum constellation size is limited to 256 points. At a high bit-rate most
of these points are in use. Therefore, there is no gain by using a maximum entropy distribution
at high transmission bit-rates since there are no extra signal points available to be added to the
constellation.

3.3.3 Channel capacity for non-uniform PAM modulation

In this section, we compute the channel capacity for the up-stream PCM voiceband channel.
The channel capacity can be defined as an upper bound for information transmission rate which
allows for transmission with an arbitrary small probability of error [37]. We compare three
3 Modulation Design for the Up-Stream PCM Channel 54

0.012

Probabilty Pave= −10 dBm0

0.008

0.004

0
0 2000 4000 6000 8000
Positive Signal Levels

Fig. 3.14 Probability distribution of the PAM symbols. The average signal power
is limited to −10 dBm0. Since the distribution is symmetrical, only positive constel-
lation points are shown. Constellation points with low probability can be eliminated
without any significant change in the average number of bits transmitted per symbol.

0.012
Pave= −12 dBm0
Probabilty

0.008

0.004

0
0 2000 4000 6000 8000
Positive Signal Levels

Fig. 3.15 Probability distribution of the PAM symbols. The average signal power
is −12 dBm0. Compared to the probability distribution shown in Fig 3.14 for Pave =
−10 dBm0, the probability distribution shown here is obtained by using a larger value
of λ in Eq. 3.30.
3 Modulation Design for the Up-Stream PCM Channel 55

−1
10

−2
10
Probabilty of Symbol Error
−3
10

−4
10

−5
10 Maximum Entropy Distribution
Uniform Probability Distribution
−6
10
40 42 44 46 48 50
Pave(−12 dBm0)/σ2 (dB)

Fig. 3.16 Probability of symbol error for PAM constellations with a maximum
entropy distribution and uniform probability distribution. Note that the average
signal power and the minimum distance between detector thresholds are the same for
both constellations.

P(Symbol Error)=10−6
58
186−PAM

56
128−PAM
Bit Rate (kbits/sec)

54
156−PAM

52
90−PAM

50
130−PAM
Uniform Probability Distribution
48 Maximum Entropy Distribution

46 58−PAM
44 46 48 50 52 54 56
Pave(−12 dBm0)/σ2 (dB)

Fig. 3.17 Transmitted bit rates of PAM modulation over a PAM channel. The
average signal power is limited to -12 dBm0 and the probability of symbol error is
set to 10−6 . Results are shown for uniform probability distributions and maximum-
entropy distribution. The symbol rate is 8000 symbols/sec.
3 Modulation Design for the Up-Stream PCM Channel 56

different bounds for the information rate. Bounds are derived based on three different channel
assumptions:

1 Continuous AWGN channel,

2 PAM signalling over AWGN channel,

3 PAM signalling over AWGN channel with a Hard-decision detector.

The capacity of a Gaussian channel with a constraint average input signal power of Pave and
a noise power of σ 2 . The capacity of this channel is computed as [37]:

1 Pave
C= log2 (1 + 2 ) bits per transmission. (3.31)
2 σ

Since the transmission rate over the PCM channel is Rs = 8000 symbols/sec, the capacity of the
Gaussian channel is shown in:

Rs Pave
Rmax = log2 (1 + 2 ) bits/sec.
2 σ

The output of a PCM up-stream channel at each sampling instant can be written as16 :

r[n] = a[n] + w[n] (3.32)

where a[n] is a PAM symbol and w[n] is assumed to be additive white Gaussian noise. The
capacity can be expressed as the maximum difference between the entropy of the received signal
and the conditional entropy of the received signal.

C = max H(R) − H(R|A) . (3.33)

P (qi )

In our discussion, the transmitted signal a[n] is chosen from a set of PAM constellation points.
We also assume that the a priori probability associated with each constellation point is given
(equally probable or maximum-entropy distribution). The capacity of a channel with M-ary PAM
modulation can be written as [40, 41]:


M  ∞
P (r|a = qi )
CPAM = P (qi ) P (r|a = qi ) log2 dr
−∞ P (r)
i=1
   (3.34)

M ∞
P (r|a = qi )
= P (qi ) P (r|a = qi ) log2 M dr
i=1 −∞ i=1 P (qi )P (r|a = qi )
16
Note that compared to Eq. (3.7), here we assume n0 = 0 and h[n0 ] = 1.
3 Modulation Design for the Up-Stream PCM Channel 57

For a given set of P (qi ), we can simplify the capacity as:

 P (qi )  ∞ 
1
M
 (r − qi )2  M
P (qj )  (r − qj )2 
CPAM = − log2 (2πσ 2 e) + √ exp log 2 √ exp dr
2 2πσ 2 −∞ −2σ 2 2πσ 2 −2σ 2
i=1 j=1
(3.35)
where σ is the standard deviation of the noise. Since the integral on the right-hand side of
Eq. (3.35) does not have a closed form, the channel capacity is computed numerically. Following
the method suggested in [40] to compute the above integral, we replace the integration by ex-
pectation over the Gaussian distributed noise variable and use Monte Carlo averaging method to
evaluate the capacity.
The capacity bound computed above does not include the hard decision detector at the re-
ceiver. For the up-stream PCM channel, the receiver front-end structure is predetermined. This
channel can be modeled as a channel with discrete input and discrete output. To compute the
capacity of this channel, we can compute the transmission probabilities P (r = rj |a = qi ) and
replace the result in Eq. (3.34). Note that the integral is replaced by a sum.
Figure 3.18 shows the maximum bit rate for PAM modulations over PCM channels. Two cases
of symbol probability distributions are considered: uniform and maximum entropy. For each PAM
modulation with a given bit-rate17 , the SNR at which the probability of symbol error reaches 10−6
is also depicted. Figure 3.18 compares three upper bounds for the channel information rate.
As shown in Fig. 3.18, at a given bit-rate there is a SNR gap between the curve associated
with the Gaussian channel capacity and the one associated with the channel capacity for a PAM
modulation. This gap is smaller when we use maximum entropy probability distribution for the
PAM constellation rather than uniform distribution.
We use the channel capacity of the PAM modulation as a baseline for our comparison. For an
acceptable probability of symbol error (e.g., 10−6 ), there is a significant SNR difference between
what we achieve by using symbol-by-symbol detection and the bound provided by the channel
capacity for the PAM modulation. In order to reduce this difference, error correction coding
techniques should be employed. Here, we do not discuss any particular error-correcting method
for the PCM channels.

3.3.4 The Huffman algorithm for bit-to-symbol mapping

In the preceding section, we showed that the maximum entropy distribution for the PAM symbols
improves the maximum achievable transmission rate. In this section, we describe a method which
will almost achieve the maximum entropy distribution by appropriate mapping of input bits to
17
The PAM constellation size along with the probability distribution of the constellation points determines the
bit-rate.
3 Modulation Design for the Up-Stream PCM Channel 58

60

58

56

54 Pe=10−6 (ME)156−PAM
Bit Rate (kbits/sec)

52
Pe=10−6 (U)90−PAM
50 −6
Pe=10 (ME)130−PAM
48

46 Pe=10−6 (U)58−PAM

44
Rs/2 log2 (1+Pave/ σ2 )
42
PAM signalling, AWGN Channel
40 PAM Signalling, Discrete Channel

38
35 40 45 50
Pave(−12 dBm0)/σ2 (dB)

Fig. 3.18 Channel capacity of a Gaussian channel with a continuous input signal
and a PAM-modulated input signal is presented. We consider two types of probability
distributions for the input PAM signal: uniform(U) and maximum entropy (ME).
The average signal power is limited to −12 dBm0. At each bit-rate, the SNR values
corresponding to a probability symbol error of 10−6 for an uncoded system are also
depicted. The symbol rate is set to 8000 symbols/sec.
3 Modulation Design for the Up-Stream PCM Channel 59

PAM constellation points. The input is assumed to be a random binary sequence of independent
and identically distributed samples with equal probabilities of zeros and ones. The input sequence
of bits is parsed into codewords and each codeword is mapped to a PAM symbol18 . Since the
probability of occurrence of a codeword of length li is 2−li , codewords with variable length can
produce a non-uniform probability distribution for the constellation points.
Suppose C(qi ) is a codeword of length li that is mapped to a PAM constellation point qi .
We consider the following two sets of conditions as design requirements for the bit-to-symbol
mapping. The first set of conditions are used to ensure that all combinations of bits are uniquely
represented by PAM symbols:

1. Different symbols should be represented by different codewords:

qi = qj ⇒ C(qi ) = C(qj )

2. A sequence of input bits should be uniquely mapped to a string of PAM symbols.

3. Each codeword C(qi ) should be instantaneously decodable to a symbol. In other words, no

codeword should be a prefix of any other codeword.

4. The set of codewords should be complete, i.e., for any random sequence of input bits there
should be one, and only one string of output symbols.

The second set of conditions ensure bit-to-symbol mapping does not change the signal mean
and average power:

5. The resulting probability distribution of the PAM constellation should be symmetrical, i.e.
P (qi ) = P (q−i ). We consider this condition to ensure the PAM signal has a zero mean.

6. The average signal power should satisfy the constraint given in (3.14).

Note that bit-to-symbol mapping can be viewed as a dual problem of encoding a discrete
memoryless source of information. In a source coding problem, we are interested in an efficient
(e.g. minimum number of bits, or minimum amount of redundancy) representation of a sequence of
symbols as a binary sequence. In bit-to-symbol mapping, we start from a sequence of bits with no
redundancy with a view to generate a set of symbols with the desired probability distributions.
Bit-to-symbol mapping is in fact equivalent to decoding an encoded source. From this point
of view, the design of a bit-to-symbol mapper is similar to the design of source coding for a
18
In a more general case, a frame of input bits is mapped to a set of PAM symbols. In our discussion in this
section, we assume that the bit-to-symbol mapping is performed in a symbol-by-symbol basis.
3 Modulation Design for the Up-Stream PCM Channel 60

memoryless discrete source with a given probability distribution. In fact, by default, a complete
prefix code [42] satisfies the first four conditions given above.
As suggested in [39], we use the Huffman algorithm [37] to generate a complete set of codewords
representing constellation points. Since a Huffman code [37, p. 92] is a prefix code, it satisfies
the first three requirements. From Kraft inequality [42], a set of codewords is complete if the
codeword lengths li satisfy the following condition:


M
2−li = 1 (3.36)
i=1

where M is the number of codewords. It is easy to show that the Huffman algorithm generates a
complete set of codewords if all the initial probabilities used in the algorithm correspond to valid
symbols.
We apply the Huffman algorithm only to positive symbols to generate a symmetrical probabil-
ity distribution. We will add an extra bit to each codeword to represent the sign of each symbol.
Adding a sign bit to each codeword is the same as duplicating the codewords that are generated
for positive symbols and adding one extra stage of bit assignment to the Huffman algorithm. The
result is still a complete prefix code.
The probability of occurrence of any codeword is a negative power of two. This probability
does not always match the desired maximum entropy probability distribution. The mismatch
between the actual probability assigned to a symbol and the desired probability can cause an
increase in the average signal power. In order to satisfy the average signal power constraint, the
parameter λ in (3.30) should be re-adjusted.
The procedure for bit-to-symbol mapping can be summarized as follows:

1. Compute the probability distribution of PAM symbols using Maxwell-Boltzmann distribu-

tions. Compute λ so that the average power constraint is satisfied.

2. Use the Huffman algorithm to generate a set of codewords for PAM constellations.

3. Compute the average signal power based on the actual probability distribution of constel-
lation points. If the power constraint is not satisfied, go to step 1 and adjust the parameter
λ to obtain a lower average signal power.

Figure 3.19 compares the achievable bit-rates using maximum entropy, the Huffman algorithm
and uniform probability distributions. As shown in this figure, the difference between bit-rates
achieved by the maximum entropy distribution and that of Huffman algorithm is not significant.
Table 3.1 shows codewords designed for a set of PAM symbols using the above procedure. Signal
3 Modulation Design for the Up-Stream PCM Channel 61

levels with a very low probability of occurrence are discarded from the constellation. The final
constellation includes 98 points. The codewords corresponding to the positive PAM symbols are
shown in this table. By altering the most significant bit of each codeword presented in Table 3.1,
we can specify a negative constellation point.

Table 3.1 Huffman algorithm for bit-to-signal mapping. The bit sequence associ-
ated with each constellation is presented. Note that for negative constellation points
only the sign bits should be altered. The constellation size is limited to 98 since the
constellation points with a very low probability (and a high energy) are not included.

i qi P (qi ) P̂ (qi ) C(qi ) i qi P (qi ) P̂ (qi ) C(qi )

−5 −7
1 26 0.021654 2 00101 25 1471 0.009422 2 0100101
2 77 0.021610 2−5 00100 26 1535 0.008750 2−7 0100001
3 127 0.021526 2−5 00011 27 1599 0.008100 2−7 0100000
4 175 0.021406 2−5 00010 28 1663 0.007475 2−7 0011010
5 227 0.021235 2−5 00001 29 1727 0.006877 2−8 01101011
6 279 0.021021 2−5 00000 30 1791 0.006306 2−8 01101010
7 327 0.020787 2−6 011110 31 1855 0.005765 2−8 01011010
8 375 0.020519 2−6 011101 32 1919 0.005253 2−8 01001001
9 423 0.020219 2−6 011100 33 1983 0.004772 2−8 00110111
10 479 0.019830 2−6 011011 34 2079 0.004107 2−8 00110110
11 543 0.019337 2−6 011001 35 2207 0.003326 2−9 010110110
12 607 0.018798 2−6 011000 36 2335 0.002659 2−9 010010000
13 671 0.018215 2−6 010111 37 2463 0.002100 2−10 0101101110
14 735 0.017596 2−6 010101 38 2591 0.001637 2−10 0100100011
15 799 0.016943 2−6 010100 39 2719 0.001261 2−11 01011011111
16 863 0.016264 2−6 010011 40 2847 0.000958 2−11 01001000101
17 927 0.015563 2−6 010001 41 2975 0.000720 2−12 010110111101
18 1007 0.014664 2−6 001111 42 3103 0.000533 2−12 010010001001
19 1087 0.013749 2−6 001110 43 3231 0.000391 2−13 0101101111001
20 1151 0.013012 2−6 001100 44 3359 0.000282 2−13 0101101111000
21 1215 0.012275 2−7 0111111 45 3487 0.000202 2−13 0100100010000
22 1279 0.011544 2−7 0111110 46 3615 0.000142 2−14 01001000100010
23 1343 0.010823 2−7 0110100 47 3743 0.000099 2−15 010010001000110
24 1407 0.010114 2−7 0101100 48 3871 0.000068 2−16 0100100010001111
49 3999 0.000046 2−16 0100100010001110

3.4 Chapter Summary

• The choice of modulation scheme for the up-stream PCM channel is baseband PAM modu-
lation. This choice is based on the assumption that the transmitter modem can be synchro-
nized to the network clock at the central office using the signal received in the downstream
3 Modulation Design for the Up-Stream PCM Channel 62

0.016
Maximum Entropy
0.014 Huffman

0.012

0.01
Probability

0.008

0.006

0.004

0.002

0
0 2000 4000 6000 8000
Positive Signal Levels
Fig. 3.19 Probability distributions of a constellation point with Huffman bit as-
signments compared to the maximum entropy distributions.

P(Symbol Error)=10−6
58
186−PAM

56
128−PAM
Bit Rate (kbits/sec)

54
156−PAM

52
90−PAM

50
130−PAM
Maximum Entropy
48 Huffman
Uniform

46 58−PAM
44 46 48 50 52 54 56
Pave(−12 dBm0)/σ2 (dB)

Fig. 3.20 Transmitted bit-rates as a function of SNR for three different probability
distributions: maximum entropy, the Huffman algorithm and uniform distributions.
3 Modulation Design for the Up-Stream PCM Channel 63

direction.

• The PCM encoder at the central office acts as the front-end of the receiver for the PCM
upstream channel. The ADC circuit can be viewed as a slicer or as a detector for the PAM
signal.

• The detector thresholds are selected as a subset of the ADC decision boundaries. Assuming
that the distortion caused by the channel ISI is not severe, the optimal signal constellation
points are midway between the adjacent detector thresholds.

• For a PAM modulation over a PCM channel with a non-linear compander, the spacing
between the constellation points is non-uniform. For the same signal and noise power, the
non-uniform PAM modulation shows some performance degradation compared to a uniform
PAM modulation.

• In the up-stream PCM channel, the PAM constellation points are not scalable. That is,
increasing the signal power does not necessarily increase the minimum distance between
the detector thresholds. As a result, for a given noise power, the performance of the PAM
modulation might remain the same even if we increase the signal power.

• In order to increase the constellation size by a factor of two and maintain the same per-
formance and average transmitted power, the signal-to-noise ratio should be increased by
6 dB.

• Non-uniform probability distribution for constellation points can reduce the energy cost
of the outer points. Non-uniform probability distribution can increase the bit-rate by
1.5 kbits/sec, at the same signal and noise power compared to that of a PAM modula-
tion with a uniform probability distribution.
 64

Chapter 4

Transmitter Design for the

Up-stream PCM Channel

In Chapter 3, we discussed a PAM modulation design for the up-stream PCM channel. As a model
for channel impairment, an additive random component was used to represent the combined effect
of noise and residual ISI at the receiver. Due to the lack of control on the receiver front-end
design in the up-stream PCM channel, the channel impairments can only be compensated for at
the transmitter. Although the transmitter has no impact on the additive noise introduced by the
channel, a proper transmitter design can eliminate or reduce the effect of ISI introduced by the
channel memory.
Since the characteristics of a subscriber loop and a receiver filter do not change rapidly, we
consider a linear time-invariant model for the PCM channel. We also assume that an equivalent
filter model for the overall channel and receiver filters is estimated at the receiver and is fed back
to the transmitter1 .
A straightforward solution for channel compensation is a linear filter structure implemented
at the transmitter. Such a transmitter filter can be specified based on different optimization
criteria. With a constraint on transmitted power, a transmitting filter may not provide adequate
ISI reduction. In particular, for channels with ISI, there is a trade-off between the average
transmitted power and ISI compensation. We introduce spectrum shaping techniques along with
the transmitter filter to control the transmitted power.
Section 4.1 describes a framework for designing optimal transmitter filters. Using this frame-
work, we consider three structures for the transmitter design. In Section 4.2, we present per-
0
This chapter has been reported in part in [43].
1
Prior to data transmission, the channel can be estimated by using a training sequence known to the receiver.
The down-stream PCM channel is used to return the channel estimate to the transmitter.
4 Transmitter Design for the Up-stream PCM Channel 65

formance results obtained by applying an optimal filter design to the up-stream PCM channels.
Section 4.3 introduces spectrum shaping and its impact on the transmitted power. In Section 4.4,
we describe precoding techniques which add redundancy to transmitted symbols. We generalize
the idea of Tomlinson-Harashima precoding to accommodate the requirements for pre-filtering
of the PCM channel. A summary of pre-filtering methods for the PCM channels is presented in
Section 4.5.

4.1 Optimal transmitter filter design

The design of an optimum transmitting filter for a fixed receiving filter and a given channel filter
has been studied in [30, 32, 44, 45]. There are several optimization criteria, such as minimum
probability of error, minimum mean-square error and minimum peak distortion error, that can be
used for the optimal transmitter filter design [30, p. 113]. The design of an optimum transmitting
filter is subject to a transmit power constraint in order to avoid a solution with an unbounded
average power.
Although different optimization criteria result in different transmitter filters, these filters share
a common structure. For any reasonable optimization criterion, an optimum linear transmitting
filter subject to an average transmitted power constraint has the following structure:
 ∗
Ht (f ) = G(f ) Hc (f )Hr (f )
(4.1)
∗
= G(f )Hcr (f )

where G(f ) is a periodic transfer function, G(f ) = G(f + 1/Ts ), that can be implemented as a
∗ (f )
discrete-time filter with an underlying sampling frequency of 1/Ts . The transfer function Hcr
represents a filter matched to the cascade of the channel filter and the receiver filter. We provide
a proof for the above statement in Appendix A, similar to that given in [46] for the receiver filter.
Figure 4.1(a) presents the arrangement of the transmitter, the channel and the receiver filter.
The matched filter Hcr∗ (f ) is a continuous-time filter. Implementing a continuous-time (analog)

matched filter in a transmitter modem is impractical since the characteristics of channels vary from
∗ (f ) has a limited bandwidth (W <4000
one subscriber line to another. Since the matched filter Hcr
Hz), this continuous-time filter can be approximated by a symbol-synchronous (a sampling fre-
quency equal to 1/Ts ) discrete-time filter. In this case, the only continuous-time filter required at
the transmitter is an interpolating (reconstruction) filter following the digital-to-analog converter.
We defer further discussion regarding the choice of the interpolating filter to Chapter 6. Here, we
assume the frequency response of the interpolating filter is accounted for as part of the channel
filter.
4 Transmitter Design for the Up-stream PCM Channel 66

ηW
+ W I = * H M π I7V
(+ FU I ) + FU I = + F I + U I

{DQ }
* H Mω7V
'LVFUHWHWR
FRQWLQXRXV
(+ FU I ) +F I +U I
Q7V
7UDQVPLWWHU &KDQQHO 5HFHLYHU)LOWHU

ηW
+ FU I = + F I + U I

{DQ }
'LVFUHWHWR
+ W H Mω7V FRQWLQXRXV
+ 1\TXLVW I +F I +U I
Q7V
7UDQVPLWWHU &KDQQHO 5HFHLYHU)LOWHU

Fig. 4.1 A general structure of an optimal transmitter filter is shown in (a) where
the transmit filter consists of an continuous-time filter matched to the channel and re-
ceiver filter, and a discrete-time filter G(ejωTs ). Since the equivalent channel-receiver
filter has a band-limited spectrum (< 1/2Ts), a sub-optimal structure for the trans-
mitter filter consists of a discrete-time filter Ht (ejωTs ), and a fixed pulse shaping filter
that replaces the analog matched filter, as shown in (b). The discrete-time transmit
filter can be determined based on different optimization criteria.

As shown in Fig. 4.1(b), the input and output of the up-stream PCM channel are discrete-
time signals while the channel itself is modeled as a continuous-time filter. When we design the
transmitter filter, it is convenient to work with a discrete-time model for the entire system. As
shown in [47], for a PAM modulation over an AWGN channel, there exists an equivalent discrete-
time model for the channel filter. Figure 4.2 shows a discrete-time model of the transmitter, the
channel and the receiver. Note that the additive noise sample η[n] represent the contribution
of thermal noise, echo and cross-talks. Although different components of the noise are added
to the transmitted signal at different points in the channel, due to linear assumption about the
channel filters, we can add the noise to the channel output. In general, the noise samples can be
correlated.
4 Transmitter Design for the Up-stream PCM Channel 67

(TXLYDOHQW&KDQQHO)LOWHU η>Q@
{DQ } KW >Q@ [>Q@ KFU >Q@ U>Q@

+ W H M πI7V + FU H M πI7V

[>Q@ =
DL KW >Q − L @
L

Fig. 4.2 A discrete-time model of the up-stream PCM channel with a discrete-time
transmitter filter.

The equivalent discrete-time model of the channel-receiver filter can be expressed as Hcr (ej2πf Ts ):

1  k k
Hcr (ej2πf Ts ) = Hc (f + )Hr (f + ) (4.2)
Ts Ts Ts
k

We discuss the optimum design of the filter Ht (ejωTs ) based on different criteria. The prob-
ability of error is the most meaningful criterion for the optimal transmitter design. However, in
many cases, there is no general solution for such an optimization problem. In our discussion, we
consider two other criteria: one being the peak distortion criterion (or zero ISI criterion) and the
other being the Mean Square Error (MSE) criterion.

4.1.1 Peak distortion criterion

The peak distortion criterion is defined as the worst-case ISI at the sampling instants. As shown
in Fig. 4.2, samples at the receiver can be written as:
∞

r[n] = a(n) h[0] + ai h[n − i] + η[n] (4.3)
i=−∞
i=0

where h[n] is the overall impulse response of the transmitter, channel and receiving filter. The
peak value of the ISI is the sum of the absolute values of the ISI terms:
∞
  

D(ht ) = |h[n]| =  ht [k]hcr [n − k] (4.4)

n=−∞ n k
n=0

The smallest value of peak distortion is obtained if all the overall impulse response coefficients,
except for the central one, are zero. This condition implies that the ISI can be completely avoided
4 Transmitter Design for the Up-stream PCM Channel 68

if the transmitting filter is a stable inverse of the overall channel-receiver filter:

1
Ht (ejωTs ) = (4.5)
Hcr (ejωTs )

The transmitter filter has a straightforward expression in terms of the channel filter. However, if
the channel filter contains spectral nulls, a signal with bounded power at the input of the inverse
filter can create an output signal with unbounded power (i.e., the inverse filter is not BIBO stable
[48, p. 209]).

4.1.2 Mean-Square Error (MSE) criterion

A commonly used criterion for an optimal linear filter design is mean-square error. According
to this criterion, filter coefficients are determined so that the mean-square value of the error at
sampling instants is minimized. In general, the mean-square value of error has two components,
one is caused by additive noise and the other is caused by the ISI. Note that the optimal transmitter
filter design can only minimize the error caused by the ISI. As a result, in our computation of
optimal transmitter filter, the contribution of the additive noise in MSE is not included. The error
caused by additive noise can always be added to the computed MSE. In the absence of noise, we
define:
    
min MSE = min E (r[k] − ak )2 (4.6)
ht ht

where E{x} denotes the mean of a random variable x. The impulse response of the transmitting
filter at sample n is denoted as ht [n]. Unless otherwise stated, we assume that the transmitting
filter has an infinite length impulse response.
We assume that the data symbols an form a wide-sense stationary random sequence [49],
taking values from a set of PAM symbols. We denote the autocorrelation function of the data
symbol sequence as:
 
φa [k] = E ai a(i+k) (4.7)

Consequently, the power spectrum density (PSD) of the input sequence is defined as:
∞

Φa (ejωTs ) = φa [k]e−jwTs k (4.8)
k=−∞

We consider a constraint on the average transmitted power as:

 
E |x[k]|2 ≤ Pt (4.9)
4 Transmitter Design for the Up-stream PCM Channel 69

where Pt is a constant and x[.] is the discrete-time sequence at the output of the transmitting
filter (see Fig. 4.2). Since the input sequence to the transmitter filter is wide-sense stationary,
the output sequence x[n] is also wide-sense stationary [49, p. 193]. The PSD of the signal at the
output of the transmitter filter is:

Φx (ejωTs ) = |Ht (ejωTs )|2 Φa (ejωTs ) (4.10)

The average power constraint is expressed in terms of Ht (ejωTs ) and Φa (ejωTs ) as follows:
 π/Ts
Ts
Φa (ejωTs )|Ht (ejωTs )|2 dω ≤ Pt (4.11)
2π −π/Ts

In Appendix B, we derive the optimum linear transmitter filter that provides the Minimum Mean-
Square Error (MMSE) solution with an average transmitter power constraint. The frequency
response of the optimal filter is:

Hcr (e−jωTs )
Ht (ejωTs ) = (4.12)
|Hcr (ejωTs )|2 + λ

where λ is a non-negative parameter that is determined based on the power constraint. From
Eq. (4.11) and Eq. (4.12), the average power constraint is

Ts π/Ts
|Hcr (ejωTs )|2
Φa (ejωTs ) dω ≤ Pt (4.13)
2π −π/Ts (|Hcr (ejωTs )|2 + λ)2

The left hand-side of Eq. (4.13) is a monotonically decreasing function of λ. We are interested
in the smallest non-negative value of λ that satisfies this condition. If λ = 0 does not satisfy the
power constraint, the minimum value of λ is determined by solving Eq. (4.13) as an equality.
As shown in Appendix B, the minimum mean-square value of error is computed as:

λ2 Ts π/Ts
Φa (ejωTs )
MMSE = dω (4.14)
2π −π/Ts (|Hcr (ejωTs )|2 + λ)2

Equation (4.12) corresponds to the transmitter filter with an infinite length impulse response.
This solution establishes the ultimate ISI reduction that can be expected from an optimal linear
transmitter filter with a given power constraint.
The actual implementation of the transmitter filter can be a Finite Impulse Response (FIR)
filter. An FIR filter can be formed as the truncation of the IIR filter impulse response. Alterna-
tively, the MMSE problem can be defined to obtain directly the optimal solution as an FIR filter.
4 Transmitter Design for the Up-stream PCM Channel 70

In this case, the filter coefficients are the solution of the following set of equations:

Copt = Γ−1 ξ (4.15)

where Γ is a square matrix and ξ is a column vector. The derivation of Eq. (4.15), and the
expression of Γ and ξ are presented in Appendix B.
The MSE criterion provides a general framework for several transmitter structures. In our
discussion, we investigate three different scenarios for the transmitter design.

(1) Optimal filtering with a flat data spectrum

For a random sequence of independent and identically distributed (i.i.d.) transmitted symbols,
the power spectral density Φa (ejωTs ) = σa2 is constant. In this case, Eq. (4.13) is simplified to:

Ts σa2 π/Ts
|Hcr (ejωTs )|2
dω ≤ Pt (4.16)
2π −π/Ts (|Hcr (ejωTs )|2 + λ)2

For a given equivalent discrete-time filter Hcr (ejωTs ), we can only reduce the MMSE by increasing
the average transmitted power Pt . Examples of transmitter filter design for the up-stream PCM
channel are presented in Section 4.2.

(2) Spectrum shaping followed by the linear filtering

The MMSE value in Eq. (4.14) is a function of the PSD of the filter input. We can shape the PSD
of the input symbols to control the average transmitted power. In Eq. (4.11), spectrum shaping
can reduce the value of parameter λ while satisfying the average power constraint.
The data symbols {an } take values from a PAM symbol alphabet. We introduce correlations
among the symbols in order to create a desired power spectrum density at the input of the
transmitting filter while maintaining the underlying PAM signal structure. Assuming the data
symbols {an } are i.i.d. random variables, we map the data symbols to a new sequence {sn } which
contains a form of redundancy in signal levels or added symbols. The new sequence {sn } has a
power spectrum density close to a desired PSD, say Φd (ejωTs ). We will discuss different methods
of creating such correlations in Section 4.3.

(3) Precoding: A combined spectrum shaping and filtering

The spectrum shaping and optimal filtering can be combined into one operation, known as pre-
coding. Tomlinson-Harashima (TH-) precoders [50, 51] are examples of channel compensation
performed at the transmitter. A TH-precoder creates an inverse channel filter whose output is
4 Transmitter Design for the Up-stream PCM Channel 71

limited to a finite range. Each symbol at the input of a TH-precoder is represented by more than
one signal level. Among all signal levels representing a symbol, the precoder chooses a signal level
that provides an output in a preset range. A further discussion regarding TH-precoders and their
limitations for use in an up-stream PCM channel appears in Section 4.4.

4.2 Optimal transmitting filter design for the up-stream PCM channels

We can apply the optimal filter design to the PCM up-stream channels to compensate for the
filtering effect of the subscriber line and the anti-aliasing filter in the PCM line card. Examples of
typical equivalent channel-receiver filter are given. In order to evaluate system performance, we
compute the MSE at the sampling instants (theoretically and also by simulation). We also use
the symbol error rate of the PAM modulation over the up-stream channel as a measure of perfor-
mance. The characteristics of telephone channels vary from line to line. Two examples of PCM
channel filters are considered. In both cases, the PCM anti-aliasing filters satisfy Recommendation
G.712 [25].

Example 1

Figure 4.3 shows the magnitude response of an equivalent discrete-time receiver-channel filter in
the up-stream PCM channel2 . The frequency response of the analog anti-aliasing filter corre-
sponding to this channel is shown in Fig 2.7. The receiver filter consists of a 5th -order elliptic
lowpass filter cascaded with a bi-quad highpass filter. Due to aliasing (caused by sampling),
the stopband attenuation of the equivalent discrete-time filter is significantly lower than that of
the analog filter. Figure 4.4 shows the impulse response of the equivalent discrete-time filter
and Fig. 4.5 shows the pole-zero locations of the equivalent channel filter transfer function. The
equivalent channel filter is non-minimum phase and contains nulls near the zero frequency.
We use a linear transmitter filter structure to compensate for the channel filter. The perfor-
mance of the transmitter filter design is evaluated based on the signal-to-interference ratio at the
receiver, the average transmitted power and the probability of symbol error in the presence of
(noise and) ISI. Note that we define the signal-to-interference ratio (SIR) as the ratio of the signal
power to the MSE caused by the ISI at the receiver.
As an example, consider a PAM modulated signal with an average power of σa2 = −12 dBm0
at the input to the transmitter filter. The signal power at the output of the transmitter filter
is determined by the power gain of the transmitter filter. Figure 4.6 shows the variation of the
2
In this example, we assume that the linear distortion in the up-stream channel is caused mainly by the receiver
filter. Signal attenuation due to the subscriber line or the hybrid circuit will further increase the required transmitter
power.
4 Transmitter Design for the Up-stream PCM Channel 72

−5
20log10 |Hcr(ej 2 π f)|

−10

−15

−20

−25

−30

−35
0 1000 2000 3000 4000
Frequency (Hz)
Fig. 4.3 The magnitude response of the channel equivalent filter as discussed in
Example 1.

0.8

0.6

0.4
Impulse response

0.2

−0.2

−0.4

−0.6

−0.8
0 1 2 3 4
Time (msec)

Fig. 4.4 The impulse response of the channel discussed in Example 1.

4 Transmitter Design for the Up-stream PCM Channel 73

Imaginary Part 0.5

−0.5

−1
−1.5 −1 −0.5 0 0.5 1
Real Part

Fig. 4.5 Pole-Zero locations of a channel-receiver filter transfer function of Fig. 4.3.

60

50 σ2a =−12 dBm0

SIR (dB)

40

30
Ex1: Simulation
Ex1: Theory

20
−10 −9.8 −9.6 −9.4 −9.2 −9
Average transmitted Power Pt (dBm0)

Fig. 4.6 Signal-to-interference ratio as a function of the average transmitted power

for the PCM channel described in Example 1. The PAM signal power at the input of
the transmitting filter is σa2 = −12dBm0.
4 Transmitter Design for the Up-stream PCM Channel 74

signal-to-interference ratio as a function of the average transmitted power Pt . The SIR at the
receiver increases by increasing the allowed transmitter power. The result shown in this figure
holds for different PAM constellation sizes, assuming that the average power of the PAM signal
is maintained at −12 dBm0.
If we assume that ISI is the dominant cause of distortion at the receiver, the probability
of symbol error can be characterized as a function of SIR. Figure 4.7 shows the probability of
symbol error as a function of SIR for different PAM constellation sizes. The PAM constellations
are designed for the up-stream PCM channels. The design procedure for the PAM constellation
was discussed in Chapter 3.
We can determine an upper bound for the probability of symbol error based on a Gaussian
approximation of the residual ISI distribution.
Figure 4.7 compares the upper bound with the simulation results. The ISI has a limited
peak and its tail distribution is far from being Gaussian. For the larger values of SIR (i.e., a
smaller values of MSE), the probability of symbol error is roughly determined by the tail of the
ISI distribution. Hence, the Gaussian assumption of the ISI distribution becomes less accurate.
While the Gaussian approximation holds at low signal-to-interference levels, it provides a loose
upper bound at higher values of SIR. There are methods to compute tighter bounds for the
probability of symbol error [41, 52, 53, 54]. However, we can use direct simulation to compute
the symbol error probabilities.
The ISI may not be the only source of distortion over the PCM channels. We consider the
combination of residual ISI and an additive Gaussian noise in the up-stream channel for a given
PAM modulation. As an example, Fig. 4.8 shows the probability of symbol error as a function
of the signal-to-noise ratio for a 98-PAM modulation with an average power of −12 dBm0 at the
input of the transmitter. This figure shows that a small change in the average transmitted power
(less than 0.1 dBm0) can significantly change the probability of symbol error, especially at a high
signal-to-noise ratio. This observation is in line with what is shown in Fig. 4.6 where the change of
average transmitter power from −9.05 dBm0 to −8.95 dBm0 increases the signal-to-interference
ratio by more than 20 dB.

Example 2

We consider a receiver filter that consists of a 7th -order elliptic lowpass filter cascaded with a
bi-quad highpass filter. Compared to the receiver filter in Example 1, the filter in this example
has a larger attenuation in the stopband. As a result, the equivalent discrete filter has greater
attenuation at DC and at 1/(2Ts ) = 4000 Hz. Figure 4.9 shows the magnitude frequency response
and the impulse response of the discrete-time filter. The pole-zero plot of the filter is shown in
4 Transmitter Design for the Up-stream PCM Channel 75

−1
10

−2
10
Symbol Error Rate

−3
10

−4
10
128−PAM
98−PAM
64−PAM
−5
10 32−PAM
Upper Bounds

25 30 35 40 45 50 55
Pave(−12 dBm0)/MSE (dB)

Fig. 4.7 The probability of symbol error is shown as a function of SIR for several
PAM signals for the channel filter discussed in Example 1. Assuming that the residual
ISI as a Gaussian random variable, we can approximate the probability of symbol
error.

−1
10

−2
10
Symbol Error Rate

−3
10

−4 Zero−ISI, Theory
10 Zero−ISI, Simulation
Reduced ISI, P =−8.95 dBm0
t
Reduced ISI, P =−8.97 dBm0
−5 t
10 Reduced ISI, P =−9.01 dBm0
t
Reduced ISI, P =−9.06 dBm0
t
−6
10
44 45 46 47 48 49 50 51
2
Pave(−12 dBm0)/σ (dB)

Fig. 4.8 Probability of symbol error is shown as a function of SNR for a 98-PAM
signal with an average power of −12 dBm0. The performance of different transmitter
filters is compared to the ideal channel filter with Gaussian additive noise.
4 Transmitter Design for the Up-stream PCM Channel 76

Fig. 4.10.
Figure 4.11 shows the signal-to-interference ratio as a function of transmitted power. The input
of the transmitter filter is a PAM signal with an average power of −12 dBm0. Due to spectral nulls
in the frequency response of the channel-receiver filter, the power gain of the transmitter filter
is impractically large. The probability of symbol error as a function of the signal-to-interference
ratio is depicted in Fig. 4.12.
Figure 4.13 shows the symbol error-rate of a 98-PAM signal transmitted over this channel. The
average signal power at the input of the transmitter filter is −12dBm0. Compared to Example 1,
the average transmitted power is significantly larger for a given symbol error rate.

4.2.1 Observations and remarks

Since the ISI does not have the same effect on the probability of error as additive noise, minimizing
the MSE does not necessarily lead to the minimum probability of error. By using the MSE criterion
for the transmitter filter design, we treat the combination of noise and residual ISI as an additive
random variable, and ignore the information that we could obtain from the ISI regarding the
transmitted data sequence. However, due to the fixed structure of the receiver front-end in the
up-stream PCM channel, the information on transmitted data cannot be extracted from the ISI.
Furthermore, the MSE criterion provides a simple way to combine the effect of noise and ISI with
a straightforward solution for the optimal filter design.
Two examples discussed above illustrate that an optimal transmitter filter can reduce the
effect of the ISI. However, there is a trade-off between the residual ISI and the transmitted signal
power. For some channel filter characteristics, such as the filter we discussed in Example 2, the
required transmitted power to obtain an acceptable performance is significantly large. The main
cause of such a requirement for the signal power is spectral nulls in the equivalent channel filter
characteristics.
Apart from the average transmitted power, spectral nulls in the frequency response of the
equivalent channel cause a slower decay in the impulse response of the transmitting filter. The
portion of transmitting filter that correspond to the spectral nulls can only be implemented as an
IIR filter.
Figure 4.11 indicates that for the SIR range in which we are interested, a small change in the
average transmitted power can drastically change the SIR. As shown in Fig. 4.7 and Fig. 4.12, a
slight increase in the average transmitted power can change the system performance. These exam-
ples suggest that for a PAM modulation with a large alphabet, the required signal-to-interference
ratio is relatively high. As a result, the transmitting filter should be designed to provide almost
4 Transmitter Design for the Up-stream PCM Channel 77

−5
)|
j2πf

−10
20log10 |Hcr(e

−15

−20

−25

−30

−35
0 1000 2000 3000 4000
Frequency (Hz)

(a) Magnitude Response

0.8

0.6

0.4
Impulse response

0.2

−0.2

−0.4

−0.6

−0.8
0 1 2 3 4
Time (msec)

(b) Impulse Response

Fig. 4.9 Channel characteristics in Example 2.

4 Transmitter Design for the Up-stream PCM Channel 78

1.5

Imaginary Part 1

0.5

−0.5

−1

−1.5

−2
−4 −3 −2 −1 0 1
Real Part
Fig. 4.10 Pole-zero locations of a channel-receiver filter transfer function discussed
in Example 2.

60
Ex2: Simulation
Ex2: Theory
50 Ex1: Theory

σ2a =−12 dBm0

SIR (dB)

40

30

20

−10 −8 −6 −4 −2 0 2
Average transmitted Power Pt (dBm0)

Fig. 4.11 Signal-to-ISI ratio as a function of the average transmitted power in

Example 2. Compared to Example 1, a significant increase in the average transmitted
power is required to obtain the same SIR.
4 Transmitter Design for the Up-stream PCM Channel 79

−1
10

−2
Symbol Error Rate 10

−3
10

−4
10
128−PAM
98−PAM
−5 64−PAM
10
32−PAM
Upper Bounds
−6
10
25 30 35 40 45 50 55
Pave(−12 dBm0)/MSE (dB)

Fig. 4.12 The probability of symbol error is shown as a function of SIR for several
PAM signals for the channel filter discussed in Example 2.

−1
10

−2
10
Symbol Error Rate

−3
10

Zero−ISI, Theory
−4
10 Reduced ISI, P = 2.86 dBm0
t
Reduced ISI, P = 2.82 dBm0
t
Reduced ISI, P = 2.77 dBm0
−5 t
10 Reduced ISI, P = 2.69 dBm0
t
Reduced ISI, P = 2.56 dBm0
t
−6
10
44 45 46 47 48 49 50 51
2
Pave(−12 dBm0)/σ (dB)

Fig. 4.13 Probability of symbol error as a function of SNR in Example 2. The

performance of different transmitter filters is compared to the ideal channel filter
with Gaussian additive noise.
4 Transmitter Design for the Up-stream PCM Channel 80

η>Q@

%LWWR6\PERO ^VQ ` [>Q@ U>Q@

,QSXWELWV 0DSSLQJ Mω7V Mω7V
+W H + FU H
6SHFWUDO6KDSLQJ

&RQWUROOLQJDYHUDJHWUDQVPLWWHG &RQWUROOLQJ006(
SRZHU

Fig. 4.14 Spectrum shaping cascaded with an optimal filtering to control MMSE
while maintaining the limit on the average transmitted power.

an ISI-free channel3 . Such a design requires a large average transmitted power. For channels with
spectral null, the required transmitted power can be unbounded.
In order to overcome above problems, we examine transmitter structures that can reduce the
average transmitted power while maintaining an acceptable level of signal-to-interference ratio.

4.3 Spectrum shaping and filtering

In this section, we investigate spectrum shaping methods that can be applied to the data sequence
in order to reduce the average transmitted power while maintaining an acceptable signal-to-
interference ratio. Figure 4.14 shows schematically a cascade of spectrum shaping and linear
filtering. The stream of incoming bits is mapped into a sequence of symbols {sn } with a power
spectrum density that is close to a desired spectrum.
Prior to describing spectrum shaping methods, we determine the desired power spectrum
density Φd (ejωTs ) that reduces the average transmitted power. There are different solutions for
Φd (ejωTs ), depending on assumptions about the channel model.

(1) An arbitrary channel frequency response

The magnitude response of the equivalent channel-receiver filter is specified as a function of fre-
quency |Hcr (ejωTs )|. In this model, no underlying structure for the transfer function is considered.
We investigate the power constraint in Eq. (4.13) and the MMSE in Eq. (4.14) to identify the
desired PSD of the input signal so that the MMSE is reduced while the power constraint on the
3
In other words, for a PCM channel, the MMSE transmitter filter solution should provide almost zero ISI that
corresponds to λ ≈ 0 in Eq. (4.12).
4 Transmitter Design for the Up-stream PCM Channel 81

transmitted signal still holds. We denote the desired power spectrum density as Φd (ejωTs ). From
Eq. (4.14), it is clear that if λ approaches zero, the MMSE will tend to zero. For λ = 0, the
transmitter filter given in Eq. (4.12) is simplified to the channel inverse filter 1/Hcr (ejωTs ).
From Eq. (4.13), we wish to determine the power spectrum density so that the power constraint
is satisfied with a smaller value of λ. Let us examine Eq. (4.13) when λ approaches zero. The left
hand side of Eq. (4.13) can be approximated as:
 
Ts π/Ts
|Hcr (ejωTs )|2 Ts π/Ts
Φd (ejωTs )
jωTs 2
Φ (ejωTs ) dω ≈
2 d
dω (4.17)
2π −π/Ts (|Hcr (e )| + λ0 ) 2π −π/Ts |Hcr (ejωTs )|2

where λ0 ≈ 0. If we choose the power spectrum density as:

Φd (ejωTs ) = K|Hcr (ejωTs )|2 , K < Pt (4.18)

the power constraint will be satisfied with a zero value of λ.

Equation(4.18) indicates that by spectrum shaping we wish to compensate for the power gain
of the transmitting filter. Equation (4.18) sets a target spectrum for the PSD at the transmitter
filter input. We aim to create a signal at the input of the transmitter filter with a PSD close to
the target spectrum4 .
Note that obtaining the target spectrum given in Eq. (4.18) may not be feasible since the
amount of redundancy added to the PAM signal to create such a power spectrum density can be
large. The constraint on the scaling factor K in Eq. (4.18) effects average power σa2 of the PAM
signal before the spectrum shaping. Such a constraint limits the data transmission rate.

(2) A rational transfer function for the channel filter

Let us assume that the transfer function of the channel-receiver filter is expressed as a rational
function: M −jωTs k
jωTs k=0 βk e B(ejωTs )
Hcr (e )=  −jωTs k
= (4.19)
1+ N k=1 αk e A(ejωTs )
4
We expect that the closer Φa (ejωTs ) is to the target spectrum, the smaller the MMSE value will be. As a
measure of the distance between the two spectra, we can use the following:

Ts π/Ts  2
D= Φa (ejωTs ) − Φd (ejωTs ) dω
2π −π/Ts
.
4 Transmitter Design for the Up-stream PCM Channel 82

It is easy to show that when λ approaches zero in Eq. (4.13), the following spectrum can reduce
the average transmitted power:

Φd (ejωTs ) = K|B(ejωTs )|2 (4.20)

From Eq. (4.13), we can show that for λ = 0, the average power constraint is satisfied if we have:

Pt Pt
K≤  = (4.21)
Ts π/Ts
2 N
|A(e jωTs
)| dω 1+ α2k
2π −π/Ts
k=1

As shown in Eq. (4.20), one solution for the desired spectrum is based on the numerator of
B(ejωTs )
the transfer function Hcr (ejωTs ) = . Suppose the polynomial B(z) is written in terms of
A(ejωTs )
its M roots:
B(z) = β0 (1 − z1 z −1 ) (1 − z2 z −1 ) · · · (1 − zM z −1 ) (4.22)

It is evident from Eq.(4.13) that for small values of λ, those roots of B(z) in the vicinity of the
unit-circle or on the unit-circle are the dominant terms that determine the average transmitted
power.
Depending on the average power constraint, it might be adequate to consider only a portion of
|B(ejωTs )|2 as the desired power spectrum. We decompose B(ejωTs ) into a product of two terms:

B(ejωTs ) = B1 (ejωTs )B2 (ejωTs ) (4.23)

where B1 (ejωTs ) contains zeros close to (or on) the unit circle. The desired power spectrum density
is written as:
Φd (ejωTs ) = K  |B1 (ejωTs )|2 (4.24)

4.3.1 Methods of Spectrum Shaping

The data symbol sequence at the input of the transmitter filter has a flat spectrum. Spectrum
shaping can be defined as adding a form of redundancy to the input sequence to create a desired
statistical correlation among the symbols. There are a variety of methods of creating such a
correlation. Line codes are typical examples of spectrum shaping that control the “running
digital sum” to create spectral nulls at DC and can be extended to create spectral nulls at
non-zero frequencies [38, Chapter 12]. There are also trellis coding techniques that can create
spectral nulls in the spectrum at desired frequencies [55]. Partial response signalling [56] [38] is an
alternative method of spectrum shaping via inverse filtering and a modulo arithmetic operation.
4 Transmitter Design for the Up-stream PCM Channel 83

Due to predetermined structure of the receiver, these spectrum shaping techniques are not directly
applicable to the up-stream PCM channel.
The spectral shaping method used in the V.90 Standard for the down-stream PCM channel
is based on Convolutional Spectral Shaping (CSS) [14]. The CSS algorithm controls the sign bits
of the transmitted symbols. The sign bits are selected on a frame by frame basis. Each frame
contains six symbols. In each frame, the sign bits are selected based on r bits of redundancy
(0 ≤ r ≤ 3) and 6 − r bits of information. The sign bit sequence is selected so that the mean
square error between the signal PSD and a desired spectrum is minimized.
Convolutional spectrum shaping is designed for the down-stream PCM channel where the
transmitter can only determine the transmitted bits. The actual PAM modulation is performed
by the PCM decoder at the central office. In [18], the application of convolutional spectrum
shaping in the up-stream PCM channel is investigated. Although the results provided in [18]
show the merits of applying spectrum shaping for the up-stream channel, the use of CSS is not
necessarily the best choice of spectrum shaping method for the up-stream PCM channel. In the
up-stream PCM channel, the analog modem controls the transmitted signal. Compared to the
down-stream PCM channel, the control on the transmitted signal in the up-stream channel can
reduce the amount of redundancy required for spectrum shaping.

4.3.2 Spectrum shaping by inserting redundant symbols

We can perform spectrum shaping by adding redundant symbols to the input sequence. The input
symbols are parsed into non-overlapping blocks of K symbols. Although the redundant symbols
can be inserted at different locations among the data symbols, we consider a set of N − K symbols
that is added to the end of each block. Values of redundant symbols are computed so that the
power spectrum density of each block is close to the target spectrum. As shown in Fig. 4.15, the
operator F (k, n) takes in a block of K data symbols and computes the redundant N − K symbols.
Note that the redundant symbols are not necessarily taken from the PAM symbol alphabet. As
we will discuss below, the redundant symbols do not carry any information; they are only added
to create the desired spectrum shaping.

A spectral null at DC

Let us assume that the desired spectrum contains a spectral null at zero frequency (DC). Each
input block of length K is padded by one symbol (N = K + 1). The output sequence s[n] is
4 Transmitter Design for the Up-stream PCM Channel 84

. 6\PEROV 1 6\PEROV

D>N @ V>Q@
) N Q

Fig. 4.15 Spectrum shaping by inserting redundant symbols into a block of data
symbols.

represented as:


 n = 0, · · · , K − 1
 a[mK + n] for
s[mN + n] = 
K−1
(4.25)

 −
 a[mK + l] for n=K
l=0

The z-transform of the output signal s[n] can be written in terms of the z-transform of its
non-overlapping blocks of length N :
∞

S(z) = s[n]z −n
n=−∞
∞
 
N −1
= s[mN + l]z −(mN +l)
m=−∞ l=0
(4.26)
∞
 
N −1
−l −mN
= ( s[mN + l]z )z
m=−∞ l=0
∞

= Sm (z)z −mN
m=−∞
4 Transmitter Design for the Up-stream PCM Channel 85

From Eq. (4.25), Sm (z) can be written as:


N −1
Sm (z) = s[mN + l]z −l
l=0

N −2
= s[mN + l]z −l (1 − z −(N −1−l) ) (4.27)
l=0

N −2
= s[mN + l]z −l (1 − z −1 )(1 + z −1 + · · · + z −(N −l−2) )
l=0

From Eq. (4.27), it is evident that Sm (z) contains a zero at DC.

We would like to compute the power spectrum density Φs (ejωTs ) as a function of the block
size. The input sequence is assumed to be i.i.d. random variable with zero mean and variance of
σa2 . Due to inserted redundant symbols, the sequence s[n] is not stationary. The autocorrelation
function of s[n] is defined as:
φs (k, l) = E{s[k]s[l]} (4.28)

From Eq. (4.25), the autocorrelation function can be characterized as:



 σa2 for k = l, k = mN − 1


 (N − 1)σ 2 for k = l, k = mN − 1
a
φs (k, l) = (4.29)

 0 for k = l, k = mN − 1, l = mN − 1


 −σ 2 for 0 < |k − l| < N, k (or l) = mN − 1
a

From Eq. (4.29), we can show that:

φs (k + N, l + N ) = φs (k, l) (4.30)

The average value of the autocorrelation function over a period N can be written as:

N −1
1 
φs (m) = φs (k, m + k) (4.31)
N
k=0

From Eq. (4.29), φs (m) is computed as:


 2N − 2 2

 σa m=0
 N
φs (m) = −1 2 (4.32)
 σ m = ±1, ±2, . . . , ±(N − 1)

 N a

0 elsewhere
4 Transmitter Design for the Up-stream PCM Channel 86

The power spectrum density of s[n] can be expressed as the discrete-time Fourier-transform of
φs (m). In this example, the power spectrum density can be computed as:
∞

Φs (ejωTs ) = φs (m)e−jωmTs
m=−∞
  (4.33)
σa2 sin(2N − 1) ωT2 s
= (2N − 1) −
N sin ωT2 s

In Eq. (4.33), Φs (ejωTs ) is a positive definite function with double zeros at DC. Figure 4.16
shows Φs (ejωTs ) as a function of frequency (with 1/Ts = 8000 Hz) for different values of block
size N . As shown in this figure, the power attenuation at frequencies near DC depends on the
block size.
Compared to the convolutional spectrum shaping results reported in [14], the insertion of
redundant symbols can provide more attenuation at DC with less redundancy. The amount of
redundancy added to the signal is controlled by the length of each block.
Figure 4.17(a) shows a transmitter with spectrum shaping followed by a linear transmitter
filter. The channel contains a spectral null at DC. The spectrum shaping is performed by inserting
a redundant symbol to each block of N − 1 data symbols. The input symbols are taken from
a 64-PAM constellation with an average power of −12 dBm0. Figure 4.17(b) shows the average
transmitted power as a function of the block size. For comparison, the required transmitter power
without spectrum shaping is also depicted. As shown, the spectrum shaping can significantly
reduce the average transmitted power. By changing the block size N we can control the trade-off
between the average transmitted power and the information symbol rate K/N .
Ideally, spectrum shaping creates nulls that cancel the poles of the transmitter filter. However,
such a pole cancellation may not be perfect due to a mismatch between the zeros created by the
spectrum shaping and the poles of the transmitter filter. A realizable transmitter filter cannot
include poles on the unit-circle. A non-zero value of λ in the optimal transmitter filter given in
Eq. (4.12) can prevent such a problem.

Nulls at a set of desired frequencies

The previous example can be extended to create nulls in more than one frequency. We use a set
of N − K redundant symbols in a block of N output symbols to create the desired spectral nulls.
Assuming Sm (z) is the z-transform of each block, we choose the redundant symbols such that the
following equations are satisfied:

Sm (zi ) = 0 for i = 1, 2, · · · , M (4.34)

4 Transmitter Design for the Up-stream PCM Channel 87

0
Normalized Power Spectrum (dB)

−10

−20

N=4
N=8
−30 N=12
N=16

0 500 1000 1500 2000 2500 3000 3500 4000

Frequency (Hz)

(a) Power spectrum in the 0–4000 Hz frequency range.

5
Normalized Power Spectrum (dB)

−5

−10

−15

−20 N=4
N=8
N=12
−25 N=16

−30
0 100 200 300 400
Frequency (Hz)

(b) Zoomed in on the 0–400 Hz frequency range.

Fig. 4.16 Power spectrum after spectrum shaping. It is evident that the spectrum
contains a spectral null at DC. The input signal power σa2 is normalized to one.
4 Transmitter Design for the Up-stream PCM Channel 88

η>Q@

D>N @ V>Q@ [>Q@ U>Q@

) N Q  − H − Mω7V
 − H − Mω7V
 −  + λ H − Mω7V + H − Mω7V

6SHFWUXPVKDSLQJ 3UH)LOWHU &KDQQHO

2SHUDWRU

(a) A transmitter design for channel with a spectral null.

10
Transmitted power (dBm0)

−5

Flat Spectrum
−10 Shaped spectrum

4 6 8 10 12 14 16 18
Block size N

(b) The average transmitted power as a function of the block

size.

Fig. 4.17 An optimal MMSE linear transmitter filter is designed to compensate

for a channel with spectral null at DC as shown in (a). The input symbols a[k] are
taken from a 64-PAM constellation with an average power of −12 dBm0. We choose
λ = 10−5 . The average transmitted power with and without spectrum shaping is
presented in (b).
4 Transmitter Design for the Up-stream PCM Channel 89

where zi corresponds to a set of desired zeros in S(z). The first K symbols in each block are data
symbols followed by N − K redundant symbols. From Eq. (4.34), we obtain the following set of
equations specifying redundant symbols:
 

K−1
 a[mK + l]z1−l 
  
 
z1−K
−(K+1)
···
−(N −1)  l=0 
z1 z1 s[mN + K]  K−1 
 −(K+1) −(N −1)     

 z2−K z2 ··· z2 
 s[mN + K + 1]  
 = − a[mK + l]z2−l 

 ···    l=0  (4.35)
 ··· ··· ···  ···   
−(K+1) −(N −1)  ··· 
−K
zM zM ··· zM s[mN + N − 1]  
 K−1
 
 −l 
a[mK + l]zM
l=0

If the desired zeros zi are distinct and their number is equal to the number of redundant symbols
per block (M = N − K), then Eq. (4.35) has a unique solution5 . For a set of distinct zeros zi ,
the determinant of this matrix is non-zero6 .
If the number of redundant symbols is less than the number of equations, the set of equations
in Eq. (4.35) is over-determined. In this case, the least square solution for the redundant symbols
can be selected [58]. As a result, Eq. (4.34) holds only in the mean-square-error sense.
The proposed arrangement of redundant symbols at the end of each block is arbitrary. Other
patterns of redundancy insertion can be analyzed the same way. However, Eq. (4.35) may not
provide a finite solution for an arbitrary pattern of inserted symbols in a block.
To show the effect of spectrum shaping in the up-stream PCM channel, we apply the redun-
dancy insertion method to a PCM channel as discussed in Example 2 of Section 4.2. Figure 4.19
shows the average transmitted power when spectral shaping is applied to data sequence. Symbols
a[k] are selected from, in this case, a 90-PAM constellation with an average power of −12 dBm0.
The spectrum shaping in this example creates zeros at DC and at 4000 Hz (1/2Ts ). Compared
to the case where the input signal has a flat spectrum, the redundancy insertion method provides
a significant reduction in the transmitted power. However, the required power to maintain an
acceptable data transmission rate is still high. In order to reduce the average power, we can
reduce the power of the average power of the input symbols7 . Figure 4.19 shows the average
transmitted power when spectral shaping is applied to data sequence. Symbols a[k] are selected
5
The determinant of the coefficient matrix in the left-hand-side of Eq. (4.35) is similar to that of a Vandermonde
Matrix [57, p.266]. It can be shown that if zi ’s are distinct, then the determinant of this matrix is non-zero.
6
In order to create repeated roots for Sm (z), the set of equations given above is not adequate. We should
consider the derivatives of the polynomials as well.
7
Note that by reducing the average signal power while maintaining the same number of constellation points,
reduces the minimum distance between adjacent constellation points. In the present of additive noise, the system
is susceptible to a higher probability of error.
4 Transmitter Design for the Up-stream PCM Channel 90

from a 90-PAM constellation with an average power of −18 dBm0.

Figure 4.20 shows the data transmission rate as a function of the transmitted power. If we
assume that ISI is the dominant source of error, the transmitted filter is designed to provide
symbol error rate close to 10−6 . The required signal-to-interference ratio is around 47 dB. Results
shown in Fig. 4.20 are only examples of the trade-off between the data transmission rate and the
average transmitted power. Depending on the channel filter, the constellation size, the value of
transmitted power constraint, the block size and the number of redundant symbols per block can
be determined.

2
σ2a = −12 dBm0
Transmitted power (dBm0)

−2

−4

−6

−8
Flat Spectrum
−10 Shaped spectrum
−12
4 6 8 10 12 14 16
Block size (N)
Fig. 4.18 Spectrum shaping reduces the average transmitted power for a 90-PAM
modulated signal with σa2 = −12dBm0.

Spectrum shaping of symbols requires adding redundancy to the input sequence. Since the
symbol rate and the maximum number of signal levels in the up-stream PCM channel are prede-
termined, adding redundancy causes a reduction in the data transmission rate over the channel.
With a constraint on the average transmitted power, several factors should be considered in order
to maximize the net bit-rate at a given probability of symbol error:

• the constellation size,

• the probability distribution of the constellation,

• the average power of constellation points and the block size.

4 Transmitter Design for the Up-stream PCM Channel 91

−2

−4
σ2a = −18 dBm0
Transmitted power (dBm0)
−6

−8

−10

−12

−14
Flat Spectrum
−16 Shaped spectrum

−18
4 6 8 10 12 14 16
Block size (N)

Fig. 4.19 Average transmitted power as a function of the block size for a 90-PAM
signal. Applying spectrum shaping to a PCM channel with spectral nulls reduces the
average transmitted power.

42

40
Data transmission rate (kbits/s)

38

36

34

32

30

28

26
−16 −14 −12 −10 −8 −6
Average transmitted power (dBm0)

Fig. 4.20 Data transmission rate as a function of transmitted power

4 Transmitter Design for the Up-stream PCM Channel 92

4.4 Precoding: combined filtering and spectral shaping

In this section, we describe precoding methods to combine linear filtering and spectrum shaping.
We explain the properties and limitations of the classical precoding techniques for use in the up-
stream PCM channels. Alternative precoding methods that are more suitable for the up-stream
PCM channels will be described.

4.4.1 Tomlinson-Harashima Precoding

Tomlinson [50] and Harashima/Miyakawa [59] invented independently a channel compensation

technique known as precoding. Basically, a precoder compensates for ISI at the transmitter. By
performing channel equalization at the transmitter, two known problems of receiver equalizers can
be avoided: noise enhancement, as in a linear equalizer, and error propagation, as in a decision
feedback equalizer [38].
Straightforward pre-filtering of channels has problems of its own; it increases the transmit
signal power, especially for channels with spectral nulls. Tomlinson-Harashima Precoding ( TH-
precoding) provides a solution for channel compensation while maintaining the transmitted signal
level in a preset interval8 . TH-precoding generalizes the idea of precoding used in partial-response
signalling. In a partial-response signalling, precoding is used to compensate for the controlled ISI
that is introduced by the transmitter. In partial response signalling, the exact model of introduced
ISI is known at the transmitter. In other applications where the ISI is introduced by channel, a
precoding can be applied if the filtering effect of the channel is known at the transmitter.
Figure 4.21 shows a model for the TH-precoding operation. In this model, an equivalent
channel filter with an FIR impulse response h[n] is considered. We assume h[n] is known at the
transmitter. The impulse response h[n] is monic (i.e., h[0] = 1) and causal. A TH-precoder acts
as an inverse filter, compensating for h[n] at the transmitter. The inverse filter is realized as
a direct form all-pole filter. In order to limit the range of transmitted signal levels, a modulo
operation maps the inverse filter output to a finite interval. The modulo operator limits the
output signal to a preset interval [−Vmax /2, +Vmax /2]. At the receiver, another modulo operation
is required to reverse the mapping performed at the transmitter. The signal x[n] at the input of
the TH-precoder is limited to ±Vmax /2 in order to maintain the one-to-one mapping between the
input x[n] and the output r[n]9 .
The operation of a TH-precoder can be interpreted as combined spectral shaping and pre-
filtering. Figure 4.22 illustrates a TH-precoder where we expand the input signal to an inverse
8
In our discussion, we only consider one-dimensional signals. The TH-precoding technique can also be used for
complex signals [60].
9
Note that for our discussion in this section x[n] is not necessarily a PAM signal.
4 Transmitter Design for the Up-stream PCM Channel 93

[>Q@ X>Q@ \>Q@ U>Q@

   9PD[    9PD[

KQ

−   9PD[ −   9PD[

K Q −δ Q

Fig. 4.21 Tomlinson-Harashima Precoder.

filter so that the output of this filter always remains in a finite range.
Taking real values in (−Vmax /2, +Vmax /2] interval, the input signal x[n] is mapped to s[n] via
the following operation:
s[n] = x[n] + Vmax kn (4.36)

where the integer kn is selected so that the filter output u[n] satisfies:

−Vmax +Vmax
< u[n] ≤
2 2

Since the inverse filter fully compensates for the channel filter, the channel output y[n] is identical
to the inverse filter input s[n]. In order to extract the original signal x[n], we use a modulo
operation at the receiver:

r[n] = y[n] − Vmax kn

  (4.37)
= mod y[n], Vmax

It is straightforward to show that r[n] is identical to x[n] [50].

A TH-precoder introduces redundancy by a multi-level representation of a signal point. The
redundancy is used to shape the signal spectrum and to control the peak and average power of
the transmitted signal. There are however, several (potentially restrictive) issues in TH-precoder
designs that require further attention:
4 Transmitter Design for the Up-stream PCM Channel 94

  9PD[   9PD[   9PD[

−   9PD[ −   9PD[ −   9PD[

[>Q@ V>Q@ X>Q@ \>Q@ U>Q@

$GGLQJ  5HPRYLQJ
UHGXQGDQW KQ UHGXQGDQW
OHYHOV  OHYHOV

K>Q@ − δ >Q@

Fig. 4.22 A TH precoder can be viewed as a combined filtering and spectrum shap-
ing. The modulo arithmetic operation is used to add (and remove) redundant signal
levels to the signal to maintain the transmitted signal in the (−Vmax /2, +Vmax /2]
range.

Increasing the Dynamic range of the received signal

As shown in Fig. 4.22, TH-precoding increases the dynamic range of the channel output y[n]. We
determine an upper bound for the absolute value of the channel output |y[n]| as a function of the
impulse response h[n]. For each sample at the output of the TH-precoder, we have:

Vmax  M
Vmax
− ≤ x[n] + Vmax kn − h[l]u[n − l] ≤ (4.38)
2 2
l=1

where M is the memory length of the channel filter impulse response h[n] and kn is an integer
number determined by the modulo operator. We assume the integer value kn has a maximum
denoted as Kmax :
−Kmax ≤ kn ≤ Kmax (4.39)

Since the input signal x[n] and the precoder output u[n] are limited to ± Vmax
2 range, Kmax satisfies
the following conditions:

Vmax 
M
Vmax Vmax Vmax
− ≤− + Vmax Kmax − |h[l]| ≤ (4.40)
2 2 2 2
l=1
4 Transmitter Design for the Up-stream PCM Channel 95

The two inequalities expressed above result in the following expression for Kmax :

"1 
M
#
Kmax = 1 + |h[l]| (4.41)
2
l=1

where x denotes the largest integer number that is less than or equal to x. At the channel
output, the received signal y[n] can be written as:

y[n] = x[n] + kn Vmax (4.42)

From Eq. (4.41), an upper bound for the received signal is determined as:

   " M

y[n] ≤ Vmax Kmax = Vmax 1 + 1 |h[l]| (4.43)
2
l=1

The maximum value of y[n] is obtained when x[n] = 0 and kn = Kmax . Assuming that the upper
bound given in Eq. (4.43) follows closely the signal, we can conclude that a TH-precoder increases
the dynamic range of the received signal at least by a factor of two. As shown in Eq. (4.41), the
upper bound is a function of the channel impulse response.

Non-minimum-phase channel filters

Although a TH-precoder can compensate for any channel with a causal and monic impulse re-
sponse, the precoding of a non-minimum phase impulse response can cause a significant increase in
the receiver dynamic range. For example, consider a channel with the following impulse response:
h[n] = δ[n] − ρ δ[n − 1]. From Eq. (4.41), we have:

Kmax = 1 + ρ/2

It is evident that increasing the value of ρ increases the dynamic range of the TH-precoding.
While the dynamic range increase for minimum-phase filters (ρ < 1) is minimal (Kmax = 1), a
non-minimum-phase filter (ρ ≥ 1) may cause a significant increase in the dynamic range of the
received signal.

Expansion of the slicer circuit for a PAM signal detection at the receiver

In our discussion, we assume the input data symbols x[n] are taken from a 2N -PAM signal al-
phabet. A TH-precoder increases the dynamic range of the input signal in order to control the
transmitted power. At the receiver, each PAM symbol alphabet si has more than one represen-
4 Transmitter Design for the Up-stream PCM Channel 96

tation. At The signal levels that correspond to the same symbol si are related as:

cij = si + kj Vmax (4.44)

where kj is an integer number chosen so that the precoder output remains in a preset range. For
a receiver with a fixed dynamic range (such as the up-stream PCM channel), Eq. (4.44) indicates
that Th-precoding reduces the effective number of constellation points that can be detected at
the receiver. A portion of threshold levels at the receiver are used to detect redundant signal
levels. If the original PAM constellation points are uniformly spaced, with an appropriate choice
of Vmax , the expanded PAM constellation can be uniformly spaced as well [38, p. 462].
If the original PAM constellation points are non-uniformly spaced, the expanded constella-
tion points are derived from Eq.(4.44) and the detection thresholds of the expanded slicer are
determined accordingly10 .

4.4.2 Precoding design for the up-stream PCM channels

The TH-precoding technique is not directly applicable to the up-stream PCM channels. The
restrictions imposed by the PCM channel on the precoder design are as follows:

1. The up-stream PCM channel has a pre-determined receiver front end. The maximum num-
ber of constellation points at the receiver is limited by the number of ADC decision levels.
The actual number of useful signal levels is smaller than the number of ADC levels in order
to create enough margin against the channel signal distortions (additive noise, echo, etc.).
As a result, a constellation expansion by a TH-precoder will reduce the PAM constellation
size and the data transmission rate over the channel.

2. As we discussed in Section 4.2, the equivalent channel filter is typically a non-minimum

phase filter.

3. In general, the estimate of a channel filter transfer function is a rational function11 . The
TH-precoder only compensates for an FIR channel filter.

4. In Chapter 2, we discussed the PAM constellation design for the up-stream PCM channel.
The constellation points are determined based on a subset of the ADC decision boundaries
in the PCM encoder. Due to the µ-Law (or A-Law) companding used in the PCM encoder,
10
For a given set of constellation points, the detector thresholds that minimize the probability of error are the
midpoints of adjacent constellation points.
11
Since the impulse response of the channel filter is relatively long, an IIR filter model reduces the number of
parameters required to specify the channel impulse response. In recommendation V.90 and in the proposed draft
of Recommendation V.92, the channel is modelled as an IIR filter.
4 Transmitter Design for the Up-stream PCM Channel 97

Equivalent impulse response h(n)

x[n] r[n]
Redundancy
Insertion
ht (n) hcr (n)

Transmitter filter Channel-Receiver Filter

h ( n ) − δ ( n)

Fig. 4.23 A precoding followed by linear filter at the transmitter.

the spacing between the constellation points is non-uniform. A TH-precoder requires a set
of “modulo-Vmax equivalent” constellation points to represent a data symbol. The PAM
constellation points designed for a PCM channel do not generally satisfy such requirements.

In order to compensate for a non-minimum phase channel filter, we consider a linear filter
cascaded with the precoder. Figure 4.23 shows the cascade structure of a precoder and a linear
filter at the transmitter. The linear transmitter filter is used to compensate for the poles of the
channel filter as well as for the zeros that are not in the vicinity of the unit circle. The equivalent
impulse response of the transmitter filter, the channel and the receiver filter is denoted as h[n].
The precoder is designed to compensate for a filter with the impulse response h[n].
We investigate an alternative method of pre-filtering to accommodate non-uniform spacing
between the PAM constellation points by inserting redundancies as extra symbols instead of
extra signal levels.

4.4.3 Linear block-by-block pre-filtering

In this section, we consider a block-by-block data transmission to allow for redundancy insertion
between data symbols. In the present approach, we combine the spectrum shaping described in
Section 4.3.2 with a linear channel inverse filter.
Figure 4.24 shows the cascade of spectrum shaping and the channel inverse filter. The equiv-
alent channel impulse response and transfer function are denoted as h[n] and H(z) respectively.
The impulse response h[n] is causal and monic. The causal inverse filter 1/H(z) is realized as
a direct form all-pole filter. The spectrum shaping is performed by a linear time-varying opera-
tor F (k, n), as introduced in Section 4.3.2. For each block of non-overlapping K data symbols,
F (n, k) specifies a block of N output symbols. The number of redundant symbols added to each
4 Transmitter Design for the Up-stream PCM Channel 98

x[k ] s[n] u[n] y[n] r[k ]

+
F ( k , n) h[n] G (n, k )
-

h[n] − δ [n]

Fig. 4.24 A block-by-block process combines spectrum shaping and channel inverse
filtering.

block is equal to the memory length of the channel filter M = N − K. At the receiver, the M
redundant symbols in each block are discarded by the G(n, k) operator. Since the inverse filter
compensates for the channel filter h[n], the input x[k] and the output r[k] are identical.
The redundant symbols in each block are determined such that the z-transform of each output
block, Sm (z), has a factor H(z):
Sm (z) = H(z)Um (z) (4.45)

where Um (z) is a finite length polynomial. In Section 4.3.2, we explained a method to compute
the redundant symbols as linear functions of the data symbols in each block (see Eq. (4.35)).
Here, we use the structure of the inverse filter to satisfy Eq. (4.45) without actually computing
the redundant symbols.
Since there are N symbols in each block, Sm (z) a polynomial of the degree N − 1 in terms
of z −1 . The transfer function H(z) is a polynomial of the degree M . We specify the redundant
symbols in each block such that Um (z) can be written as a polynomial of the degree N − M − 1:

Sm (z)
Um (z) =
H(z)
N −M
−1 (4.46)
−n
= um [n]z
n=0

We use series expansion of 1/H(z) to compute the first K coefficients of Um (z). The causal inverse
of H(z) can be expressed as:
 ∞
1
= hinv [k]z −k (4.47)
H(z)
k=0

where the sequence hinv [k] will not be convergent if H(z) is a non-minimum phase filter. The
4 Transmitter Design for the Up-stream PCM Channel 99

first K symbols of sm [n] are taken from the input data symbols:

sm [n] = x[Km + n] for n = 0, . . . , K − 1 (4.48)

From Eq. (4.46), Eq. (4.47) and Eq. (4.48), the first K = N − M coefficients of Um (z) are
determined as:


n 
n
um [n] = hinv [k] sm [n − k] = hinv [k] x[Km + n − k] for 0≤n<K (4.49)
k=0 k=0

The last M redundant symbols in each block are specified such that the output of the inverse
filter satisfies the following condition:

um [n] = 0 for K≤n (4.50)

Redundant symbols can be computed from the difference equation that describes the inverse filter:


M
sm [n] − h[k]um (n − k) = um [n] = 0 for K ≤n<N (4.51)
k=1

or equivalently:

M
sm [n] = h[k]um [n − k] for K≤n<N (4.52)
k=1

Equation (4.52) specifies M = N − K redundant symbols in each block. Equation (4.52), together
with Eq. (4.48) completely specifies the input s[n] to the inverse filter. It is evident that only M
redundant symbols in each block are required to create the desired spectrum shaping for Sm (z).
From Eq. (4.49) and Eq. (4.50), we can identify u[n] as:
 
n
k=0 hinv [k] x[Km + n − k] ∀m and 0≤n<K
u[mN + n] = (4.53)
0 ∀m and K≤n<N

It is clear from Eq. (4.53) that the actual values of the redundant symbols at the input of the
inverse filter do not contribute to the inverse filter output. This observation suggests that we
should be able to perform spectrum shaping without computing the redundant symbols.
Figure 4.25 shows an implementation of the block-by-block linear inverse filtering. The non-
uniform up-sampling block parses the input symbols into blocks of K symbols and pads M zero
symbols to the end of each block. Switch S1 disconnects the feedback loop and enforces the last
M values of um [n] to be zero. In other words, at the end of each block, Switch S1 resets the
4 Transmitter Design for the Up-stream PCM Channel 100

memory of the feedback loop one symbol at a time. As a result, at the beginning of each block
the inverse filter starts from a zero initial state and at the end of each block the inverse filter
returns to a zero-state.

x[k ] s[n] u[n] y[n] r[k ]

"0"
+
(K , N ) h[n] (N , K )
- S1

h[n] − δ [n]

Fig. 4.25 Redundancy insertion in a block of symbols to control the transmitted

power.

The output of the inverse filter is characterized by Eq. (4.53). At the receiver, the channel
output y[n] is identical to the inverse filter input s[n]. Such as for s[n], the channel output y[n]
contains non-overlapping blocks of K data symbols and M redundant symbols. To obtain the
final output symbols r[n], we discard the M redundant symbols at the end of each block using a
non-uniform down-sampling process.
We illustrate the effect of the block-by-block inverse filtering by an example. Suppose that
the equivalent channel transfer function is:

H(z) = 1 − z −2

Since H(z) is not minimum-phase, the inverse filter is not BIBO stable. However, a block-by-block
inverse filtering can be used to control the transmitted output power.
In this example, we choose a PAM input signal x[.] with an average power of σx2 = −12
dBm0. The block-by-block process requires two redundant symbols per block to reset the feedback
memory. Figure 4.26 shows the average signal power σu2 at the output of the inverse filter as a
function of the block size N . Note that there are N − 2 data symbols in each block. Reducing
the block size will reduce the average transmitted power as well as the data symbol rate over the
channel.
The signal power at the output of the inverse filter can be expressed in terms of the input
power, the impulse response hinv [n] of the causal inverse filter 1/H(z) and the number of data
symbols K in each block. Assuming the input signal x[n] forms an i.i.d. sequence with an average
4 Transmitter Design for the Up-stream PCM Channel 101

−7

−8

Transmitted power (dBm0)

−9

−10

−11

−12

−13
Theory
−14
Simulation
−15
4 6 8 10 12 14
Block size (N)

Fig. 4.26 The average signal power at the output of the inverse filter is shown as a
function of the block size. The input is a PAM signal with an average power of −12
dBm0. The channel filter is specified as: H(z) = 1 − z −2 . There are 2 redundant
symbols in each block. The values of signal power computed from Eq. (4.55) and the
simulation results are similar.

power of σx2 , we have:

σu2 = E{|u[n]|2 }
N −1 (4.54)
1
= E{ |um [n]|2 }
N
n=0

Replacing um [n] from (4.49) and (4.50), we can express the signal power σu2 as:

σx2 
K−1
σu2 = (K − n) |hinv [n]|2 (4.55)
N
n=0

Equation (4.55) indicates that the impulse response of the causal inverse filter has a major
influence on σu2 . The block-by-block process restricts the contribution of the impulse response to
its first K samples. A linear weighting function (K − n|) tapers off the effect of the |hinv [n]|2
when n increases.
It is important to note that for a non-minimum phase channel, hinv [n] can grow exponentially
causing a significant power gain σu2 /σx2 for the inverse filter. For channels with spectral nulls the
location and the order of the spectral nulls affect the power gain of the inverse filter significantly.
4 Transmitter Design for the Up-stream PCM Channel 102

In particular, if the channel has a repeated zero at a certain frequency, the impulse response
grows linearly and creates a larger power gain. To illustrate this point, we compare the power
gain of the block-by-block inverse filters for H1 (z) = 1 − z −2 and H2 (z) = (1 − z −1 )2 . Figure 4.27
illustrates the power gain as a function of the block size. In both cases, the number of redundant
symbols per block is 2.

25
z1=1, z2=1
z =1, z =−1
20 1 2
Power gain σ2u/σ2x (dB)

15

10

−5
4 6 8 10 12 14
Block size (N)
Fig. 4.27 the power gain of two inverse filters are compared. It is evident that
filters with repeated nulls require a larger transmitted power.

4.5 Concluding Remarks

In this chapter, we developed a framework for pre-equalization techniques applicable for the up-
stream PCM channels. It was shown that a linear pre-equalizer should provide almost an ISI free
condition in order to support the transmission of 2N -PAM signals with a desired constellation
size (2N > 64). For typical PCM channel filters, implementing a zero-ISI pre-equalizer requires
a large transmitted power.
We investigated power spectrum methods to reduce the required transmitted power while
maintaining a pseudo ISI-free solution for the pre-equalizer. A spectrum shaping technique based
on redundant symbol insertion was investigated. It was shown that spectrum shaping can sig-
nificantly reduce the average transmitted power. However, this technique requires to compute
redundant symbols per each block of transmitted symbols.
As an alternative method, we proposed a block-by-block pre-filtering. The implementation
4 Transmitter Design for the Up-stream PCM Channel 103

of the pre-filter employs a single switch in the feed-forward path of the transmitter filter to
reset the channel memory. Assuming that the number of redundant symbols per block is the
same as the memory length of the (FIR) channel filter, we can show that the block-by-block
pre-filtering has the same performance as the spectrum shaping method obtained by redundancy
insertion. Compared to a TH-precoder, the block-by-block pre-filtering does not expand the
dynamic range of the data symbols, instead it adds appropriate redundancies to the input signal
to cancel undesired poles of the pre-filter. The price to be paid for stabilizing the system is a
reduction in the effective symbol rate.
We developed a theoretical ground for linear block-by-block pre-filtering. The average power
gain of this structure can be determined in terms of the channel impulse response (Eq. (4.55)).
We also determined an upper bound for the required signal level redundancy for a TH-precoder
as a function of the impulse response of the channel filter (Eq. (4.55)). The power gain of a
TH-precoder is almost one especially for PAM signals with a large alphabet [38, 60]. For a
given channel impulse response and a fixed transmitted power, we can compare the effective bit-
rate that can be obtained by a TH-precoder and a block-by-block pre-filter. In many cases the
result is in the favor of the TH-precoder, especially if the channel filter includes repetitive zeros
(e.g., H(z) = 1 − z −2 ). This observation suggests that having redundant signal levels can be more
effective than adding redundant symbols. However, as we discussed, a TH-precoder is not directly
applicable to a PCM up-stream channel since the spacing between the constellation points are
not uniform in general. It is important to note that the modulo arithmetic is only one way to
create redundant signal levels. Other methods of creating redundant levels is a subject for our
future studies.
 104

Chapter 5

Filterbank Structures for ISI Channel

Pre-Equalization

As described in Chapter 4, the up-stream PCM channel requires channel pre-equalization in

order to avoid performance degradation due to ISI. A straightforward inverse filter is not feasible
since spectral nulls of the channel filter can cause a high power gain in the pre-equalizer. We
presented methods of adding redundancy to the transmitted signal in order to control the average
transmitted power while compensating for the channel filter.
In this chapter, we discuss channel pre-equalization methods based on filterbank structures.
A filterbank structure can provide a natural way of creating patterns of redundancy in a block
of transmitted symbols. The filterbank pre-equalizer can compensate for channel filters even if
they have a non-minimum phase impulse response. We compare the performance of the filterbank
structure with that of the inverse pre-filtering methods presented in Chapter 4.

5.1 Signal design based on non-uniform sampling

Nyquist’s first criterion for signal design implies that for an ISI-free data transmission over a
bandlimited channel with a one-side bandwidth of W Hz, the symbol rate should be less than
2W samples/sec. For an up-stream PCM channel, the symbol timing is determined by the fixed
sampling rate of the A/D converter set to fs = 8000 Hz at the central office. Since the effective
bandwidth of an up-stream PCM channel is less than 4000 Hz, the basic Nyquist signal design
for this channel cannot prevent ISI distortion.
In a conventional Nyquist signalling, modulated waveforms carrying data symbols are trans-
5 Filterbank Structures for ISI Channel Pre-Equalization 105

mitted regularly at uniformly distributed time instants:

∞

x(t) = ak g(t − kTs + τ ) (5.1)
k=−∞

where ak are transmitted symbols and g(.) is a Nyquist waveform. There is a fixed timing interval
of Ts sec between two consequent transmissions.
For a signalling scheme in general, the intervals between consequent transmissions do not
need to be the same. According to the dimensionality theorem [61, p. 294], the maximum number
of independent symbols transmitted over a channel with W Hz bandwidth is 2W symbols/sec,
regardless of the arrangement of symbols in a time interval. The conventional Nyquist signal
design is only a special case of signalling over bandlimited channels. We review other signalling
methods that can provide non-uniform symbol timing.
Inspired by non-uniform sampling methods such as those described in [62], we design a set of
bandlimited waveforms that can be modulated by data symbols at non-uniformly distributed time
instants. Data symbols are expected to be detectable at the receiver without any ISI distortion.
For an up-stream PCM channel in particular, we would like to have symbol intervals of integer
multiples of Ts = 1/8000 sec. However, only a subset K of every N consecutive timing instants
is permitted in order to maintain an average transmission rate of less than or equal to 2W
symbols/sec. We choose K and N such that:

K
≤ 2W Ts (5.2)
N

Figure 5.1 shows the transmitter structure based on the set of K linear time-invariant filters
with K distinct impulse responses p0 (t) to pK−1 (t). The transmitted signal u(t) can be written
as:
∞ K−1
 
u(t) = sk [n] pk (t − nN Ts ) (5.3)
n=−∞ k=0

where s0 [n] to sK−1 [n] are K data symbols transmitted in a time interval of N Ts sec. At the
receiver, sampling instants are integer multiples of Ts . However, out of every N consecutive
samples, K samples correspond to data symbols and N − K samples are redundant.
Each waveform pk (t) has a limited bandwidth of K/(2N Ts ) Hz and is designed to avoid ISI at
certain sampling instants. For an ISI-free signalling, these waveforms should satisfy the following
conditions: 
1 n=k
pk (nTs ) = (5.4)
0 n = Nm + l
5 Filterbank Structures for ISI Channel Pre-Equalization 106

V > Q@

%LWWR6\PEROPDSSLQJ
'& S W

,QSXWELWV V>Q@ XW Q 7V U Q7V

'& S W Σ KW
Q = 1L+ M

V . −>Q@
'& S . − W

'LVFUHWHWLPHWR&RQWLQXRXVWLPH
FRQYHUVLRQ

Fig. 5.1 Signal design for a bandlimited channel with non-uniform symbol timing
distribution.

where the index N m + l points to a subset K of each N consecutive samples. Compared to a

Nyquist pulse that has regular zero-crossings, pk (t) has no constraints at a subset of sampling
instants. By relaxing the constrains, we allow for ISI at particular sampling instants while reducing
the required bandwidth of each waveform1 .

Example

Let us assume that an up-stream PCM channel filter is modeled as an ideal brick-wall lows pass
filter, strictly bandlimited to 3500 Hz. We would like to design a signalling scheme for this channel
that is compatible with channel bandwidth limitation and the fixed sampling rate at the receiver.
By choosing K = 7 and N = 8 in Eq. (5.3), seven waveforms that construct the transmitted
signal are determined as follows [62, 5]:

t − kTs $
K−1
π(t − iTs )
pk (t) = Ck sinc( ) sin for k = 0, 1, . . . , 6 (5.5)
8Ts 8Ts
i=0
i=k

where Ck is a normalization factor to ensure pk (kTs ) = 1. Figure 5.2 shows two waveforms p0 (t)
and p1 (t) that satisfy conditions given in Eq. (5.4). As shown, both these waveforms share the
same redundant sampling instants2 .
1
Note that waveforms pk (t) are different from signals with controlled ISI used in partial-response signalling. The
important feature of signals pk (t) is that the required bandwidth for these signals is less than 1/(2Ts ).
2
Note that other redundant samples occur outside the range shown in these figures.
5 Filterbank Structures for ISI Channel Pre-Equalization 107

1.2
1
0.8
0.6
0.4
Amplitude

0.2
0
−0.2
−0.4
−0.6 Data sample
Redundant Samples
−0.8

−9 −8 −7 −6 −5 −4 −3 −2 −1 0 1 2 3 4 5 6 7 8 9
t/Ts

(a) p0 (t)

1.2
1
0.8
0.6
0.4
Amplitude

0.2
0
−0.2
−0.4
−0.6 Data sample
−0.8 Redundant Samples

−9 −8 −7 −6 −5 −4 −3 −2 −1 0 1 2 3 4 5 6 7 8 9
t/Ts

(b) p1 (t)

Fig. 5.2 Signal design for an up-stream PCM channel with W = 3500 Hz bandwidth
and sampling rate of 1/Ts =8000 Hz.
5 Filterbank Structures for ISI Channel Pre-Equalization 108

5.1.1 Pre-equalizer design for the up-stream PCM channel

Signal design with non-uniform sampling intervals is proposed in [4] for signalling over PCM
channels. The work on non-uniform signalling for PCM up-stream and down-stream channels is
further studied in [5]. Assuming a brick-wall frequency response for the channel filter, the work
presented in [5] describes a condition for permissible patterns non-uniform symbol intervals. In
practice, the effect of the up-stream PCM channel on the transmitted signal is far from a brick-wall
filter. In [5], the ideal continuous filters are replaced by discrete-time FIR filters. The coefficients
of these filters are computed at the receiver by using training sequence sent by the transmitter.
Filter coefficients are then fed back to the transmitter. However, such training is problematic
since there is no direct access to the channel output at the up-stream receiver. Furthermore,
there are no results reported on the required transmitted power for such a filterbank structure.
In the rest of this chapter we discuss filterbank structures that are specifically designed to
compensate for a known FIR channel filter. We evaluate the required transmitted power for
the filterbank structure and compare the results with those obtained for the block-by-block pre-
filtering method described in the previous chapter.

5.2 Non-maximally decimated filterbanks

In this section, We discuss a filterbank structure to pre-compensate for the channel filter at
the transmitter. To obtain zero ISI, the pre-equalizer should act as channel inverse filter. As
discussed in Chapter 4, a direct inverse filtering for a non-minimum phase channel may require
an unacceptably large transmitted power. A filterbank can provide a simple method of adding
redundancy to the transmitted signal. Such filterbank structure is known as non-maximally
decimated filterbanks [63, 64]. We determine conditions to obtain FIR filters for a non-maximally
decimated filterbank.
Assuming that the channel is FIR and is known at the transmitter, we consider a filterbank
structure that can accommodate redundant symbols along with the information-carrying sym-
bols in the transmitted signal. Figure 5.3 shows the filterbank structure at the transmitter, the
equivalent discrete-time channel with impulse response h[n], and the sample selection process at
the receiver. As shown, the input signal a[.] is parsed into K non-overlapping sequences a0 [.] to
aK−1 [.]. Sequence ai [.] is up-sampled by a factor of N (> K). We design filters such that ISI in
one branch and Inter-Channel Interference (ICI) between different branches are avoided.
Each branch of the filterbank contains a filter gk [n]. We will show that these filters can be
chosen to be FIR, even though the inverse filter 1/H(z) is an IIR filter. Since only a subset of the
received samples carry data symbols, the system can allow ISI at redundant sampling instants.
Figure 5.4 illustrates the contribution of each branch of the filterbank in the overall system
5 Filterbank Structures for ISI Channel Pre-Equalization 109

7UDQVPLWWHU &KDQQHO 5HFHLYHU

D>N @ D > N @ V  > Q@ X>Q@ \ > Q @ U>N @

. 1 J > Q@ K>Q@ 1 . ] −. +
] − D>N @ V >Q@ ] − \>Q@
. 1 J>Q@ 1 . ] −. + 
] − ] −

] − D . −>N @ V. −>Q@ ] − \ . −>Q@

. 1 J . −>Q@ 1 . ]

Fig. 5.3 A non-maximally decimated filterbank structure is used to pre-equalize the

channel filter. The redundant symbols are discarded at the receiver by a non-uniform
down-sampler. Note that the filterbank structure at the receiver is only used to show
a non-uniform down-sampling process which discards N − K redundant symbols from
a block of N symbols.

response. Note that structure shown in Fig. 5.4 is functionally equivalent to the filterbank struc-
ture. Conditions imposed on the kth filter to avoid ISI and ICI distortion can be formulated
as: 




1 n=k
gk [n] ⊗ h[n] = dk (i, j) n = N i + j, 0 ≤ i ≤ L, j∈S (5.6)



0 otherwise

where S is a set of N − K distinct indices selected between 0 and N − 1. Note that redundant
symbols dk (i, j) are spread over L+1 symbol blocks. Figure 5.4 shows an example where redundant
symbols are at the end of block of N symbols. In this example we have:

S = {K, K + 1, . . . , N − 1}

Coefficients dk (i, j) correspond to the sampling instants that no data symbol is transmitted.
We wish to determine coefficients dk (i, j) such that pre-equalizer filters have FIR impulse
5 Filterbank Structures for ISI Channel Pre-Equalization 110

1 6\PEROV 1 6\PEROV

V >Q@ J  > Q@ K>Q@ \  > Q@

V>Q@ J > Q @ K>Q@ \>Q@

V . −>Q@ J . −>Q@ K>Q@ \ . −>Q@

\>Q@
Σ

Fig. 5.4 A set of FIR filters can partially invert the channel filter. The redundant
samples are used to allow for FIR inverse filters.

response. From Eq. (5.6), we obtain:


L 
z −k + dk (i, j)z (−N i−j)
i=0 j∈S
Gk (z) =

M
1+ h[i]z −i
i=1

L
z −k + dk (i, j)z (−N i−j)
i=0 j∈S (5.7)
=
$
M
(1 − zi z −1 )
i=1

M k −1

= gk (i)z −i
i=0

In order to have an FIR filter in each branch, Gk (z) should be a finite length polynomial in z −1 .
Since H(z) has a finite number of roots, say M , the number of coefficients dk (i, j) required for
pole-zero cancellation is at least M . Assuming that polynomial H(z) has no repetitive roots,
5 Filterbank Structures for ISI Channel Pre-Equalization 111

coefficients dk (i, j) should satisfy the following set of equations:

 −jN−K −(N L+jm )
  
z1−j1 z1−j2 · · · z1 ··· z1 z1−k
 −j1 −jN−K −(N L−jm )   −k 
 z2
 z2−j2 · · · z2 ··· z2  
 dk = −  z2


 ···   ···  (5.8)
 ··· ··· ··· ··· ···   
−j1 −j2 −jN−K −(N L−jm ) −k
zM zM ··· z2 ··· zM zM
% &' (
≥M

where {j1 , j2 , . . . , jN −K } are indices of redundant symbols in each block. There are at least M
unknown values in the system of M equations in Eq. (5.8). As a result, there can be more than
one solution for the coefficients dk (i, j). Among all these solutions we are interested in one that
can provide a minimum transmitted power. Depending on the roots of H(z) and the pattern of
redundancy, there may or may not be a solution for these equations. Below, we discuss examples
to illustrate these cases.

Example 1: H(z) = 1 + z −1

Consider a channel filter with a signal null at z1 = −1. We examine a filterbank structure with
K=3 branches and a block size of N =4. Since the channel filter has one root, only one redundant
symbol per block is required to obtain the FIR filter in each branch. Filters of different branches
are determined as follows:

1 + z −3
G0 (z) = = 1 − z −1 + z −2
1 + z −1
z −1 − z −3
G1 (z) = = z −1 − z −2 (5.9)
1 + z −1
z −2 + z −3 −2
G2 (z) = −1 = z
1+z

For a channel filter with a single root, Eq. (5.8) has always a solution for the coefficient dk (1).

Example 2: H(z) = 1 − z −2

We investigate different patterns of added redundancy for different block sizes. First, we consider
symbols blocks with two redundant symbols. In this case, Eq. (5.8) has always a solution for dk
since the determinant of the matrix in the left-hand side is always non-zero. Filter coefficients for
5 Filterbank Structures for ISI Channel Pre-Equalization 112

K = 3 and N = 5 can be determined as:

1 − z −4
G0 (z) = = 1 + z −2
1 − z −2
z −1 − z −3 −1
G1 (z) = −2 = z
(5.10)
1−z
z −2 − z −4
G2 (z) = = z −2
1 − z −2

Next, we consider one redundant symbol at the end of each block (N = K + 1). Depending on
the number of symbols N in each block, there may or may not be solution for dk . Equation (5.8)
can be written as:
) * ) *
(1)N −2 (1)2N −2 ··· (1)N L−2 (1)−k
dk = − (5.11)
(−1)N −2 (−1)2N −2 · · · (−1)N L−2 (−1)−k
% &' (
≥2

When N is an even number, Eq. (5.11) does not provide a solution for dk . If the block length N
is an odd number, there is at least one solution for the filters. For example, the set of filters for
N = 5 and K = 4 can be computed as:

1 − z −4
G0 (z) = = 1 + z −2
1 − z −2
z −1 − z −9
G1 (z) = = z −1 + z −3 + z −5 + z −7
1 − z −2
(5.12)
z −2 − z −4 −2
G2 (z) = −2 = z
1−z
z −3 − z −9
G3 (z) = = z −3 + z −5 + z −7
1 − z −2

This example shows for a non-maximally decimated filterbanks there exists FIR solutions for the
branch filters even if the number of redundant symbols per block is smaller than the number of
zeros of the channel filter.

Example 3: Minimum redundancy per block

We are interested in maintaining a minimum amount of redundancy per symbol block since it
corresponds to a smaller reduction in the data symbol rate. For a block of K data symbols, we
consider adding only one redundant symbol N = K + 1. As shown in the previous example,
there are cases where no solution exists for branch filters. Here, we derive conditions under which
5 Filterbank Structures for ISI Channel Pre-Equalization 113

Eq. (5.8) has at least a solution for minimum added redundancy. Assuming that one redundant
symbol is added to the end of each block of N − 1 symbols, we can write Eq. (5.8) as:
 −(N −2) −(2N −2) −(N L−2)
  
z1 z1 ··· z1 z1−k
 −(N −2) −(2N −2) −(N L−2)   −k 
 z2 ···   
 z2 z2  dk = −  z2  (5.13)
 ··· ··· ··· ···   ··· 
   
−(N −2) −(2N −2) −(N L−2) −k
zM zM ··· zM zM
% &' (
L≥M

Note that each block of N symbols contains only one redundant symbols. We need at least M
redundant symbols dk (i, j) in order to compensate for M roots of H(z). To ensure that at least
one solution for dk exists, the matrix in the left-hand side of Eq. (5.13) should be a full rank
matrix. A condition for this matrix to be full-rank is that:

ziN = zjN for 1 ≤ i, j ≤ M (5.14)

The above statement can be proved by examining the determinant of the first M columns of the
left-hand side matrix in Eq. (5.13). Using the properties of the determinant of Vandermonde
matrix, we can show that if Eq. (5.14) holds, then the determinant of this matrix is non-zero.

5.2.1 IIR implementation of the FIR filters

Equation (5.7) indicates that the coefficients of each FIR filter Gk (z) are computed based on the
values of dk (i, j). It is possible to implement filters in each branch without actually computing
the coefficients gk [i] of these FIR filters.
Let us assume that the inverse filter 1/H(z) has an impulse response hinv [n]. Equation (5.7)
can be written in time domain as:


L 
gk [n] = hinv [n − k] + dk (i, j)hinv [n − N i − j] (5.15)
i=0 j∈S

The value of gk [n] is forced to zero after a finite number of samples. To eliminate the tail of the
inverse channel impulse response hinv [n], the impulse response is delayed, scaled and subtracted
from the original impulse. In the context of FIR filter implementation, this technique has been
used to implement an FIR filter as a tail-cancelling IIR filter [65].
5 Filterbank Structures for ISI Channel Pre-Equalization 114

The recursive difference equation that describes each IIR filter can be written as:


L  
M
uk [n] = sk [n − k] + dk (i, j)sk [n − N i − j] − uk [n − i] (5.16)
i=0 j∈S i=1

For an IIR implementation of the filter gk [n], the actual coefficients of the filter are not required.

5.2.2 Average transmitted power

In this section, we compute the average signal power at the output of the filterbank as a function
of the input signal power σa2 and filter coefficients. Input signal a[n] is assumed to be an i.i.d.
random variable. As shown in Fig. 5.3, the output of the filterbank is a sum of signals from
different branches. Since the input signals to these branches are independent, the output power
can be written as a sum of the power at the output of each branch:


K−1
E{u2 [n]} = E{u2k [n]} (5.17)
k=0

The average power at the output of each branch can be written as:

Mk −1
σa2 
E{u2k [n]} = |gk (i)|2 (5.18)
N
i=0

where Mk is the length of the filter impulse response gk in the kth branch. Due to the up-sampling
in each branch, the sum of squared coefficients is normalized by N . As a result, the average signal
power at the output of the filterbank can be written as:

K−1 Mk −1
σa2  
E{u2 [n]} = |gk (i)|2 (5.19)
N
k=0 i=0

This equation relates the average output power to the input signal power, filter coefficients and
the block size N .

Example

Let us consider the following channel filter:

H(z) = (1 − z −1 )(1 + z −1 )(1 − ej0.8π z −1 )(1 − e−j0.8π z −1 ) (5.20)

5 Filterbank Structures for ISI Channel Pre-Equalization 115

We design a pre-equalizer using the filterbank structure. Several block sizes N and symbol sizes
K are considered. We would like to gain the maximum throughput from the channel. The
decimation ratio K/N is directly related to the data symbol-rate. As a measure of transmitted
power, we use the power gain derived from Eq. (5.19). Table 5.1 shows the results. Since the
filterbank structure contains only FIR filters, the transmitter power is always bounded. However,
depending on the channel filter and the choice of up-sampling and down-sampling factors, the
total filter gain varies widely. As shown in the previous section, for some combinations of N and
K, there is no solution for the filters. From the results shown in Table 5.1, it is evident that if the
number of redundant symbols per block is reduced, the average transmitted power can increase
significantly.

Table 5.1 Comparison of decimation rates and the average filterbank power gain.
K−1 Mk
1 
K N Decimation Rate |gk (i)|2
N i=0
k=0
4 8 0.500 4.05
5 8 0.625 35.9
6 8 0.750 36.2
7 8 0.875 no solution
5 9 0.556 6.28
6 10 0.600 8.33
7 11 0.637 10.10

5.3 Performance Comparison

The filterbank structure is capable of creating a zero-ISI pre-equalizer for a channel with spectral
nulls. A non-maximally decimated filterbank can include different patterns of redundancy in a
block of data symbols. In many cases, the number of redundant symbols can be chosen to be less
than the length of the channel filter. A block-by-block pre-filtering, as discussed in Chapter 4,
requires the same number of redundant symbols per block as the number of roots of the channel
filter. In this section we investigate whether the more flexible pattern of redundancy provided by
the filterbank can improve the performance in terms of the average transmitted power for a given
data symbol-rate.
As a measure of the effective data symbol-rate, the system sampling rate is normalized by
effective ratio of data symbols per block. For a system with a sampling rate of 1/Ts , the effective
symbol rate is K/(N Ts ) symbols/sec. In examples below, we use the sampling rate corresponding
to a PCM channel, 1/Ts = 8000 Hz.
5 Filterbank Structures for ISI Channel Pre-Equalization 116

For a channel transfer function with a single root, it is expected that the filterbank struc-
ture and block-by-block pre-filtering have equivalent performance since the number of redundant
symbols per block is minimal. As an example, we consider a channel with a null at zero fre-
quency (H(z) = 1 − z −1 ). Figure 5.5 shows the power gain of both filters as a function of the
effective symbol-rate. The power gain for the block-by-block filter structure is computed based
on Eq. (4.55). As shown, the required power gain of both structures is the same at a given data
symbol rate.

9
Pre−filtering N−K=1,
8 Filterbank Structure, N−K=1
7
Filter power gain (dB)

4
N=5
3
N=4
2

0
5000 5500 6000 6500 7000 7500
Effective symbol−rate (Symbols/Sec)

Fig. 5.5 Power gain as a function of the effective symbol rate for H(z) = 1 − z −1 .
The power gain for filterbank structure is similar to that of the pre-filtering structure.

When the memory length of the channel filter is larger than one, a filterbank structure can
accommodate more than one pattern of redundancy. For example for a channel with a transfer
function H(z) = 1 − z −2 , we can have one or two redundant symbols per block. Figure 5.6 shows
the power gain of the filterbank structure for these two cases and compares it with that of a
block-by-block pre-filtering structure. In this particular example, the power gain of the filterbank
is the same for two different patterns of redundancy.
The same analysis is performed for a channel filter with transfer function:

H(z) = (1 + z −1 )(1 − ej0.8π z −1 )(1 − e−j0.8π z −1 )

Figure 5.7 shows that the overall effect of reducing the number of redundant symbols per block
to one symbol is an increase in required power for a given data symbol rate.
5 Filterbank Structures for ISI Channel Pre-Equalization 117

7
Pre−filtering N−K=2,
6 Filterbank Structure, N−K=2
Filterbank Structure, N−K=1
Filter power gain (dB)
5

0
5000 5500 6000 6500 7000 7500
Effective symbol−rate (Symbols/Sec)

Fig. 5.6 Power gain as a function of the effective symbol rate for H(z) = 1 − z −2 .
The power gain for filterbank structure is similar to that of the pre-filtering structure.

9
Pre−filtering N−K=3,
Filterbank Structure, N−K=3
8
Filterbank Structure, N−K=1
Filter power gain (dB)

2
5800 6000 6200 6400 6600 6800 7000
Effective symbol−rate (Symbols/Sec)

Fig. 5.7 Power gain as a function of the effective symbol rate for H(z) = (1 +
z −1 )(1 − ej0.8π z −1 )(1 − e−j0.8π z −1 ).
5 Filterbank Structures for ISI Channel Pre-Equalization 118

The examples above suggest that although the filterbank structure is capable of accommo-
dating more flexible patterns of redundancy in a block of symbols, the overall effect of different
redundancy pattern does not reduce the required transmitted power at a given data symbol rate.

5.4 Remarks

We investigated pre-equalizer structures for the up-stream PCM channel. The pre-equalizer is
implemented as FIR filters with in a filterbank structure. The non-maximally decimated filterbank
provides a natural way of inserting redundant symbols in blocks of data symbols. Compared to
block-by-block redundancy insertion discussed in Chapter 4, the filterbank structure offers more
flexibility in choosing the pattern of redundancy. However, in terms of the required transmitted
power, there is no gain in having patterns different than those we discussed in Chapter 4.
 119

Chapter 6

A New Family of Pulse Shaping

Filters

Data transmission over bandlimited channels requires pulse shaping to eliminate or control Inter-
Symbol Interference (ISI). Nyquist filters provide ISI-free transmission. Here, we introduce a
phase compensation technique to design Nyquist filters. Phase compensation can be applied to
the square-root of any zero-phase bandlimited Nyquist filter with a normalized excess bandwidth
of less than or equal to one. The resulting phase-compensated square-root filter is also a Nyquist
filter. In the case of a raised-cosine spectrum, the phase compensator has a simple piecewise
linear form. Such a technique is particularly useful to accommodate two different structures for
the receiver, one with a filter matched to the transmitting filter and one without a matched filter.
In this chapter, a general relationship between the phase and amplitude responses of a ban-
dlimited Nyquist filter is presented. We also show that a bandlimited zero-phase Nyquist filter
can always be split into two cascaded Nyquist filters matched to one another. The special case of
the square-root raised-cosine spectrum is investigated. We quantify the SNR degradation due to
replacing the matched filter with a lowpass filter.
We also introduce a new family of Nyquist filters which subsumes raised-cosine filters. These
“generalized raised-cosine filters” offer more flexibility in filter design. Design examples are pro-
vided to illustrate the applications of the new Nyquist filters.

6.1 Pulse shaping filters for voiceband PCM channels

In Section 4.1, we explained that the transmitter filter can be implemented as a discrete time filter
and the only continuous-time filter required is an interpolating filter. In order for an interpolating
Parts of this chapter have been reported in [66] and [67]
6 A New Family of Pulse Shaping Filters 120

filter not to add any ISI to the transmitted signal, the impulse response of the filter should have
regular zero crossings at integer multiples of the sampling times. As we describe in the next
section, filters with this property are known as Nyquist filters.
Here, we consider two scenarios for data transmission over a PCM channel: the up-stream
channel, and the end-to-end PCM channel that consists of a cascade of an up-stream and a
down-stream channel. Figure 6.1 shows these two scenarios. For an up-stream PCM channel, the
Nyquist pulse shaping should be implemented entirely at the transmitter since there is no access
to the receiver front end of this channel. In Fig. 6.1(a) the pulse shaping filter is shown as GT (f ).
For an end-to-end PCM channel1 , we assume that the signal conversion from continuous-time to
discrete-time and back to continuous-time, performed in the CODEC in the central office, does
not cause any significant distortion. This assumption is valid only if the quantization error in the
CODEC can be avoided. Figure 6.1(b) shows an end-to-end PCM channel. For the end-to-end
PCM channel, we use the same filter GT (f ) at the transmitter2 . We would like to use a filter
matched to GT (f ) at the receiver front-end in order to maximize the signal-to-noise ratio at the
sampling instants. At the same time, the cascade filter GT (f )GR (f ) should be a Nyquist filter.
The phase compensation technique described in this chapter, provides a method of performing
pulse shaping at the transmitter of an analog PCM modem, complying with two different channels:
the end-to-end PCM channel where a matched filter can be implemented at the receiver, and the
up-stream PCM channel where the receiver filter is fixed.

6.2 Nyquist filters

A conventional baseband Pulse Amplitude Modulation (PAM) signal can be represented as

∞

x(t) = ak g(t − kTs ) (6.1)
k=−∞

where ak ’s are the transmitted symbols and g(·) is a real-valued “Nyquist pulse” which satisfies
Nyquist’s first criterion,

1 for k=0
g(kTs ) = (6.2)
0 for k = ±1, ±2, · · ·

g(·) represents the overall impulse response of the transmitting filter, the receiving filter and the
communication channel. Each transmitted symbol ak can be recovered from the received signal
1
Note that for an end-to-end PCM channel, we assume that sample timing and channel estimates are known.
2
Note that the digital transmitter filter H
t (f ) can be different from the filter Ht (f ) used in the up-stream
channel.
6 A New Family of Pulse Shaping Filters 121

7UDQVPLWWHU &KDQQHO5HFHLYHU)LOWHU
{DQ } {DÖQ }
+ W H Mω7V 'LVFUHWHWR
FRQWLQXRXV *7 I + FU I
7V

7V
D 7KHXSVWUHDP3&0FKDQQHO

7KH8SVWUHDP(TXLYDOHQW
7UDQVPLWWHU &KDQQHO)LOWHU
{DQ }
'LVFUHWHWR
+ W′ H Mω7V FRQWLQXRXV *7 I + FU I &2'(&

7V 7V

7KH'RZQVWUHDP
(TXLYDOHQW&KDQQHO
)LOWHU 5HFHLYHU
{DÖQ }
+ FU′ I *5 I + U H Mω7V
7V

E (QGWRHQG3&0FKDQQHO

Fig. 6.1 The pulse shaping design for a PCM channel should consider two scenarios:
(a) the up-stream PCM channel, where the pulse shaping is performed entirely at the
transmitter as a Nyquist filter GT (f ) and, (b) an end-to-end PCM channel, where
the the receiving filter GR (f ) is matched to the transmitting filter GT (f ) and, at the
same time, the cascaded filter of the two GT (F )GR (f ) is also a Nyquist filter.
6 A New Family of Pulse Shaping Filters 122

x(t), by taking samples of x(t) at the time instants kTs . In other words, choosing g(·) as a
Nyquist pulse avoids Inter-Symbol-Interference (ISI) and allows sample-by-sample detection at
the receiver. In the frequency domain, Nyquist’s first criterion is written as:
∞
 n
G(f − ) = Ts (6.3)
n=−∞
Ts

where G(f ) is known as a Nyquist filter. A particular Nyquist filter with wide practical applica-
tions is the raised-cosine filter

 1−α

 Ts |f | ≤


 + πT  1 − α  , 1 − α 2Ts 1+α
Ts cos2
s
Grc (f ) = |f | − ≤ |f | ≤ (6.4)

 2α 2T 2T 2Ts


s s
1 + α

 0 |f | >
2Ts

where α is called the roll-off factor and takes values between zero and one. The parameter α also
represents the normalized excess bandwidth occupied by the signal beyond the Nyquist frequency
1/2Ts .
In practical applications, the overall magnitude response of the Nyquist filter is split evenly
between the transmitter and receiver. The phase response of the receiving filter compensates for
the transmitting filter phase so that the overall filter has a linear phase:

G(f ) = GT (f )GR (f )


|GT (f )| = |GR (f )| = |G(f )| (6.5)
 GR (f ) = −2πf τ0 −  GT (f )

GT (f ) and GR (f ) are the transfer functions of the transmitting and receiving filters accordingly.
The receiving filter in this case is matched to the transmitting filter to maximize the signal-to-noise
ratio (SNR) at the sampling time instants at the receiver [32]. The transmitting and receiving
filters are typically considered to be linear phase with a nominal delay of τ0 /2 that is required to
make the filters physically realizable.

6.2.1 Nyquist Filters with compensated phase

For Nyquist filter design a constraint on the magnitude is adequate to provide regular zero-crossing
property. The phase response of the filter can be designed independently.
We consider more general phase responses for the transmitting and receiving filters. The phase
6 A New Family of Pulse Shaping Filters 123

compensated transmitting filter satisfies Nyquist’s first criterion. As a result, with or without a
matched filter at the receiver, we obtain ISI free transmission. The ability to use a simple receiver
filter (not a matched filter) is particularly useful to reduce the cost of the receiver [68].

6.3 Extension of Nyquist’s first criterion

The transfer function of a filter G(f ) with real impulse response g(t) can be expressed in terms
of its magnitude and phase responses:

G(f ) = |G(f )| · ejφ(f ) (6.6)

where φ(·) is a real-valued odd function and |G(·)| is a real-valued function with even symmetry.
In our discussion we assume G(f ) is bandlimited to |f | < 1/Ts . We consider the relationship
between |G(f )| and φ(f ) such that G(f ) satisfies Nyquist’s first criterion. For a Nyquist filter
with normalized excess bandwidth less than or equal to one, Nyquist’s first criterion is written as:

1
|G(f )| · ejφ(f ) + |G(f − 1/Ts )| · ejφ(f −1/Ts ) = K 0≤f ≤ (6.7)
Ts

where K is a real constant. Following Gibby and Smith [69], we decompose Eq. (6.7) into the
real and imaginary parts:

|G(f )| · cos φ(f ) + |G(f − 1/Ts )| · cos φ(f − 1/Ts ) = K (6.8)

|G(f )| · sin φ(f ) + |G(f − 1/Ts )| · sin φ(f − 1/Ts ) = 0 (6.9)

As shown in [69], one can combine Eq. (6.8) and Eq. (6.9) to express φ(f ) in terms of the magnitude
response of G(f ):

K 2 + |G(f )|2 − |G(f − 1/Ts )|2 1

φ(f ) = arccos 0≤f ≤ (6.10)
2 K|G(f )| Ts

Note that for −1/Ts ≤ f ≤ 0 the phase response φ(f ) is determined using its odd symmetry
property φ(−f ) = −φ(f ). Since the argument of the arccos(·) should be limited to one in absolute
value, there may not be a real solution for φ(f ).

6.3.1 Square-root Nyquist filters

For the case of a zero-phase Nyquist filter, we can design a phase compensator for the square-root
filter such that the compensated square-root filter also satisfies Nyquist’s criterion. To show this,
6 A New Family of Pulse Shaping Filters 124

consider the zero-phase Nyquist filter |G(f )|2 :

1
|G(f )|2 + |G(f − 1/Ts )|2 = Ts 0≤f ≤ (6.11)
Ts

The phase compensated square-root filter is:

G(f ) = |G(f )| · ejθ(f ) (6.12)

The phase compensator θ(f ) is chosen such that G(f ) is Nyquist filter. From Eq. (6.10), θ(f ) can
be expressed in terms of |G(f )| as:

A2 + |G(f )|2 − |G(f − 1/Ts )|2 1

θ(f ) = arccos 0≤f ≤ (6.13)
2 A|G(f )| Ts
√
where A is a real constant. If we choose A= Ts , the phase compensator simplifies:

 |G(f )|  1
θ(f ) = arccos √ 0≤f ≤ (6.14)
Ts Ts

|G(f )|
From Eq. (6.11) it is evident that 0 ≤ √ ≤ 1, therefore there is always a real solution for
Ts
θ(f ) in Eq. (6.14).
The phase compensator can fully characterize the zero-phase Nyquist filter and the phase
compensated square-root filter:

|G(f )|2 = Ts cos2 θ(f ) (6.15)

G(f ) = Ts cos θ(f ) · ejθ(f ) (6.16)

Note that |G(f )|2 is to within a scaling factor, the real part of G(f ):

- .
|G(f )|2 = Ts Re G(f ) (6.17)

6.3.2 Phase compensated square-root raised-cosine filter

The standard raised-cosine spectrum as defined in Eq. (6.4) is zero phase and satisfies Nyquist’s
criterion. To find the phase compensator θ(f ), we substitute the square-root of the raised-cosine
spectrum into Eq. (6.14). The resulting phase response θ(f ) is a piecewise linear function, as
6 A New Family of Pulse Shaping Filters 125

θ(f)

1 π Τs
2α

- 1−α 1−α 1
π Τs f
2Τs 2Τs 2α

Fig. 6.2 Piece-wise linear phase compensator for square-root raised-cosine filter.

shown in Fig. 6.2 and expressed below:


 πTs 1−α

 −f − for − 1+α 1−α
2Ts ≤ f ≤ − 2Ts

 2α 2Ts
1−α
θ(f ) = 0 for |f | < (6.18)


2Ts

 πTs 1−α 1−α 1+α
 −f + for 2Ts ≤f ≤ 2Ts
2α 2Ts

Note that −θ(f ) can also be used for phase compensation. Therefore, for the transmitting and



receiving filters, we can use GT (f )= |Grc (f )| ejθ(f ) and GR (f )= |Grc (f )|e−jθ(f ) which are indi-
vidually Nyquist filters and matched to one another. Throughout our discussion here, we assume
the time delay τ0 , introduced in Eq. (6.5), is zero. A time delay can always be added to an
appropriately truncated filter response to make it causal and physically realizable.
The impulse response of the phase-compensated square-root raised-cosine filter has a simple
closed form:

π t αt 1
g(t) = F −1 { |Grc (f )| ejθ(f ) } = √ sinc( ) · sinc( − ) (6.19)
2 Ts Ts Ts 2

where F −1 is the inverse Fourier transform and sinc(x) = sin(πx)/πx. The filter impulse response
is expressed as a product of two terms where the first term sinc(t/Ts ), provides the regular zero
6 A New Family of Pulse Shaping Filters 126

crossings at integer multiples of Ts . In general, adding the phase compensator to the square-
root filter changes the filter impulse response and the pulse shape may no longer be symmetric.
The zero-phase square-root raised-cosine filter has a closed form impulse response with an even
symmetry around the center:
   
−1

sin π (1 − α) t + 4αt cos π (1 + α) t
F { |Grc (f )|} =   (6.20)
π 1 − (4αt)2 t

Using the inverse Fourier transform of the Eq. (6.17), we can show that the impulse response
of a standard raised-cosine filter is within to a scaling factor the even part of g(t):
√
Ts + ,
grc (t) = g(t) + g(−t)
2
π t + αt 1 αt 1 ,
(6.21)
= sinc( ) · sinc( − ) + sinc( + )
4 Ts Ts 2 Ts 2

which is consistent with the result given in [70, p. 62].

6.3.3 Special Case: Full excess bandwidth

In the case of full excess bandwidth, i.e. α = 1, θ(f ) is a linear function over the whole frequency
range of the filter:
πTs 1
θ(f ) = − f for |f | ≤ (6.22)
2 Ts
The zero-phase square-root raised-cosine filter (no phase compensation) has the following impulse
response:

π t 1 t 1
F −1 { |Grc (f )|} = √ sinc( + ) · sinc( − ) (6.23)
2 Ts Ts 4 Ts 4

The zero-crossings of the impulse response do not occur in the correct places. Adding the phase
response θ(f ) to the filter delays the impulse response by a quarter of the sampling period Ts :

π t t 1
F −1 |Grc (f )| ejθ(f ) = √ sinc( ) · sinc( − ) (6.24)
2 Ts Ts Ts 2
√
Figure 6.3 shows the two impulse responses where both pulses are normalized by 1/ Ts . Since
θ(f ) causes only a pure time delay, the phase compensation does not change the pulse shape. Thus,
in the case of full excess bandwidth, with or without a matched filter at the receiver, one can
achieve ISI free transmission using the square-root full raised-cosine filter by merely shifting the
sampling points at the receiver.
6 A New Family of Pulse Shaping Filters 127

1.4

1.2

1
normalized amplitude

0.8

0.6

0.4

0.2

−0.2
−4 −3 −2 −1 0 1 2 3 4
t/Ts

Fig. 6.3 Normalized impulse responses of the square-root raised-cosine filter with
α=1, (dashed line). The phase compensated filter has a delayed impulse response
(solid line).

6.3.4 Eye pattern diagram

Eye pattern diagrams provide a simple and effective way to measure and visualize the noise
immunity of a pulse shaping scheme. Using an eye pattern, one can also assess the effect of
errors in the timing phase and the sensitivity to phase jitter [38]. Here, we use the eye pattern
diagrams to compare conventional raised-cosine filters with the phase-compensated square-root
raised-cosine filters.
First consider a raised-cosine filter with α= 1; Figure 6.4(a) shows the eye pattern of the pulse
shape modulated by binary data. At integer multiples of the sampling period, each transmitted
symbol can be recovered without any ISI. Figure 6.4(b) shows the eye pattern of a square-root
full raised-cosine spectrum. Note that the center of the eye in Fig. 6.4(b) coincides with peak
of the impulse response. For zero ISI, the sampling points should be shifted by one quarter of
sampling time. However, for the special case of binary data, the center of the eye has the widest
vertical opening. In fact, in Appendix C we show that the lower boundaries of the eye pattern stay
constant, equal to 1 and −1 for half of the sampling period around the center. The boundaries
are shown in Fig. 6.4(b) as dotted lines. Compared to the eye pattern of the conventional full
raised-cosine filter, the eye pattern of the square-root filter shows that it is insensitive to sampling
6 A New Family of Pulse Shaping Filters 128

phase error over a large interval.

Note however, that the above results are based on the binary PAM signaling and do not
generalize to a multi-level PAM signaling. Figure (6.5) illustrates the eye patterns of the same
filters with 4 level input data. To avoid ISI, the sampling points for the square-root raised-cosine
filter should be shifted by a quarter of sampling period.
As another example, consider the raised-cosine filter with α=0.5; Figure 6.6 shows the eye
diagram for the raised-cosine and phase-compensated square-root raised-cosine filters. Due to
the non-symmetric impulse response of the filter, the eye pattern of the modified square-root
raised-cosine is not symmetric around the sampling points.

6.3.5 SNR degradation

It is well known that the use of matched filter at the receiver of an additive white Gaussian noise
(AWGN) channel maximizes the signal-to-noise ratio at the sampling instants [32]. Here, we
quantify the SNR degradation due to the use of a non-matched filter at the receiver.
Assume that the transmitter uses a modified square-root raised-cosine filter. The received
signal is passed through a filter H(f ) and is sampled at integer multiples of Ts . Let us also
assume the channel adds only white Gaussian noise to the transmitted signal. The noise power
at the output of the receiving filter is calculated as:
 ∞
N0
σn2 = |H(f )|2 df, (6.25)
2 −∞

where N0 /2 is the power spectral density of the noise. The signal power at the sampling instant
can be written as:  2
 ∞ 
σs2 =  GT (f )H(f )df  (6.26)
−∞

where GT (f ) is the transmitting filter. We consider two different receiving filters:

1. A filter matched to the transmitting filter: H(f ) = G∗T (f ).

1+α
2. A brick-wall filter with a pass-band region |f | ≤ .
2Ts
Using the fact that the transmitting filter is normalized, the SNR after sampling is

2
SNR (matched) = (6.27)
N0
2
SNR (brick-wall) = (6.28)
(1 + α)N0
6 A New Family of Pulse Shaping Filters 129

1.5

0.5
amplitude

−0.5

−1

−1.5
−0.5 0 0.5
time offset (τ)

(a) Raised-cosine filter with α = 1.

1.5

0.5
amplitude

−0.5

−1

−1.5
−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5
time offset (τ)

(b) Square-root raised-cosine filter with α = 1.

Fig. 6.4 Binary eye patterns for a raised-cosine filter and a square-root raised-cosine
filter.
6 A New Family of Pulse Shaping Filters 130

1
amplitude

−1

−2

−3

−4

−5
−0.5 0 0.5
time offset (τ)

(a) Raised-cosine filter with α = 1.

1
amplitude

−1

−2

−3

−4

−5
−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5
time offset (τ)

(b) Square-root raised-cosine filter with α = 1.

Fig. 6.5 Eye patterns (4 level PAM) for a raised-cosine filter and a square-root
raised-cosine filter.
6 A New Family of Pulse Shaping Filters 131

1.5

0.5
amplitude

−0.5

−1

−1.5

−0.5 0 0.5
time offset (τ)

(a) Raised-cosine filter with α = 0.5.

1.5

0.5
amplitude

−0.5

−1

−1.5

−0.5 0 0.5
time offset (τ)

(b) phase-compensated square-root raised-cosine filter with α = 0.5.

Fig. 6.6 Binary eye patterns for a raised-cosine filter and a phase compensated
square-root raised-cosine filter.
6 A New Family of Pulse Shaping Filters 132

Equations (6.27) and (6.28) show that there will be a 10 log10 (1 + α) dB loss if the matched filter
is replaced by an ideal brick-wall filter. For instance, for α = 0.5 the loss is 1.76 dB.

6.4 Generalized Raised-Cosine Filters

We use the phase compensation technique, discussed in Section 6.3, to characterize a more general
family of Nyquist filters. Compared to the conventional raised-cosine filters, the new filters offer
more flexibility in filter design. For instance, the rate of asymptotic decay of the filter impulse
response may be increased, or the residual ISI introduced by truncation of the impulse response
may be minimized.

6.4.1 Nyquist filter design using phase compensation

We derive a sufficient condition for a real-valued odd function θ(f ) so that the zero-phase bandlim-
ited filter in Eq. (6.15) and the phase compensated square-root filter in Eq. (6.16) satisfy Nyquist’s
first criterion. Note that the filters are bandlimited to |f | < 1/Ts . The ISI-free condition for both
filters can be expressed as:



1
Ts cos θ(f ) · ejθ(f ) + Ts cos θ(f − 1/Ts ) · ejθ(f −1/Ts ) = Ts 0≤f ≤ (6.29)
Ts

Solving for θ(f ), we obtain the following condition:

π 1
θ(f ) + θ(1/Ts − f ) = nπ + 0≤f ≤ (6.30)
2 Ts

where n is an integer. Note that we use the odd symmetry of θ(f ) to obtain this result.
Using the condition developed for the phase compensator in Eq. (6.30), we introduce a general
family of Nyquist filters. The standard raised-cosine filter for any normalized excess bandwidth
can be regarded as a special case of this family of filters.
Let us define a real-valued monotonic odd function V (x) which satisfies the following condi-
tions:

V (x) = − V (−x)
(6.31)
V (x) = 1 x≥1

Consider a real-valued odd function φ(f )= − φ(−f ) defined as:

π + 2Ts 1 , π
φ(f ) = − V (f − ) − f ≥0 (6.32)
4 α 2TS 4
6 A New Family of Pulse Shaping Filters 133

It can be verified that φ(f ) satisfies Eq. (6.30) therefore, the following filters satisfy Nyquist’s
first criterion:

HG (f ) = Ts cos2 φ(f )

(6.33)
Hsqrt(f ) = Ts cos φ(f ) ejφ(f )

We call HG (f ) a generalized raised-cosine filter. If V (x) is a increasing function for −1 < x < 1,
HG (f ) corresponds to a low-pass filter with monotonically decreasing spectrum. The phase-
compensated square-root of HG (f ) is denoted by Hsqrt (f ). The standard raised-cosine filter,
defined in Eq. (6.4), is a special case of HG (f ) where

 −1 x < −1

V (x) = x −1 ≤ x ≤ 1 (6.34)


+1 x > +1

The impulse response of Hsqrt (f ) does not have a closed form in general. In Appendix D, we
show that:

hsqrt (t) = F −1 {Hsqrt (f )}

 1
1 t πtα παt π πtα  (6.35)
= √ sinc( ) · cos( )+ cos V (x) − x dx
Ts Ts Ts Ts 0 2 Ts

The first term, sinc(t/Ts ), provides regular zero crossings at integer multiples of Ts except at
t = 0. In the case of conventional raised-cosine filter, the impulse response has a closed form as
shown in Eq.(6.19). In more general cases, the impulse response must be evaluated numerically.
Decomposing the impulse response into two terms as shown in Eq. (6.35) facilitates the numerical
evaluation.
The impulse response of HG (f ), denoted by hG (t), is to within a scaling factor the even part
of hsqrt (t) (see Eq. (6.17)):
√
Ts + ,
hG (t) = hsqrt (t) + hsqrt (−t)
2
 (6.36)
t πtα παt 1  π   πtα 
= sinc( ) · cos( )+ sin V (x) · sin x dx
Ts Ts Ts 0 2 Ts

A family of Nyquist filters is also presented in [68] which with or without a matched filter at the
receiver satisfy the ISI-free conditions. Each filter in this family is characterized by a real-valued
continuous function ν(x) which “controls the transfer band of the filters”. We can rewrite the
family of filters presented in [68] in the form of a phase-compensated square-root filter, described
6 A New Family of Pulse Shaping Filters 134

in Eq. (6.16). In fact, Xia’s Nyquist filters belong to the family of generalized raised-cosine filters
in Eq. (6.33) with α≤1/3.

6.5 Design examples

In this section, we provide examples of Nyquist filter design using generalized raised-cosine filters.
In practice, it is important that the transmitting and receiving filters be well approximated with
short impulse responses [71, 72]. From this point of view, the standard raised-cosine filter is not
necessarily the best choice for Nyquist filter design. We consider two design examples. First, a
family of Nyquist pulses is designed such that these pulses have faster asymptotic decay than the
raised-cosine filter impulse responses. In the second example, Nyquist filters are designed such
that the inter-symbol interference (ISI) caused by truncation of the transmitting and receiving
filters is minimized.

6.5.1 Nyquist filters with smoother spectra

It is a well known result that if the amplitude response of a filter along with its first (K−1)
derivatives are all continuous but its Kth derivative is discontinuous, the filter impulse response
asymptotically decays as 1/|t|K+1 [56]. Based on this result, the impulse response of a raised-cosine
filter decays asymptotically as 1/|t|3 . Using the generalized raised-cosine spectrum, we can design
filters with higher rates of decay of their impulse responses. We use a family of polynomials Pn (x)
to design the generalized raised-cosine spectra of Eq. (6.33) with higher degrees of continuity:


 −1 x < −1
V (x) = Pn (x) −1 ≤ x ≤ 1 (6.37)


1 x>1

where V (x) specifies the phase compensator in Eq. (6.32). To satisfy condition Eq. (6.31), Pn (x)
should be an odd function of x:

n
Pn (x) = ak · x2k−1
k=1

where ak ’s are real-valued coefficients and an takes on a non-zero value. Furthermore, the co-
efficients ak ’s are determined such that V (x) and its first (n − 1) derivatives are continuous at
6 A New Family of Pulse Shaping Filters 135

x = ±1 which results in the following expression:

 x
(1 − u2 )n−1 du
0
Pn (x) =  1 n = 1, 2, 3, · · · (6.38)
2 n−1
(1 − u ) du
0

The first derivative of Pn (x) is a polynomial of degree (2n−2) with (n−1) repeated roots at x=1
and by symmetry (n−1) repeated roots at x = −1. Since the first derivative of Pn (x) has no
roots in −1 < x < 1 interval, Pn (x) is monotonic in this interval. Figure 6.7 illustrates the first
five polynomials of this family3 . The actual polynomials Pn (x) are given in Table 6.1. Note that
P1 (x) generates the standard raised-cosine filter. Using a Taylor series, we can show that the
generalized raised-cosine spectrum corresponding to Pn (x) has (2n−1) continuous derivatives and
its impulse response decays asymptotically as 1/|t|2n+1 .
The rate of decay describes only the asymptotic behavior of the impulse response and does not
characterize the impulse response for the first few lobes around the center. Figure 6.8 compares
the impulse responses of two generalized raised-cosine filters generated from polynomials P1 (x)
and P2 (x). The excess bandwidth in this case corresponds to α=0.5. Figure 6.8(a) shows that for
the first few lobes |h2 (t)| is larger than |h1 (t)|. As shown in Fig. 6.8(b), only for the lobes farther
away from the center is h2 (t) smaller than h1 (t).
Apart from the rate of decay of the filter impulse response, there are other issues to be
considered. Since timing recovery at the receiver is not always perfect, we encounter timing phase
jitter. The eye pattern diagram of a pulse shape can be used to assess the immunity of the pulse to
timing phase jitter. Using polynomials given in Table 6.1, we design five generalized raised-cosine
filters with α = 1 and normalized sampling period. Figure 6.9 shows the inner boundaries of the
binary eye diagrams associated with the impulse responses of these filters. As we increase the
number of continuous derivatives of the spectrum, the width of the eye diagram decreases (see
Table 6.1).

Table 6.1 Comparison of several generalized raised-cosine filters with α = 1.

n Pn (x) Decay Eye-width

1 x |t|−3 1.000
2 2 (−x3 + 3x)
−1
|t|−5 0.911
3 2 (3x5 − 10x3 + 15x)
−3
|t|−7 0.843
4 2 (−5x7 + 21x5 − 35x3 + 35x)
−4
|t|−9 0.791
5 2−7 (35x9 − 180x7 + 378x5 − 420x3 + 315x) |t|−11 0.750

3
P1 (x) and P3 (x) correspond to examples presented by Xia [68]
6 A New Family of Pulse Shaping Filters 136

0.8

0.6

0.4

0.2

−0.2

−0.4 n=1
n=2
−0.6 n=3
n=4
−0.8 n=5
−1
−1 −0.5 0 0.5 1

Fig. 6.7 Polynomials Pn (x) to generate the generalized raised-cosine filter.

6.5.2 Nyquist filters with reduced ISI due to truncation

As shown in Section 1, pulse shaping can be split between the transmitter and the receiver so
that the overall response satisfies Nyquist’s first criterion (see Eq. (6.5)):

g(t) = gT (t) ∗ gR (t) (6.39)

Truncating a Nyquist pulse does not affect its Nyquist zero-crossing property. However, the
convolution of the truncated responses will in general no longer satisfy Nyquist’s first criterion.
Here we use generalized raised-cosine filters to design Nyquist filters with minimized ISI due
to impulse response truncation. To study the effect of truncating the impulse response of the
square-root filter, we consider the following conditions:

• We assume that the compensated square-root generalized raised-cosine filters are used at the
transmitter and the receiver. In this example we assume the channel is ideal with additive
noise.
6 A New Family of Pulse Shaping Filters 137

1.2
n=1
1 n=2
0.8

0.6

0.4

0.2

−0.2

−0.4
−4 −3 −2 −1 0 1 2 3 4

(a) The main lobes of the impulse responses

0.01
n=1
n=2
0.005

−0.005

−0.01
4 5 6 7 8 9 10 11

(b) Extended scale view of the tails

Fig. 6.8 Impulse responses of generalized raised-cosine filters generated by P1 (x)

and P3 (x) are compared.
6 A New Family of Pulse Shaping Filters 138

0.8

0.6

0.4

0.2
n=1
0
n=2
n=3
n=4
−0.2 n=5
−0.4

−0.6

−0.8

−1
−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5

Fig. 6.9 Inner boundaries of the eye diagram for several generalized raised-cosine
filters with α = 1.

• To specify the generalized raised-cosine filter of Eq. (6.33), we define V (x) as follows:


 −1 x < −1
V (x) = P (x) −1 ≤ x ≤ 1 (6.40)


1 x>1

where P (x) is a polynomial defined as:

 x
(1 + p1 u2 )(1 + p2 u2 ) du
P (x) = 0 1 (6.41)
2 2
(1 + p1 u )(1 + p2 u ) du
0

To obtain a monotonic function V (x), both p1 and p2 should be greater than or equal to
−1. Note that for p1 = p2 = 0, P (x)=x which generates the phase compensated standard
raised-cosine filter. For p1 = p2 = −1, P (x)=P3 (x) as defined in Eq. (6.38).

• To calculate samples of the impulse response of the transmitting and receiving filters, we
evaluate Eq. (6.35) numerically. Each impulse response is calculated for N · M points where
M is the up-sampling ratio and N is the number of lobes in each truncated impulse response.
Numerical results are presented for M =20.
6 A New Family of Pulse Shaping Filters 139

• The overall impulse response is determined by convolving the impulse responses of the
truncated transmitting and receiving filters.

• To measure the ISI caused by truncation, we use the peak distortion criterion [41]:

1 
N
Dpeak = |h(nTs )| (6.42)
h(0) n=−N
n=0

where h(t) is the overall impulse response.

Table 6.2 shows the results for several values of normalized excess bandwidth α. In each case
the parameters p1 and p2 are found using an optimization procedure to minimize Dpeak . The
corresponding polynomial P (x) is given in the second column of the table. For each α two values
of Dpeak are presented. The “Minimized Dpeak ” value corresponds to the filter generated by P (x)
and the “Standard Dpeak ” value corresponds to the filter generated by P (x) = x which is a phase
compensated square-root of the standard raised-cosine filter. In all cases the optimized filters
produce less ISI due to truncation of the impulse responses.
To calculate probabilities of error, we assume a white Gaussian additive noise corrupts the
transmitted signal. An upper bound for probability of error for binary data can be found in terms
of Dpeak [41]:
+1 − D ,
P/e = Q
peak
(6.43)
σ
0∞ 2 /2
where σ is the standard deviation of the noise and Q(x)= √12π x e−t dt. For example, consider
the second row of Table 6.2. If the probability of error with zero ISI (Dpeak =0 for N =∞) is 10−6 ,
then with Dpeak =0.104, the worst case bound for the probability of error is P/e =1.02 × 10−5 . For
Dpeak =0.192, we obtain P/e =6.13 × 10−5 .
Table 6.2 also shows that for smaller values of α, truncating the filter impulse response causes
a larger distortion. Comparing different rows of the table, we notice that filters with smaller excess
bandwidth have larger ISI due to truncation even though they have larger truncation lengths.

6.6 Remarks

We have shown that any zero-phase bandlimited Nyquist filter with normalized excess bandwidth
can be split into two cascaded Nyquist matched filters. Each filter consists of the square-root of
the overall response along with a phase compensator. We have expressed the phase compensator
in terms of the square-root magnitude response of the overall filter.
The phase compensation can be applied to the square-root of a standard raised-cosine spec-
trum. The required phase response in this case is a simple piecewise linear function. For standard
6 A New Family of Pulse Shaping Filters 140

Table 6.2 Nyquist filter design to reduce ISI due to truncation.

Truncation effects
α P (x) N Minimized Standard
Dpeak Dpeak
0.10 0.4799x5 + 0.3827x3 + 0.1373x 14 0.125 0.216
0.15 0.3147x5 + 0.4259x3 + 0.2594x 10 0.104 0.192
0.25 0.2951x5 + 0.4269x3 + 0.2780x 6 0.089 0.194
0.50 −0.0105x5 − 0.1775x3 + 1.1879x 6 0.015 0.046
0.75 −0.0145x5 − 0.2122x3 + 1.2267x 4 0.011 0.052
1.00 0.0192x5 + 0.1844x3 + 0.7964x 4 0.004 0.010

raised-cosine filter with α=1, the square-root filter satisfies Nyquist’s first criterion, provided an
appropriate time delay is added to the impulse response.
Using the phase compensation technique, we have extended the family of raised-cosine filters
to a more general family of Nyquist filters. Compared to the standard raised-cosine spectrum, the
family of generalized raised-cosine filters provides more flexibility for designing Nyquist filters.
As an example, we designed a family of Nyquist filters with smoother spectra. The impulse
responses of these filters have higher asymptotic rates of decay. We also designed transmitting
and receiving filters such that when we truncate the impulse responses of these filters, the overall
impulse response has a reduced ISI.
The work reported in [66, 67] on generalized raised-cosine filters was followed by other research
groups for different applications such as the relationship between orthogonal wavelet functions
and Nyquist pulses [73], window design for Harmonic analysis [74], and data transmission appli-
cations [75].
 141

Chapter 7

Concluding Remarks

This thesis has presented design methods which allow the data transmission rate of a PCM
voiceband channel in a public switched telephone network to approach channel capacity. These
methods take into account the underlying structure of the PCM encoder/decoder to avoid or
reduce the distortion due to the signal conversion at the central office. PCM voiceband channels
can be categorized into three different types of digital communication channels where the prede-
termined structural constraints appear in the transmitter back-end (as in the down-stream PCM
channels), the receiver front-end (as in the up-stream PCM channels), or the tandem connection
of two channels (as in the end-to-end PCM channels).
Our particular interest is the up-stream PCM channel in which the communication system
designer has no control on the receiver front-end. Such a constraint creates many unconven-
tional questions in the theory and practice of the modem design. We tackle several problems in
modulation, channel equalization and pulse shaping filter design for this channel.
In the PCM voiceband channel, the appropriate choice of modulation scheme is a baseband
PAM modulation that matches the structure of the PCM encoder at the receiver. We have
described constellation design methods to satisfy the average signal power constraint and minimize
the probability of symbol error. For a reliable communication over the up-stream channel the
average number bits per information symbol should be less than 7.1 bits/symbol.
We have investigated the problem of index mapping and its potential contribution in the
probability of bit error. Our new design method of index mapping can reduce the bit error rate
without reducing the transmission data rate. As an example, the proposed hybrid bit-to-symbol
mapping algorithm can reduce the probability of bit error by a factor of 2-3 for a frame of 7
symbols.
We have considered a non-equally-probable signal constellation design to obtain a shaping gain
in the constellation design. We have also investigated the application of the Huffman algorithm
7 Concluding Remarks 142

to implement the bit-to-symbol mapping to obtain a shaping gain. The resulting system has a
variable rate due to the variable number of bits assigned to symbols. The overall effect of non-
equally probable constellation design is an increase in the transmission rate in the range of 1–2
kbits/sec.
A PCM up-stream channel has a limited bandwidth and a fixed sampling rate at the receiver.
The sampling rate is larger than twice the effective bandwidth of the channel. The combination
of these constraints leads to inter-symbol interference at the sampling instants at the receiver.
We have investigated design techniques to compensate for the channel at the transmitter. With
an average power constraint, a linear transmitter filter alone is not adequate to fully compensate
for the channel in all cases. We have proposed methods to add redundancy to the transmitted
data.
The redundancy is added so that the power spectrum density of the transmitted symbols at
the input of the transmitter filter closely follows a desired shaping function. We have also iden-
tified three different solutions for the desired shaping function in terms of the channel frequency
response. Spectral shaping is used to control the average signal power by appropriately distribut-
ing the signal power density at different frequencies. We have also proposed precoding techniques
to combine spectral shaping and filtering.
As an alternative method for adding redundancy to the transmitted signal, we have consid-
ered non-maximally decimated filterbanks. A pre-equalizer design based on the non-maximally
decimated filterbanks can be used to compensate for channels. The channel filter can be non-
minimum phase and its frequency response can contain special nulls. Adding redundancy using
pre-filtering or non-maximally decimated filterbank is an effective way to avoid or reduce ISI. For
a typical up-stream PCM channel, the required rate of redundancy is 1-2 symbols in a block of 8
transmitted symbols.
By using a baseband PAM modulation scheme, an optimum constellation design, an appro-
priate bit-to-symbol mapping, and the block-by-block pre-filtering the maximum achievable rate
over the up-stream PCM channel can be increased to 49 kbits/sec which is 50% higher than that
provided by recommendation V.90 in the up-stream direction. Many of the proposed methods
can be used as part of the Recommendation V.92 to improve the performance and increase the
maximum bit-rate of the up-stream PCM channel.
In Chapter 6, we introduced a phase compensation for the family of square-root raised-cosine
filters so that the resulting filters satisfy Nyquist’s first criterion. As a result, a raised-cosine
filter can be used in two different scenarios: in the up-stream PCM channel as an interpolating
filter, and in an end-to-end PCM channel as a pair of transmitter/receiver filters matched to one
another.
We also introduced a new family of Nyquist filters that includes the raised-cosine filters.
7 Concluding Remarks 143

Using this new family of filters, we can characterize filters with smoother spectra (compared to
the standard raised-cosine spectra) which result in a faster decay of the pulse shape. This family
of filters can provide more flexibility in the design of digital Nyquist filters as an approximation
to the truncated analog filters.

7.1 Contributions

• Three types of PCM voiceband channels are characterized. The structural constraints of
the up-stream channel are identified and a communication model for the up-stream channel
is established (Chapter 2 and [43])

• The optimal constellation design for the up-stream PCM channel is presented and the mod-
ulation performance under the average signal power constraint is analyzed. Two methods
of selecting a subset of PCM decision boundaries as detector thresholds are presented. The
criterion for the subset selection is to maximize the minimum distance between adjacent
thresholds while maintaining a constraint on the average signal power (Section 3.1).

• A new algorithm for index mapping over a frame of symbols is described. Its performance
of the new method is compared to the existing index mapping algorithm used in the V.90
Standard and in the proposed V.92 Standard (Section 3.2.5).

• A non-equally-probable constellation design is presented. The performance improvement

due to this probability assignment is analyzed. The maximum achievable data rate of a
PAM modulation with non-uniform spaced and non-equally probable constellation points is
computed (Section 3.3).

• A bit-to-symbol mapping method for the up-stream PCM channels based on the Huffman
algorithm is presented. The shaping gain obtained via this method is compared to the
maximum shaping achievable on the up-stream PCM channel (Section 3.3.4).

• A framework of the transmitter design for the up-stream PCM channel is presented. In
this work, we consider MMSE as the optimization criterion and describe three different
structures for the transmitter as a combination of filtering and spectral shaping (Section
4.1).

• The performance of the optimal filter design for an up-stream PCM channel is analyzed. It
is shown that a reliable data transmission over the up-stream PCM channel requires almost
ISI-free channel.
7 Concluding Remarks 144

• A new spectral shaping method based on redundant symbol insertion is proposed (Section
4.3).

• A new block-by-block pre-filtering structure for the up-stream PCM channel is presented
(Section 4.4 and [43]).

• A non-maximally filterbank structure for adding redundancies to transmitted symbols is

proposed. The non-maximally decimated filterbank structure provides a natural way of
adding redundancy to the symbols. The filterbanks can be composed of FIR filters even if
the channel filter contains spectral nulls. The complexity of such a filter is compared with
the spectral shaping and precoding techniques (Section 5.2 and [76]).

• A phase compensation technique for pulse shaping filter designs is investigated. The result-
ing pulse shaping filters act both as an interpolating filter in the transmitter back-end of an
up-stream PCM channel and as a pair of matched filters in the end-to-end PCM channels.
(Section 6.3 and [66, 67]).

• A family of generalized raised-cosine filters is presented. The new family of filters provides
more flexibility in the design of pulse shaping filters, particularly in the design of digital
filters as an approximation of the continuous pulse shaping filters (Section 6.4 and [66, 67]).

7.2 Future Work

We introduced an index mapping algorithm that can reduce the Hamming distances between
adjacent points. However, this algorithm does not necessarily provide the minimum Hamming
distance. Further studies regarding a generalized Gray encoding in general and a possible modifi-
cation to the proposed algorithm in particular should be considered. The link between the index
assignment problem in vector quantization and the modulation index mapping should be studied
further. The proposed solutions for either of these problems may provide insight for the other.
In Section 3.3.3, the maximum achievable rate for an up-stream PCM channels was presented.
The performance gap between the maximum achievable rate and the actual PAM performance is
significant. The use of appropriate channel coding techniques over the up-stream PCM channel
can reduce this gap. Furthermore, channel coding can be particularly useful to combat the residual
echo that is not cancelled by the echo canceller.
The non-equally-probable distribution of constellation points results in variable length code-
words for transmitted symbols. A variable-rate data transmission can be more sensitive to symbol
errors. The actual number of output bits can change even if one symbol error occurs. There are
7 Concluding Remarks 145

possible solutions to reduce this sensitivity. For example, the transmitted bits can be catego-
rized into two separate channels, one with a fixed-rate as a primary channel and another with a
variable-rate as secondary channel. The impact of variable rate bit-to-symbol mapping obtained
via the Huffman algorithm requires further investigation.
The modulation design and channel equalization work described here is mainly concerned with
the up-stream PCM channel. Many of proposed techniques can also be applied to the end-to-
end PCM channels. At the end points of an end-to-end PCM channel, only a partial knowledge
regarding the up-stream and the down-stream channel might be available. The sensitivity of
channel equalization techniques to a mismatch between the actual channel characteristics and the
estimated channel should be further investigated.
 146

Appendix A

Optimal Linear Transmitter

Structure

For a given channel filter Hc (f ) and receiving filter Hr (f ), we derive the structure of the linear
transmitter filter which for a given sequence of sampled outputs minimizes the average transmitted
power. The steps are similar to those described in [46] for the receiver design.
We assume that the sequence of the transmitted symbols {ak }, form a wide-sense stationary
random sequence. The power spectrum density of the input sequence is denoted as Φa (f ). We
show that among all transmitter filters, a filter of the following form minimizes the average
transmitted power,
 ∗
Ht (f ) = G(f ) Hc (f )Hr (f )
(A.1)
∗
= G(f )Hcr (f ).

The overall filtering effect of the transmitter, channel and receiver filters at the sampling
instants can be combined as an equivalent filter. The equivalent filter at the output of the
sampler can be expressed as:
∞
 k k
Xeq (f ) = Z(f − ) Hcr (f − ) (A.2)
Ts Ts
k=−∞

where Hcr (f ) = Hc (f )Hr (f ) and Z(f ) is an arbitrary transmitter filter.

Here we show that, for any transmitting filter Z(f ), there exists a transmitter filter in the form
given in Eq. (A.1) that can produce the same output as Z(f ) at the sampling instants. Replacing
A Optimal Linear Transmitter Structure 147

the transmitter filter from Eq. (4.1) into Eq. (A.2), we have:
∞

 k 2
Xeq (f ) = G(f ) |Hcr (f − )| (A.3)
Ts
k=−∞

Note that in the above equation, G(f ) is periodic with a period of 1/Ts , hence it can be factored
 (f ). If we choose G(f ) as:
out from the sum. Now we choose G(f ) so that Xeq (f ) = Xeq

∞
 k k
Z(f − )Hcr (f − )
Ts Ts
k=−∞
G(f ) = ∞ (A.4)
 k
|Hcr (f − )|2
Ts
k=−∞

the equivalent filters given by Eq. (A.2) and Eq. (A.3) will be the same.
Now we compare the average transmitted power for both filters and show that the average
transmitted power for the structure given in Eq. (A.1) is less than or equal to that of Z(f ). The
average transmitted power are computed as:
 1 ∞

Ts k 2
P1 = Φa (f ) |Z(f − )| df (A.5)
− T1 Ts
s k=−∞

and  1 ∞

Ts k 2
P2 = Φa (f )|G(f )|2 |Hcr (f − )| df (A.6)
− T1 Ts
s k=−∞

Replacing G(f ) from Eq. (A.4) and using the Schwarz inequality, we have
∞
 2
k k
Z(f − )Hcr (f − )
∞
 Ts Ts
k 2
|G(f )|2
k=−∞
|Hcr (f − )| = ∞
Ts  k 2
k=−∞ |Hcr (f − )| (A.7)
Ts
k=−∞
k 2
≤ |Z(f − )|
Ts

From Eq. (A.7), the average power for the system of Eq. (A.1) is less or equal to the average
power for general filter of Eq. (A.2). Since the channel output for both filters is the same, any
reasonable optimization criterion would choose the filter with a lower average transmitted power.
 148

Appendix B

Optimal MMSE Transmitting Filters

Here we derive the minimum mean square error (MMSE) solution for a discrete time transmitter
filter. Figure 4.2 shows a discrete time model of the filters. As we discussed in Chapter 4, the
contribution of the additive noise η[.] in MSE is independent of the transmitter filter design.
Therefore, we do not include the additive noise in our computation. In the absence of noise, the
MMSE criterion for minimizing the ISI is stated as:
 
MMSE = min E (r[k] − ak )2 (B.1)
ht

We assume input symbols ak form a stationary random sequence. The autocorrelation function
and power spectrum density of ak ’s are defined in Eq. (4.7) and Eq. (4.8) respectively. We derive
the MMSE solution of the transmitting filter, subject to an average transmit power constraint.

B.1 Transmitting filter with infinite length impulse response

We consider an infinite length impulse response ht [n] for the transmitting filter. Using the La-
grange multiplier method, we determine the optimal transmitting filter based on the following
objective function:
     
Fht = E (r[k] − ak )2 + λ E (x[k])2 − Pt (B.2)

Expanding the right-hand side of Eq. (B.2), we obtain:

 
Fht = φa [i − j]h[i] h[j] − 2 φa [i] h[i] + φa [0]
i j i
   (B.3)
+λ φa [i − j] ht [i] ht [j] − Pt
i j
B Optimal MMSE Transmitting Filters 149

where the overall transmitter, channel and receiver filter impulse response is:

h[n] = ht [i]hcr [n − i]
i

To compute the MMSE, we take the derivative of the objective function with respect to the
transmitting filter coefficients ht [k]. For all values of k, we have1 :
∞
 ∞

∂ ∂  
(Fht ) = φa [i − j] h[i] h[j]
∂ht [k] ∂ht [k]
i=−∞ j=−∞
∞
∂  
−2 φa [i] h[i] (B.4)
∂ht [k]
i=−∞
∞ ∞
 ∂  
+λ φa [i − j] ht [i] ht [j]
∂ht [k]
i=−∞ j=−∞

To minimize the objective function, the derivatives with respect to the coefficients ht [k] should
be zero. The resulting equations for all values of k are:

2φa [k] ⊗ ht [k] ⊗ hcr [k] ⊗ hcr [−k] − 2φa [k] ⊗ hcr [−k] + 2λφa [k] ⊗ ht [k] = 0 (B.5)

where “⊗” denotes the discrete convolution. Equation B.5 can be simplified to:

ht [k] ⊗ hcr [k] ⊗ hcr [−k] + λ = hcr [−k] (B.6)

In the frequency domain, Eq. (B.6) corresponds to the following expression for the transmitting
filter transfer function:
Hcr (e−jωTs )
Ht (ejωTs ) = (B.7)
|Hcr (ejωTs )|2 + λ
where λ is determined from the average power constraint. Replacing Ht (ejωTs ) in the power
constrain equation Eq. (4.11), we obtain:

1 π
|Hcr (ejωTs )|2
Φa (ejωTs )dω = Pt (B.8)
2π −π (|Hcr (ejωTs )|2 + λ)2

The MSE can be written in terms of the transfer function of the filters and power spectrum density
1
Note that in our discussion h[n] is an impulse response with real coefficients. Optimization of real function
with respect to complex coefficients is discussed in [77]
B Optimal MMSE Transmitting Filters 150

of the input data:

 
MSE = E (r[k] − ak )2
 π
1
= Φa (ejωTs )H(ejωTs )H(e−jωTs ) dω (B.9)
2π −π
  π
1 π 1
− Φa (ejωTs )H(ejωTs ) dω + Φa (ejωTs ) dω
π −π 2π −π

where H(ejωTs ) = Ht (ejωTs )Hcr (ejωTs ). Substituting the optimal transmitting filter of Eq. (B.7)
into this expression, we compute the minimum mean square error (MMSE) as:

λ2 π
Φa (ejωTs )
MMSE = dω (B.10)
2π −π (|Hcr (ejωTs )|2 + λ)2

B.2 Finite Impulse Response

We consider a finite impulse response of length 2K + 1 for the transmitting filter. The filter
coefficients are denoted as ck .

ck for |k| ≤ K
ht [k] = (B.11)
0 elsewhere

We also assume the channel impulse response hcr [n] has finite length with L non-zero coefficients:

hcr [n] = 0 for n<0 & n≥L (B.12)

The objective function given in Eq. (B.2) is given by:

 K+L−1
K+L−1  
K+L−1
FC = φa [i − j]h[i] h[j] − 2 φa [i] h[i]
i=−K j=−K i=−K
(B.13)

K 
K

+ φa [0] + λ φa [i − j] ci cj − Pt
i=−K j=−K

where the overall impulse response h[n] can be computed as:


K
h[n] = ci hcr [n − i] for −K ≤ n ≤L+K −1 (B.14)
i=−K
B Optimal MMSE Transmitting Filters 151

The derivative of Eq.( B.2) with respect to ck for |k| ≤ K can be written as:

∂ 
K  K+L−1
K+L−1 
(FC ) = 2 φa [i − j]hcr [j − k] hcr [i − l]
∂ck
l=−K i=−K j=−K
(B.15)

K 
K
−2 φa [l]hcr (l − k) + 2λ φa [k − l]cl
l=K l=−K

These equations can be simplified to:


K
Γkl cj = ξk for |k|, |l| ≤ K (B.16)
l=−K

where
 K+L−1
K+L−1 
Γkl = φa [i − j]hcr [j − k] hcr [i − l] + λφa [k − l] (B.17)
i=−K j=−K

and

K
ξk = hcr [m − k]φa [m] for |k| ≤ K (B.18)
m=−K

Equation (B.16) has the following matrix form

Γ Copt = ξ (B.19)
 152

Appendix C

Inner boundaries of the eye pattern

for the square-root filter

To find the lower boundaries of the eye pattern diagram of the square-root full raised-cosine filter,
we first determine the maximum ISI for a PAM signal. From Eq. (6.1), we obtain:

∞

x(τ ) = a0 g(τ ) + ak g(τ − n) (C.1)
n=−∞
n=0

where τ is the time offset relative to the sampling instances nTs . The second term in this equation
is due to ISI. In the case of binary PAM signal with ak = ±1, the maximum ISI for a Nyquist
pulse is calculated as:
∞

D(τ ) = |g(τ − n)| (C.2)
n=−∞
n=0

therefore, the eye-pattern boundaries are:

+ ,
Lower Boundaries: = ± g(τ ) − D(τ )
+ , (C.3)
Upper Boundaries: = ± g(τ ) ± D(τ )

Substituting Eq. (6.23) into Eq. (C.2), we obtain

∞  
− cos(2πτ ) cos(2πτ )   1 

g(τ ) − D(τ ) = −  (τ + n + 1/4)(1/4 − τ − n)  (C.4)
4π(τ + 1/4)(τ − 1/4) 4π n=−∞
n=0
C Inner boundaries of the eye pattern for the square-root filter 153

For |τ | ≤ 1/4, the above expression simplifies to:

∞
cos(2πτ )  1 1
g(τ ) − D(τ ) = − for |τ | ≤ (C.5)
4π n=−∞
(τ + n + 1/4)(τ + n − 1/4) 4

Consider the following identity

∞
 1 −4π
= . (C.6)
(x + k + 1/4)(x + k − 1/4) cos(2πx)
k=−∞

Therefore, the lower boundaries of the eye pattern are

+ , 1
± g(τ ) − D(τ ) = ±1 for |τ | ≤ (C.7)
4
 154

Appendix D

The impulse response of Hsqrt(f )

To obtain the impulse response given in Eq. (6.35), we note that Hsqrt (f ), as defined in Eq. (6.33),
√
has Hermitian symmetry. The inverse Fourier transform of Ts Hsqrt (f ) can be written as:

 1−α  1+α
+ , + ,
2Ts 2Ts
F −1 { Ts Hsqrt (f )} =2Ts cos(2πf t) df + 2Ts cos φ(f ) · cos φ(f ) + 2πf t df
1−α
0 2Ts
 1−α
2Ts
=2Ts cos(2πf t) df
0
 1+α  1+α + ,
2Ts 2Ts
+ Ts cos(2πf t) df + Ts cos 2φ(f ) + 2πf t df
1−α 1−α
2Ts 2Ts

(D.1)

where φ(f ) is defined in Eq. (6.32). The first two terms of Eq. (D.1) can be combined as follows:
 1−α  1+α
2Ts 2Ts t πtα
2Ts cos(2πf t) df + Ts cos(2πf t) df = sinc( ) · cos( ) (D.2)
0 1−α Ts Ts
2Ts

2Ts 1
By introducing a new variable x = α (f − 2Ts ), we simplify the third integral in Eq. (D.1) to:

 1+α + , 
2Ts α 1 π πtα πt 
Ts cos 2φ(f ) + 2πf t df = sin V (x) − x+ dx
1−α 2 −1 2 Ts Ts
2Ts (D.3)
 1
πt π πtα 
= α sin( ) cos V (x) − x dx
Ts 0 2 Ts

Combining Eq. (D.2) and Eq. (D.3), we obtain the result as given in Eq. (6.35).
 155

References

[1] K. Pahlavan and J. L. Holsinger, “Voiceband data communication modems – a historical

review 1919–1988,” IEEE Commun. Mag., vol. 26, pp. 16–27, Jan. 1988.

[2] ITU-T Recommendation V.34, “A Modem Operating at Data Signalling Rates of Up to

28,800 bit/s for Use on the General Switched Telephone Network and on Leased Point-to-
Point 2-Wire Telephone-Type Circuits,” ITU-T V Series Recommendations, 1994.

[3] G. D. Forney, L. Brown, M. V. Eyuboglu, and J. L. Moran III, “The V.34 high-speed modem
standard,” IEEE Commun. Mag., vol. 34, pp. 28–33, Dec. 1996.

[4] I. Kalet, J. E. Mazo, and B. R. Saltzberg, “The capacity of PCM voiceband channels,” Proc.
IEEE Int. Conf. Commun., pp. 507–511, 1993.

[5] E. Ayanoglu, N. R. Dagdeviren, G. D. Golden, and J. E. Mazo, “An equalizer design technique
for the PCM modem: A new modem for the digital public switched network,” IEEE Trans.
Commun., vol. 46, pp. 763–774, June 1998.

[6] H. Herzberg and B. R. Saltzberg, “Coding for a channel with quantization in the presence
of an estimable interference,” IEEE Trans. Commun., vol. 45, pp. 45–51, Jan. 1997.

[7] E. Ayanoglu, N. R. Dagdeviren, J. E. Mazo, and B. R. Saltzberg, “A high-speed modem

synchronized to a remote codec,” United States Patent, no. 5 394 437, Feb. 1995.

[8] E. Ayanoglu, G. D. Golden, R. K. Jones, and J. E. M. D. G. Shaw, “High speed quantization-

level-sampling modem with equalization arrangement,” United States Patent, no. 5 528 625,
June 1996.

[9] N. R. Dagdeviren, “Modem with received signals and transmitted signals comprising signal
sets,” United States Patent, no. 540 6 583, Apr. 1995.

[10] H. Herzberg and B. R. Saltzberg, “Communication arrangement with improved echo and
noise suppression in a channel containing quantization,” United States Patent, no. 571 0 790,
Jan. 1998.

[11] B. Townshend, “High speed communications system for analog subscriber connections,”
United States Patent, no. 580 1 695, Sept. 1998.
References 156

[12] P. A. Humblet and M. G. Troulis, “The information driveway,” IEEE Communications Mag-
azine, vol. 34, pp. 64–68, Dec. 1996.

[13] ITU-T Recommendation V.90, “A digital modem and analogue modem pair for use on the
public switched telephone network (PSTN) at data signalling rates of up to 56 000 bit/s
downstream and up to 33 600 bits upstream,” ITU-T V Series Recommendations, 1998.

[14] D. Kim and V. Eyuboglu, “Convolutional spectral shaping,” IEEE Commun. Lett., vol. 3,
pp. 9–11, Jan. 1999.

[15] B. Barazesh, “PCM upstream considerations for the US Interim Standard,” Telecommunica-
tion Industry Association (TIA), Technical Committee TR-30 Meetings, January 1997.

[16] N. Dagdeviren, “Estimated Coverage with PCM Upstream,” Telecommunication Industry

Association (TIA), Technical Committee TR-30 Meetings, January 1997.

[17] V. Eyuboglu, “Upstream transmission in PCM modems,” Telecommunication Industry As-

sociation (TIA), PCM Modem Ad-Hoc Group Contribution, January 1997.

[18] N. Warke and M. Ali, “On the next generation of high-speed voiceband modems,” Proc.
IEEE Int. Conf. Commun., pp. 256–260, 1999.

[19] W. D. Reeve, Subscriber Loop Signaling and Transmission Handbook, Analog. IEEE Press,
New York, NY, 1992.

[20] W. D. Reeve, Subscriber Loop Signaling and Transmission Handbook, Digital. IEEE Press,
New York, NY, 1995.

[21] T. Starr, J. M. Cioffi, and P. Silverman, Understanding Digital Subscriber Line Technology.
Prentice-Hall, Upper Saddle River, NJ, 1999.

[22] Members of Technical Staff, Transmission Systems for Communications. Bell Telephone
Laboratories, New York, NY, Fifth ed., 1982.

[23] S. Ramo, J. R. Whinnery, and T. Van Duzer, Fields and Waves in Communication Electron-
ics. Wiley, New York, NY, Second ed., 1984.

[24] ITU-T Recommendation V.56bis, “Network transmission model for evaluating modem per-
formance over 2-wire voice grade connect,” ITU-T V Series Recommendations, August 1995.

[25] Recommendation G.712, “Transmission performance characteristics of Pulse Code Modula-

tion channels,” ITU-T G series Recommendations, November 1996.

[26] A. V. Oppenheim and R. W. Schafer, Discrete-Time Signal Processing. Prentice-Hall, En-

glewood Cliffs, NJ, 2nd ed., 1998.

[27] Recommendation G.711, “Pulse code modulation (PCM) of voice frequencies,” ITU-T G
Series Recommendations, 1988.
References 157

[28] J. E. Mazo, “Quantization noise and data transmission,” The Bell System Technical Journal,
vol. 47, pp. 1737–1753, Oct. 1968.

[29] C. E. Shannon, “A mathematical theory of communication,” Bell System Technical Journal,

vol. 61, pp. 379–423 and 623–656, July and Oct. 1948.

[30] R. Lucky, J. Salz, and E. Weldon, Principles of Data Communication. McGrow-Hill, Book
Company, New York, NY, 1968.

[31] N. Warke and M. Ali, “Optimum CODEC companding for high-speed PCM data transmission
in telephoe networks.,” IEEE Int. Conf. on Acoustics, Speech, Signal Processing, pp. 2679–
2682, March 1999.

[32] J. G. Proakis, Digital Communications. McGraw-Hill, New York, NY, third ed., 1995.

[33] N. Farvardin, “A study of vector quantization for noisy channels,” IEEE Trans. Inform.
Theory, vol. 36, pp. 799–809, July 1990.

[34] K. Zeger and A. Gersho, “Pseudo-Gray coding,” IEEE Trans. Commun., vol. 38, pp. 2147–
2158, Dec. 1990.

[35] P. Knagenhjelm and E. Agrell, “The Hadamard transform – a tool for index assignment,”
IEEE Trans. Inform. Theory, vol. 42, pp. 1139–1151, July 1996.

[36] P. Knagenhjelm, “How good is your index assignment?,” IEEE Int. Conf. on Acoustics,
Speech, Signal Processing, pp. 423–426, vol. II, Apr. 1993.

[37] T. M. Cover and J. A. Thomas, Elements of Information Theory. Wiley, New York, NY,
1991.

[38] E. A. Lee and D. G. Messerschmitt, Digital Communication. Kluwer Academic Publishers,

Boston, MA, 1994.

[39] F. R. Kschischang and S. Pasupathy, “Optimal nonuniform signaling for gaussian channels,”
IEEE Trans. Inform. Theory, vol. 39, pp. 913–929, May 1993.

[40] G. Ungerboeck, “Channel coding with multilevel/phase signals,” IEEE Trans. Inform. The-
ory, vol. 28, pp. 55–67, Jan. 1982.

[41] R. D. Gitlin, J. F. Hayes, and S. B. Weinstein, Data Communications Principles. Plenum

Press, New York, NY, 1992.

[42] R. E. Blahut, Digital Transmission of Information. Addison-Wesley, Reading, MA, 1990.

[43] N. S. Alagha and P. Kabal, “Linear time varying precoder applied to ISI channel,” IEEE
Pacific Rim Conf. Commun., Computers and Sig. Processing, pp. 36–39, Aug. 1997.

[44] D. W. Tufts, “Nyquist’s problem–The joint optimization of transmitter signal and receiver
in pulse amplitude modulation,” Proc. IEEE, vol. 53, pp. 248–259, Mar. 1965.
References 158

[45] J. W. Smith, “The joint optimization of transmitted signal and receiving filter for data
transmission systems,” Bell System Technical Journal, vol. 44, pp. 2363–2392, Dec. 1965.
[46] T. Ericson, “Structure of optimum receiving filters in data transmission systems,” IEEE
Trans. Inform. Theory, vol. 17, pp. 352–353, May 1971.
[47] T. Ericson, “Optimum PAM filters are always band limited,” IEEE Trans. Inform. Theory,
vol. 19, pp. 570–573, July 1973.

[48] J. G. Proakis and D. G. Manolakis, Digital Signal Processing. Prentice-Hall, Englewood

Cliffs, NJ, Third ed., 1996.

[49] W. Gardner, Introduction to Random Processes. McGraw-Hill, New York, NY, Second ed.,
1989.

[50] M. Tomlinson, “New automatic equaliser employing modulo arithmetic,” Electron. Lett.,
vol. 7, pp. 138–139, Mar. 1971.
[51] H. Harashima and H. Miaykawa, “Matched–transmission technique for channels with inter-
symbol interference,” IEEE Trans. Commun., vol. 20, pp. 774–780, Aug. 1972.
[52] B. R. Saltzberg, “Intersymbol interference error bounds with application to ideal bandlimited
signaling,” IEEE Trans. Inform. Theory, vol. 14, pp. 563–568, July 1968.
[53] K. Yao and R. M. Tobin, “Moment space upper and lower bounds for digital systems with
intersymbol interference,” IEEE Trans. Inform. Theory, vol. 22, pp. 65–74, Jan. 1976.
[54] F. E. Glave, “An upper bound on the probability of error due to intersymbol interference for
correlated digital signals,” IEEE Trans. Inform. Theory, vol. 18, pp. 356–263, May 1972.
[55] A. Calderbank, T. Lee, and J. E. Mazo, “Baseband trellis codes with spectral null at zero,”
IEEE J. Selected Areas Communications, vol. 34, pp. 425–434, May 1988.
[56] P. Kabal and S. Pasupathy, “Partial–response signaling,” IEEE Trans. Commun., vol. 23,
pp. 921–934, Sept. 1975.
[57] F. A. Graybill, Matrices with Applications in Statistics. Wadsworth International Group,
Belmont, California, Second ed., 1983.
[58] G. H. Golub and C. F. Van Loan, Matrix Computations. John Hopkins University Press,
Baltimore, 1996.
[59] H. Harashima and H. Miyakawa, “Matched–transmission technique for channels with inter-
symbol interference,” IEEE Trans. Commun., vol. 20, pp. 774–780, Aug. 1972.
[60] J. E. Mazo and J. Salz, “On the transmitted power in generalized partial response,” IEEE
Trans. Commun., vol. 24, pp. 348–352, Mar. 1976.
[61] J. M. Wozencraft and I. M. Jacobs, Principles of Communication Engineering. Wiley, New
York, NY, 1965.
References 159

[62] J. L. Yen, “On the non–uniform sampling of bandwidth–limited signals,” IRE transactions
on Circuit Theory, vol. CT-3, pp. 251–257, Dec. 1956.

[63] P. Vaidyanathan, Multirate Systems and Filter banks. Prentice-Hall, Englewood Cliffs, NJ,
1993.

[64] T. Li and Z. Ding, “Joint transmitter-receiver optimization for partial response channels
based on nonmaximally decimated filterbank precoding technique,” IEEE Trans. Commun.,
vol. 47, pp. 2407–2414, Sept. 1999.

[65] A. Wang and J. Smith, “On fast FIR filters implemented as tail-canceling IIR filters,” IEEE
Trans. Signal Processing, vol. 45, pp. 1415–1427, June 1997.

[66] N. S. Alagha and P. Kabal, “Generalized raised-cosine filters,” IEEE Trans. Commun.,
vol. 45, pp. 989–997, July 1999.

[67] N. S. Alagha and P. Kabal, “A family of Nyquist filters based on generalized raised-cosine
spectra,” Proc. 19th Biennial Symp. Commun., pp. 131–135, July 1998.

[68] X.-G. Xia, “A family of pulse-shaping filters with ISI-free matched and unmatched filter
properties,” IEEE Trans. Commun., vol. 45, pp. 1157–1158, Oct. 1997.

[69] R. A. Gibby and J. W. Smith, “Some extensions of Nyquist’s telegraph transmission theory,”
vol. 44, pp. 1487–1510, Sept. 1965.

[70] J. A. C. Bingham, The Theory and Practice of Modem Design. Wiley, New York, NY, 1988.

[71] P. Kabal and E. Dubois, “Interpolating and Nyquist filters with constraints at certain fre-
quencies,” Signal Processing, vol. 24, pp. 117–126, Aug. 1991.

[72] R. P. Ramachandran and P. Kabal, “Minimax design of factorable Nyquist filters for data
transmission systems,” Signal Processing, vol. 18, pp. 327–339, Nov. 1989.

[73] C. J. Zarowski, “Orthogonal wavelet packet functions and Nyquist pulses,” Proc. 19th Bien-
nial Symp. Commun., pp. 136–139, June 1998.

[74] José A. de la O, “On the use of amplitude shaping pulses as windows for harmonic analysis,”
IMTC, pp. 1388–1393, 2000.

[75] C. Tan and N. Beaulieu, “An investigation of transmission properties of Xia Pulses,” Proc.
IEEE Int. Conf. Commun., 1999.

[76] N. S. Alagha and P. Kabal, “A filterbank structure for voice-band PCM channel pre-
equalization,” IEEE Int. Conf. on Acoustics, Speech, Signal Processing, pp. 2793–2796, June
2000.

[77] S. Haykin, Adaptive Filter Theory. Prentice-Hall, Englewood Cliffs, NJ, 3 ed., 1996.