Linear System Theory

Dr. Vali Uddin

Hamdard University

Vali.uddin@hamdard.edu.pk

Lecture 4 1
Norms
 Want to add more structures to a linear space
 Norms: Generalization of the idea of length
– Key points?
– From here, can define distance, x2 x
convergence, derivative, etc.
x1
 Norm. ||x||: (X, R)  R (or (X, C)  R) such
that
– ||x||  0 and ||x|| = 0 iff x = 0
– ||x|| = || ||x|| for all   R (or C)
– ||x1 + x2||  ||x1|| + ||x2|| ~ Triangular inequality
Lecture 4 2
 Give some norms for (Rn, R)
– ||x||1  |xi|
 [|xi|2]1/2
– ||x||2
 [|xi|p]1/p
– ||x||p
 maxi |xi|
– ||x||
• Do they satisfy the conditions in the definition?
• Find the various norms for x = (2, -3, 1)T
 |xi| = 2 + 3 + 1 = 6
– ||x||1
 [|xi|2]1/2 = (4 + 9 + 1)1/2 = 141/2 = 3.74
– ||x||2
 [|xi|3]1/3 = (8 + 27 + 1)1/3 = 361/3 = 3.302
– ||x||3
 maxi |xi| = 3
– ||x||
Lecture 4 3
 Consider the set of real-valued, piece-wise
continuous functions over [a, b]
 What are their norms?
– ||x||1, ||x||2, ||x||p, ||x||
b 1
x 1   x ( t ) dt b 2  2
x 2    x ( t ) dt 
a a 
1
b p  p
x p    x ( t ) dt  x   max x ( t )
a  a tb

Lecture 4 4
Example. x(t) = e-2t over [0, 1]
1
2 t  1 2t 1
x 1  e dt  e  0.432
0 2 0
1 1
 
1
1   2  1
 1  2 1 4  2  0.495
x 2    e dt    e
4 t 4 t
   1 e 
0   4 0 4 
x   max x( t )  1
a t b

 A normed vector space is a vector space (X,

F) with a norm ||x|| defined on X
 It has a sense of length or distance

Lecture 4 5
 Inner Product
 More structure: A sense of orientation ~ X 
X F
– Suppose x = (1 0)T, y = (4 3)T, and z = (0 2)T in R2
z y

x
– What is the inner product of x and y, i.e., <x, y>? What
is <x, z>?
• <x, y> = xyT = x1y1 + x2y2 = |x| |y| cos 
– <x, y> = (1 0) (4 3)T = 4 ~ Projection of y onto x
– <x, z> = (1 0) (0 2)T = 0, as x and z are "orthogonal"
• The above is for R2. How to generalize it?
Lecture 4 6
Example. x = (1 - j, 2)T, y = (3, 1 + 2j)T
 <x, y> = ? <x, x> = ?
*
n

x y  x y   x i yi for Cn , C  *: Complex conjugate transpose
i 1
n  3 
x y   x i yi  1  j 2     5 7j
i 1 1  2 j
n 1  j
x x   xixi  1  j 2   6 ~ Always real
i 1  2 
Q. Why defined in this way?
– Want <x, x>  R and <x, x> > 0 when x  0
Q. General definition?
Lecture 4 7
Inner Product. <x, y>: X  X F such
that
 x, y   y, x  Bar ~ Complex conjugate
1x1   2 x 2 , y  1 x1, y   2 x 2 , y

x, x  0 for all x  0

• Note that from the first two conditions, we have

x, 1y1   2 y 2  1y1   2 y 2 , x

 1 y1, x  2 y 2 , x  1 x, y1  2 x, y 2
Also, x  y, x  y  x, x   x, y   y, x   y, y

x Ay  x*Ay
*
 
* *
A x y  A x y  x*Ay  x Ay A*x y  x Ay
Lecture 4 8
• It can be easily shown that the previous inner
product definitions for (Rn, R) and (Cn, C) satisfy the
above 3 conditions
Example. Real-valued, piece-wise continuous
function over [a, b]
b b
*
x y   x ( t ) y( t )dt   x( t ) y( t )dt
a a
 A pre-Hilbert space is a vector space (X, F) with
a inner product <x, y> defined on X  X
 What properties does inner product have?
The Cauchy-Schwarz Inequality
12 12
x, y  x, x  y, y
Lecture 4 9
 12 12
x, y  x, x  y, y

Proof. If y = 0, the above is obviously satisfied. Now

assume y  0. Then for an arbitrary scalar  C,
0  x  y, x  y  x, x   x, y   y, x   y, y

x y x y y x
Set   then   
y y y y y y
2 2 2
x y y x x y
and 0  x, x   
y y y y y y
2
or 0  x, x y y  x y • What is the relationship
between norm and inner
12 12
or x, y  x, x  y, y product?
Lecture 4 10
Theorem: ||x||  <x, x>1/2 is a norm
Proof. Recall that norm should satisfy
– ||x||  0 and ||x|| = 0 iff x = 0
– ||x|| = || ||x|| for all   R (or C)
– ||x1 + x2||  ||x1|| + ||x2|| ~ Triangular inequality
Now 12 12
x x  0, and x x  0 iff x  0
   1 2  x x
12 12 12
x x   x x
x1  x 2 x1  x 2  x1 x1  x1 x 2  x 2 x1  x 2 x2

 x1 x1  x 2 x 2  2 x1 x 2 x1 x 2
12 12
 x1 x1  x 2 x 2  2 x1 x1 x2 x2


 x1 x1
12
 x2 x2 
12 2  ||x1 + x2||  ||x1|| + ||x2||
 ||x||  <x, x>1/2 is a norm
Lecture 4 11
Example. Consider (R2[t], R) ~ Polynomials with
real coefficients of degrees less than 2, 0  t  1
– How to define the inner product? Norm?
12
1 1 2 
x y   x ( t ) y( t )dt x    x ( t )dt 
0 0 
– Let x(t) = t + 3, y(t) = 2t - 1. Compute <x, y>, ||x||,
and ||y||

 
1 1
2  2 3 5 2 
x y   2 t  5t  3 dt   t  t  3t  
1
0 3 2 0 6

Lecture 4 12
 12
 
12
1  1 3 1
x    t  6t  9 dt 
2
   t  3t 2  9 t  
0   3  
 0 
12
   3  9 
1
 3.512
3 
12
 
12
1   4 3 
1
y    4 t  4 t  1 dt 
2
   t  2t  t  
2
0   3  
 0 
12

   2  1
4
 0.577
3 

1 12 12
x y   x x  y y  2.028
6

Lecture 4 13
Orthogonality
 The concept of perpendicularity
 In a pre-Hilbert space, x and y are orthogonal (written as
x  y) iff <x, y> = 0
– {x1, x2, .., xn} is an orthogonal set iff xi  xj  i  j
– x is orthogonal to a set S  X if x  s  s  S
– Inner product extends the  concept of R2 and R3 to
general pre-Hilbert spaces
– If several nonzero vectors are orthogonal to each
other, then are they linearly independent?
– An orthogonal set of nonzero vectors is a linearly
independent set. How to show this?
Lecture 4 14
 Suppose that {x1, x2, .., xm} is an orthogonal set of
nonzero vectors and iixi = 0, then
<xk, iixi> = <xk, 0> = 0
= ii <xk, xi> = k <xk, xk>
 k = 0  k
 {x1, x2, .., xm} are linearly independent
– If m = n, then {x1, x2, .., xn} is good candidate for a
basis in view of its orthogonality
 {x1, x2, .., xm} is an orthonormal set iff xi  xj  i  j
and ||xi|| = 1  i
x2

– An even better candidate for a basis x1

x3
Lecture 4 15
– Basis, Representation, and Orthonormalization
 Relationship among a set of vectors: Linear
dependence and linear independence
 Dimension of a linear space

 The base of a linear space: Basis

 Representations of a vector in term of a basis

 Relationship among representations for different

bases
 Generalization of the idea of length: Norms

 A sense of orientation: Inner Product

 The concept of perpendicularity: Orthogonality

• Gram-Schmidt Process to obtain orthonormal vectors

 Projection and Orthogonal Projection Theorem

– Linear Operators and Representations

Lecture 4 16
Gram-Schmidt Process
 The problem: Given a set of LI vectors which are
not orthogonal, derive an orthonormal set
– How?
– Example: (R2, R) T
x2 e2 = (1, 2)
v2 T
e1 = (2, 1)
x1
v1 = e1
– The component of e2 that is  to v1:
v1 e 2 1 2  2  2    0.6 
v 2  e2  v1       
v1 v1  2 4  1  1   1.2 
Lecture 4 17
 {v1 , v2} is an orthogonal set:
<v1 , v2> = -1.2 + 1.2 = 0
– What is next?
– Normalization. How?
 2 
v1  
e1   5
v1  1 

 5 x2
 1 _ _
v2  5  e2 e1
e2   
v2  2  x1
 
 5

Lecture 4 18
 General Procedure:
– Let {e1, e2, .., en} be a linearly independent set
– Form an orthogonal set by subtracting from
each vector the components that align with
v1 e 2
previous
v1  e1 vectorsv 2  e2  v1
v1 v1
v1 e3 v2 e3
v 3  e3  v1  v2
v1 v1 v2 v2
n 1
v k en
v n  en   vk
k 1 v k v k

– Normalize the new set

vi
ei  i
vi

Lecture 4 19
 Orthogonal Projection Theorem
 Extension of Pythagorean theorem to linear
spaces
Lemma. If x  y, then
2 2
xy  x  y
2 y

~ Pythagorean theorem x
• How to prove it?
2
xy  xy xy

 x x  x y  y x  y y

2 2
 x  y
Lecture 4 20
Projection
– What is a projection? What properties does
it have?
– Consider projecting x  X onto a subspace M
x
m2
m0
M
m1

– m0 is the point in M that is closest to x

– (x - m0)  M
Lecture 4 21
Orthogonal Projection Theorem
Let (X, F) be a pre-Hilbert space, M be a
subspace, and x be an arbitrary element in
X. Then x
m2
m0  arg  
 min x  m  m0
mM  M
iff (x - m0)  M m1
m0: Projection of x on M
m0 is unique
– If we want to find the best m0 to minimize the
distance, find the projection which is characterized
by (x - m0)  M
– How to prove it?
Lecture 4 22
 If m0 is the minimizing vector, show that (x - m0)  M
 Prove by contradiction
– Suppose that there  m  M s.t. <x - m0, m> =   0
– Without loss of generality, let ||m|| = 1
– m1  m0 +m, where is the complex conjugate of 
– Then want to show that ||x - m1||2 < ||x - m0||2
2
x  m1  x  m0  m, x  m0  m
 x  m0 , x  m0  x  m0 , m  m, x  m0  m, m
2 2 2 2 2
 x  m0    m x
2 2 2
 x  m0    x  m0 , contradict ion
m0 m1
Lecture 4 23
 If (x - m0)  M, show that m0 is the
minimizing vector
2 2
x  m  x  m0  m0  m , m  m0

 x  m0 , x  m0  m0  m, m0  m
 x  m0 , m0  m  m0  m, x  m0

 x  m0
2 = 0 in view that (x - m0)  M

 We can also see the uniqueness:

2 2
x  m  x  m0  m  m0

Lecture 4 24
 Example t 2 f(t) = a + bt

1
 
2
Error :  t  a  bt  dt
2

t 1
-1 1

• Best approximate t2 by a linear function f(t) = a + bt for t  [-1,

1] to minimize the square of error
– How to look at this problem?
– Find the projection of t2 onto (R2[t], R)
– What is (X, F)? What is M? What is <x, y>?
 X: R3[t] (the set of polynomials with n < 3),
-1  t  1
 F: R
1
 M: R2[t], -1  t  1
x y   x( t )  y( t )dt
1
Lecture 4 25
• The problem: Find m0 such that (x - m0)  M
1
 ( x  m0 )  mdt  0  m  M
1
1
  
 t  bt  a  c1t  c0  dt  0  c0 , c1  R
2
1
1
 
  c1t 3  c0  bc1 t 2  ac1  bc 0 t  ac 0 dt
1
1
c  bc1 3 ac1  bc 0 2
  1 t 4  0 t  ac 0 t 
c
t 
4 3 2  1

 c0  bc1   2  2b
  2  ac 0  2    2a c0  c1 = 0
 3  3  3

Lecture 4 26
 1 1
a ,b0 Or , m 0 
3 3

t2

-1 1 t

– There are definitely better ways to solve this

problem. The approach just presented,
however, illustrated several key concepts on
linear spaces
Lecture 4 27
 Linear Operators and Representations
(X, F) (Y, F)
• Functions L
– What is a "function"?
x
– Which one below is a function? y
y y

x x
– A function f is a mapping from domain X to codomain Y
that assigns each x  X one and only one element of Y
– Range: {y  Y|  x  X, s.t. f(x) = y}  Y
– What is a "linear function"?
Lecture 4 28
– A function L that maps from (X, F) to (Y, F)
is said to be a linear operator (linear
function, linear mapping, or linear
transformation) iff
L(1x1 + 2x2) = 1L(x1) + 2L(x2)
 1, 2  F, and  x1, x2  X
– Which of the following is a linear function?
y f0

x
f1
– The is relatively simple. Consider a more
complicated case of a linear time-invariant system:
Lecture 4 29
 u y
g(t)

– Assume that the initial conditions are zero, and u() is a

real-valued, piece-wise continuous function
• What is the mapping from u() to y()?
• Is it a linear mapping?
t
y( t )   g( t  )u( )d, t  0, T
0
– L: (U, R)  (Y, R), where U and Y are the set of real-
valued, piece-wise continuous functions
– L satisfies the linearity property
• Representation of a mapping in terms of bases in X/Y?
Lecture 4 30
 Introduction to System Theory and Linear
Algebra
 A set of LI vectors {e1, e2, .., en} of (X, F) is said to be a
basis of X if every vector in X can be expressed as a
unique linear combination of them   1
 
x  e1 e 2 ... e n   2 
: 
  ~  
 n
e1 e2 ... en    e1 e2 ... en   
 p11   p1i 
   
e1  e1 e2 ... en   p12 
ei  e1 e2 ... en   p1i 
 :   : 
   
 p1n   p1i 
~ p1 ~ pi
Lecture 4 31
 e1 e2 ... en   e1 e2 ... en  P
x  e1 e2 ... e n    e1 e2 ... en  P

 e1 e2 ... en      P

ith column of P: Representation of ei

w.r.t. the set of new basis
– Conversely, ith column of Q: Representation ofei
  Q w.r.t. the set of existing basis
P  Q 1
Inner Product. <x, y>: X  X F such
that
 x, y   y, x  Bar ~ Complex conjugate
1x1   2 x 2 , y  1 x1, y   2 x 2 , y
x, x  0 for all x  0
Lecture 4 32
 Matrix Representation of Linear
Operators
 How can we represent a linear operator?
– Every L with finite dimensional X and Y has a
matrix representation with coefficients in F
Theorem. Suppose that (X, F) (Y, F)
L

– Dim X = n, Dim Y = m x
y
– {x1, x2, .., xn} is a basis of X
– {w1, w2, .., wm} is a basis of Y
Then L: (X, F)  (Y, F) is uniquely determined
by n pairs of mapping
yi  Lxi, i = 1, 2, .., n
Lecture 4 33
 What is the representation?
– With yi  Lxi, i = 1, 2, .., n, let
 ai be the representation of yi w.r.t. {w1, w2, .., wm}
 A be the matrix formed as [a1, a2, .., an]

– Then for any x  X represented by  w.r.t {x1,

x2, .., xn}, the representation  of y = Lx w.r.t.
{w1, w2, .., wm} is given by  = A
Proof. (i) Uniqueness
 n  n n
Lx  L  i x i    i Lx i    i yi
 i 1  i 1 i 1

– L is uniquely determined by yi
Lecture 4 34
– To show that  = A, let a1i 
a 2i
Lx i  yi  w1 w 2 .. w m   
 : 
– Then a  ai
 mi 
y  Lx  Lx1 x 2 .. x n  

 Lx1 x 2 .. x n  

 y1 y2 .. y n  

 w1 w 2 .. w m a1 a 2 .. a n  

 w1 w 2 .. w m  A  A
=
– ith column of A: Representation of yi  Axi
w.r.t. {w1, w2, .., wm}
Lecture 4 35
 Example. Rotating counter-clock-wise in
R2 by  e2 1   0 1  0
Le e1   , e 2    w1   , w 2   
2 Le1 0 1   0 1

e1 How to proceed?
x  1e1  2e2 Lx  L1e1  2e2   1Le1  2 Le 2  1y1  2 y2

– What is y1? What is y2? a1 cos   sin 

 cos   A 
y1  cos  w1  sin  w 2  w1 w 2    sin  cos  
 sin     sin  
y 2   sin  w1  cos  w 2  w1 w 2  
 cos   a
– If  = (1, 1)T, and  = 90, then 2
e2
0  1 1   1 y x
  A       
1 0  1  1 
e1
Lecture 4 36
Change of Basis
– L: x  y ~ The mapping is independent of bases
–  = A ~ The ith column of A is the representation of
Lxi (= yi) w.r.t. {w1, w2, .., wm}
 The representation A depends on the bases for X
and Y
– Consider a special case where L: (X, F)  (X, F)
– With {e1, e2, .., en} as a basis, L:    ( = A, and
A is n  n)
– Supposed the basis is changed to {e1,e2, ..,en}.
We now have =A
– How to get?
– How are A andA related?
Lecture 4 37
• Recall that = P and = P, where the ith
column of P is the representation of ei w.r.t.
{e1,e2, ..,en}

  P  PA  A   AP

 AP  PA, or A  PAP 1 or A  P 1AP  QAQ 1

– This is called the similar transformation, and A
andA are similar matrices
– Of course, the ith column ofA is the
representation of Lei w.r.t. {ei}i from 1 to n

Lecture 4 38
  0 1
Example. Consider x  Ax , A   
  2 3
– What are the natural modes?
– What is the dynamics for a new representation with
 2  1
P   x  Ax , with A  PAP 1
 1 1 

1  2  1  0 1 1 1 1 0
A  PAP        
  1 1   2 3 1 2   0 2 
1 0 
x    x  Natural modes?
0 2 
 x1   c1e t   c11e t  c12e2 t 
1
x    2t  xP x t 2t 
 2   c2e  c21e  c22e 
Lecture 4 39
Norm of a Linear Operator
(X, F), || ||x (Y, F), || ||y
A

x y = Ax

• Want to define the norm for the linear operator A

– How?
– ||A|| should be based on ||||x and ||||y , and how A
"magnifies" x during the transformation
Ax y
A  sup , or , A  sup Ax y
x 0 x x x x 1

Lecture 4 40
  3 2 4
Example. A   4 0 6
 
1 3 2
– With ||x||1 and ||y||1, what is ||A||1?
– Recall that ||x||1 = i|xi| and A  sup Ax y
x x 1
– Try x = e1, e2, and e3
 3  2  4
Ae1  4, Ae 2  0, Ae 3  6
     
1  3 2
y1  8, y2  5, y3  12,  ||A||1 = 12
m 
• In general, A 1  max   a ij 
j  i 1  ~ Add up column-wise

Lecture 4 41
• Now with ||x|| and ||y||, what is ||A||?
– Recall that ||x|| = maxi|xi|
 3 2 4
A   4 0 6 , A  sup Ax y
  x x 1
1 3 2

 3 2 4 1  9  ||x|| = 1, ||y|| = 10

4 0 6 1  10,
    
1 3 2 1  6   ||A|| = 10
 n 
• In general, 
A   max   a ij 
i  j1  ~ Add up row-wise
• With ||x||2 and ||y||2, what is ||A||2?

Lecture 4 42
  
12
n 2 2
x 2    xi  , Ax  Ax Ax  x*A*Ax  x* A*A x
 i 1 
– It can be shown that ||A||2 is the largest eigenvalue of
(A*A)
– A is said to be bounded iff ||A|| < 
• The concept of ||A|| is needed in Chap. 5

Lecture 4 43
THE END

Lecture 4 44