‫ﺷﺒﮑﻪ ﻫﺎي ﻋﺼﺒﯽ‬

‫)‪(Neural Networks‬‬

‫‪Signal and Weight Vector Spaces‬‬

‫داﻧﺸﮕﺎه آزاد اﺳﻼﻣﯽ واﺣﺪ ﻋﻠﻮم و ﺗﺤﻘﯿﻘﺎت‬

 Notation

Vectors in ℜn. Generalized Vectors.

x1
x
x = 2

xn

2
 Linear Vector Space
1. An operation called vector addition is defined such that if
x ∈ X and y ∈ X then x+y ∈ X.

2. x + y = y + x

3. (x + y) + z = x + (y + z)

4. There is a unique vector 0 ∈X, called the zero vector, such

that x + 0 = x for all x ∈ X.

5. For each vector there is a unique vector in X, to be called

(-x ), such that x + (-x ) = 0 .

3
 Vector Space (Cont.)
6. An operation, called multiplication, is defined such that
for all scalars a ∈ F, and all vectors x ∈ X, a x ∈ X.

7. For any x ∈ X , 1x = x (for scalar 1).

8. For any two scalars a ∈ F and b ∈ F, and any x ∈ X,

a (bx) = (a b) x .

9. (a + b) x = a x + b x .

10. a (x + y) = a x + a y

4
 Examples (Decision Boundaries)
To illustrate these conditions, let’s investigate a few sample sets
and determine whether or not they are vector spaces. First
consider the standard two-dimensional Euclidean space, ,
shown in figure. This is clearly a vector space, and all ten
conditions are satisfied for the standard definitions of vector
addition and scalar multiplication.

5
 Examples (Decision Boundaries)
What about subsets of ? What subsets of are also vector
spaces (subspaces)? Consider the boxed area (X) in the figure.
Does it satisfy all ten conditions? No. Clearly even condition 1 is
not satisfied. The vectors and shown in the figure are in X,
but is not. From this example it is clear that no bounded
sets can be vector spaces.

6
 Examples (Decision Boundaries)
Are there any subsets of that are vector spaces?

Assume that the line extends to infinity in both directions. Well,

any line that passes through the origin will work. If it does not
pass through the origin then condition 4, for instance, would not
be satisfied.

7
 Linear Independence

If

8
 Example (Banana and Apple)
–1 1
p1 = 1 p2 = 1
–1 –1

Let
a 1 p 1 + a2 p2 = 0

– a1 + a2 0
a1 + a2 = 0
– a1 + (– a 2 ) 0

This can only be true if

a1 = a2 = 0

9 Therefore the vectors are independent.

 Example

10
 Spanning a Space
A subset spans a space if every vector in the space can be
written as a linear combination of the vectors in the subspace.

x = x1u1 + x2u2 + ... + xmum

11
 Basis Vectors
 A set of basis vectors for the space X is a set of vectors
which spans X and is linearly independent.

 The dimension of a vector space, Dim(X), is equal to the

number of vectors in the basis set.

 Let X be a finite dimensional vector space, then every

basis set of X has the same number of elements.

12
 Example
Polynomials of degree 2 or less.

Basis A:
2
u1 = 1 u2 = t u3 = t

Basis B:
2
u1 = 1– t u2 = 1+t u3 = 1+t+t

Any three linearly independent vectors in the space will work.

13
 Inner Product / Norm
A scalar function of vectors x and y can be defined as an
inner product, (x,y), provided the following are satisfied (for
real inner products):

• (x ,y) = (y,x)
• (x , ay1+by2) = a(x ,y1) + b(x ,y2)
0 , where equality holds iff x = 0
• (x ,x) =

14
 Inner Product / Norm
A scalar function of a vector x is called a norm, ||x||, provided
the following are satisfied:

• ||x|| ≥
0.
• ||x|| = 0 iff x = 0 .
• ||a x|| = |a| ||x|| for scalar a .
• ||x + y|| ≤ ||x|| + ||y|| .

15
 Example
Standard Euclidean Inner Product

x T y = x1 y1 + x 2 y2 + … + x n yn

Standard Euclidean Norm

||x|| = (x , x)1/2

||x|| = (xTx)1/2 = (x12 + x22 + ... + xn2) 1/2

Angle
cos(θ) = (x ,y)/(||x|| ||y||)
16
 Orthogonality
Two vectors x,y ∈X are orthogonal if (x,y) = 0 .

For example, any vector in the p2, p3 plane is orthogonal to the

weight vector.

17
 Gram-Schmidt Orthogonalization
Independent Vectors Orthogonal Vectors
y1 , y2 , … , yn v 1 , v 2, … , v n

Step 1: Set first orthogonal vector to first independent vector.

v1 = y1
Step 2: Subtract the portion of y2 that is in the direction of v1.
v2 = y 2 – av 1

Where a is chosen so that v2 is orthogonal to v1:

(v 1 ,v 2) = (v 1, y2 – a v1 ) = (v 1, y2) – a (v 1 ,v 1) = 0

(v , y )
a = -------1--------2----
18
(v 1,v 1 )
 Gram-Schmidt (Cont.)
Projection of y2 on v1:

(v 1, y2)
------------------ v 1
(v1 ,v 1)

Step k: Subtract the portion of yk that is in the direction of all

previous vi

k–1
vk = y k – ∑ -(--v------i,--y----k--)v i
i = 1 (v i,v i)

19
 Example
y1 = 1 y2 = – 1
1 2

Step 1. v1 = y1 = 1
1

20
 Example (Cont.)
Step 2.

–1
T 11
v 1 y2 2
v2 = y 2 – ------------ v 1 = – 1 – ------------------------ 1 = – 1 – 0.5 = – 1.5
T
v 1 v1 2 1 1 2 0.5 1.5
11
1

21
 Column of Numbers
The vector expansion provides a meaning for writing a vector as a
column of numbers.

x1
x
x = 2

xn

To interpret x, we need to know what basis was used for the

expansion.
22
 Vector Expansion
If a vector space X has a basis set {v1, v2, ..., vn}, then any x∈X
has a unique vector expansion:

n
x = ∑ x i v i = x1 v 1 + x2 v 2 + … + xn v n
i=1

If the basis vectors are orthogonal, and we take the inner product
of vj and x:

23
 Vector Expansion
n n
(v j,x ) = (v j, ∑ x iv i) = ∑ x i(v j,v i) = x j (v j,v j)
i=1 i=1

Therefore the coefficients of the expansion can be computed:

(v j,x )
x j = ----------
--------
(v j,v j)

24
 Reciprocal Basis Vectors
Definition of reciprocal basis vectors, ri :

(r i,v j) = 0 i≠ j
= 1 i = j

where the basis vectors are {v1, v2, ..., vn}, and the reciprocal
basis vectors are {r1, r2, ..., rn}.

For vectors in ℜn we can use the following inner product:

T
(r i,v j) = r i v j
25
 Reciprocal Basis Vectors
Therefore, the equations for the reciprocal basis vectors become:

RT B = I T
R = B
–1

R = r 1 r2 … r n

B = v1 v2 … vn

26
 Vector Expansion
x = x1 v 1 + x 2 v2 + … + x n vn

Take the inner product of the first reciprocal basis vector with the
vector to be expanded:

(r 1 ,x ) = x 1 (r 1 ,v 1) + x 2 (r 1, v2 ) + … + x n (r 1 ,v n)

By definition of the reciprocal basis vectors:

(r 1,v 2) = (r 1,v 3) = … = (r 1,v n) = 0

(r 1,v 1) = 1

27
 Vector Expansion
Therefore, the first coefficient in the expansion is:

x1 = (r 1 ,x )

In general, we then have (even for nonorthogonal basis vectors):

x j = (r j,x )

28
 Example
Basis Vectors:

vs1 = 1 v2 = 2
s

1 0

Vector to Expand:

xs = – 1
2

29
 Example (Cont.)
Reciprocal Basis Vectors:
–1
T
R = 12 = 0 1 r1 = 0 r2 = 0.5
10 0.5 – 0.5 1 – 0.5

Expansion Coefficients:
v
x1 = r 1 x =
T s –1 = 2
01
2

v T s –1
x 2 = r2 x = 0.5 –0.5 = – 1.5
2

Matrix Form:
v T s –1 s
x =R x = B x = 0 1 –1 = 2
0.5 –0.5 2 – 1.5
30
 Example (Cont.)
x = ( –1 ) s 1 + 2 s 2 = 2 v1 - 1.5 v2

x = –1
s v 2
x =
2 – 1.5

The interpretation of the column of numbers

31 depends on the basis set used for the expansion.