Dr.

Manju, Associate Professor, CSE , IIUC

CHAPTER FIVE

REGRESSION ANALYSIS

Dr. Mohammad Manjur Alam (Manju)

Associate Professor
Department of Computer Science and Engineering
International Islamic University Chittagong.
Email: manjuralam44@yahoo.com

Regression is the functional relationship between two variables and of the two variables one may
represent cause and the other may represent effect. The variable representing cause is known as
independent variable and is denoted by X. The variable X is also known as predictor variable or
repressor. The variable representing effect is known as dependent variable and is denoted by Y.
Y is also known as predicted variable. The term “regression” was used by a famous Biometrician
Sir. F. Galton (1822-1911) in 1877.

Example: The productions of paddy of amount y is dependent on rainfall of amount x. Here x is

independent variable and y is dependent variable.

Assumptions

1. The x’s are non-random or fixed constants

2. At each fixed value of X the corresponding values of Y have a normal distribution about a
mean.

3. For any given x, the variance of Y is same.

4. The values of y observed at different levels of x are completely independent.

Objectives of Regression Analysis:

i. To estimate the relationship that exits, on the average, between the dependent variable and
independent variables.

ii. To determine the effect of each independent variable on the dependent variable, controlling
the effects of the others independent variables.
 Dr. Manju, Associate Professor, CSE , IIUC

iii. To predict the value of the dependent variable for a given value of the explanatory variables

Simple Linear Regression Model

The simplest form of the regression model that displays the relation between X and Y is a
straight line, which appears as follows: 𝐘=a + bx

here denotes the predicted value of Y, a is the intercept and b is the slope of the straight line.
In regression terminology, b is the regression coefficient of Y on X. This straight line is called
the fitted line of Y.

The least-Squares Method:

The least-squares method is a technique for minimizing the sum of the squares of the differences
between the observed values and estimated values of the dependent variable. That is the least-
squares line is the line that minimizes Σ𝐞𝐢𝟐 =Σ (𝐘𝐢−𝐛𝐗𝐢−𝐚)𝟐

Here Σei2 is called sum of squares of errors (SSE).

 Dr. Manju, Associate Professor, CSE , IIUC

To minimizes SSE with respect to a, and b, from calculus we know that the partial derivatives of


2
SSE with respect to a, and b must be 0. Then ei =−2Σ (Yi−bXi−a) =0
a



2
ei =−2Σ (Yi−bXi−a) Xi=0
b

hich concludes ΣYi=n a +b Σ Xi

and ΣXiYi = aΣXi + bΣXi2

 (x
i 1
i  x )( y i  y )
b yx  n

 x
i 1
i  x
2

and 𝐚=𝐘−𝐛𝐗

Regression coefficient

The mathematical measures of regression are called the coefficient of regression.

Let, (x1,y1), (x2,y2)……….. (xn,yn) be the pairs of n observations. Then the regression coefficient
of y on x is denoted by byx and defined by
n

 (x
i 1
i  x )( y i  y )
b yx  n

 x
i 1
i  x
2

Again, the regression coefficient of x on y is denoted by bxy and defined by

n

 (x
i 1
i  x )( y i  y )
b xy  n

 y
i 1
i  y
2

Regression lines:

If we consider two variables X and Y, we shall have two regression lines as the regression line of
Y on X and the regression line of X on Y. The regression line of Y on X gives the most probable
 Dr. Manju, Associate Professor, CSE , IIUC

values of Y for given values of X and The regression line of X on Y gives the most probable
values of X for given values of Y. Thus we have two regression lines. However, when there is
either perfect positive or perfect negative correlation between the two variables, the two
regression lines will coincide i.e, we will have one line.

Regression equation:

The regression equation of y on x is expressed as follows:

y = a + bx, where y is the dependent variable to be estimated and x is the independent variable, a
is the intercept term (assume mean) and b is the slope of the line.
n

 y b x  ( x  x )( y
i i  y)
Here, a  y  bx  and b  i 1
n

 x  x 
n n 2
i
i 1

 n  n 
  xi   y i 
 xi y i   i 1 n i 1 
 2
 n 
  xi 
 i 1 
n


i 1
xi 
2

Similarly, the regression equation of x on y is expressed as follows:

x = a + by, where x is the dependent variable to be estimated and y is the independent variable, a
is the intercept term (assume mean) and b is the slope of the line.

a  x  by 
x b  y
n n

 n  n 
  xi   yi 
 xi yi   i1 n i1 
n

 ( x  x )( y
i i  y)
And b i 1
n
 2
 n 
  yi  y 2   yi 
yi   i1 
i 1 n


i 1
2

n
 Dr. Manju, Associate Professor, CSE , IIUC

Properties of regression coefficient.

1. Regression coefficient is independent of change of origin but not of scale.

2. Regression coefficient lies between -  to +  . i.e, -<  byx <  .

3. Regression coefficient is not symmetric. i.e, bxy  byx

4. The geometric mean of regression coefficients is equal to correlation coefficient

i.e, rxy  byx  bxy

5. The arithmetic mean of two regression coefficient is greater than correlation

b b 
Coefficient. i.e,  yx xy   rxy
 2 

6. If one of regression coefficient is greater than unity the other must be less than

unity. i.e, bxy  1 and b yx <1

7. Regression coefficient is not pure number.

Coefficient of Determination R2 :

The coefficient of determination, r 2, is useful because it gives the proportion of the variance
(fluctuation) of one variable that is predictable from the other variable. It is a measure that allows
us to determine how certain one can be in making predictions from a certain model/graph. The
coefficient of determination is the ratio of the explained variation to the total variation.
2
The coefficient of determination is such that 0 < r < 1, and denotes the strength of the
linear association between x and y.
The coefficient of determination represents the percent of the data that is the closest to the line
2
of best fit. For example, if r = 0.922, then r = 0.850, which means that 85% of the total
variation in y can be explained by the linear relationship between x and y (as described by the
regression equation). The other 15% of the total variation in y remains unexplained.
The coefficient of determination is a measure of how well the regression line represents the
data. If the regression line passes exactly through every point on the scatter plot, it would be
able to explain all of the variation. The further the line is away from the points, the less it is able
to explain.
 Dr. Manju, Associate Professor, CSE , IIUC

Theorem: Show that correlation coefficient is the geometric mean of regression coefficients. i.e,
rxy= byx  bxy

Proof: Let, Let, (x1,y1), (x2,y2)……….. (xn,yn) be the pairs of n observations. Then the correlation
coefficient between x and y is denoted by rxy and defined as,
n

 ( x  x )( y  y ) i i
rxy  i 1
……………..(1)
n n

 x  x    y  y 
i 1
i
2

i 1
i
2

 ( x  x )( y  y )
i i
Again, the regression coefficient of y on x is, byx  i 1
n

 x  x 
i 1
i
2

 (x
i 1
i  x )( y i  y )
b xy  n

 y
i 1
i  y
2

Again, the regression coefficient of x on y is,

n n

 (x
i 1
i  x )( y i  y )  x
i 1
i  x  y i  y 
b yx  b xy  n
 n

 x i  x 
i 1
2
 y
i 1
i  y
2

 ( x  x )( y  y )
i i
byx  bxy  i 1
= rxy (proved)
n n

 x  x    y  y 
i 1
i
2

i 1
i
2

Theorem: The arithmetic mean of two regression coefficient is greater than correlation
b b 
coefficient. i.e,  yx xy   rxy
 2 

Proof: Let, (x1,y1), (x2,y2)……….. (xn,yn) be the pairs of n observations. Then the regression
coefficient of y on x is denoted by byx and the regression coefficient of x on y is denoted by bxy.

b b 
The arithmetic mean of byx and bxy is  yx xy  and the geometric mean is byx  bxy
 2 
 Dr. Manju, Associate Professor, CSE , IIUC

We know, Correlation coefficient is the geometric mean of regression coefficients.

i.e, rxy = byx  bxy

Since, A.M  G.M

b b 
or,  yx xy   byx  bxy
 2 

b b 
or,  yx xy   r (proved
 2 

Uses of regression:

(i) Whether a relationship exists or not.

(ii) To find the strength of relationship.

(iii) Determination of mathematical equation.

(iv) Prediction the values of the dependent variables.

Distinguish between correlation coefficient and regression coefficient.

Correlation coefficient Regression coefficient.

1. The numerical value by which we measure the 1. The mathematical measures of regression
strength of linear relationship between two or are called the coefficient of regression.
more variables is called correlation coefficient.

2. Correlation coefficient is independent of 2. Regression coefficient is independent of

change of origin and scale of measurement. change of origin but not of scale.

3. Correlation coefficient lies between -1 to +1. 3. Regression coefficient lies between -  to

i.e, -1< rxy < 1. +  . i.e, -  < byx <  .

4. Correlation coefficient is symmetric. i.e, rxy= 4. Regression coefficient is not symmetric.

ryx i.e, bxy  byx
5. It is always unit free. 5. Regression coefficient is not pure number.
6. When r=0 then the variables are correlated. 6. When r=0 then two lines of regression are
perpendicular to each other.

Application problem-1: A researcher wants to find out if there is any relationship between the
ages of husbands and the ages of wives. In other words, do old husbands have old wives and
 Dr. Manju, Associate Professor, CSE , IIUC

young husbands have young wives? He took a random sample of 7 couples whose respective
ages are given below:

Age of Husband(in years):x 39 25 29 35 32 27 37

Age of wife(in years):y 37 18 20 25 25 20 30

i. Compute the regression line of y on x.

ii. Predict the age of wife whose husband’s age in 45 years.
iii. Find the regression line of x on y and estimate the age of husband if the age of his wife is
28 years.
iv. Compute the value of correlation coefficient with the help of regression coefficients.

Solution: The equation of the best –fitted regression line of y on x is ŷ = a + bx

 n  n 
  xi   yi 
 xi yi   i1 n i1 
Where, b = 2
and a = y - b x
 n 
  xi 
xi   i 1 
n


i 1
2

Computation table

x y x2 y2 xy
39 37 1521 1369 1443
25 18 625 324 450
29 20 841 400 580
35 25 1225 625 875
32 25 1024 625 800
27 20 729 400 540
37 30 1369 900 1110
 x  224  y  175  x 2 7334  y 2 4643  xy 5798
 n  n 
  xi   yi 
 xi yi   i1 n i1  5798 
224 175 
Here, b = = 7 = 1.193
 n

2
224 
2

  xi  7334 
7
xi   i 1 
n


i 1
2

n
 Dr. Manju, Associate Professor, CSE , IIUC

And a = y - b x =
y -b
x
n n

=
175
-(1.193)
224  = 25-38.176 = -13.176
7 7

Hence the fitted regression line is ŷ = a + bx = -13.176 + 1.193x

Hence, if the age of husband is 45, the probable age of wife would be

ŷ = -13.176 + 1.193x = -13.176 + 1.193  45 = 40.51 years.

The equation of the best –fitted regression line of y on x is x̂ = a + by

 n  n 
  xi   yi 
 xi yi   i1 n i1 
Where, b = 2
 n 
  yi 
yi   i1 
n


i 1
2

5798 
224 175 
= 7 = 0.739
4643 
175 
2

And a = x -by =
x b  y
n n

224 175
=  0.739 = 13.525
7 7

Hence the fitted regression line is x̂ = a + by = 13.525+ 0.739y

Hence, if the age of wife is 28 years, the estimate age of husband is

x̂ = a + by

= 13.525+ (0.739)(28) = 34.22 years.

Application problem-2: A research physician recorded the pulse rates and the temperatures of
water submerging the faces of ten small children in cold water to control the abnormally rapid
 Dr. Manju, Associate Professor, CSE , IIUC

heartbeats. The results are presented in the following table. Calculate the correlation coefficient
and regression coefficients between temperature of water and reduction in pulse rate.

Temperature of water 68 65 70 62 60 55 58 65 69 63
Reduction in pulse rate. 2 5 1 10 9 13 10 3 4 6
b b 
Also show that (i)  yx xy   rxy
 2 

Solution: Calculating table of correlation coefficient and regression coefficients.

xi yi xi2 yi2 xiyi

68 2 4624 4 136
65 5 4225 25 325
70 1 4900 1 70
62 10 3844 100 620
60 9 3600 81 540
55 13 3025 169 715
58 10 3364 100 580
65 3 4225 9 195
69 4 4761 16 276
63 6 3969 36 378
x i 
635 y i 
63 x i
2

40537 y i
2

541  x y =3835
i i

 n  n 
  xi   yi 
 xi yi   i1 n i1 
We know, rxy =
  n  
2
 n  
2

n   xi   n   yi  
     i1  
 xi   yi 
2 i 1 2

 i1 n  i1 n 
  
  

635  63
3835 
= 10 = -0.94
 635  541  63
2
  2

40537   
 10  10 
 Dr. Manju, Associate Professor, CSE , IIUC

 ( x  x )( y  y )
i 1
i i
We know, the regression coefficient of y on x is, byx = n

 x  x 
i 1
i
2

 n  n 
  xi   yi 
 xi yi   i1 n i1  3835 
635  63
 1655
 = 10 =  -0.77
 n

2
635 
2
2145
  xi  40537 
10
xi   i1 
n


i 1
2

 ( x  x )( y  y )
i 1
i i
Regression coefficient of x on y is, bxy = n

 y  y
i 1
i
2

 n  n 
  xi   yi 
 xi yi   i1 n i1  3835 
635  63
 1655
= = 10 =  -1.1
 n 
2
63
2
1441
  yi  541 
10
yi   i1 
n


i 1
2

b b 
(i)  yx xy   rxy
 2 

 b  b   0.77    1.1
Here,  yx xy   = -0.94 = rxy
 2  2

Assignment Problem-1: The following data give the test scores and sales made by nine
salesmen during the last year of a big departmental store:

Test Scores: y 14 19 24 21 26 22 15 20 19
Sales(in lakh Taka) 31 36 48 37 50 45 33 41 39
i. Find the regression equation of test scores on sales. Ans: ŷ = -2.4 + 0.56x
ii. Find the test scores when the sale is Tk. 40 lakh. Ans: 20 lakh
iii. Find the regression equation of sales on test scores. Ans: x̂ = 7.8 + 1.61y
iv. Predict the value of sale if the test score is 30. Ans: 56.1 lakh
v. Compute the value of correlation coefficient with the help of regression coefficients.
 Dr. Manju, Associate Professor, CSE , IIUC

Assignment Problem-2: The following table gives the ages and blood pressure of 10 women:

Age in years 56 42 36 47 49 42 72 63 55 60
x
Blood pressure 147 125 118 128 125 140 155 160 149 150
y
Obtain the regression line of y on x. Ans: ŷ = 83.76+ 1.11x

Estimate the blood pressure of a women whose age is 50 years. Ans: 139.26

Obtain the regression line of x on y.

Find correlation coefficient between x and y and comment.

Assignment Problem-3: Consider the following data set on two variables x and y:

x:1 2 3 4 5 6

y:6 4 3 5 4 2

Find the equation of the regression line y on x. Ans: ŷ = 5.799-0.541x

Graph the line on a scatter diagram.

Estimate the value of y when x = 4.5 Ans: ŷ = 3.486

Predict the value of y when x = 8. Ans: ŷ =1.687

Assignment Problem-4: Cost accountants often estimate overhead based on production. At the
standard knitting company, they have collected information on overhead expenses and units
produced at different plants and what to estimate a regression equation to predict future
overhead.

Units 56 40 48 30 41 42 55 35

Overhead 282 173 233 116 191 171 274 152

(i)Draw a scatter diagram and comment

(ii)Fit a regression equation.

(iii)Estimate overhead when 65 units are produced.

Assignment Problem-5: The following data refer to information about annual sales

( Tk.’000) and year of experience of a super store of 8 salesmen:

 Dr. Manju, Associate Professor, CSE , IIUC

Salesmen 1 2 3 4 5 6 7 8

Annual sales (Tk.’000) 90 75 78 86 95 110 130 145

Year of experience 7 4 5 6 11 12 13 17

(i)Fit two regression lines.

(ii)Estimate sales for year of experience is 10

(iii)Estimate year of experience for sales 100000