Overview
❑ Unifying theme: problem-solving and analysis
using interesting algorithms and data
structures.
Review: algorithms, analysis, sorting (2 weeks)
Dynamic programming and greedy algorithms (2 weeks)
Advanced analysis: amortization (1 week)
Graph and digraph problems (4 weeks)
linear programming (2 weeks)
String algorithms (1 week)
NP-Completeness (1 week)
© 2020 Shermer Introduction 5
Analysis of Algorithms
Input Algorithm Output
© 2010 Goodrich, Tamassia Analysis of Algorithms 6
Pseudocode Ideas
(not prescriptive)
❑ Control flow ❑ Method call
◼ if … then … [else …] var.method (arg [, arg…])
◼ while … do … ❑ Return value
◼ repeat … until … return expression
◼ for … do … ❑ Expressions
◼ Indentation can replace Assignment
braces (like = in C++)
= Equality testing
❑ Method declaration (like == in C++)
Algorithm method (arg [, arg…]) n2 Superscripts and other
Input … mathematical
Output … formatting allowed
© 2010 Goodrich, Tamassia Analysis of Algorithms 12
Seven Important Functions
❑ Seven functions that
often appear in algorithm 1E+30
1E+28 Cubic
analysis: 1E+26
◼ Constant 1 1E+24 Quadratic
Logarithmic log n 1E+22
◼ Linear
1E+20
◼ Linear n 1E+18
N-Log-N n log n
T (n )
◼ 1E+16
1E+14
◼ Quadratic n2
1E+12
◼ Cubic n3 1E+10
◼ Exponential 2n 1E+8
1E+6
1E+4
❑ In a log-log chart, the 1E+2
slope of the line 1E+0
corresponds to the 1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
growth rate n
© 2010 Goodrich, Tamassia Analysis of Algorithms 13
Functions Graphed
Using “Normal” Scale
g(n) = n lg n
g(n) = 1 g(n) = 2n
g(n) = n2
g(n) = lg n
g(n) = n
g(n) = n3
© 2010 Stallmann Analysis of Algorithms 14
The Random Access Machine
(RAM) Model
❑ A CPU
❑ An potentially unbounded
bank of memory cells, 2
1
each of which can hold an 0
arbitrary number or
character
Memory cells are numbered and accessing
any cell in memory takes unit time.
© 2010 Goodrich, Tamassia Analysis of Algorithms 15
Primitive Operations
❑ Basic computations
❑ Examples:
performed by an algorithm
◼ Evaluating an
❑ Identifiable in pseudocode expression
❑ Largely independent from the ◼ Assigning a value
to a variable
programming language
◼ Indexing into an
❑ Exact definition not important array
(we will see why later) ◼ Calling a method
Returning from a
❑ Assumed to take a constant ◼
method
amount of time in the RAM
model
© 2010 Goodrich, Tamassia Analysis of Algorithms 16
Counting Primitive Operations
❑ By inspecting the pseudocode, we can determine the
maximum number of primitive operations executed by
an algorithm, as a function of the input size
Algorithm arrayMax(A, n) # operations
currentMax A[0] 2
for i 1 to n − 1 do 2n
if A[i] currentMax then 2(n − 1)
currentMax A[i] 2(n − 1)
{ increment counter i } 2(n − 1)
return currentMax 1
Total 8n − 3
© 2010 Goodrich, Tamassia Analysis of Algorithms 17
Estimating Running Time
❑ Algorithm arrayMax executes 8n − 3 primitive
operations in the worst case. Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
❑ Let T(n) be worst-case time of arrayMax. Then
a (8n − 3) T(n) b(8n − 3)
❑ Hence, the running time T(n) is bounded by two
linear functions
© 2010 Goodrich, Tamassia Analysis of Algorithms 18
Growth Rate of Running Time
❑ Changing the hardware/ software
environment
◼ Affects T(n) by a constant factor, but
◼ Does not alter the growth rate of T(n)
❑ The linear growth rate of the running
time T(n) is an intrinsic property of
algorithm arrayMax
© 2010 Goodrich, Tamassia Analysis of Algorithms 19
Comparison of Two Algorithms
insertion sort is
n2 / 4
merge sort is
2 n lg n
sort a million items?
insertion sort takes
roughly 70 hours
while
merge sort takes
roughly 40 seconds
This is a slow machine, but if
100 x as fast then it’s 40 minutes
versus less than 0.5 seconds
© 2010 Stallmann Analysis of Algorithms 20
Constant Factors
1E+26
❑ The growth rate is 1E+24 Quadratic
Quadratic
not affected by 1E+22
1E+20 Linear
◼ constant factors or 1E+18 Linear
◼ lower-order terms
1E+16
1E+14
❑ Examples T (n )
1E+12
1E+10
◼ 102n + 105 is a linear
1E+8
function 1E+6
◼ 105n2 + 108n is a 1E+4
quadratic function 1E+2
1E+0
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
n
© 2010 Goodrich, Tamassia Analysis of Algorithms 21
Big-Oh Notation
10,000
❑ Given functions f(n) and 3n
g(n), we say that f(n) is 2n+10
1,000
O(g(n)) if there are
n
positive constants
c and n0 such that 100
f(n) cg(n) for n n0
10
❑ Example: 2n + 10 is O(n)
◼ 2n + 10 cn
1
◼ (c − 2) n 10
1 10 100 1,000
◼ n 10/(c − 2) n
◼ Pick c = 3 and n0 = 10
© 2010 Goodrich, Tamassia Analysis of Algorithms 22
Big-Oh Example
1,000,000
n^2
❑ Example: the function 100n
100,000
n2 is not O(n) 10n
◼ n2 cn 10,000 n
◼ nc
◼ The above inequality 1,000
cannot be satisfied
since c must be a 100
constant
10
1
1 10 100 1,000
n
© 2010 Goodrich, Tamassia Analysis of Algorithms 23
More Big-Oh Examples
7n-2
7n-2 is O(n)
need c > 0 and n0 1 such that 7n-2 c•n for n n0
this is true for c = 7 and n0 = 1
◼ 3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0 1 such that 3n3 + 20n2 + 5 c•n3 for n n0
this is true for c = 4 and n0 = 21
◼ 3 log n + 5
3 log n + 5 is O(log n)
need c > 0 and n0 1 such that 3 log n + 5 c•log n for n n0
this is true for c = 8 and n0 = 2
© 2010 Goodrich, Tamassia Analysis of Algorithms 24
Big-Oh Rules
❑ If is f(n) a polynomial of degree d, then f(n) is
O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors
❑ Use the smallest possible class of functions
◼ Say “2n is O(n)” instead of “2n is O(n2)”
❑ Use the simplest expression of the class
◼ Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”
© 2010 Goodrich, Tamassia Analysis of Algorithms 25
Asymptotic Algorithm Analysis
❑ The asymptotic analysis of an algorithm determines
the running time in big-Oh notation
❑ To perform the asymptotic analysis
◼ We find the worst-case number of primitive operations
executed as a function of the input size
◼ We express this function with big-Oh notation
❑ Example:
◼ We determine that algorithm arrayMax executes at most
8n − 2 primitive operations
◼ We say that algorithm arrayMax “runs in O(n) time”
❑ Since constant factors and lower-order terms are
eventually dropped anyhow, we can disregard them
when counting primitive operations
© 2010 Goodrich, Tamassia Analysis of Algorithms 26
Math you need to Review
Summations
Logarithms and Exponents
❑ properties of logarithms:
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
logbxa = alogbx
logba = logxa/logxb
❑ properties of exponentials:
a(b+c) = aba c
abc = (ab)c
Proof techniques
ab /ac = a(b-c)
Basic probability b = a logab
bc = a c*logab
© 2010 Goodrich, Tamassia Analysis of Algorithms 27
Relatives of Big-Oh
big-Omega
◼ f(n) is (g(n)) if there is a constant c > 0
and an integer constant n0 1 such that
f(n) c•g(n) for n n0
big-Theta
◼ f(n) is (g(n)) if there are constants c’ > 0 and c’’
> 0 and an integer constant n0 1 such that
c’•g(n) f(n) c’’•g(n) for n n0
© 2010 Goodrich, Tamassia Analysis of Algorithms 28
Intuition for Asymptotic
Notation
Big-Oh
◼ f(n) is O(g(n)) if f(n) is asymptotically
less than or equal to g(n)
big-Omega
◼ f(n) is (g(n)) if f(n) is asymptotically
greater than or equal to g(n)
big-Theta
◼ f(n) is (g(n)) if f(n) is asymptotically
equal to g(n)
© 2010 Goodrich, Tamassia Analysis of Algorithms 29
Example Uses of the
Relatives of Big-Oh
◼ 5n2 is (n2)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1
such that f(n) c•g(n) for n n0
let c = 5 and n0 = 1
◼ 5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1
such that f(n) c•g(n) for n n0
let c = 1 and n0 = 1
◼ 5n2 is (n2)
f(n) is (g(n)) if it is (n2) and O(n2). We have already seen the former,
for the latter recall that f(n) is O(g(n)) if there is a constant c > 0 and an
integer constant n0 1 such that f(n) < c•g(n) for n n0
Let c = 5 and n0 = 1
© 2010 Goodrich, Tamassia Analysis of Algorithms 30
Algorithm Analysis –
Tom’s Rules of Thumb
◼ Start by defining a function that represents the time
for the thing you are trying to analyze.
◼ Often T(n)
◼ Be sure to state what n is.
◼ Time is worst-case if not specified.
quicksort(A, i, j) { Let T(n) be the time to complete
… quicksort on n array elements,
} where n = j-i+1.
bubblesort(A) { Let S(n) be the time to complete
… bubblesort on an array of n
} elements.
© 2019 Shermer Analysis of Algorithms 31
Algorithm Analysis –
Tom’s Rules of Thumb
◼ Work from inner blocks of (pseudo-)code to outer
blocks.
quicksort(A, i, j) {
pivot = A[i];
for(k = i+1 to j) {
if(pivot > A[k]) {
…
}
Start with these Then do this
else { Then this …
…
}
…
}
© 2019 Shermer Analysis of Algorithms 32
Assignments and Function
Calls
◼ An assignment with no function calls is O(1).
◼ A function call to a known algorithm (not the one you
are analyzing) takes the known time for that algorithm.
foo(A, n) {
…
k = (j+91)/3; O(1)
m = max(A, n); O(n) (finding maximum of an array takes linear time)
p = (j – 17) + max(B, n) O(1) + O(n) = O(n)
…
}
© 2019 Shermer Analysis of Algorithms 33
Recursive Function Calls
◼ A function call to the algorithm you are analyzing takes
(the function you defined)(some function of n) time.
◼ For example, T(n/2)
foo(A, n) { Define S(n) to be time taken by foo with second
parameter n
…
foo(A, n-2); S(n-2)
}
bar(A, i, j) { Define T(n) to be time taken by bar with n = j-i+1
m = (i + j) / 2;
bar(A, i, m); T(n/2)
…
}
© 2019 Shermer Analysis of Algorithms 34
Conditionals
◼ Add up the time in each branch of the conditional.
◼ The conditional takes the time taken by the condition,
plus the maximum of the branch times.
foo(A, n) {
…
if( p < A[i] ) { O(1)
… O(n)
} The whole if statement
else { takes time
O(1) + max(O(n), O(n2))
… O(n2)
= O(n2)
}
}
© 2019 Shermer Analysis of Algorithms 35
Loops
◼ Add up the time in the body of the loop.
◼ Determine how many times t the loop will be executed,
as a function of your n. Use worst-case estimate.
◼ The time for the loop is t * (time for body)
for(i = 1 to n ) { n iterations The whole for loop takes
… O(n) time
} n * O(n) = O(n2)
i = 0;
while(p < A[i]) { n iterations
…
i++;
}
© 2019 Shermer Analysis of Algorithms 36
Triangular Loops
◼ Triangular loops have an inner index that counts up to
the outer index.
◼ Assume the inner loops have the same number of
iterations as the outer loop.
for(i = 1 to n ) { n iterations
…
for(j = 1 to i) { n iterations
…
}
}
❑ This also works for more than two nested loops
© 2019 Shermer Analysis of Algorithms 37
At the end
◼ Set (the function of n you defined) = the summed-up
cost of the entire algorithm.
◼ Along the way, don’t absorb T(…) factors into the big-
Oh notation.
◼ If you end up with a recurrence, solve it.
© 2019 Shermer Analysis of Algorithms 38
At the end
int find(A, i, j, target) { Let T(n) be the time for find where n = j-i+1.
if( i == j) {
if(A[i] == target)
return i O(1)
O(1) O(1)
else
return -1 O(1)
}
m = (i + j) / 2 O(1) O(1) + 2T(n/2)
f = find(A, i, m, target) O(1) + T(n/2)
if(f > 0) {
return f O(1) O(1) T(n) = O(1) +
} 2T(n/2)
return find(A, m+1, j, target) O(1) + T(n/2)
T(n) is O(n)
}
© 2019 Shermer Analysis of Algorithms 39
