designing analyzing algorithms Algorithm logarithm! Finiteness: Definiteness: Input: Output: Effectiveness: Design Analysis

{+, −, ∗, / \atop >}

? f(n)

g(n) n f g n

lim_{n→∞} \frac{f(n)}{g(n)} = c,

c 0 ∞ f(n) ∈

θ(g(n)) f(n) g(n)

f(n) ∈ Θ(g(n))f(n) ≡ g(n)0 < lim_{n→∞} \frac{f(n)}{g(n)} < ∞
f(n) ∈ O(g(n))f(n) ≤ g(n)0 ≤ lim_{n→∞} \frac{f(n)}{g(n)} < ∞
f(n) ∈ Ω(g(n))f(n) ≥ g(n)0 < lim_{n→∞} \frac{f(n)}{g(n)}
f(n) ∈ o(g(n)) f(n) < g(n)lim_{n→∞} \frac{f(n)}{g(n)} = 0
f(n) ∈ ω(g(n)) f(n) > g(n)lim_{n→∞} \frac{f(n)}{g(n)} = ∞

T (n) = ∑_{x=1}^{n} x^{2} ∈ Θ(n^{3})

lim_{n→∞} \frac{T (n)}{n^{3}} = lim_{n→∞} \frac{(n^{3} + 3n^{2} + 2n)/6}{n^{3}} = 1/6,

0 < 1/6 < ∞ T (n) ∈

O(n^{4}) T (n) = n^{3}/3+ O(n^{2}).

O

O(g(n)) = {f(n) : ∃c, n_{0} > 0f(n) ≤ cg(n)∀n ≥ n_{0}} O(·)upper bound O(·)worst-case

1/3n^{2}− 3n ∈ O(n^{2}) 1/3n^{2}− 3n ≤

cn^{2} c ≥

1/3−

3/n c = 1/3 n > 1 p(n) =

∑_{i=0}^{d} a_{i}n^{i}

d a_{d} > 0 p(n) ∈

O(n^{k})

k ≥ d c n_{0}

O(n^{2}) worstcase O(n^{2}) f(n) = O(g(n)) f(n) ∈ O(g(n)) O(n)

2n^{2}+ 3n+ 1 = 2n^{2}+ O(n)

2n^{2}+

3n+ 1 = 2n^{2}+ f(n)

f(n) O(n) O(1)

Ω

Ω(g(n)) = {f(n) : ∃c, n_{0} > 0cg(n) ≤ f(n)∀n ≥ n_{0}}

Ω(·)lower bound 1/3n^{2}−

3n = Ω(n^{2})

1/3n^{2}− 3n ≥

cn^{2} c ≤

1/3−

3/n c = 1/6 n > 18 p(n) =

∑_{i=0}^{d} a_{i}n^{i}

d a_{d} > 0 p(n) ∈

Ω(n^{k})

k ≤ d c n_{0}

g(n) = Ω(f(n)) f(n) = O(g(n)) Ω(n^{2}) best case Ω(n^{2})

Θ

Θ(g(n)) = {f(n) : ∃c_{1}, c_{2}, n_{0} > 0c_{1}g(n) ≤ f(n) ≤ c_{2}g(n)∀n ≥ n_{0}} Θ(·)tight bound

f(n) = Θ(g(n))if and only iff(n) = O(g(n))f(n) = Ω(g(n)) 6n log n+ \sqrt{n} log^{2} n =

Θ(n log n) n_{0}, c_{1}, c_{2}

c_{1}n log n ≤ 6n log n+ \sqrt{n} log^{2} n ≤

c_{2}n log n n > n_{0} c_{1}n log n ≤

6n log n+ \sqrt{n} log^{2} n ⇒

c_{1} ≤ 6+ \frac{log n}{\sqrt{n}}

c_{1} = 6 n_{0} = 1 6n log n+

\sqrt{n} log^{2} n ≤

c_{2}n log n ⇒ 6+

\frac{log n}{\sqrt{n}} ≤

c_{2} c_{2} = 7 log n ≤

\sqrt{n}

n ≥ 2 c_{1} = 6 c_{2} = 7 n_{0} = 2 p(n) =

∑_{i=0}^{d} a_{i}n^{i}

d a_{d} > 0 p(n) ∈

Θ(n^{k})

k = d a ≥ 1, b > 1 c > 0 T (n)

T (n) = aT (\frac{n}{b})+cn^{k},

n ≥ 0 a >

b^{k} T (n) =

Θ(n^{log_{b} a}). a = b^{k} T (n) =

Θ(n^{k} log_{b} n). a <

b^{k} T (n) =

Θ(n^{k}). a n/b

cn^{k} a^{2} n/b^{2} a∗ c(n/b)^{k}

a^{3} n/b^{3}

a^{2}∗ c(n/b^{2})^{k}

a^{log_{b} n} a^{log_{b} n}c(\frac{n}{b^{log_{b} n}})^{k}

T (n) = O(n^{log_{b} a})+ ∑_{i=0}^{log_{b} n} a^{i}c (\frac{n}{b^{i}} )^{k}.

? ? A B C n× n Z = X· Y n n n^{2} X =

\begin{cases}x_{11} & x_{12} & ⋯ & x_{1n} \\ x_{21} & x_{22} & ⋯ & x_{1n} \\ ⋯ & ⋯ & ⋯ & ⋯ \\ x_{n1} & x_{n2} & ⋯ & x_{nn} \end{cases}\}

Y =

\begin{cases}y_{11} & y_{12} & ⋯ & y_{1n} \\ y_{21} & y_{22} & ⋯ & y_{1n} \\ ⋯ & ⋯ & ⋯ & ⋯ \\ y_{n1} & y_{n2} & ⋯ & y_{nn} \end{cases}\}

Z = X· Y

z_{ij} = ∑_{k=1}^{n} x_{ik}·y_{kj}.

n^{2} Z n n− 1 Θ(n^{3})

O(n^{log_{2} 7})

T (n) = 7T (n/2)+cn^{2},

c n T (constant) =

O(1) a = 7 b = 2 c = c k = 2 a >

b^{k} T (n) =

O(n^{log{7 \atop 2}}) n/2× n/2 Z =

\begin{cases}A & B \\ C & D \end{cases}\}·

\begin{cases}E & F \\ G & H \end{cases}\} =

\begin{cases}(S_{1} + S_{2} − S_{4} + S_{6}) & (S_{4} + S_{5}) \\ (S_{6} + S_{7}) & (S_{2} + S_{3} + S_{5} − S_{7}) \end{cases}\}

S_{1}=(B − D) · (G + H)
S_{2}=(A + D) · (E + H)
S_{3}=(A − C) · (E + F)
S_{4}=(A + B) · H
S_{5}=A · (F − H)
S_{6}=D · (G − E)
S_{7}=(C + D) · E

S_{4}+ S_{5} AF + BH

S_{4} + S_{5}=(A + B) · H + A · (F − H)
=AH + BH + AF − AH
=AF + BH

T (n) = 7T (n/2)+ Θ(n^{2})

7 Θ(n^{2})

n/2× n/2 Θ(n^{2}) Ω(n^{2})

Z Ω(n^{2})

O(n^{2.3...}) n× n A B C AB = C ? A B C n× n F AB ≠

C r {0, 1}^{n}

P r[ABr = Cr] ≤ 1/2 ? sample space {1, 2, 3, 4, 5, 6} event {2, 4, 6} n 2^{n} A B S A∪ B Ā A∩ B A B mutually exclusive A∩ B = ∅ S A ⊆ S 0 ≤ P (A) ≤ 1 probability

A P (A)

P (S) =

1 A B P (A∪

B) = P (A)+ P (B) P (A)∩ P (B) =

0 P (Ā) =

1− P (A)

Ā = S\

A A B P (A∪

B) = P (A)+ P (B)− P (A∩ B)

A ⊂ B P (A) ≤

P (B) A B P (B|A) conditional probability

B A P (B)

B 2 P (B) =

1/6

A P (A) =

1/2 P (B|A) 1/3 {2, 4, 6}

2 1/3

P (B|A) =

\frac{P (A∩B)}{P (A)}

P (A∩ B) = P (B|A)P (A) P (A|B) =

\frac{P (A∩B)}{P (B)}

P (A∩ B) = P (A|B)P (B)

A B P (A|B) =

P (B|A) =

0 A ⊂ B P (B|A) =

1 B A 2 A B Bayes

The- o- rem B_{1}, B_{2}, ⋯, B_{k}

S A S

P (B_{i}|A) = \frac{P (A ∩ B_{i})}{P (A)} = \frac{P (A|B_{i})P (B_{i})}{∑_{i=1}^{k} P (A|B_{i})P (B_{i})}.

? A = {

} B_{1} = { }

B_{2} B_{3}

P (B_{1}) = 1/2 P (B_{2}) = P (B_{3}) = 1/4 P (A|B_{1}) = P (A|B_{2}) =

0.02 P (A|B_{3}) =

0.04 P (A) = P (A|B_{1})P (B_{1})+P (A|B_{2})P (B_{2})+P (A|B_{3})P (B_{3}) = 0.025.

P (B_{1}|A) = \frac{P (A|B_{1})P (B_{1})}{∑_{i=1}^{k} P (A|B_{i})P (B_{i})} = \frac{0.02 ∗ 1/2}{0.02 ∗ 1/2 + 0.02 ∗ 1/4 + 0.04 ∗ 1/4} = 0.40.

A B in- de- pen-

dent P (A∩

B) = P (A)P (B)

A B ? A ={

} B ={ } C ={ } P (A) = P (B) = P (C) = 1/2 P (A∩ B) = 1/4 = P (A)P (B) P (A∩ C) = 1/4 = P (A)P (C) P (B∩ C) = 1/4 = P (B)P (C) P (A∩

B∩ C)

P (A∩

B∩ C) =

0 ≠ P (A)P (B)P (C) A, B C mutually indepen-

dent P (A∩

B∩ C) =

P (A)P (B)P (C) n X random variable X : S → ℜ X n discrete P (x_{i}) = P (X = x_{i}) P (x_{i}) ≥

0 ∑_{i=1}^{...} P (x_{i}) =

1 p

probability function (x_{i}, P (x_{i})) probability distribution X X X {0, 1, 2, 3} P (0) = P (3) = 1/8 P (1) = P (2) =

3/8 a b a 2/3

b 1/3

b X X 1, 2, 3, ...

P (X =

1) = 1/3 P (X =

2) = 2/3∗ 1/3 P (X =

i) = (2/3)^{i−1}1/3 ∑_{i=1}^{∞}(2/3)^{i} = 1 Binomial distribution A P (A) = p

P (Ā) = q = 1− p n X A X = k 1 ≤ k ≤ n P (X =

k) = \binom{n}{k}p^{k}q^{n−k} X binomial random variable p n Bernoulli trials X continuous f f(x) ≥

0 x

∫_{−∞}^{+∞} f(x)dx =

1 −∞ < a < b < +∞ P (a < x < b) =

∫_{a}^{b} f(x)dx f(x) x = a x = b

Expected value of a discrete r.v. X E[X] = ∑_{i=1}^{...} x_{i}P (x_{i}) E[X] X

Expected value of a continuous r.v. X E[X] = ∫_{−∞}^{+∞} xf(x)dx

X [a, b]

f(x) =

\frac{1}{b−a}

a ≤ x ≤ b E[X] =

∫_{a}^{b} \frac{x}{b−a}dx = (a+ b)/2 variance X V [X] = E[X− E[X]]^{2} [a, b] E[X] = (a+ b)/2 E[X^{2}] = ∫_{a}^{b} \frac{x^{2}}{b−a}dx =

\frac{b^{3}−a^{3}}{3(b−a)}

V [X] = E[X^{2}]− [E[X]]^{2}

V [X] = \frac{b^{3}−a^{3}}{3(b−a)}−

(\frac{a+b}{2})^{2} =

\frac{(b−a)^{2}}{12}

X c E[X−

c]^{2} ϵ > 0

P (|X−c| ≥ ϵ) ≤ \frac{1}{ϵ^{2}} E[X−c]^{2}.

X R = {x :

|x− c| ≥ ϵ} P (|X− c| ≥ ϵ) = ∫_{R} f(x)dx |x−

c| ≥ ϵ

\frac{(x−c)^{2}}{ϵ^{2}} ≥ 1

P (|X−c| ≥ ϵ) ≤ ∫_{R} \frac{(x − c)^{2}}{ϵ^{2}} f(x)dx ≤ ∫_{−∞}^{+∞} \frac{(x − c)^{2}}{ϵ^{2}} f(x)dx = \frac{1}{ϵ^{2}} E[X−c]^{2}.

X normal distribution

f(x) = \frac{1}{\sqrt{2π}σ} e^{− \frac{1}{2} ⌈ \frac{x−μ}{σ} ⌉^{2}}, −∞ < x < ∞.

N (μ, σ^{2}) μ σ f(x) >

0 x ∫ f(x)dx =

1 f x = μ x = μ± σ N (0, 1)

standardized normal distribution X N (μ, σ^{2})

Y = aX+ b Y N (aμ+

b, a^{2}σ^{2})

A n n_{A} A f_{A} =

n_{A}/n

P (A) = p ϵ > 0

P (|f_{A}−p| ≥ ϵ) ≤ \frac{p(1 − p)}{nϵ^{2}},

P (|f_{A}−p| < ϵ) ≥ 1−\frac{p(1 − p)}{nϵ^{2}}.

n_{A}

E[n_{A}] = np

V [n_{A}] = np(1− p) f_{A} = n_{A}/n

E[f_{A}] = p

V [f_{A}] = p(1− p)/n

?? P (|X−

μ| < ϵ) ≥

1− \frac{V ar(X)}{ϵ^{2}}

X = f_{A} p = μ

P (|f_{A}−p| < ϵ) ≥ 1−\frac{V [f_{A}]}{ϵ^{2}}.

ϵ = k\sqrt{p(1 − p)/n} V [f_{A}] = p(1− p)/n

P (|f_{A}−p| < ϵ) ≥ 1−\frac{p(1 − p)}{nϵ^{2}}.

lim_{n→∞} P (|f_{A}−p| < ϵ) = 1.

P (A) f_{A}

P (A) n ϵ = 0.01 1−

\frac{p(1−p)}{nϵ^{2}} =

0.9 n = \frac{p(1−p)}{0.1ϵ^{2}} p =

1/2 n = 25000 ? n n X_{1}, X_{2}, ⋯, X_{n} n μ =

E[X_{i}]

σ^{2} = V [X_{i}]

S = ∑_{i=1}^{n} X_{i}

E[S] = nμ

V [S] =

nσ^{2} n T_{n} = \frac{S−nμ}{\sqrt{n}σ}

N(0, 1) n n ? 1

X_{i} i M_{i} =

\frac{1}{i} ∑_{k=1}^{i} X_{k}

i X_{i} 0, 1, 2 0.2, 0.3 0.5 M_{1} = X_{1} 0, 1 2 0.2, 0.3 0.5 M_{2}

\begin{matrix}m = \frac{X_{1}+X_{2}}{2} \\ P (M_{2} = m) \end{matrix}

\begin{matrix}m = \frac{X_{1}+X_{2}+X_{3}}{3} \\ P (M_{3} = m) \end{matrix}

n M_{n} ? X s > 0 P (X ≥

s) ≤ E[X]/s E[X] = ∑_{i=0}^{∞} i.P (X =

i) ≥ ∑_{i=s}^{∞} i.P (X =

i) ≥ s ∑_{i=s}^{∞} P (X =

i) = sP (X ≥ s) P (X ≥

s) ≤ E[X]/s X_{1}, ⋯ X_{n} P (X_{i} =

1) = p_{i}

P (X_{i} =

0) =

1− p_{i}

X = ∑_{i=1}^{n} X_{i} m = ∑_{i=1}^{n} p_{i}

E[X] = E[∑_{i=1}^{n} X_{i}] = ∑_{i=1}^{n} E[X_{i}] = ∑_{i=1}^{n} p_{i} = m.

n X_{1}, ⋯, X_{n}

X_{i} = 1 i X_{i} = 0 P (X_{i} =

0) = P (X_{i} =

1) = 1/2

X = ∑_{i=1}^{n} X_{i}

E[X] =

n/2 = m X (1±

ϵ)m

0 < ϵ < 1 P (X >

(1+ ϵ)m) ≤ 1/(1+ ϵ) Chernoff bounds P (X ≥ (1+ϵ)m) ≤ exp(−ϵ^{2}m/3)

P (X ≤ (1−ϵ)m) ≤ exp(−ϵ^{2}m/2).

t > 0 P (X ≥

(1+ ϵ)m) ≤ exp(−ϵ^{2}m/3) ≥ (1+ ϵ)m) = P (e^{tX} ≥

e^{t(1+ϵ)m}) =

e^{−t(1+ϵ)m}e^{t(1+ϵ)m}P (e^{tX} ≥ e^{t(1+ϵ)m}) ≤ e^{−t(1+ϵ)m}E[e^{tX}]

?? P (e^{tX} ≥

e^{t(1+ϵ)m}) ≤ e^{−t(1+ϵ)m}E[e^{tX}] E[e^{tX}] = E[e^{t ∑_{i=1}^{n} X_{i}}] = E[Π_{i=1}^{n}e^{tX_{i}}] = Π_{i=1}^{n}E[e^{tX_{i}}] X_{i} E[e^{tX_{i}}] = p_{i}e^{t}+ (1− p_{i})e^{0} E[e^{tX}] = Π_{i=1}^{n}(p_{i}e^{t}+ (1− p_{i})) = Π_{i=1}^{n}(1+ p_{i}(e^{t}−

BTW, I never ‘really’ understood the proof of this theorem, except that I can verify each of the steps in the proof. I hope one day I will get it!