Lecture Notes for COMP 5703

(A Course on Advanced Topics in Algorithm Design)

Anil Maheshwari School of Computer Science

Carleton University anil@scs.carleton.ca

December 11, 2013

2

Contents

1 Preliminaries 1 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Model of Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Asymptotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3.1 O-notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3.2 Ω-notation (big-Omega) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3.3 Θ-notation (Big-Theta) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.4 How to analyze Recurrences? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.5 Strassen’s Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.5.1 Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.5.2 Strassen’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.6 Elementary stuff from probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.6.1 Chernoff Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2 Introduction to Graphs and their Representation 17 2.1 Introduction and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 How to represent graphs in a computer? . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3 Graph traversal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.3.1 Breadth-first search (BFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.4 Topological sort and DFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.4.1 Topological Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4.2 Depth First Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.4.3 Computation of low(v) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.4.4 Biconnectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.4.5 Equivalence Relation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.4.6 Biconnectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3 Minimum Spanning Trees 33 3.1 Minimum Spanning Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2 Kruskal’s Algorithm for MST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.3 Prim’s MST algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.4 Randomized Algorithms for Minimum Spanning Trees . . . . . . . . . . . . . . . . . 37

3.4.1 Boruvka’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.4.2 Heavy Edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.4.3 Randomized Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.5 MST Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.5.1 Overview of verification algorithms . . . . . . . . . . . . . . . . . . . . . . . . 41 3.5.2 Komlós’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.5.3 Dixon et al.’s Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.5.4 King’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.5.5 Hagerup’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.5.6 Putting it all together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3.6 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4 Network Flow 53 4.1 What is a Flow Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.2 Ford and Fulkerson’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.3 Edmonds-Karp Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.4 Applications of Network Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

5 Separators in a Planar Graph 63 5.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.2 Proof of the Planar Separator Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 64 5.3 Generalizations of the Planar Separator Theorem . . . . . . . . . . . . . . . . . . . . 67

5.3.1 Weighted Separators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.3.2 r-Divisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.3.3 Edge Separators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

5.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

6 Lowest Common Ancestor 73 6.1 LCA → RMQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.2 Range Minima Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

6.2.1 A naive O(n^{2}) algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.2.2 An O(n log n) algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.2.3 An O(n) algorithm with ±1 property . . . . . . . . . . . . . . . . . . . . . . 75

6.3 RMQ → LCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

7 Locality-Sensitive Hashing 77 7.1 Similarity of Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 7.2 Similarity-Preserving Summaries of Sets . . . . . . . . . . . . . . . . . . . . . . . . . 78 7.3 LSH for Minhash Signatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 7.4 Theory of Locality Sensitive Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 82 7.5 LSH Families . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

7.5.1 Hamming Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 7.5.2 Cosine Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 7.5.3 Euclidean Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 7.5.4 Fingerprint Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 7.5.5 Image Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

7.6 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 7.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

8 Dimensionality Reduction - Under Construction 95 8.1 Johnson and Lindenstrauss Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 8.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

8.2.1 Sort of the Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 8.3 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

9 Stable Matchings 101 9.1 Stable Marriages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 9.2 Hospitals/Residents (College Admissions) . . . . . . . . . . . . . . . . . . . . . . . . 104 9.3 Stable Roommates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 9.4 Alternate Algorithm for Stable Roommates . . . . . . . . . . . . . . . . . . . . . . . 112 9.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Preface + Acknowledgments

These notes extensively use material from the course notes of Lars Arge, David Mount, COMP 2805/3803 Notes of myself and Michiel Smid [50], CLRS book [12], Knuth’s Art of Computer Programming [43], Kleinberg and Tardos Algorithms book [42], Ullman et al’s book on Algorithms for Massive Data Sets [58]. These notes are updated occasionally. A substantial update was done in the Fall Term of 2013. Chapters on Elementary probability, Locality Sensitive Hashing, Dimensionality Reduction, and several exercises have been added. Moreover, as part of the offering of COMP 5703 in the Fall of 2013, several students have contributed significantly. Gregory Bint updated the chapter on the Minimum Spanning Trees, and has added a completely new Section 3.5 on the Spanning Tree Verification. Alexis Beingessner has provided Section 5.3 on extension of the planar separator theorem. Andrew Wylie has updated the chapter on Locality Sensitive Hashing and has added a Section ?? on Image Similarities and provided an extensive bibliography on locality sensitive hashing and its applications. Michael St. Jules has provided an entire chapter on Stable Matchings. I thank the Fall 2013 batch of COMP 5703 for providing valuable contributions.

I have used parts of this material for the graduate course COMP 5703 at Carleton. The aim of these notes is mainly to summarize the discussion in the lectures. They are not designed as stand alone chapters or a comprehensive coverage of a topic. The exercises are from numerous sources and most of them have been asked in either an exam or in an assignment! Since these chapters have been written over time, and many of the TeX tools weren’t available in olden times, you will see that the initial chapters don’t have elegant figures or texts. Hopefully, volunteers in future will modernize this part!

If you spot any errors, or have suggestion which can help me to improve, I will be glad to hear them. If you wish to add material to these notes, including exercises, please do get in touch. Thanks in advance!

Anil Maheshwari anil@scs.carleton.ca

Chapter 1

Preliminaries

1.1 Introduction

• Class^{-1-} is about designing and analyzing algorithms [1]: Keep Asking questions!

- handwriting!- especially to slow me down and to understand Indian English!

– Algorithm:

∗ Mis-spelled logarithm!.

∗ The first most popular algorithm is the Euclid’s algorithm for computing GCD of two numbers.

∗ A well-defined procedure that transfers an input to an output.

∗ Not a program (but often specified like it): An algorithm can often be implemented in several ways.

∗ Knuth’s, Art of Computer Programming, vol.1, is a good resource on the history of algorithms!. He says that an algorithm is a finite set of rules that gives a sequence of operations for solving a specific type of problem. Algorithm has five important features: Finiteness: must terminate after finite number of steps. Definiteness: each step is precisely described. Input: algorithm has zero or more inputs. Output: has at least one output!. Effectiveness: Each operation should be sufficiently basic such that they can be done in finite amount of time using pencil and paper.

– Design: The focus of this course is on how to design good algorithms and how to analyze their efficiency. We will study methods/ideas/tricks for developing fast and efficient algorithms.

– Analysis: Abstract/mathematical comparison of algorithms (without actually implementing, prototyping and testing them).

• This course will require proving the correctness of algorithms and analyzing the algorithms.

Therefore MATH is the main tool. Math is needed in three ways:

– Formal specification of problem

– Analysis of correctness

– Analysis of efficiency (time, memory use,...)

Revise mathematical induction, what is a proof?, logarithms, sum of series, elementary number theory, permutations, factorials, binomial coefficients, harmonic numbers, Fibonacci numbers and generating functions [Knuth vol 1. or his book Concrete Mathematics is an excellent resource].

• Hopefully the course will show that algorithms matter!

1.2 Model of Computation

• Predict the resources used by the algorithm: running time and the space.

• To analyze the running time we need mathematical model of a computer:

Random-access machine (RAM) model:

– Memory consists of an infinite array of cells.

– Each cell can store at most one data item (bit, byte, a record, ..).

– Any memory cell can be accessed in unit time.

– Instructions are executed sequentially

– All basic instructions take unit time:

∗ Load/Store

∗ Arithmetic’s (e.g. +, −, ∗, /)

∗ Logic (e.g. >)

• Running time of an algorithm is the number of RAM instructions it executes.

• RAM model is not realistic, e.g.

– memory is finite (even though we often imagine it to be infinite when we program)

– not all memory accesses take the same time (cache, main memory, disk)

– not all arithmetic operations take the same time (e.g. multiplications are expensive)

– instruction pipelining

– other processes

• But RAM model often is enough to give relatively realistic results (if we don’t cheat too much).

1.3 Asymptotics

We do not want to compute a detailed expression of the run time of the algorithm, but rather will like to get a feel of what it is like? We will like to see the trend - i.e. how does it increase when the size of the input is increased - is it linear in the size of the input? or quadratic? or exponential? or who knows? The asymptotics essentially capture the rate of growth of the underlying functions describing the run-time. Asymptotic analysis assumes that the input size is large (since we are interested how the running time increases when the problem size grows) and ignores the constant factors (which are usually dependent on the hardware, programming smartness or tricks, compiletime-optimizations).

David Mount suggests the following simple definitions based on limits for functions describing the running time of algorithms. We will describe the formal definitions from [12] later.

Let f(n) and g(n) be two positive functions of n. What does it mean when we say that both f and g grow at roughly the same rate for large n (ignoring the constant factors), i.e.

lim_{n→∞} \frac{f(n)}{g(n)} = c,

where c is a constant and is neither 0 or ∞. We say that f(n) ∈ θ(g(n)), i.e. they are asymptotically equivalent. What about f(n) does not grow significantly faster than g(n) or grows significantly faster? Here is the table of definitions from David Mount.

Asymptotic Form Relationship Definition
f(n) ∈ Θ(g(n))  f(n) ≡ g(n) 0 < lim_{n→∞} \frac{f(n)}{g(n)} < ∞
f(n) ∈ O(g(n))  f(n) ≤ g(n) 0 ≤ lim_{n→∞} \frac{f(n)}{g(n)} < ∞
f(n) ∈ Ω(g(n))  f(n) ≥ g(n) 0 < lim_{n→∞} \frac{f(n)}{g(n)}
f(n) ∈ o(g(n))  f(n) < g(n) lim_{n→∞} \frac{f(n)}{g(n)} = 0
f(n) ∈ ω(g(n))  f(n) > g(n) lim_{n→∞} \frac{f(n)}{g(n)} = ∞

Example: T (n) = ∑_{x=1}^{n} x^{2} ∈ Θ(n^{3}). Why?

lim_{n→∞} \frac{T (n)}{n^{3}} = lim_{n→∞} \frac{(n^{3} + 3n^{2} + 2n)/6}{n^{3}} = 1/6,

and 0 < 1/6 < ∞. Just for fun show that T (n) ∈ O(n^{4}) or T (n) = n^{3}/3 + O(n^{2}).

1.3.1 O-notation

O(g(n)) = {f(n) : ∃ c, n_{0} > 0 such that f(n) ≤ cg(n) ∀n ≥ n_{0}}

• O(·) is used to asymptotically upper bound a function.

• O(·) is used to bound worst-case running time (see Figure 1.1).

• Examples:

– 1/3n^{2} − 3n ∈ O(n^{2}) because 1/3n^{2} − 3n ≤ cn^{2} if c ≥ 1/3 − 3/n which holds for c = 1/3 and n > 1.

– Let p(n) = ∑_{i=0}^{d} a_{i}n^{i} be a polynomial of degree d and assume that a_{d} > 0. Then p(n) ∈ O(n^{k}), where k ≥ d is a constant. What are c and n_{0} for this?

Figure 1.1: Illustration of O() notation.

• Note:

– When we say “the running time is O(n^{2})”, we mean that the worst-case running time is O(n^{2}) — best case might be better.

– We often abuse the notation:

∗ We write f(n) = O(g(n)) instead of f(n) ∈ O(g(n))!

∗ We often use O(n) in equations: e.g. 2n^{2} + 3n + 1 = 2n^{2} + O(n) (meaning that 2n^{2} + 3n + 1 = 2n^{2} + f(n) where f(n) is some function in O(n)).

∗ We use O(1) to denote a constant.

1.3.2 Ω-notation (big-Omega)

Ω(g(n)) = {f(n) : ∃ c, n_{0} > 0 such that cg(n) ≤ f(n) ∀n ≥ n_{0}}

• Ω(·) is used to asymptotically lower bound a function (see Figure 1.2).

• Examples:

– 1/3n^{2} − 3n = Ω(n^{2}) because 1/3n^{2} − 3n ≥ cn^{2} if c ≤ 1/3 − 3/n which is true if c = 1/6 and n > 18.

– Let p(n) = ∑_{i=0}^{d} a_{i}n^{i} be a polynomial of degree d and assume that a_{d} > 0. Then p(n) ∈ Ω(n^{k}), where k ≤ d is a constant. What are c and n_{0} for this?

– Prove or disprove: g(n) = Ω(f(n)) if and only if f(n) = O(g(n)).

• Note:

– When we say “the running time is Ω(n^{2})”, we mean that the best case running time is Ω(n^{2}) — the worst case might be worse.

Figure 1.2: Illustration of Ω() notation.

1.3.3 Θ-notation (Big-Theta)

Θ(g(n)) = {f(n) : ∃ c_{1}, c_{2}, n_{0} > 0 such that c_{1}g(n) ≤ f(n) ≤ c_{2}g(n) ∀n ≥ n_{0}} • Θ(·) is used to asymptotically tight bound a function.

Figure 1.3: Illustration of Θ() notation.

f(n) = Θ(g(n)) if and only if f(n) = O(g(n)) and f(n) = Ω(g(n)) (see Figure 1.3)

• Examples: – 6n log n + \sqrt{n} log^{2} n = Θ(n log n):

∗ We need to find n_{0}, c_{1}, c_{2} such that c_{1}n log n ≤ 6n log n + \sqrt{n} log^{2} n ≤ c_{2}n log n for n > n_{0} c_{1}n log n ≤ 6n log n + \sqrt{n} log^{2} n ⇒ c_{1} ≤ 6 + \frac{log n}{\sqrt{n}}. Ok if we choose c_{1} = 6 and n_{0} = 1. 6n log n + \sqrt{n} log^{2} n ≤ c_{2}n log n ⇒ 6 + \frac{log n}{\sqrt{n}} ≤ c_{2}. Is it ok to choose c_{2} = 7?

Yes, log n ≤ \sqrt{n} if n ≥ 2.
∗ So c_{1} = 6, c_{2} = 7 and n_{0} = 2 works.

– Let p(n) = ∑_{i=0}^{d} a_{i}n^{i} be a polynomial of degree d and assume that a_{d} > 0. Then

p(n) ∈ Θ(n^{k}), where k = d is a constant.

1.4 How to analyze Recurrences?

There are many ways of solving recurrences. I personally prefer the recursion tree method, since it is visual! Here the recurrence is depicted in a tree, where the nodes of the tree represent the cost incurred at the various levels of the recursion. We illustrate this method using the following recurrence (so called the recurrence used in the Masters method).

Let a ≥ 1, b > 1 and c > 0 be constants and let T (n) be the recurrence

T (n) = aT (\frac{n}{b}) + cn^{k},

defined for integer n ≥ 0. Then Case 1 a > b^{k} then T (n) = Θ(n^{log_{b} a}). Case 2 a = b^{k} then T (n) = Θ(n^{k} log_{b} n). Case 3 a < b^{k} then T (n) = Θ(n^{k}).

The proof is fairly simple (of course I would have made some errors on what I have written below). We need to sort of visualize it using the recursion tree. Level 1: a subproblems are formed, each of size n/b, and the total cost is cn^{k}. Level 2: a^{2} subproblems are formed, each of size n/b^{2}, and the total cost is a ∗ c(n/b)^{k}. Level 3: a^{3} subproblems are formed, each of size n/b^{3}, and the total cost is a^{2} ∗ c(n/b^{2})^{k}. ... Level log_{b} n: a^{log_{b} n} subproblems are formed, each of constant size and the total cost is about

a^{log_{b} n}c(\frac{n}{b^{log_{b} n}})^{k}. Therefore the total cost is

T (n) = O(n^{log_{b} a}) + ∑_{i=0}^{log_{b} n} a^{i}c (\frac{n}{b^{i}} )^{k}.

[2]: Ah! This guy leaves the Now apply the various cases^{-2-}

hard work for me. I will get him next time.

1.5 Strassen’s Matrix Multiplication

1.5.1 Matrix Multiplication

To illustrate the analysis of recurrences as well as the divide and conquer method, we consider Strassen’s matrix multiplication method [46, 62]. Let A, B and C be three n × n matrices, where Z = X · Y. There are n rows, n columns and n^{2} entries in each of the matrices.

• X = \{ \begin{matrix}x_{11} & x_{12} & ⋯ & x_{1n} \\ x_{21} & x_{22} & ⋯ & x_{1n} \\ ⋯ & ⋯ & ⋯ & ⋯ \\ x_{n1} & x_{n2} & ⋯ & x_{nn} \end{matrix} \}

• Y = \{ \begin{matrix}y_{11} & y_{12} & ⋯ & y_{1n} \\ y_{21} & y_{22} & ⋯ & y_{1n} \\ ⋯ & ⋯ & ⋯ & ⋯ \\ y_{n1} & y_{n2} & ⋯ & y_{nn} \end{matrix} \}

• We want to compute Z = X · Y, where

z_{ij} = ∑_{k=1}^{n} x_{ik} · y_{kj}.

• How many operations we require?

• In all we generate n^{2} entries in the matrix Z and each entry requires n multiplications and n − 1 additions. So the total number of operations can be bounded by Θ(n^{3}).

• Next we want to discuss a divide and conquer solution by Strassen which requires only O(n^{log_{2} 7}) operations.

• Let’s first analyze the recurrence

T (n) = 7T (n/2) + cn^{2},

where c is a constant, n is a positive integer, and T (constant) = O(1).

• Using the simplified master method, a = 7, b = 2, c = c, k = 2 and a > b^{k}. Hence T (n) = O(n^{log^{7}^{2}}).

1.5.2 Strassen’s Algorithm

• Divide each of the matrices into four sub-matrices, each of dimension n/2 × n/2. Strassen observed the following:

Z = \{ \begin{matrix}A & B \\ C & D \end{matrix} \} · \{ \begin{matrix}E & F \\ G & H \end{matrix} \} = \{ \begin{matrix}(S_{1} + S_{2} − S_{4} + S_{6}) & (S_{4} + S_{5}) \\ (S_{6} + S_{7}) & (S_{2} + S_{3} + S_{5} − S_{7}) \end{matrix} \}

where^{-3-} [3]: For sure I am not supposed to remember these seven S’s.

S_{1} = (B − D) · (G + H)
S_{2} = (A + D) · (E + H)
S_{3} = (A − C) · (E + F)
S_{4} = (A + B) · H
S_{5} = A · (F − H)
S_{6} = D · (G − E)
S_{7} = (C + D) · E

– Lets test that for S_{4} + S_{5}, which is supposed to be AF + BH.

S_{4} + S_{5} = (A + B) · H + A · (F − H)
= AH + BH + AF − AH
= AF + BH

• This leads to a divide-and-conquer algorithm with running time T (n) = 7T (n/2) + Θ(n^{2}), since

– We only need to perform 7 multiplications recursively. Additions/Subtractions only take Θ(n^{2}) time, and we need to do 18 of them for n/2 × n/2 matrices for each step of the recursion.

– Division/Combination can still be performed in Θ(n^{2}) time.

Matrix multiplication is a fundamental problem and it arises in almost all branches of Sciences, Social Sciences and Engineering. For example, high energy physicists multiply monstrous matrices. Strassen’s is not the currently fastest known algorithm, there have been numerous improvements over that method. It is obvious that any algorithm for matrix multiplication needs to perform Ω(n^{2}) operations, since the output matrix Z has that many entries. But only lower bound that is known for this problem is the trivial one, i.e. the Ω(n^{2}) bound. The currently best known upper bound is significantly larger than this (its about O(n^{2.3...})). So a major open problem, whose solution will be of immense importance will be either to raise the lower bound or drop down the upper bound!

[4]: Is it really interesting? Here is something which may be interesting to look into.^{-4-} This is regarding verifying given three

Field? Probability? Uniform Distribution? Random? Why the hell I am doing this course if the first class is so bad? Grrrrr..

n × n matrices A, B and C, whether AB = C? It turns out that there is a nice randomized algorithm that can do this and it is stated in the following Theorem (See Motwani and Raghavan [55] for details).

Theorem 1.5.1 Let A, B, and C be n × n matrices over a Field F such that AB ≠ C. Then for r chosen uniformly at random from {0, 1}^{n}, probability that P r[ABr = Cr] ≤ 1/2.

1.6 Elementary stuff from probability

This material is from the book of Meyer [52]. (BTW, this was my textbook for the first undergraduate course in probability - way back in the winter of 1982-83!) With each probabilistic experiment, the sample space is the set of all possible outcomes of that experiment. For example, for rolling a dice the set of all possible outcomes are {1, 2, 3, 4, 5, 6}. An event is also a set of possible outcomes, and is a subset of the sample space. For example, for rolling a dice and getting an even number, the events are {2, 4, 6}. If the sample space consists of n elements, then the total number of all possible events are 2^{n}. Since events are sets, we can use associated operations on sets. For example, if A and B are events for a sample space S, then we can define A ∪ B, Ā, A ∩ B, with the usual meaning.

Two events A and B are said to be mutually exclusive if they cannot occur simultaneously, i.e. A ∩ B = ∅. Let S be a sample space associated with an experiment. For each event A ⊆ S, associate a real number 0 ≤ P (A) ≤ 1, called as the probability of A, and P (A) satisfies following natural conditions

1. P (S) = 1,

2. If events A and B are mutually exclusive, P (A ∪ B) = P (A) + P (B) and P (A) ∩ P (B) = 0.

3. P (Ā) = 1 − P (A), where Ā = S \ A.

4. In general, for two events A and B, P (A ∪ B) = P (A) + P (B) − P (A ∩ B).

5. If A ⊂ B, P (A) ≤ P (B).

Let A and B be two events associated with an experiment. We say P (B|A) to be the conditional probability of occurrence of event B given that A has occurred. Intuitively, computation of P (B) is done with respect to a reduced sample space. For example, let B be the event of getting 2 after rolling a dice. Then P (B) = 1/6. Let A be the event of getting an even number after rolling a dice. Then P (A) = 1/2. What about P (B|A)?. It is 1/3, since the reduced sample space is \{2, 4, 6\}, and the probability of drawing a 2 is 1/3 from this space. Note that P (B|A) = \frac{P (A∩B)}{P (A)} or

equivalently P (A ∩ B) = P (B|A)P (A). Similarly we can define P (A|B) = \frac{P (A∩B)}{P (B)}, or equivalently P (A ∩ B) = P (A|B)P (B). Note that if two events A and B are mutually-exclusive than P (A|B) = P (B|A) = 0. If A ⊂ B, than P (B|A) = 1. (For example, if B is the event of obtaining an even number and A is the event of getting a 2 on rolling a dice, than if A has occurred, than for sure B has occur ed.) Now a classical theorem on Conditional Probabilities, called Bayes Theorem.

Let B_{1}, B_{2}, ⋯, B_{k} represent a partition of sample space S. Let A be an event in S. Then

P (B_{i}|A) = \frac{P (A ∩ B_{i})}{P (A)} = \frac{P (A|B_{i})P (B_{i})}{∑_{i=1}^{k} P (A|B_{i})P (B_{i})}.

Here is a classical example from Meyer [52] showing an application of this theorem.

Example 1.6.1 An item is produced by three factories - F1, F2, and F3. F2 and F3 produce the same number of items. F1 produces twice many items as F2 and F3. 2% of items produced by F1 and F2 are defective, and 4% of items produced by F3 are defective. All of these items are indistinguishable in terms of which factory they come from. First lets understand the partitioning by the following.

a) What is the probability that an item is defective? Here A = {item is defective}. B_{1} = {item came from F1} and likewise define B_{2} and B_{3}. Note that P (B_{1}) = 1/2 and P (B_{2}) = P (B_{3}) = 1/4. P (A|B_{1}) = P (A|B_{2}) = 0.02 and P (A|B_{3}) = 0.04. Then

P (A) = P (A|B_{1})P (B_{1}) + P (A|B_{2})P (B_{2}) + P (A|B_{3})P (B_{3}) = 0.025.

Hence there is a 2.5% chance that an item is defective.

b) Suppose that an item is defective. What is the chance that it is made in F1? Let us apply Bayes Theorem.

P (B_{1}|A) = \frac{P (A|B_{1})P (B_{1})}{∑_{i=1}^{k} P (A|B_{i})P (B_{i})} = \frac{0.02 ∗ 1/2}{0.02 ∗ 1/2 + 0.02 ∗ 1/4 + 0.04 ∗ 1/4} = 0.40.

Hence there is a 40% chance.

We say two events A and B are independent if P (A ∩ B) = P (A)P (B). Intuitively, this implies that occurrence and nonoccurence of A has no effect on the occurrence or non-occurrence of B. An example from Meyer [52] - its insightful.

Example 1.6.2 We are rolling two dices and have three types of events. A ={The first die shows an even number} B ={The second die shows an odd number} C ={Both show an odd or both show an even number} Observe that P (A) = P (B) = P (C) = 1/2. Note that P (A ∩ B) = 1/4 = P (A)P (B). P (A ∩ C) = 1/4 = P (A)P (C). P (B ∩ C) = 1/4 = P (B)P (C). But what about P (A ∩ B ∩ C)? Note that P (A ∩ B ∩ C) = 0 ≠ P (A)P (B)P (C).

Hence three events A, B and C are said to be mutually independent if and only if they are pairwise independent as well as P (A∩B ∩C) = P (A)P (B)P (C). In general n events are said to be mutually independent, if and only if all combinations of them are mutually independent.

A function X assigning every element of a sample space of an experiment to a real number is called a random variable (r.v.), i.e. X : S → ℜ. For example, X may count number of 1’s in a random bit string of length n. A r.v. is called discrete if the number of possible values it can take is either finite or countably infinite. For each of those values associate a real value between 0 and 1, i.e. its probability P (x_{i}) = P (X = x_{i}). Each P (x_{i}) ≥ 0 and ∑_{i=1}^{...} P (x_{i}) = 1. The function p is called the probability function and the collection of pairs (x_{i}, P (x_{i})) is called the probability distribution of X. For example, if X is a r.v. describing number of heads in three tosses of a coin, than the range of X is {0, 1, 2, 3} and P (0) = P (3) = 1/8, P (1) = P (2) = 3/8. Consider the following example.

Example 1.6.3 Suppose you want to generate several strings of as followed by a single b. The random character generator spits out an a with probability 2/3 and b with probability 1/3. The way the string is generated is by repeatedly invoking the ‘sputter’ till it outputs the first b. Let X be the random variable denoting the number of times the sputter is invoked. Note that X can take values 1, 2, 3, .... Observe that P (X = 1) = 1/3, P (X = 2) = 2/3 ∗ 1/3, and P (X = i) = (2/3)^{i−1}1/3. Note that ∑_{i=1}^{∞}(2/3)^{i} = 1.

The classical Binomial distribution is defined as follows. Consider an experiment, where A is an event. Let P (A) = p and P (Ā) = q = 1 − p. Let us repeat the experiment n times and define a random variable X indicating the number of times A occurs. It is assumed that each experiment is independent of others. What is probability that X = k, for some 1 ≤ k ≤ n? It is easy to see that P (X = k) = \binom{n}{k}p^{k}q^{n−k}. X is called a binomial random variable with parameters p and n. Individual experiments (trials) are called Bernoulli trials.

A r.v. X is said to be continuous if there exists a function f satisfying (a) f(x) ≥ 0, for all x.

(b) ∫_{−∞}^{+∞} f(x)dx = 1 and (c) For all −∞ < a < b < +∞, P (a < x < b) = ∫_{a}^{b} f(x)dx, i.e. the area of the curve under f(x) between x = a and x = b.

Expected value of a discrete r.v. X is defined as E[X] = ∑_{i=1}^{...} x_{i}P (x_{i}). If this series converges, than E[X] is also called the mean value of X.

Expected value of a continuous r.v. X is defined analogously, i.e., E[X] = ∫_{−∞}^{+∞} xf(x)dx. For example, consider X to be uniformly distributed over the interval [a, b]. We know that the probability distribution function f(x) = \frac{1}{b−a}, a ≤ x ≤ b. Hence E[X] = ∫_{a}^{b} \frac{x}{b−a}dx = (a + b)/2.

The variance of a r.v. X is defined to be V [X] = E[X − E[X]]^{2}. Let us calculate the variance of the r.v. that is uniformly distributed over the interval [a, b]. We know that E[X] = (a + b)/2. Note that E[X^{2}] = ∫_{a}^{b} \frac{x^{2}}{b−a}dx = \frac{b^{3}−a^{3}}{3(b−a)}. Since V [X] = E[X^{2}] − [E[X]]^{2} (see Exercises), we obtain

V [X] = \frac{b^{3}−a^{3}}{3(b−a)} − (\frac{a+b}{2})^{2} = \frac{(b−a)^{2}}{12} .

Now let us look at a famous inequality due to Chebyshev.

Theorem 1.6.4 Let X be a random variable and let c be a real number. If E[X − c]^{2} is finite and ϵ > 0, then

P (|X − c| ≥ ϵ) ≤ \frac{1}{ϵ^{2}} E[X − c]^{2}.

Proof. Let X be a continuous r.v. Let R = {x : |x−c| ≥ ϵ}. Note that P (|X −c| ≥ ϵ) = ∫_{R} f(x)dx. Observe that |x − c| ≥ ϵ implies \frac{(x−c)^{2}}{ϵ^{2}} ≥ 1. Thus

P (|X − c| ≥ ϵ) ≤ ∫_{R} \frac{(x − c)^{2}}{ϵ^{2}} f(x)dx ≤ ∫_{−∞}^{+∞} \frac{(x − c)^{2}}{ϵ^{2}} f(x)dx = \frac{1}{ϵ^{2}} E[X − c]^{2}.

Next, lets talk about Normal Distributions, as we will need them in the chapter on Dimensionality Reduction. Random variable X has a normal distribution if its probability density function is of the form

f(x) = \frac{1}{\sqrt{2π}σ} e^{− \frac{1}{2} ⌈ \frac{x−μ}{σ} ⌉^{2}}, −∞ < x < ∞.

Usuall it is denoted by N (μ, σ^{2}), where μ is the mean (expected value) and σ is the standard deviation, i.e. positive squareroot of the variance. It is also referred to as Gaussian or bell-shaped distribution. It is easy to see that it is a valid distribution, as f(x) > 0 for all values of x and ∫ f(x)dx = 1. The function f is symmetric around x = μ. If we trace the boundary of the function, it changes from being convex to concave, and these points of inflection are at x = μ ± σ. The distribution N (0, 1) is referred to as a standardized normal distribution. If X is N (μ, σ^{2}) and Y = aX + b than Y is N (aμ + b, a^{2}σ^{2}).

Lastly, a bit on law of large numbers. Suppose we repeat an experiment multiple times. Then the relative frequency of occurrence of an event should converge to its actual probability, if the experiment is repeated for sufficiently large number of times. For example, a factory is producing items. We don’t know what is the failure probability. Then one way to estimate this probability is to take a large sample and see what percentage are faulty. This percentage will be a good estimate of the failure probability. Of course this should be taken with a grain of salt. This really depends upon what kind of sample is chosen, etc. Essentially what we are heading towards is that if the elements in the sample are chosen randomly, than the percentage of faulty items will be a true indicator of failure probability.

Consider a particular event A in an experiment. This experiment is repeated n times, and each run is independent of other runs. Let n_{A} be the number of times A occurs in the runs. Let f_{A} = n_{A}/n. Let P (A) = p. For a positive ϵ > 0,

P (|f_{A} − p| ≥ ϵ) ≤ \frac{p(1 − p)}{nϵ^{2}},

or equivalently,

P (|f_{A} − p| < ϵ) ≥ 1 − \frac{p(1 − p)}{nϵ^{2}}.

This can be shown as follows. Observe that n_{A} is a binomially distributed random variable. E[n_{A}] = np and V [n_{A}] = np(1 − p). Since f_{A} = n_{A}/n, E[f_{A}] = p and V [f_{A}] = p(1 − p)/n. Recall Chebyshev’s inequality (see Exercise 1.16) which states that P (|X − μ| < ϵ) ≥ 1 − \frac{V ar(X)}{ϵ^{2}}. Substituting X = f_{A}, p = μ, we obtain

P (|f_{A} − p| < ϵ) ≥ 1 − \frac{V [f_{A}]}{ϵ^{2}}.

Set ϵ = k\sqrt{p(1 − p)/n} and V [f_{A}] = p(1 − p)/n, and we obtain

P (|f_{A} − p| < ϵ) ≥ 1 − \frac{p(1 − p)}{nϵ^{2}}.

Note that

lim_{n→∞} P (|f_{A} − p| < ϵ) = 1.

This is sort of the meaning that the relative frequency converges to P (A), when the experiment is repeated for a large number of times.

Here is a different type of question. How many times should we repeat the experiment, so that the relative frequency f_{A} differs from P (A) by at most 0.01 with probability at least 0.9? We need to choose n so that for ϵ = 0.01, 1 − \frac{p(1−p)}{nϵ^{2}} = 0.9. This implies that n = \frac{p(1−p)}{0.1ϵ^{2}}. For example if p = 1/2, than n = 25000. What this means is that if we toss a coin 25000 times, than we are 90% sure that the relative frequency of getting a head is within 0.01 of the theoretical probability.

Lastly, we state a theorem (without proof), which is a special case of the Central Limit Theorem (see Meyer [52]). Intuitively, the following theorem states that the arithmetic mean of n observations from the same random variable, has for large n, a normal distribution.

Theorem 1.6.5 Let X_{1}, X_{2}, ⋯, X_{n} be n independent r.v. all of which have the same distribution. Let μ = E[X_{i}] and σ^{2} = V [X_{i}] be their common expectation and variance, respectively. Define a r.v. S = ∑_{i=1}^{n} X_{i}. Then E[S] = nμ and V [S] = nσ^{2}. Moreover, for large n, T_{n} = \frac{S−nμ}{\sqrt{n}σ} has essentially the distribution N (0, 1).

Intuitively, the above theorem states that the arithmetic mean of n observations from the same random variable, has for large n, a normal distribution. Here is an example from Meyer’s [52] to get some idea of this theorem.^{1}

Example 1.6.6 Consider a box containing three types of balls - 20 balls labelled with a zero, 30 with a one and 50 with a two. In this experiment, we draw a random ball, note its label, and then place it back. We will repeatedly pick the balls, and eventually report the average of their labels. We are interested in understanding the distribution of the averages when we repeat this experiment for a large number of rounds. The above theorem claims that the average behaves like a normal distribution. Let X_{i} be the label of the ball drawn in Round i of this experiment. We are interested in the random variable M_{i} = \frac{1}{i} ∑_{k=1}^{i} X_{k}, denoting the averages of the labels drawn up to and including the Round i. Note that each X_{i} takes values 0, 1, and 2, with probabilities 0.2, 0.3 and 0.5, respectively. Observe that M_{1} = X_{1} and it takes values 0, 1 or 2 with probability 0.2, 0.3 and 0.5, respectively. Next let us look at M_{2}.

\begin{matrix}m = \frac{X_{1}+X_{2}}{2} & 0 & 1/2 & 1 & 3/2 & 2 \\ P (M_{2} = m) & 0.04 & 0.12 & 0.29 & 0.30 & 0.25 \end{matrix}

\begin{matrix}m = \frac{X_{1}+X_{2}+X_{3}}{3} & 0 & 1/3 & 2/3 & 1 & 4/3 & 5/3 & 2 \\ P (M_{3} = m) & 0.008 & 0.036 & 0.114 & 0.207 & 0.285 & 0.225 & 0.125 \end{matrix}

Hopefully, this example convinces us that for large values of n, M_{n} converges to a Normal distribution.

1.6.1 Chernoff Bounds

Suppose we toss a fair coin 1000 times. We expect about 500 tails. What is the probability that we will get over 750 tails? or over 900 tails? Chernoff bounds help us in determining probabilities of extreme events in a large collection of independent events. In this section we will establish these bounds. This section is derived from [27] and notes of Michiel Smid. First let us establish Markov’s inequality.

Theorem 1.6.7 Let X be a non-negative discrete r.v. and s > 0 be a constant. Then P (X ≥ s) ≤ E[X]/s.

Proof. Note that E[X] = ∑_{i=0}^{∞} i.P (X = i) ≥ ∑_{i=s}^{∞} i.P (X = i) ≥ s ∑_{i=s}^{∞} P (X = i) = sP (X ≥ s). Hence P (X ≥ s) ≤ E[X]/s.

Let X_{1}, ⋯ X_{n} be identical 0-1 independent r.v’s, such that P (X_{i} = 1) = p_{i} and P (X_{i} = 0) = 1 − p_{i}. Define X = ∑_{i=1}^{n} X_{i} and let m = ∑_{i=1}^{n} p_{i}. Note that

E[X] = E[∑_{i=1}^{n} X_{i}] = ∑_{i=1}^{n} E[X_{i}] = ∑_{i=1}^{n} p_{i} = m.

1BTW, I never ‘really’ understood the proof of this theorem, except that I can verify each of the steps in the proof. I hope one day I will get it!

Suppose we have a fair coin, which we toss n times. We define 0-1 r.v’s X_{1}, ⋯, X_{n}, where X_{i} = 1 if the i-th toss was a head otherwise X_{i} = 0. P (X_{i} = 0) = P (X_{i} = 1) = 1/2. Let X = ∑_{i=1}^{n} X_{i}. Now E[X] = n/2 = m. We are interested in estimating the probability of X deviating from (1 ± ϵ)m, for 0 < ϵ < 1. From Markov’s inequality we obtain P (X > (1 + ϵ)m) ≤ 1/(1 + ϵ). We will show the following, which are collectively known as Chernoff bounds in the algorithms community.

P (X ≥ (1 + ϵ)m) ≤ exp(−ϵ^{2}m/3)
P (X ≤ (1 − ϵ)m) ≤ exp(−ϵ^{2}m/2).

Let t > 0. Let us first deal with proving P (X ≥ (1 + ϵ)m) ≤ exp(−ϵ^{2}m/3).

P (X ≥ (1 + ϵ)m) = P (e^{tX} ≥ e^{t(1+ϵ)m})  (1.1)
= e^{−t(1+ϵ)m}e^{t(1+ϵ)m}P (e^{tX} ≥ e^{t(1+ϵ)m})  (1.2)
≤ e^{−t(1+ϵ)m}E[e^{tX}]  (1.3)

Equation 1.3 follows from Markov inequality applied to P (e^{tX} ≥ e^{t(1+ϵ)m}) ≤ e^{−t(1+ϵ)m}E[e^{tX}]. Observe that E[e^{tX}] = E[e^{t ∑_{i=1}^{n} X_{i}}] = E[Π_{i=1}^{n}e^{tX_{i}}] = Π_{i=1}^{n}E[e^{tX_{i}}], as X_{i}’s are independent. E[e^{tX_{i}}] = p_{i}e^{t} + (1 − p_{i})e^{0}. Hence E[e^{tX}] = Π_{i=1}^{n}(p_{i}e^{t} + (1 − p_{i})) = Π_{i=1}^{n}(1 + p_{i}(e^{t} − 1)) ≤ Π_{i=1}^{n}e^{p_{i}(e^{t}−1)} = e^{∑_{i=1}^{n} p_{i}(e^{t}−1)} = e^{m(e^{t}−1)}. (Note that for this we used the fact that e^{x} ≥ 1 + x.) Hence,

P (X ≥ (1 + ϵ)m) ≤ e^{−t(1+ϵ)m+m(e^{t}−1)}.

This expression holds for all values of t ∈ R. It is minimized for t = ln(1 + ϵ). Thus,

P (X ≥ (1 + ϵ)m) ≤ (1 + ϵ)^{−(1+ϵ)m}e^{mϵ} = (\frac{e^{ϵ}}{(1 + ϵ)^{1+ϵ}} )^{m} ≤ e^{−ϵ^{2}m/3}.

To show the last inequality, one needs to prove that ϵ − (1 + ϵ) ln(1 + ϵ) ≤ −\frac{1}{3}ϵ^{2}, which is left as an exercise.

Next, we show that P (X ≤ (1 − ϵ)m) ≤ exp(−ϵ^{2}m/2). The proof is along the same lines as the previous one.

P (X ≤ (1 − ϵ)m) = P (m − X ≥ ϵm)  (1.4)
= P (e^{t(m−X)} ≥ e^{tϵm})  (1.5)
≤ e^{−tϵm}E[e^{t(m−X}]  (1.6)
= e^{tm(1−ϵ)}E[e^{−tX}]  (1.7)

Note that

E[e^{−tX}] = Π_{i=1}^{n}E[e^{−tX_{i}}]  (1.8)
= Π_{i=1}^{n}[p_{i}e^{−t} + (1 − p_{i})]  (1.9)
. = Π_{i=1}^{n}[1 − p_{i}(1 − e^{−t})]  (1.10)
≤ Π_{i=1}^{n}[e^{−p_{i}(1−e^{−t})}]  (1.11)
= e^{− ∑_{i=1}^{n} p_{i}(1−e^{−t})}  (1.12)
= e^{−m(1−e^{−t})}  (1.13)

Therefore,

P (X ≤ (1 − ϵ)m) ≤ e^{−m(1−e^{−t})}e^{tm(1−ϵ)}.

This is minimized for t = − ln(1 − ϵ). Hence,

P (X ≤ (1 − ϵ)m) ≤ (1 − ϵ)^{−m(1−ϵ)}e^{−mϵ} = [(\frac{1}{1 − ϵ})^{1−ϵ}e^{−ϵ}]^{m} ≤ e^{−ϵ^{2} \frac{m}{2}} .

1.7 Exercises

1.1 Provide a proof for the simplified Masters theorem.

1.2 Present examples for each of the three cases and present an example where this theorem is not applicable.

1.3 This is from [12] and is based on Divide-and-Conquer Multiplication. (Do not use FFTs as such for this)

1. Show how to multiply two polynomials of degree 1, namely ax + b and cx + d using only three multiplications. (Note that (a + b).(c + d) is considered as 1 multiplication.)

2. Give a divide-and-conquer algorithm for multiplying two polynomials of degree n that runs in Θ(n^{log 3}). You may think of dividing the coefficients into a high half and a low half, or in terms of whether the index is even or odd.

3. Show that two n-bit integers can be multiplied in On^{log 3}) steps, where each step operates on at most a constant number of 1-bit values.

1.4 Show that 1^{d} + 2^{d} + 3^{d} + .... + n^{d} is O(n^{d+1}) for a constant d? What is the value of c and n_{0}.

1.5 Solve the recurrence relation

T (n) = T (xn) + T ((1 − x)n) + cn

in terms of x and n where x is a constant in the range 0 < x < 1. Is the asymptotic complexity the same when x = 0.5, 0.1 and 0.001? What happens to the constant hidden in the O() notation.

1.6 Analyze the recurrence relation

T (n) = \sqrt{n}T (\sqrt{n}) + n.

(This is called as the rootish-divide-and-conquer and for example plays an important role in parallel computing.)

1.7 V. Pan has discovered a way to multiply two 70∗70 matrices using only 143640 multiplications. Ignoring the additions, what will the asymptotic complexity of Pan’s algorithm for multiplying two n ∗ n matrices? Is it better than Strassen’s? Justify your answer.

1.8 Suppose we have n real numbers, where no two are the same. We want to report the smallest i numbers in the sorted order, where i < n. Which algorithm you think is the best option (Justify your answer)? A. Sort the numbers and list the first i. B. Build a priority queue and then call Extract-Min i times. C. Use the order statistics to find the i-th smallest number, partition the set according to this value, and then sort the i smallest numbers.

1.9 Let A and B be two arrays, each consisting of n distinct elements in sorted order (in an increasing order). Report the median of the set A ∪ B in O(log n) time.

1.10 Given two binary strings a = a_{0}a_{1} … a_{p} and b = b_{0}b_{1} … b_{q}, where each a_{i} and b_{j} are either 0 or 1. We say that a ≤ b if either of the following holds

(1) there exists an integer j, 1 ≤ j ≤ min(p, q), such that a_{i} = b_{i} for all i = 0, 1, ..., j − 1 and a_{j} < b_{j}.

(2) p < q and a_{i} = b_{i} for all i = 0, 1, 2, , , , p. Let A ⊆ Σ^{∗} be a set of distinct binary strings whose lengths sums up to n. Present an algorithm that can sort the binary strings in O(n) time. (All the strings are not of the same length!)

1.11 Let X be a binomially distributed r.v. with parameters n and p. Then E[X] = ∑ k = 0^{n}k\binom{n}{k}p^{k}(1 − p)^{n−k} = np.

1.12 Let X and Y be two r.v. Show that E[X + Y] = E[X] + E[Y].

1.13 Let X and Y be two independent random variables. Show that E[XY] = E[X]E[Y].

1.14 Show that the variance of a r.v. V [X] = E[X^{2}] − [E[X]]^{2}.

1.15 If X and Y are independent r.v. than V [X + Y] = V [X] + V [Y].

1.16 Let E[X] = μ and ϵ > 0. Show that p(|X − μ| ≥ ϵ) ≤ \frac{V [X]}{ϵ^{2}}.

1.17 For any value of x, 0 < x < 1, show that x − (1 + x) ln(1 + x) + \frac{1}{3}x^{2} ≤ 0.

Introduction to Graphs and their Representation

2.1 Introduction and Definitions

Graphs were discovered by Euler (Königsberg bridge problem)^{-5-}, Kirchoff (electrical networks)^{-6-} and [5]: Leonhard Euler,

1707-1783

[6]: Gustav Kirchoff,

1824-1887: At every node in an electrical circuit the sum of all currents should be equal to zero, i.e., the charge cannot accumulate at a node - or what comes in must go out!

Cayley (enumeration of organic chemical isomers)^{-7-} in different contexts. Graphs are combinatorial

[7]: Arthur Cayley,

1821-1895

structures used in computer science. Lists, Trees, Directed Acyclic Graphs, Flow Charts, Control Flow Graphs, and Planar Graphs are examples of graphs that are widely used in computer science. Most often, practical problems, can be cast into some sort of graph problem. Examples include the Traveling Salesperson problem (finding a route of the cheapest cost through many cities), or coloring a map so that no two neighboring countries receive the same color or finding shortest path from Carleton to National Art Gallery, or navigating hyperlinks in webpages. There are excellent books and thousands of papers discussing several aspects of graphs (definitions, connectivity, coloring, independent sets, matchings, Kuratowski’s theorem, four color theorem, ...). Some of the classical books include the book by Harary [28], Bollobas [5], Bondy and Murty [6], Lovasz [49]. We need to get used to some of the basic definitions.

Figure 2.1: An example of a undirected graph. The edge {1,4} is incident to the vertex labelled 1 and to the vertex labelled 4. The degree of vertex 1 is 2, degree of vertex 2 is 4. A path from the vertex u = 1 to the vertex v = 5 consists of vertices < u = 1, 4, 2, 5 = v >. This graph is connected and has only one connected component. This graph is simple.

Figure 2.2: An example of a directed graph. This graph is not strongly connected since there is no way to reach from the vertex labelled 3 to the vertex labelled 4 or from the vertex labelled 1 to the vertex labelled 6. The strongly connected components are {{1,2,5,4},{3},{6}}.

Graph A graph G = (V, E) consists of a finite set of vertices V and a finite set of edges E.

• Undirected graph: E is a set of unordered pairs of vertices {u, v} where u, v ∈ V (see Figure 2.1).

• Directed graphs: E is a set of ordered pairs of vertices (u, v) where u, v ∈ V (see Figure 2.2).

Incidence An edge {u, v} is incident to u and v.

Degree of vertex in undirected graph is the number of edges incident to it.

In (Out) degree of a vertex in directed graph is the number of edges entering (or leaving) it.

Path A path from u to v is a sequence of vertices < u = v_{0}, v_{1}, v_{2}, ⋯, v_{k} = v > such that (v_{i}, v_{i+1}) ∈ E (or {v_{i}, v_{i+1}} ∈ E)

• We say that v is reachable from u

• The length of the path is k

• It is a cycle if u = v

Connected An undirected graph is connected if every pair of vertices are joined by a path.

Component The connected components are the equivalence classes of the vertices under the “reachability” relation.

Strongly Connected A directed graph is strongly connected if every pair of vertices are reachable from each other.

Strongly connected components The strongly connected components are the equivalence classes of the vertices under the “mutual reachability” relation.

Simple connected undirected graph An undirected connected graph is called simple, if between every pair of vertices there is at most one edge, and no vertex contains a self loop (i.e. a vertex connected to itself by an edge). In this course, a graph specified without any qualifications is an undirected, connected, and a simple graph!

Complete Graphs An undirected graph is called a complete graph, if every pair of (distinct) vertices are joined by an edge. Examples include K_{1} (just a single vertex), K_{2} (a pair of vertices joined by an edge), K_{3} (a triangle), K_{5} (graph on five vertices), ... K_{5} is the smallest (in terms of vertices) non-planar graph (i.e. no matter how one draws it in the plane, there is a crossing). See Figure 2.3.

Figure 2.3: A complete bipartite graph K_{33} and a complete graph K_{5}.

Bipartite Graphs A graph is called bipartite if the vertex set V can be partitioned into two subsets S ∪ T = V, such that for any edge {a, b}, a ∈ S and b ∈ T. A bipartite graph is complete if every vertex in S is connected to each vertex in T by an edge. For example, K_{mn} refers to a complete bipartite graph consisting of vertices V = S ∪ T, where |S| = m and |T | = n. Interestingly K_{3,3} is the smallest graph (in terms of edges) which is non-planar.

Kuratowski’s Theorem A graph is planar if and only if it has no subgraph homeomorphic to

K_{5} or K_{3,3}.^{-8-} Two graphs are homeomorphic if both can be obtained from the same graph by [8]: This is one of the

fundamental theorems in Graph Theory.

a sequence of subdivisions of edges (insertion of a vertex on an edge). For example any two

cycles are homeomorphic.

2.2 How to represent graphs in a computer?

There are two standard ways of representing graphs in computers: Adjacency list and Adjacency Matrix. Let G = (V, E) be the graph under consideration (assume that it is undirected - for directed the same representation works as well).

In the adjacency matrix representation, we form a |V | × |V | matrix A of 0s and 1s, where ijth entry is 1 if and only if there is an edge from vertex i to vertex j. It is easy to see that this matrix will be symmetric for undirected graphs. Also given a pair of vertices v_{i} and v_{j}, it takes constant time to check whether there is an edge joining them by inspecting the ijth entry in the matrix A. Moreover, this is independent of the sizes of V and E. But the main drawback is that this representation requires O(|V |^{2}) memory space whereas the graph G may have very few edges! Just for fun and to get some insight, try to see what it means by taking products A × A or A × A × A,..., where the ‘.’ refers to the boolean AND and ‘+’ refers to the boolean OR? Try it for small graphs. We will come back to that later!

The other most common representation is the adjacency list representation. The adjacency list for a vertex v is a list of all vertices w that are adjacent to v. To represent the graph we have in all |V | lists, one for each vertex. This representation requires optimal storage, i.e. O(|V |+|E|). But to check whether the two vertices v and w are connected, we need to check in lists of v (or w) whether the vertex w (resp. v) is present. Searching in a list requires, in the worst case, time proportional to the size of the list, and hence searching will require O(min{degree(v), degree(w)}) time. This is because we can either search in the list of v or the list of w, and hence its advantageous to search in the smaller list.

2.3 Graph traversal

Once we have a graph, represented inside the computer, what do we do with it? When we go to a new (small!) city what is the best way to explore all the bars? What is the best way to search for a particular web page just following the hyperlinks (assume no search engine and assume that we can go from any web page to any other web page)? How to solve those maze problems?

• There are two standard and simple ways of traversing all vertices/edges in a graph in a

[9]: There is another one, systematic way^{-9-} called scan first search, which is combination of bfs and dfs.

– Breadth-first search (bfs)

– Depth-first search (dfs)

• They are used in many fundamental algorithms as a preprocessing step.

2.3.1 Breadth-first search (BFS)

• Main idea (see Figure 2.4):

– Start at a source vertex and visit,

– All vertices at distance 1,

– Followed by all vertices at distance 2,

– Followed by all vertices at distance 3, ⋮

• BFS corresponds to computing shortest path distance (number of edges) from s to all other vertices in the graph.

Here is the BFS algorithm.

procedure BFS(v)
Input: A simple connected undirected graph G=(V,E) and the start vertex v
       represented as adjacency lists.
Output : A BFS tree T, where each vertex u knows its parent p(u).
Variables: Q: Queue of vertices, maintained in the FIFO order
           x, y: vertex.
Begin
   T := empty; // BFS Tree initialized as empty.

Figure 2.4: Illustration of BFS.

   mark[v] := visited;
   Enqueue (v,Q); //Insert v in the Queue Q.
   While not empty (Q) do
      x:=front(Q); // Let x be in the front of Q.
      Dequeue(Q); // Remove the element in the front of Q.
      For each vertex y adjacent to x do
          if mark[y]=unvisited then
              mark[y]:= visited;
              Enqueue(y,Q);
              Insert((x,y),T);//Insert the directed edge (x,y) in the tree T.
          end if
      end for
    end while
End

It is easy to see that this algorithm runs in O(|V | + |E|) time since each vertex is inserted (enqueued) once in the queue Q and each edge {x, y} is explored twice, once when the vertex x is dequeued from Q and once when the vertex y is dequeued. Insertion and Deletion of a vertex in Q can be achieved in constant time since Q is a FIFO Queue, and can be maintained as a doubly-connected list. Also adjacency list representation will suffice and the correctness is left as an assignment problem.

2.4 Topological sort and DFS

2.4.1 Topological Sort

Let G = (V, E) be a directed acyclic graph (DAG) on the vertex set V and directed edge set E. A topological sort of a DAG is a linear ordering of all its vertices such that if G contains a directed edge (u, v), then u appears before v in the ordering. One can think of this process as assigning a number f : V → {1, ⋯, |V |} to each vertex such that for every directed edge (u, v), f(u) < f(v) (see Figure 2.5). We will sketch two algorithms, first a slower and then the optimal one. You need

Figure 2.5: Illustration of a Topological Sort

to verify the correctness as well as the complexities of both of them. Try to decide yourself what should be a suitable graph representation for these algorithms. Algorithm 1: Topological sort in O(|V |^{2}) time using adjacency matrix representation.

Step 1: Start from any vertex and follow edges backwards until a vertex v is found having no incoming edges.

Step 2: Make v the next vertex in the total order.

Step 3: Delete the vertex v and all of its outgoing edges and go to Step 1.

Observe that in Step 1 we will find a vertex v since DAGs have no cycles and they are finite graphs. Moreover a vertex is assigned a number when it has no incoming edges. This should be sufficient to prove the correctness. Algorithm 2: Topological sort in O(|V | + |E|) time using adjacency list representation, where for each vertex maintain a separate list of incoming edges and outgoing edges.

Step 1: Form a queue Q of vertices which have no incoming edges.

Step 2: Pick a vertex v from Q, and make v the next vertex in the order.

Step 3: Delete v from Q and delete all of its outgoing edges. Let (v, w) be an outgoing edge. If the list of incoming edges of w becomes empty then insert w in Q.

Step 4: If Q is not empty then GOTO Step 2.

Which invariant(s) are maintained by Algorithm 2? Why is it correct? Why does it run in linear

[10]: When we say that an O(|V | + |E|) time?^{-10-} algorithm takes linear time for a graph G = (V, E), we mean it takes O(|V | + |E|) time - linear in the size of the input.

2.4.2 Depth First Search

DFS is another way of exploring the graph. Like BFS, DFS traversal will take linear time, will produce a DFS spanning tree and this tree will posses very interesting, useful and beautiful properties.

Assume that we have an undirected connected simple graph G = (V, E). We will discuss DFS in G. (Similar ideas generalize to disconnected as well as directed graphs). Informally DFS on G can be described as follows:

• Select a vertex v of G which is initially unvisited.

• Make v visited.

• Each unvisited vertex adjacent to v is searched in-turn using DFS recursively.

DFS partitions the edges in G into two sets, the set of DFS spanning tree edges, say T, and the set of back edges, say B, where E = T ∪ B, and T ∩ B = ∅. Here is the formal description of the algorithm (taken from Aho, Hopcroft, Ullman [1]). Each vertex in G will be assigned a DFS-number, i.e., the order in which they are first visited in the DFS (see Figure 2.6).

Figure 2.6: Illustration of dfs and low-numbers.

Algorithm: DFS on G = (V, E) represented by adjacency list L[v] for each vertex v ∈ V. Output: Partition of E = T ∪ B. Tree edges are given as directed edges from a child to its parent. All edges not in T are considered to be in B. begin

1. T := ∅; COUNT:=1;

2. for all v ∈ V do mark v as unvisited;

3. while there exists an unvisited vertex do SEARCH(v)

end

procedure SEARCH(v) begin

1. mark v as visited; 2. DSF-number[v]:=COUNT;

3. COUNT:=COUNT+1;

4. for each vertex w on L[v] do if w is unvisited then

(a) add (w, v) to T; /*edge (w, v) is a DFS tree edge */

(b) SEARCH(w); /* Recursive call */

end

Complexity Analysis: Why does the above algorithm runs in O(|V | + |E|) time? We call the procedure SEARCH(v), |V | times, once for each vertex. The total running time of SEARCH(v) exclusive of recursive calls is proportional to the degree of v. Hence the total time complexity is O(|V | + ∑_{v∈V} (degree(v))) = O(|V | + |E|).

Property of Back edges: If {w, v} ∈ B is a back edge, then either w is an ancestor of v or v is an ancestor of w in the DFS tree T. Why? Suppose, without loss of generality, v has a lower DFS-number than w, i.e., the vertex v is visited before the vertex w. Therefore when SEARCH(v) is invoked, the vertex w is labeled unvisited. All the unvisited vertices visited by SEARCH(v) will become descendants of v in the DFS tree. Therefore w will become descendant of v since, w ∈ L[v] and each vertex in L[v] is looked at while executing SEARCH(v).

2.4.3 Computation of low(v)

Here we introduce a quantity called low(v) for each vertex v ∈ V with respect to the DFS tree T and the back edges B. This quantity will be used in checking whether the graph is biconnected, as well as computing its biconnected components. (This will the topic of the next section.) What is low(v)? First relabel the vertices by their DFS-number. For each vertex v ∈ V define low(v) as follows: low(v) = MIN({v} ∪ {w|there exists a back edge (x, w) ∈ B such that x is a descendant of v and w is an ancestor of v in the DFS tree})

Here is the intuition. Consider the subtree T_{v} of the DFS-tree T for the vertex v. What is the vertex closest to the root of T which we can reach just using the back edges from T_{v}? If there are no back edges going out of T_{v}, then low(v) = v; otherwise it is the minimum (i.e. closest to the root) among the set of ancestor vertices of v, which are joined by the back edges from the vertices in T_{v}. In order to compute low(v), we need to compute three values and we will sketch the modification in the DFS algorithm to compute them. The three values are

1. w = v; i.e. the case when there are no back edges going out of the subtree T_{v}.

2. w = low(c) and c is a child of v; i.e. the case when low(v) is the same as low value of one of its children.

3. (v, w) is a back edge in B; i.e. the back edges associated to vertex v itself.

The final low(v) value is given by low(v) = MIN({v} ∪ {low(c)| c is a child of v} ∪ {w|(v, w) ∈ B}).

Here is the revised SEARCH(v) procedure that computes the low values as well.

procedure SEARCH(v) begin

1. mark v as visited;

2. DSF-number[v]:=COUNT;

3. COUNT:=COUNT+1;

4. low(v):=DFS-number[v]; /* low(v) is at least equal to the DFS-number of v */

5. for each vertex w on L[v] do if w is unvisited then

(a) add (w, v) to T; /*edge (w, v) is a DFS tree edge */

(b) SEARCH(w); /* Recursive call */

(c) low(v) := min(low(v), low(w)) /*Compare the low value of v with its child w */

else if w is not the parent of v then

low(v) := min(low(v), DF S − number[w]); /* (v, w) is a back edge */

end

Given that the DFS algorithm runs in linear time (O(|V | + |E|)) time, it is easy to see that this algorithm runs within the same time complexity.

2.4.4 Biconnectivity

2.4.5 Equivalence Relation

Before we talk about biconnectivity, we need to recall the concept and definitions of equivalence relation.

Relation Let A and B be finite sets. A binary relation, R, from A to B is a subset of the cross product of A and B, i.e. R ⊆ A × B. A relation on a set A is a relation from A to A. Example : Let A = {1, 2, 3, 4}. Let R = {(a, b)|a divides b, where a, b ∈ A}, i.e. R = {(1, 1), (2, 2), (3, 3), (4, 4), (1, 2), (1, 3), (1, 4), (2, 4))}.

Reflexive A relation R on A is reflexive if (a, a) ∈ R for every element a ∈ A. The relation in the divide example is reflexive.

Symmetric A relation R on A is called symmetric if (b, a) ∈ R whenever (a, b) ∈ R, where a, b ∈ A. The relation in the divide example is not symmetric.

Transitive A relation R on A is called transitive if whenever (a, b) ∈ R and (b, c) ∈ R, then (a, c) ∈ R, for a, b, c ∈ A. The relation in the divide example is transitive.

Equivalence Relation A relation on a set A is an equivalence relation if it is reflexive, symmetric, and transitive.

Equivalence Classes Let R be an equivalence relation on a set A. The set of all elements that are related to an element a ∈ A is called the equivalence class of a. denoted by [a].

Property of Equivalence Classes Let R be an equivalence relation on A. Then if (a, b) ∈ R, then [a] = [b], where a, b ∈ A.

Partition of A Let R be an equivalence relation on A. Then the equivalence classes of R form a partition of A.

An example of an Equivalence Relation: Let R be the relation on the set of integers such that (a, b) ∈ R if and only if a = b or a = −b. The equivalence class of integer 4, denoted by [4] will be [4] = {4, −4}. Similarly [7] = {−7, 7}. Observe that since (4, −4) ∈ R, then [4] = [−4] = {4, −4}. Also observe that the set of integers can be partitioned by R as follows: {(−1, 1), (0), (−2, 2), (−3, 3), ...}.

There are many books in Discrete Mathematics discussing equivalence relations, for example, see Rosen [60].

2.4.6 Biconnectivity

Most of the material in this section is from Kozen [46] and Aho, Hopcroft and Ullman [1]. Assume that the graph G = (V, E) is undirected and connected. We start with definitions.

Articulation vertex A vertex v ∈ V is called an articulation vertex, if its removal disconnects the graph. Equivalently, vertex v is an articulation vertex if there exists vertices a and b, so that every path between a and b goes through v and a, b, and v are all distinct.

Biconnected A connected graph is biconnected if any pair of distinct vertices lie on a simple cycle (one with no repeated vertices). Equivalently, for every distinct triple of vertices v, a, and b, there exists a path between a and b not containing v. Observe that G is biconnected if and only if it has no articulation vertices. Note that a graph just consisting of a single edge (two vertices joined by an edge) is biconnected!

Relation on edges For edges e, e′ ∈ E, define that the two edges are equivalent, e ≡ e′, if e and e′ lie on a simple cycle.

Lemma 2.4.1 The relation ≡ defined above is an equivalence relation.

Proof. [46]: To prove that it is an equivalence relation, we need to show that it is reflexive, symmetric and transitive. Reflexive: Obviously e ≡ e, ∀e ∈ E, since an edge in itself lie on a cycle.

Figure 2.7: Illustration of Transitivity

Symmetric: If e ≡ e′, then e′ ≡ e, since they are on the same cycle.

Transitivity: Suppose that e = (u, v) ≡ e′ = (u′, v′) and e′ = (u′, v′) ≡ e′′ = (u′′, v′′). We want to show that e ≡ e′′. Suppose that the simple cycle c contains e and e′ and the simple cycle c′ contains e′ and e′′. See Figure 2.7. Suppose that u, u′, v′, v appear in that order around c. Let x be the first vertex in c from u to u′ that also is in c′. Similarly let y be the first vertex in c from v to v′ which also lies in c′. The vertices x and y exists, since u′ and v′ are in c′ as well. Construct a new cycle c′′ consisting of

(a) Path from x to y in c containing uv, and

(b) Path from x to y in c′ containing u′′v′′. Observe that c′′ is a simple cycle containing e and e′′, and hence e ≡ e′′.

Remark 2.4.2 Professor Danny Sleator pointed out that the above proof is not complete. Here is an excerpt below (bit modified) from his e-mail: “... you claim to prove that the edges of a bi-connected component form an equivalence relation by the equivalence defined as edge (u, v) and (u′, v′) are equivalent if there is a simple cycle through them.

The proof is not really complete, at least from my perspective. The vertices x and y as you define them may not exist. E.g. suppose that the simple cycle using (u′, v′) and (u′′, v′′) only overlaps the other cycle on the bottom path from v to v′. (At the very least there are cases you are not considering.)

Here’s how to correct the proof. Consider edges a, b, and c. Say a and b are on a simple cycle X and a and c are on a simple cycle Y. If c is also on the cycle X then we’re done because b is on that cycle too. Otherwise, follow the edges of Y from one end of c until it hits X. Call that intersection vertex p. Do the same for the other end of c, and call that intersection vertex q. Now p and q are distinct and there are two disjoint paths between them using edges of X. One of these paths contains edge b. Combing this path with the ones of Y we used to get from c to X gives us a simple cycle containing both b and c.”

Now we have an equivalence relation. What are the equivalence classes of this relation with respect to the edge set of G.

Biconnected Components The equivalence classes of the relation ≡ are the biconnected components of G.

Now we discuss critical lemmas, which relate articulation vertices, biconnected components, DFS number and Low values. These lemmas are ‘if and only if’ type - or ‘necessary and sufficient’ type. These lemmas are extremely useful (especially in computer science) since they give a complete characterization of the object/structure under consideration, and often lead to an algorithm.

Lemma 2.4.3 A vertex v is an articulation vertex if and only if it is contained in at least two distinct biconnected components.

Proof. Suppose v is an articulation vertex. Then its removal disconnects G. That means that there are two vertices a and b neighboring v so that each path between a and b goes through v. See Figure 2.8. Then the edges (a, v) and (v, b) cannot lie on a simple cycle, and hence they belong to two distinct biconnected components. This implies that v is contained in at least two distinct biconnected components.

Figure 2.8: An articulation vertex in two distinct biconnected components.

Now suppose that v is contained in two distinct biconnected components, and is adjacent to vertices a and b in these components, respectively. Then (va) ≢ (vb). Then all paths between a and b goes through v, and hence removing v disconnects G. So v is an articulation vertex.

Lemma 2.4.4 Let (uv) and (vw) be two adjacent tree edges in a DFS of G. Then (uv) ≡ (vw) if and only if there exists a back edge from some descendant of w to some ancestor of u.

Proof. Recall that the descendants of w are w and all the vertices in the subtree rooted at w. Similarly ancestors of u include u and all vertices on the path from u to the root of the DFS tree.

If there exists a backedge from some descendant w′ of w to some ancestor u′ of u, then (uv) ≡ (vw), since there is a simple cycle consisting of the tree path between u′ and w′ and the backedge w′u′. See Figure 2.9.

Suppose (uv) ≡ (vw). By definition, there is a simple cycle consisting them. The edges (uv) and (vw) must appear in this order in the cycle, as the vertex v appears exactly once on the cycle. This implies that there is an edge (actually a backedge) from some vertex w′ in the subtree rooted

Figure 2.9: Illustration of existence of a back edge

at w to some ancestor u′ of u. (This ancestor could be the vertex u or a vertex on the path u to the root of the DFS tree.)

Lemma 2.4.5 Vertex v is an articulation vertex if and only if either

(a) v is the root of the DFS tree and has more than one child.

(b) v is not the root, and for some some child w of v there is no backedge between any descendant of w (including w) and a proper ancestor of v.

Part (a) is easy to prove and part (b) follows from the previous lemma! Here is the modification to the DFS procedure to compute the biconnected components.

Figure 2.10: Illustration of biconnected components

See Figure 2.10 for an illustration for the biconnected components computed by the following search procedure. procedure SEARCH(v) begin

1. mark v as visited;

2. DSF-number[v]:=COUNT;

3. COUNT:=COUNT+1;

4. low(v):=DFS-number[v]; /* low(v) is at least equal to the DFS-number of v */

5. for each vertex w on L[v] do if w is unvisited then

(a) add (w, v) to T; /*edge (w, v) is a DFS tree edge */

(b) SEARCH(w); /* Recursive call */

(c) If low(w) ≥ DF S − number[v] then a biconnected component has been found;

(d) low(v) := min(low(v), low(w)) /*Compare the low value of v with its child w */

else if w is not the parent of v then

low(v) := min(low(v), DF S − number[w]); /* (v, w) is a back edge */

end

Why does the above algorithm compute biconnected components? Actually we will need a STACK to figure out the edges in a biconnected component! How do we do that? When a vertex w is encountered in the SEARCH procedure, put the edge (v, w) in the STACK if it is already not there. After discovering a pair (v, w) such that w is a child of v and low(w) ≥ DF S − number[v], then POP all the edges from the STACK up to and including (v, w). These edges form a biconnected component. This extra step can be accomplished in linear time as well. To prove that the above SEARCH procedure indeed computes the biconnected components, we need to argue by induction on the number of biconnected components (A problem for the next assignment).

2.5 Exercises

2.1

This problem is related to the representation of graphs. Assume that the number of edges in the graph G = (V, E) is small, i.e., it is a sparse graph. In the adjacency matrix representation of G, the normal tendency is to first initialize the matrix, requiring O(|V |^{2}) time. Is there any way we can initialize the adjacency matrix in time proportional to O(|E|) and still have O(1) adjacency test?

2.2 Provide a clear, concise, and complete proof for the correctness of the BFS algorithm.

2.3 Let G=(V,E) be a directed acyclic graph with two designated vertices, the start and the destination vertex. Write an algorithm to find a set of paths from the start vertex to the destination vertex such that (a) no vertex other than the start or the destination vertex is common to two paths. (b) no additional path can be added to the set and still satisfy the condition (a). Note that there may be many sets of paths satisfying the above conditions. You are not required to find the set with the most paths but any set satisfying the above conditions. Your algorithm should run in O(|V | + |E|) time. State the algorithm, its correctness and analyze the complexity.

2.4 Clearly describe the modifications you need to make in the SEARCH procedure to compute and output the biconnected components. Prove that your algorithm is correct, i.e. it computes all the biconnected components.

2.5 How can we find in O(|V | + |E|), whether a graph G = (V, E) is a bipartite graph (Hint: Use BFS).

2.6 An Euler circuit for an undirected graph is a path that starts and ends at the same vertex and uses each edge exactly once (vertices might be repeated). A connected, undirected graph G has an Euler circuit if and only if every vertex is of even degree. Give an O(|E|) algorithm to find an Euler circuit in G, if it exists.

2.7 Assume that you are given n positive integers, d_{1} ≥ d_{2} ≥ ⋯ ≥ d_{n}, each greater than 0. You need to design an algorithm to test whether these integers form the degrees of an n vertex simple undirected graph G = (V, E) (Think of a greedy algorithm.)

2.8 Show that in a depth-first search, if we output a left parenthesis ‘(’ when a node is accessed for the first time and output a right parenthesis ‘)’ when a node is accessed for the last time, then resulting parenthesization (or bracketing sequence) is proper. Each left ‘(’ is properly matched with each right ‘)’.

32

Minimum Spanning Trees

3.1 Minimum Spanning Trees

Let G = (V, E) be an undirected connected graph with a cost function w mapping edges to positive real numbers. A spanning tree is an undirected tree connecting all vertices of G. The cost of a spanning tree is equal to the sum of the costs of the edges in the tree. A minimum spanning tree (MST) is a spanning tree whose cost is minimum over all possible spanning trees of G. It is easy to see that a graph may have many MSTs with the same cost (e.g., consider a cycle on 4 vertices where each edge has a cost of 1; deleting any edge results in a MST, each with a cost of 3).

As in the CLRS book[13], we will describe the two main algorithms for building MSTs, Kruskal’s and Prim’s. Both of these algorithms are greedy algorithms and are based on the following generic algorithm (Algorithm 3.1). The algorithm maintains a subset of edges A, which is a subset of some MST of G.

Algorithm 3.1: Generic-MST Input: Graph G, cost function w Output: A minimum spanning tree of G

1 A ← 0 2 while A ≠ MST do 3 find a safe edge {u, v} for A 4 A ← A ∪ {u, v} 5 end 6 return A

Intuitively this algorithm is straight-forward except for two pressing questions: What is a safe edge, and how do we find one? To answer these questions, we first need a few definitions.

Cut A cut (S, V \ S) of G = (V, E) is a partition of vertices of V.

Edge crossing a cut An edge (u, v) ∈ E crosses the cut(S, V \ S) if one of its end point is in the set S and the other one in the set (V \ S).

Cut respecting A A cut (S, V \ S) respects the set A if none of the edges of A crosses the cut.

Light edge An edge which crosses the cut and which has the minimum cost of all such edges.

Theorem 3.1.1 Let A be a subset of the edges of E which is included in some MST, and let (S, V \ S) be a cut which respects A. Let (u, v) be a light edge crossing the cut (S, V \ S), then (u, v) is safe for A.

Proof.

Assume that T is a MST that includes A (similarly, you may think of A as being a subset of the edges of T, or being “the makings of” a MST). If T includes the edge (u, v) then (u, v) is safe for A. If T does not include (u, v), then we will show that there is another MST, T′, that includes A ∪ {(u, v)}, and this will prove that {(u, v)} is safe for A. Since T is a spanning tree, there is a path, say P_{T} (u, v), from the vertex u to vertex v in T. By inserting the edge (u, v) in T we create a cycle. Since u and v are on different sides of the cut, there is at least one edge (x, y) ∈ P_{T} (u, v) that crosses the cut(S, T \ S). Moreover (x, y) ∉ A, since the cut respects A. But the cost of the edge (x, y) is at least the cost of the edge (u, v), since edge (u, v) is a light edge crossing the cut. Construct a new tree T′ from T by deleting the edge (x, y) in T and inserting the edge (u, v). Observe that the cost of the tree T′ is at most the cost of the tree T since the cost of (x, y) is at least the cost of (u, v). Moreover A ∪ {(u, v)} ⊂ T′ and (x, y) ∉ A, hence edge (u, v) is safe for A.

The above theorem leads to the following corollary, where we fix a particular cut (i.e. the cut(C, V \ C)).

Figure 3.1: An example of Corollary 3.1.2. The edge (u, v) connects C to some other component of G_{A} and is a light edge; it is therefore safe to add to the MST.

Corollary 3.1.2 Let A ⊂ E be included in some MST. Consider the forest consisting of G_{A} = (V, A), i.e., the graph with the same vertex set as G but restricted to the edges in A. Let C = (V_{C}, E_{C}) be a connected component of G_{A}. Let (u, v) be a light edge connecting C to another connected component in G_{A}, then (u, v) is safe for A (See Figure 3.1).

3.2 Kruskal’s Algorithm for MST

Proposed by Kruskal in 1956, this algorithm follows directly from Corollary 3.1.2. Here are the main steps. To begin with the set A consists of only isolated vertices, and no edges (so, |V | “connected” components in all).

1. Sort the edges of E in non-decreasing order with respect to their cost.

2. Examine the edges in order; if the edge joins two components then add that edge (a safe edge) to A.

To implement Step 2, we do the following. Let e_{i} be the edge under consideration, implying that all edges with a lesser cost than e_{i} = (a, b) have already been considered. We need to check whether the endpoints a and b are within the same component or whether they join two different components. If the endpoints are within the same component, then we discard the edge e_{i}. Otherwise, since it is the next lightest edge overall, it must be the lightest edge between some pair of connected components, and so we know from Corollary 3.1.2 that it is safe to add to A. We will need to merge these two components to form a bigger component.

To accomplish all of this, we will need some data structure which supports the following operations:

• Make-Set(v) - create a new set containing only the vertex v.

• Find(v) - Find the set which presently contains the vertex v.

• Merge(V_{x}, V_{y}) - Merge the two sets V_{x} and V_{y} together such that Find will work correctly for all vertices in merged set.

We can implement this data structure as follows. For each vertex we keep track of which component it lies in using a label associated with the vertex. Initially each vertex belongs to its own component, which is done with Make-Set. During the algorithm the components will be merged, and the labels of the vertices will be updated. Assume that we need to merge the two components V_{a} and V_{b} corresponding to the end points a and b of the edge e_{i} = (a, b). We use Find(a) and Find(b) to get the sets V_{a} and V_{b} respectively. We then call Merge which will relabel all of the vertices in one of the components to have the same labels as the vertices of the other. The component which we relabel will be the one which is smaller in size. Given such a data structure, we can implement Kruskal’s algorithm as in Algorithm 3.2.

Let us analyze the complexity of Kruskal’s algorithm. Sorting the edges takes O(|E| log |E|) time. The test for an edge, whether it joins two connected components or not, can be done in constant time. (In all O(E) time for all edges.) What remains is to analyze the complexity of merging the components which can be bounded by the total complexity of relabeling the vertices. Consider a particular vertex v, and let us estimate the maximum number of times this will be relabeled. Notice that the vertex gets relabeled only if it is in a smaller component and its component is merged with a larger one. Hence after merging, the size of the component containing v becomes at least double. Since the maximum size of a component is |V |, this implies that v can be relabeled at most log_{2} |V | times. Therefore, the total complexity of the Step 2 of the algorithm is O(|E| + |V | log |V |) time. These results are summarized in the following theorem.

Theorem 3.2.1 (Kruskal) A minimum (cost) spanning tree of an undirected connected graph G = (V, E) can be computed in O(|V | log |V | + |E| log |E|) time.

Algorithm 3.2: Kruskal-MST Input: Graph G = (V, E), cost function w Output: A minimum spanning tree of G

1 A ← ∅ 2 foreach v ∈ V do 3 MakeSet(v) 4 end 5 sort the edges of E in non-decreasing order w.r.t. w 6 foreach e = {a, b} ∈ E, where e is taken in sorted order do 7 V_{a} ← Find(a) 8 V_{b} ← Find(b) 9 if V_{a} ≠ V_{b} then

{10  A ← A ∪ \{e\} \atop 11  Merge(V_{a}, V_{b})}

12 end 13 end 14 return A

3.3 Prim’s MST algorithm

Prim’s algorithm is very similar to Dijkstra’s single source shortest path algorithm[17], and, in fact, their complexity analysis will be the same. Here the set A at any stage of the algorithm forms a tree, rather than a forest of connected components as in Kruskal’s. Initially the set A consists of just one vertex. In each stage, a light edge is added to the tree connecting A to a vertex in V \ A.

The key to Prim’s algorithm is in selecting that next light edge efficiently at each iteration. For each v ∈ V \ A, we keep track of the least cost edge which connects v to A, and the cost of this edge is used as the “key” value of v. These key values are then used to build a priority queue Q. See Figure 3.2 for an example of these sort of light edges.

Figure 3.2: Example of Prim’s Algorithm showing the set A (encircled) and the least cost edges associated with each vertex in V \ A.

In each step of the algorithm, the vertex v with the least priority is extracted out of Q. Suppose that corresponds to the edge e = {u, v}, where u ∈ A, then observe that e is a safe edge since it is the light edge for cut(A, V \ A). We update A := A ∪ {e}. Finally, after extracting v out of Q, we need to update Q. The details are explained in Algorithm 3.3.

Algorithm 3.3: PRIM-MST Input: Graph G = (V, E), cost function w, root vertex r Output: A minimum spanning tree of G

1 foreach v ∈ V do 2 key(v) ← ∞

 3  π(v) ← nil /* π keeps track of the parent of a vertex in the tree.        */

4 end 5 key(r) ← 0

 6 Q ← V /* Priority queue consists of vertices with their key values         */

7 while Q ≠ ∅ do 8 u ← Extract-Min(Q) 9 foreach v adjacent to u do 10 if v ∈ Q and w(u, v) < key(v) then

{11  π(v) ← u \atop 12  key(v) ← w(u, v)}

13 end 14 end 15 end

The vertices that are in the set A at any stage of the algorithm are the vertices in V \ Q, i.e., the ones that are not in Q. kev(v) is the weight of the light edge {v, π(v)} connecting v to some vertex in the MST A. Notice that the key value for any vertex starts at infinity, when it is not adjacent to A via any edge, and then keeps decreasing.

Let us analyze the complexity of the algorithm. The main steps are the priority queue operations, namely decrease-key and extract-min. We perform |V | extract-min operations in all, one for each vertex. We also perform O(|E|) decrease-key operations, one for each edge. The following table shows the complexity of these operations depending on the type of priority queue you choose. These complexities are per operation, although the complexities of Fibonacci Heaps are amortized (kind of an average over the worst possible scenario! - more on that later).

Binary Heaps Fibonacci Heaps
Extract-min O(log n)  O(log n)
Decrease-key O(log n)  O(1)

3.4 Randomized Algorithms for Minimum Spanning Trees

Here we discuss some results related to randomized algorithms for computing minimum spanning trees. These results are based on Section 10.3 of Raghavan and Motwani’s book on Randomized Algorithms [56] and T. Chan’s simplified analysis from [11]. Assume that all edge weights are distinct and hence there is a unique MST in the given graph G = (V, E). The randomized algorithm uses a few concepts which are discussed in the following subsections followed by the actual algorithm itself in Section 3.4.3. The first concept is an algorithm due to Boruvka from 1926[7, 8] which helps us to reduce the number of vertices in the graph. The second is about heavy and light edges with respect to a spanning tree.

3.4.1 Boruvka’s Algorithm

Observe that for any vertex v ∈ V, the edge, say {v, w}, with the minimum weight incident to that vertex will be included in the MST, as per Corollary 3.1.2. This leads to a simple way to compute the MST as given in Algorithm 3.4.

Algorithm 3.4: Boruvka-MST Input: Graph G, cost function w (all costs distinct) Output: A minimum spanning tree T of G

1 T ← ∅

   // Each iteration of this loop is called a Phase

2 while G contains more than a single vertex do 3 Mark the edges with the minimum weight incident to each vertex. Add these edges to T 4 Identify the connected components of the marked edges 5 Replace each connected component by a vertex 6 Eliminate all self loops. Eliminate all multiple edges between a pair of vertices, except

the edge with the minimum weight 7 end 8 return T

A few observations about this algorithm:

• Let G′ be the graph obtained from G after contracting the edges in a single phase of the algorithm. Then the MST of G is the union of the contracted edges from that phase and the edges in the MST of G′.

• In each contraction phase, the number of vertices in the graph is reduced by at least a half. Hence there will only be O(log |V |) phases in all.

• Each phase can be implemented in O(|V | + |E|) time and so we obtain yet another MST finding algorithm. Total running time for this algorithm is O(|E| log |V |).

3.4.2 Heavy Edges

We’ve already seen the definition of a light edge. Now we examine edges which are not light.

Let F be any spanning tree of G (in particular, F may or may not be a minimum spanning tree). Consider any two vertices, say u and v, of G and there is a unique path P (u, v) between them in F. Let w_{F} (u, v) be the edge with the maximum weight along this path.

We say an edge {u, v} is F -heavy if the weight of this edge is larger than the weight of each of the edges on the unique path between u and v in F. More formally, we define an edge {u, v} ∈ E to be F -heavy if w(u, v) > w_{F} (u, v), otherwise it is F -light. Observe that by this definition, all edges of F are F -light (every edge of a path in F is at most as heavy as the heaviest edge in that path of F).

Lemma 3.4.1 Let F be a spanning tree of G which is not necessarily minimum. If an edge of G is F -heavy then it does not lie in the Minimum Spanning Tree of G.

Proof. It is left for you to prove it formally. The proof proceeds by contradiction. Assume that the edge e = {u, v} ∈ E is F -heavy, that T is an MST of G, and that e ∈ T (so, we are talking about two trees here, T, which is known to be an MST, and F, which may be one as well). Consider the edges on the path P (u, v) in F. Add all these edges to T and remove e from T, to obtain a graph G′ (that is, G′ = T ∪ P (u, v) \ {e}, but since it may have some cycles, we cannot call it a tree). G′ is still connected. Remove edges from P (u, v) one-by-one from G′ until G′ is once again a tree. Observe that this new tree is still a spanning tree, but it must have weight lower than that of T, contradicting the minimality of T.

It is very important to note that because of the way F -light is defined, an edge which is F -light may or may not be in the MST. For example, if we construct F such that the heaviest edge of E is in F, that edge will be counted as F -light, even though it may not be present in any minimum spanning tree.

3.4.3 Randomized Algorithm

Algorithm 3.5 gives the main steps of the randomized algorithm we have been discussing. The analysis is based on Timothy Chan’s paper[11].

Algorithm 3.5: Randomized-MST Input: Connected Graph G = (V, E) with distinct edge weights Output: A minimum spanning tree T

1 Execute 3 phases of Boruvka’s algorithm (reduces number of vertices). Let the resulting graph be G_{1} = (V_{1}, E_{1}), where |V_{1}| ≤ |V |/8 and |E_{1}| ≤ |E|. Let C be the set of contracted edges. (Running time: O(|V | + |E|))

2 Random Sampling: Choose each edge in E_{1} with probability p = 1/2 to form the set E_{2} and

obtain the sampled graph G_{2} = (V_{2} = V_{1}, E_{2}). 3 Compute Recursively the Minimum Spanning Tree of G_{2}, and let it be F_{2}. (T (|V |/8, |E|/2)) 4 Verification: Compute the set of F_{2}-light edges in E_{1}, and let this set be E_{3}. (O(|V_{1}| + |E_{1}|)

time) 5 Final MST: Compute MST, F_{3}, of the graph G_{3} = (V_{3} = V_{1}, E_{3}). (T (|V |/8, |V |/4)) 6 return MST of G as C ∪ F_{3}.

Theorem 3.4.2 Algorithm 3.5 correctly computes the MST of G in O(|V | + |E|) time.

Proof. The correctness of the algorithm is straightforward. To estimate the complexity, the crux is in estimating the size of the set E_{3}, i.e., the size of the set of F_{2}-light edges in E_{1}. We will prove in the Sampling Lemma (Lemma 3.4.4) that for a random subset R ⊂ E, the expected number of edges that are light with respect to MST(R) is at most (|E| · |V_{1}|)/|R|. In our case, the expected value of |R| = |E_{1}|/2 ≤ |E|/2, and hence the expected number of F_{2}-light edges will be at most 2|V_{1}| ≤ |V |/4. Hence the running time of this algorithm is given by the recurrence

T (|V |, |E|) = O(|V | + |E|) + T (|V |/8, |E|/2) + T (|V |/8, |V |/4),

which magically solves to O(|V | + |E|).

Before we describe the Sampling Lemma here are some technicalities. Consider that we are sampling the edges of the graph G = (V, E), and the sampled edges form the subgraph R.

We use the notation R for both the set of edges as well as the sampled graph. Since we are sampling the edges, it is possible that the sampled graph R of the graph G is not connected, and hence there will not be any minimum spanning tree. To ensure connectedness, we will fix any spanning tree T_{0} of G, consisting of |V | − 1 edges and we will consider the minimum spanning tree of R ∪ T_{0}, denoted as MST (R).

Lemma 3.4.3 (Observation about light edges) An edge e ∈ E is light with respect to MST (R) if and only if e ∈ MST (R ∪ {e}).

Proof. If e = (u, v) is light then there is some edge e′ on the unique path between u and v in MST (R) such that its weight, w(e′) = w_{MST (R)}(u, v), and hence e can be added to MST (R) and e′ can be removed to obtain MST of R ∪ {e}. Therefore e ∈ MST (R ∪ {e}).

Now suppose e ∈ MST (R ∪ {e}). We need to show that e is light with respect to MST (R). Since e is part of the MST, by definition it is light with respect to that MST.

Lemma 3.4.4 (Sampling Lemma) For a graph G = (V, E) and a random subset R ⊂ E, of edges, the expected number of edges that are light with respect to MST (R) is at most (|E|·|V |)/|R|.

Proof. Pick a random edge e ∈ E (this choice is independent of the edges in R). We will prove that e is light with respect to MST (R) with probability at most |V |/|R|. From Lemma 3.4.3 we see that this is equivalent to finding the bound on the probability that e ∈ MST (R ∪ {e}). Let R′ = R∪{e}. We will use a technique called backward analysis. First we analyze the probability on a fixed set R′, and then we will show that the expression obtained is not dependent on the elements of R′, but just the cardinality, and hence the probability holds unconditionally as well.

Instead of adding a random edge to R, we will think of deleting a random edge from R′. This is an easier proposition since we know the elements of R′, having just fixed it. MST (R′) has |V | − 1 edges, and e is a random edge of R′, hence the probability that e is an edge from MST (R′), given a fixed choice of R′, is (|V | − 1)/|R′| ≤ |V |/|R|. This bound is independent upon the choice of the set R′, and holds unconditionally as well.

3.5 MST Verification

1

If I give you any tree F derived from a graph G = (V, E), can you identify whether F is a minimum spanning tree of G?

A trivial method for determining this would be to run a known-correct algorithm such as Kruskal’s or Prim’s on G and compare the output to F, however there are two main drawbacks to this approach:

1. It is too slow 2. The MST may not be unique, making a direct comparison difficult.

1This section is contributed by Gregory Bint.

What we would like is to be able to calculate this in linear time with respect to the graph. The following lemma has been shown to be very useful in this respect, and it is used by virtually every MST verification algorithm.

Lemma 3.5.1 Let F be a spanning tree of G, then F is a minimum spanning tree of G if and only if every edge in E \ F is F -heavy.

Proof. Let P (u, v) be the unique tree path between u and v in F, and let w_{F} (u, v) be the weight of the heaviest edge along that path. w(e) or w(u, v) is the weight of the edge e having endpoints u and v.

We first show that if F is a MST, then every edge in E \ F is F -heavy. Let e be any edge in E \ F with endpoints u and v and assume that e is F -light. Note that P (u, v) ∪ {e} is a cycle. Let e′ = {x, y} be the edge corresponding to w_{F} (u, v). Since e is F -light, w(e′) > w(e), meaning we could replace e′ by e in F to obtain a lighter tree overall, contradicting the minimality of F.

For the other half of the proof, we show that if every edge in E \ F is F -heavy, then F is a MST of G. Again, we proceed by contradiction. Suppose that F is not a MST of G, then we should be able to lower the weight of the tree by replacing some edges in F with those from E \ F. But, for every einE \ F with endpoints u and v, we have that w(u, v) > w_{F} (u, v), so exchanging e for any other edge in P (u, v) will increase the weight of F.

Given the above lemma, a natural idea for an algorithm would be to try to classify every edge in the graph, and then check if each non-tree edge is in fact F -heavy. This turns out to be something which is possible: given a graph G = (V, E) and a tree F, we can partition the edges of G in two sets, the set of heavy edges and the set of light edges, with respect to F in O(|V | + |E|) time.

We will see that many of the algorithms for doing so fairly complex, although there has been recent progress in simplifying it somewhat.

3.5.1 Overview of verification algorithms

Here we look at a brief history of the literature on MST Verification. As hinted at above, every single one of the following methods uses Lemma 3.5.1 as its underpinning. This problem can also be restated as the following:

Problem 3.5.2 (The Tree Path Maxima Problem) Let F be a spanning tree of G, then we want to identify the cost of the heaviest edge along each tree path P (u, v).

Given an answer to Problem 3.5.2, we can perform a simple linear scan through the the edges of G \ F. For each edge e ∈ G \ F, we compare the cost of e with the tree path of its endpoints. If every edge e is heavier than its corresponding tree path maxima, then F is a MST of G. We look at this sort of translation of the problem in more detail in Section 3.5.6.

Here is a timeline of some results:

• In 1979, Tarjan introduced a method which uses path compression[65] of trees to achieve a near-linear time of O(mα(m, n)) where α is the Inverse Ackermann function.

• In 1984, Komlós’s provided an algorithm[44, 45] which showed that only a linear number of comparisons of the edge costs would be sufficient to solve the problem, however the algorithm itself has significantly more than linear overhead.

• In 1992, Dixon et al.[18] combine methods from Tarjan’s 1979 algorithm and Komlós’s 1984 algorithm to produce the first linear time MST verification algorithm. The problem is divided into one large problem and several small problems, with the larger being attacked with path compression, and the smaller with a lookup scheme which is bounded in size by Komlós’s algorithm.

• In 1994, Karger et al. present an algorithm for computing the MST of a graph in expected linear time.[38, 41] While not a verification algorithm in itself, its output could be useful to help verify another tree, or to take the place of the other tree altogether (e.g., why bother verifying a potential MST in linear time when you can just create one!)

• In 1995, King produced another linear time MST verification method[40, 39] which is a great deal simpler than that of Dixon et al.. In this method, Boruvka’s algorithm is used to reduce a general tree down to one which can be handled entirely by the full branching tree base case of Komlós’s algorithm, which is simpler than his algorithm for general trees.

• In 2010, Hagerup simplifies King’s method[26] even further and provides an implementation in the D programming language. Like King’s method, Hagerup continues to use Komlós’s full branching tree case, but eschews complex edge encoding schemes in favour of a richer logical data type.

We will walk through parts of Komlós’s algorithm, Dixon et al.’s algorithm, King’s algorithm, and finally Hagerup’s algorithm as we piece together the tools needed for a reasonably simple approach to solving this problem.

3.5.2 Komlós’s Algorithm

In 1984, Komlós[44, 45] gives an algorithm of sorts which can solve Problem 3.5.2 in O(n + m) comparisons. He does not provide an implementable algorithm, however, and there are other factors of overhead in his method which would drive up the actual cost of a straight implementation. Nevertheless, this method of breaking down the problem is built upon by later papers, notably Dixon et al. in 1992, and King, which we cover in Section 3.5.4.

Komlós begins by considering two special cases of spanning trees. In each case, we consider F to be a directed tree with edges oriented away from the root. Additionally, we shift the edge costs down to their lower endpoint vertices, as this simplifies the conceptual model.

The first case occurs when the tree is a path. For the path, we construct a symmetric order heap, H, which a tree with both the binary search property on the ordering determined by the path, and the (maximum) heap property determined by the vertex costs. The root of H represents the heaviest vertex, and the heaviest vertex of any path from u to v is found at LCA(u, v). Determining the LCA of two vertices in a tree can be accomplished in several ways and is covered elsewhere in these notes, but Komlós cites Harel[30] specifically.

The second case is somewhat more interesting and is concerned with processing full branching trees. Not to be confused with full binary trees, a full branching tree is defined as one where every leaf is at the same level, and every internal vertex has at least 2 children. Let F be our full branching tree with root r and all edges directed away from r.

We need to calculate the maximum cost edge of every path through F. Given a vertex y, let A(y) be the set of all paths through F which contain y. That is A(y) = {P (x, z)|x ≧ y ≧ z} where u ≧ v denotes that u is a predecessor of v (or, u may equal v). Since F is directed away from the root, this means that u is at least as close to the root than v. Note that if F was not directed, then it might be possible for x ≧ y ≧ z to hold even though y ∉ P (x, y). Given A(y), let A^{∗}(y) be the set of all paths through y, but restricted to just the interval [r, y]; that is, just the subpath from the root down to y.

We process F one level at a time, starting from the root, finding the maximum weights of all paths in the sets A^{∗}(y). To do this, assume that we have calculated the maximum costs in all such paths up to level i, and that we are now trying to process some vertex y on level i + 1. Let ȳ be the parent of y. Since ȳ resides on level i, we know the maximum cost of all paths in A^{∗}(ȳ).

The key observation to make here is the following.

Property 3.5.3 Consider two paths P (x, ȳ) and P (x′, ȳ). If x is a predecessor of x′ then the maximum cost in P (x, ȳ) is at least as large as the maximum cost in P (x′, ȳ) (since, under these conditions, P (x′, ȳ) is a subpath of P (x, ȳ)).

In A^{∗}(ȳ), the shortest path is P (ȳ, ȳ) while the longest is P (r, ȳ). By the above property, the maximum costs form a non-decreasing sequence with respect to the length of the path. That is, we can order the maximum costs by considering the path length. This observation about the ordering helps us while building A^{∗}(y), as we can use a binary search insertion of f(y) to compare f(y) against all path cost maximums in A^{∗}(ȳ) simultaneously.

By now you should be asking “How are all these sets like A(y) and A^{∗}(ȳ) created, copied, and updated?” As far as Komlós’s paper is concerned, the answer is “slowly”. Essentially what Komlós shows us is that a linear number of comparisons are sufficient to determine maximum path costs, however finding those comparisons is left open.

The remainder of Komlós’s paper details how these two primitive cases can be applied to any general tree, however this method is fairly complex, and as he states, results in too much overhead. No implementable algorithm is given in this paper.

3.5.3 Dixon et al.’s Technique

This technique has the distinction of being the first to achieve a linear running time, requiring O(m) time on a graph with n vertices and m edges. The underlying process is fairly complex, however, and involves first preprocessing the graph into a suitable form.

The preprocessing itself is interesting as it shows a method of massaging a graph into a more attractive form for the problem at hand without affecting anything about the spanning tree that we wish to verify. The preprocessing involves the following steps.

For a given graph G = (V, E) and spanning tree F, not necessarily minimum, we choose an arbitrary vertex r to be the root of F. Now consider any non-tree edge {v, w} with cost c(v, w) and lowest common ancestor u. If u is not one of v or w, then this implies that v and w are not related; that is, the v is neither ancestor nor descendant of w. In such a case, we replace the edge {v, w} by {v, u} and {u, w}, each with cost c(v, w). F is unchanged by this process and, more importantly, the maximum weight along P (v, w) is also unchanged, which preserves the current minimality of F. There are several linear time algorithms for finding lowest common ancestors in a tree (e.g., Harel & Tarjan, 1984[29] or Schieber & Vishkin, 1988[61]).

Taken over the entire graph, this will at most double the number of non-tree edges. When completed, every non-tree edge in F is a backedge.

In the second stage of preprocessing, we will mark several vertices. We can imagine these marks as subdividing the tree into edge-disjoint subtrees where a marked vertex represents a “root”, and any marked descendants are ignored.

Figure 3.3: An example tree with marked vertices. The marked subtree defined by u is enclosed in red.

The choice of which vertices to mark is based on subtree size. Using a post-order traversal, for each vertex v we calculate h = 1 + ∑{s(w)|w is a child of v}. Let g be a small integer, then if h ≤ g we assign s(v) := h, otherwise s(v) := 1 and v becomes marked. We will look more at the specific choice of g later. Note the following important properties resulting from this process:

1. The number of marked vertices, and hence the number of subtrees, is at most (n − 1)/g + 1.

2. Considering any subtree, if its root (marked vertex) is deleted, along with incident edges, we get a collection of disjoint trees, each with size at most g. We call each of these a microtree.

Following that process, r is also marked, although it will probably not have Property 2.

A final phase of edge replacements will ensure that all backedges either span two vertices belonging to the same microtree, or span between microtrees and marked vertices only (i.e., the edge {u, v} in Figure 3.3 will be replaced).

To accomplish this, we first build the tree F′ whose vertex set consists of all of the marked vertices of F, and where, for two vertices s and t in F′, s is the parent of t (i.e, there is a tree edge between them) if s is the first marked vertex that we encounter when walking from t to r. We call T′ the macrotree.

We can now eliminate the “long” edges, like {u, v}, by doing the following. Let p(v) be the nearest marked vertex to v which is a proper ancestor of v. Note that if v is marked then p(v) ≠ v. We also assume that p(r) is undefined (but it won’t be needed anyway). We can calculate p(v) for the entire tree using a depth-first search. For every non-tree edge {u, v}, assume w.l.o.g. that u is an ancestor of v and find p(u) and p(v). If p(u) = p(v), then u and v are part of the same microtree (recall that the root of a microtree is not marked).

Otherwise, if p(u) ≠ p(v), then we know that there is at least one marked vertex between them. Let r_{1} = u if u is marked, or r_{1} = p(u) if u is not marked. Similarly, let r_{3} = v if v is marked, or r_{3} = p(v) if v is not marked. Let r_{2} be the child of r_{1} in F′ (note: F′, not F). We then replace {u, v} by {u, r_{2}}, {r_{2}, r_{3}}, and {r_{3}, v}, skipping any edge that creates either a self-loop or which duplicates a tree edge. For edges which we did not skip, assign the cost c(u, v). As in the first phase of edge replacements, assigning this cost preserves the current minimality of F.

With this preprocessing finished, we have now divided the problem into one large tree rooted at r with several microtrees around the periphery. The authors complete the process by using Tarjan’s Path Compression on the large tree.

The microtrees are processed in a very different way. Essentially, the authors precalculate all possible minimum spanning trees on graphs containing at most g vertices. Leveraging Komlós’s result, they show that for any such input, the corresponding decision tree for comparing edges and determining minimality is not too big. The choice of g is such that the total size of these precalculations is only O(n), which places g in the neighbourhood of O(log log n) [39].

3.5.4 King’s Method

Presented by King[40, 39] in 1995, this method is not the first MST verification algorithm to achieve linear time (that falls to Dixon et al.[18]), however it is quite a bit simpler. King’s method uses Boruvka’s algorithm in a clever way to change any input tree into a full binary tree, which can then be entirely processed by the appropriate case presented by Komlós. This method requires linear time and space in the unit-cost RAM model with Θ(log n) word size.

Boruvka Tree Property

The first step is to take our input tree F and convert it to a full binary tree. This is accomplished by running Boruvka’s algorithm on the tree F (we usually would run Boruvka’s on an entire graph, but not in this case). As Boruvka’s runs on F, we can build a new tree B which represents the execution of the algorithm on F, rather than a modification of F itself.

Algorithm 3.6 details the construction of B. In the first step, a leaf is added to B for each vertex of F, so we already know that |B| ≥ |F |. In fact, B will have at most twice as many vertices as F when we are finished. The algorithm proceeds by colouring the vertices and edges to represent subtrees within F, such that any vertices connected along a coloured (blue) path is considered part of the same subtree.

Figure 3.4: An example of the Full Binary Tree construction given by Algorithm 3.6. F is shown on the left and B on the right. Edge weights are not shown, so imagine that each tree chooses its minimum weight edge at each step.

Refer to Figure 3.4 for an example of the algorithm’s execution. Note the following important properties which ensure that B is a full branching tree.

1. In each step of Loop 1, an edge joins two blue trees into one.

2. In each phase of the while loop, every blue tree is combined by some edge with another blue tree. Thus, from every level of B, every vertex has a parent in the next level.

For every v in F there is a vertex f(v) in B, and by construction we also have that for every path F (x, y) there is a path B(f(x), f(y)). However, to show that there is any meaningful correspondence between these paths beyond their existence, we need the following lemma, presented as Theorem 1 in King’s paper.

Lemma 3.5.4 Let F be any spanning tree and let B be the tree constructed by Algorithm 3.6. For any pair of vertices x, y ∈ F, the cost of the heaviest edge in F (x, y) equals the cost of the heaviest edge in B(f(x), f(y)).

Algorithm 3.6: FullBranchingTree Input: A spanning tree F = (V, E) with distinct edge weights Output: A full branching tree B satisfying Lemma 3.5.4

1 Initialize B as an empty tree 2 foreach vertex v of V do 3 Colour v blue, considering it as a singleton tree 4 Add the leaf f(v) to B 5 end

6 while there is more than one blue tree do

     // Loop ``1'', joins blue trees together

7 foreach blue tree a do 8 Select a minimum cost edge e incident to a and colour it blue 9 end

     // Loop ``2'', updates B

10 foreach new blue tree t do 11 Add f(t) to B 12 Let A be the set of trees joined into t in Loop 1 13 Add an edge {f(t), f(a)} for each a ∈ A 14 Set the cost of {f(t), f(a)} to that of edge selected by a in Loop 1 (i.e., e) 15 end 16 end 17 return B Proof. Let the cost of an edge e be denoted by w(e). For every edge e ∈ B(f(x), f(y)), we will show that there is an edge e′ ∈ F (x, y) such that w(e′) ≥ w(e).

Suppose that e = {a, b} such that a is the endpoint of e which is farthest from the root. As a is in B, a = f(t) for some blue tree t, and t must contain either x or y, but not both. Similarly, b = f(t′) which is new blue tree consisting of f(t) and others from the previous phase of the algorithm. Since e ∈ B, e was selected by t.

Let e′ be the edge in F (x, y) with exactly one endpoint in t. Since e′ is adjacent to t, t would have considered e′. Since t ultimately chose e, it must be that w(e′) ≥ w(e) since t chooses the edge with minimum cost.

To finish the proof we also need to show the following: The cost of the heaviest edge in F (x, y) is the cost of the heaviest edge in B(f(x), f(y)). Let e be the heaviest edge in F (x, y) (for simplicity, assume that there is a unique such edge). If e is ever selected by a blue tree which contains either x or y, then B(f(x), f(y)) contains an edge with the same weight.

Otherwise, assume that e is selected by some other blue tree t′ not containing x or y. We know that e is on the path from x to y in F, so t′ contained at least one intermediate vertex on that path. But since F is a tree, if it contains an intermediate vertex of F (x, y), it must be incident to at least two edges of F (x, y). By our assumption, e is the heaviest edge on this path, so t′ would have selected the other edge, giving a contradiction.

The intuition with the last part of the above proof is that, since e is the heaviest edge along F (x, y), any blue tree which includes part of that path, but which does not yet include x or y always has another edge to select which brings it “closer” to x or y.

King’s algorithm now continues with B rather than F, which maintains the path maximum cost property for each path in F, implying that if Lemma 3.5.1 holds for B it will also hold for F.

The remainder of King’s paper shows a bit-wise labeling scheme from for the vertices and edges of B which exploits Property 3.5.3 of the full binary tree case presented by Komlós. We will now jump to Hagerup’s method to conclude our verification method, which he wrote specifically to simplify away from this labeling scheme, but which otherwise picks up at exactly this point of the algorithm.

3.5.5 Hagerup’s Method

Hagerup presents an algorithm for solving the Tree Path Maxima problem (TPM) rather than MST Verification, per se, but as we have mentioned, a solution to TPM implies a solution to MST Verification. A sketch of such a translation is given in the next section.

The input to Hagerup’s method assumes that we are given a tree on n vertices and a list of pairs (u_{1}, v_{1}), …, (u_{m}, v_{m}) such that in each pair u_{i} is a proper ancestor of v_{i}. At most, this list would describe the endpoints of every root to leaf path in B, and every subpath of such a root to leaf path. Any subset is also permissible. In practice, we choose a subset equivalent to the non-tree edges of G, the graph containing the spanning tree F we are trying to verify.

The basic algorithm involves collecting several types of information about each vertex. For every vertex u in B, we store the depth d(u), and, if u is not the root r, we let w(u) represent the cost of the edge from u to its parent. For each u we also build the following set:

D_{u} = {d(u_{i})|u_{i} is a proper ancestor of u and v_{i} is a descendant of u}

Simply put, D_{u} stores the set of depths corresponding to proper ancestors of u such that u is in the subpath represented by some pair (u_{i}, v_{i}) from the input.

We would also like to create the set M_{u} for each u, which stores a subset of the ancestors of u indicated by D_{u}. The choice of which ones are stored again exploits Property 3.5.3.

Consider any two successive ancestors d and d′ of u which are indicated by D_{u} such that d is closer to the root, and d′ is closer to u. Then d ∈ M_{u} if the path maximum cost of the path d → u is greater than that of d′ → u. Put another way, we store only those ancestors of u where there is an actual increase in path maximum cost between it and the previous (closer) ancestor.

This can still work out to be a lot of entries, however, and a lot of copying between vertices, which breaks linear time. Fortunately, Hagerup was able to find an alternate, yet equivalent set representation which does satisfy our needs, and our desired running time, using the set infix operator. The details of this operator and its equivalence to M_{u} take a few pages to discuss and can be found in his paper.

3.5.6 Putting it all together

One way of applying all of the tools we have seen so far to build a complete MST Verification algorithm is as follows.

Taking a graph G = (V, E) with spanning tree F, let U = E \F be the set of non-tree edges. We use King’s method of using Boruvka’s method to convert F to the full branching tree B. Translate U onto B so that ∀e = {x, y} ∈ U we create e′ = {f(x), f(y)} and call the resulting graph G′. We next apply Dixon et al.’s first preprocessing step to G′ to replace all cross edges with back edges. Let U′ be the set of non-tree edges in G′ after all of this.

The set U′ corresponds to the pairs (u_{i}, v_{i}) that we need to input into Hagerup’s algorithm. After that algorithm has run, MST Verification is completed by examining every non-tree edge in G, translating it to the equivalent one or two edges in U′, querying B, and determining whether the non-tree edge is costlier than the tree path maximum.

The extra steps required to find U, translate it to B, and then find U′ all take time linear in the number of edges.

3.6 Bibliographic Notes

Kruskal’s algorithm, presented in Section 3.2 makes use of a data structure known as Union-Find or Disjoint-Set. A near-linear time implementation was first described by Tarjan[64].

Boruvka’s algorithm is quite old[7, 8], and not originally published in English. Neetil et al. published a translation from the original Czech in 2001 along with some comments.[57]

Komlós mentions that his method of using symmetric order heaps for processing paths is something of a well-known method by the time he covers it in his own paper. However, he was unable to find a reference to it in any other literature, which is why he took the time to write about it.

The way that King uses Boruvka’s algorithm is first described by Tarjan in 1983[63].

3.7 Exercises

3.1 Let S=(V,T) be a minimum cost spanning tree, where |V | = n + 1. Let c_{1} ≤ c_{2} ≤ ... ≤ c_{n} be the costs of the edges in T. Let S’ be an arbitrary spanning tree with edge costs d_{1} ≤ d_{2} ≤ .... ≤ d_{n}.

Show that c_{i} ≤ d_{i}, for 1 ≤ i ≤ n.

3.2 Assume all edges in a graph G have distinct cost. Show that the edge with the maximum cost in any cycle in G cannot be in the Minimum Spanning Tree of G. Can you use this to design an algorithm for computing MST of G by deletion of edges, and what will be its complexity?

3.3 Recall that Dijkstra’s SSSP algorithm was for directed (or undirected) graphs where the weights of the edges are positive and we need to compute shortest paths from the source vertex to all other vertices in the graph. What happens when some of the edges have negative weights. Try to consider the cases where the algorithm will fail and where the algorithm will still work.

3.4 Design an efficient algorithm to find a spanning tree of a connected, (positive) weighted, undirected graph G = (V, E), such that the weight of the maximum-weight edge in the spanning tree is minimized (Justify your answer).

3.5 Let G = (V, E) be a weighted directed graph, where the weight of each edge is a positive integer and is bounded by a number X. Show how shortest paths from a given source vertex s to all vertices of G can be computed in O(X|V | + |E|) time (Justify your answer).

3.6 Prove that if all edge weights are distinct then the minimum spanning tree of a simple undirected graph is unique.

3.7 Provide a formal proof of Lemma 3.4.1.

3.8 Suppose all edge weights are positive integers in the range 1..|V | in a connected graph G = (V, E). Devise an algorithm for computing Minimum Spanning Tree of G whose running time is better than that of Kruskal’s or Prim’s algorithm.

3.9 Consider a connected graph G = (V, E) where each edge has a non-zero weight. Furthermore assume that all edge weights are distinct. Show that for each vertex v ∈ V, the edge incident to v with minimum weight belongs to a Minimum Spanning Tree.

(Bonus Problem: Can you use this to devise an algorithm for MST - the above step identifies at least |V |/2 edges in MST - you can collapse these edges (by identifying the vertices and then recursively apply the same technique - the graph in the next step has at most half of the vertices that you started with - and so on!)

3.10 Prove that the distance values extracted from the priority queue over the entire execution of Dijkstra’s single source shortest path algorithm, in a directed connected graph with positive edge weights, is a NON-Decreasing sequence. Where is this fact used in the correctness of the algorithm?

3.11 Can you devise a faster algorithm for computing single source shortest path distances when all edge weights are 1? (Think of an algorithm that runs in O(|V | + |E|) time on a graph G = (V, E).)

3.12 Execute Dijkstra’s SSSP algorithm on the following graph on 7 vertices and 18 edges starting at the source vertex s. The edges and their weights are listed in the following (the entry (xy, 10) means the edge directed from the vertex x to the vertex y with edge weight 10):

(sb, 5), (sa, 10), (sf, 5), (bf, 6), (ba, 3), (be, 5), (bc, 5), (fs, 2), (fe, 4), (ca, 3), (ce, 2), (cd, 5), (df, 1), (de, 1), (ef, 1), (ec, 1), (af, 1), (ae, 2).

3.13 Recall that Dijkstra’s SSSP algorithm only computes distances from source vertex to all the vertices. What modifications we should make to the algorithm so that it reports the shortest paths as well (in fact the collection of all these paths can be represented in a directed tree rooted at the source vertex).

3.14 Suppose in place of computing shortest path distance from a vertex to every other vertex, we are interested in finding the shortest path distances between every pair of vertices. Then one way to do this is to run Dijkstra’s algorithm |V | times, where each vertex in the graph G = (V, E) is considered as a source vertex once. Can you devise an algorithm that is asymptotically faster than just running Dijkstra’s algorithm O(|V |) times?

3.15 Which of the following algorithms result in a minimum spanning tree? Justify your answer. Assume that the graph G = (V, E) is connected.

1. Sort the edges with respect to decreasing weight. Set T := E. For each edge e taken in the order of decreasing weight do, if T − {e} is connected, then discard e from T. Set MST (G) = T.

2. Set T := ∅.

For each edge e, taken in arbitrary order do, if T ∪ {e} has no cycles then

T := T ∪ {e}. Set MST (G) = T.

3. Set T := ∅.

For each edge e, taken in arbitrary order do

begin

T := T ∪ {e}.

If T has a cycle c then let e′ be a maximum weight edge on c.

Set T := T − {e′}.

end

Set MST (G) = T.

3.16 A spanning tree T of a undirected (positively) weighted graph G is called a minimum bottleneck spanning tree (MBST) if the edge with the maximum cost is minimum among all possible spanning trees. Show that a MST is always a MBST. What about the converse?

3.17 Design a linear time algorithm to compute MBST. (Note that an edge with medium weight can be found in linear time. Consider the set of edges whose weight is smaller than the weight of the ‘median edge’. What happens if this graph is connected? disconnected?

3.18 Consider an undirected (positively) weighted graph G = (V, E) with a MST T and a shortest path π(s, t) between two vertices s, t ∈ V. Will T still be an MST and π(s, t) be a shortest path if a) Weight of each edge is multiplied by a fixed constant c > 0.

b) Weight of each edge is incremented by a fixed constant c > 0.

Network Flow

4.1 What is a Flow Network

A flow network consists of the following:

1. A simple finite directed graph G = (V, E).

2. Two specified vertices, namely source s and target t.

3. For each edge e ∈ E, a non-negative number c(e) called the capacity. If a pair of vertices u

and v are not joined by an edge, then c(u, v) = 0. Flow: A flow function f in G is a real-valued function

f : V × V → ℜ

that satisfies the following three properties.

1. Capacity Constraint: For all u, v ∈ V, f(u, v) ≤ c(u, v).

2. Skew Symmetry (A tough constraint to see!): For all u, v ∈ V, f(u, v) = −f(v, u). This is for notational purposes, and basically says that flow from a vertex u to vertex v is the negative of the flow in the reverse direction.

3. Flow conservation: For all u ∈ V − {s, t}, ∑_{v∈V} f(u, v) = 0. This uses the skew symmetry property, otherwise we have to sum up the flow values coming into a vertex and that should be equal to the sum of the outgoing flow values from that vertex. This is same as the Kirchoff law for current in an electrical circuit, i.e. no node can hold the current, or no node can hold the flow, or whatever comes in goes out. There is no reservoir at a node.

The value of the flow is defined to be the flow out of the source s or the flow into the target t, i.e.

|f| = ∑_{v∈V} f(s, v) = ∑_{v∈V} f(v, t).

The Maximum Flow Problem to find the flow of maximum value in a given flow network. See Figure 4.1 for an example of a network flow.

Figure 4.1: An example of a network flow. Example is taken from Even [20]

4.2 Ford and Fulkerson’s Algorithm

This is an iterative method for computing the flow. Ford-Fulkerson-Method (G, s, t)

1. Initialize the flow f to 0.

2. While there exists an augmenting path p augment the flow f along p.

3. Return f.

An augmenting path is a path from s to t along which additional flow can be sent. This path is found using the concept of residual networks. The residual network consists of those edges which can admit more flow. The residual capacity c_{f}(u, v) of an edge (u, v) in a flow network is given by

c_{f}(u, v) = c(u, v) − f(u, v).

In our example c_{f}(a, b) = 12−4 = 8, c_{f}(s, c) = 4−4 = 0, c_{f}(b, a) = c(b, a)−f(b, a) = 0−(−4) = 4. Given a flow network G and the flow function f, the residual network G_{f} = (V, E_{f}) consists of the same vertex set and the edges E_{f} are defined as follows:

E_{f} = {(u, v) ∈ V × V : c_{f}(u, v) > 0}.

The residual network of our example is given in Figure 4.2.

As we can see that there is an augmenting path in this network (the red path), and the flow can be augmented along this path, by a value of 3. Hence we get a new flow network with the total flow value equals to 7 = 4 + 3 given in Figure ??.

The new residual network that we obtain for the flow corresponding to the flow network in Figure 4.3 is given in Figure 4.4. Note that there exists an augmenting path that can further increases the flow value by 3.

The new flow graph is shown in Figure 4.5 and the corresponding residual network is shown in Figure 4.6.

This will continue for two more iterations and after that there is no path between s and t in the residual network. The corresponding figures are Figure 4.7 and Figure 4.8.

Figure 4.2: Residual network corresponding to the flow in Figure 4.1.

Figure 4.3: Flow network after augmenting the flow from Figures 4.1 and 4.2.

Now consider the residual network in Figure 4.8, where there are no paths joining s and t. As can be seen from the figure, there is a path from s to every vertex in the set {s, a, b}, and there are paths from vertices {c, d, t} to t. This automatically partitions the set of vertices into two, call it a s − t cut {S, T }, where s ∈ S and t ∈ T (in our example, S = {s, a, b} and T = {c, d, t}). See Figure 4.9. Define the capacity of a cut as follows

c(S, T) = ∑_{u∈S,v∈T} c(u, v).

In other words consider the edges crossing the cut, and sum up the capacities of the edges which

Figure 4.5: Flow network after augmenting the flow from Figures 4.3 and 4.4.

go from a vertex in the set S to a vertex in the set T. Define the net flow across the cut to be

f(S, T) = ∑_{u∈S,v∈T} f(u, v).

In other words the net flow is the sum of the positive flow on edges going from S to T minus the sum of the positive flows on edges going from T to S (recall the skew symmetry property). Amazingly in our example f(S, T) = c(S, T). Is it always true or just a luck! Before we get to this, a few observations.

Observation 4.2.1 For any s − t cut S, T, and flow f

|f| ≤ c(S, T).

This follows from the definition of the flow across the cut. The flow f(S, T) is defined to be the sum of the positive flows along the edges in the forward direction, i.e. the ones going from vertices in S to vertices in T minus the sum of the positive flows along the edges in the reverse direction. If we ignore the reverse direction, then clearly the flow along each edge in the forward direction is bounded by the capacity of the edge. Sum of these capacities is the capacity of the cut and hence the observation.

Figure 4.7: Resulting Flow network.

The following observation explains why the flow f′ found using the augmenting paths in the residual graph G_{f}, can be augmented with the flow f in G, to obtain a new flow in G of a higher value |f + f′| ≥ |f|.

Observation 4.2.2 Let G be the flow network with flow f and G_{f} be the corresponding residual network and let f′ be the flow in G_{f}. Then the flow sum f + f′ is a flow in G and its value is |f + f′| = |f| + |f′|.

Proof: To prove that f +f′ is a flow in G, we need to prove that the three conditions are satisfied. I will prove here the capacity constraint, and others can be verified similarly. The capacity constraint follows from

(f + f′)(u, v) = f(u, v) + f′(u, v) ≤ f(u, v) + (c(u, v) − f(u, v)) = c(u, v).

Observe that

|f + f′| = ∑_{v∈V}(f + f′)(s, v) = ∑_{v∈V} f(s, v) + ∑_{v∈V} f′(s, v) = |f| + |f′|.□

Figure 4.9: An illustration of a cut.

Theorem 4.2.3 Let f be a valid flow in the flow network G = (V, E) from the source s to the target t, then the following statements are equivalent.

1. Flow f is a maximum flow.

2. Residual network G_{f} does not contain an augmenting path.

3. There exists some cut c(S, T) such that |f| = c(S, T).

This is the famous max flow min cut theorem.

Proof: Recall that to prove that the three statements are equivalent we need to show that 1 ⇒ 2 ⇒ 3 ⇒ 1.

1 ⇒ 2: Let f be a maximum flow and, for contradiction, assume that there exists an augmenting path in G_{f}. Then we can increase the flow along the path using Observation 4.2.2 and contradicting that f is a maximum flow. 2 ⇒ 3: Define the set

S = {v ∈ V |there is a path from s to v in G_{f}

and

T = V − S.

Also observe that s ∈ S and t ∈ T, so it is a valid s − t cut. Moreover for all edges (u, v) crossing the cut, where u ∈ S and v ∈ T, f(u, v) = c(u, v), otherwise (u, v) ∈ E_{f} and v ∈ S, which is not possible. Net flow across the cut (S, T) is |f|! Why? (Think about this yourself!) So we have shown a cut where 3 holds. 3 ⇒ 1: We know that the capacity of any cut is an upper bound to the value of the flow. If for a cut we obtain the equality, then we have attained the max-flow. (In other words the capacity of minimum cut is the value of the maximum flow!). □

This proves the correctness of the Ford-Fulkerson algorithm. The algorithm iteratively increases the value of the flow using augmenting paths and return the value of the flow, the maximum flow, when it is not able to find augmenting paths in the residual graph. How do we analyze the complexity of this algorithm?

First a special case where all capacities are integers. Observe that value of all flows computed during the algorithm are integers. In each iteration of the algorithm, the value of flow increases by at least 1. If f^{∗} is a maximum flow, then the number of iterations in the algorithm are bounded by |f^{∗}|. It is easy to see that each iteration requires O(|E|) time; this involves computing residual graph (i.e. capacities on at most 2|E| edges), and computing a path between s and t (directed dfs or bfs). Hence the algorithm runs in O(|f^{∗}||E|) time - this is a strange complexity since the running time depends upon the value of the output! Is there a better way to analyze this algorithm!

4.3 Edmonds-Karp Algorithm

In this variation, the computation of the augmenting path is done using a BFS tree rooted at s. Augmenting path is a shortest path (in the unweighted residual graph) from s to t. It turns out that this variation leads to an algorithm that runs in O(|V ||E|^{2}) time. Here is the main lemma - let δ_{f}(s, v) denote the shortest path distance between s and v in the unweighted residual graph G_{f}, corresponding to the flow network G with flow function f.

Lemma 4.3.1 Shortest path distance for all vertices v ∈ V − {s, t} in G_{f} increases monotonically with each flow augmentation.

Proof. By contradiction. Caution: This is a little bit strange proof, and the proof in generic terms goes as follows. To prove the statement P, the contradictory proof assumes that ¬P is true. Inside the proof we need to prove a claim C, which in turn is proved using the contradiction. Say ¬C is true. The contradiction is arrived by showing that ¬C is true only if P is true, but since ¬P is assumed to be true, implying that C is true. Once we show that C is true, the contradiction to the original assumption is arrived at. (Wow! Pay attention)

Assume that for a vertex v ∈ V − {s, t}, the shortest path decreases after a flow augmentation. Let f be the flow before the augmentation and f′ be the flow after the augmentation. Let v be the vertex with minimum δ_{f′}(s, v) whose distance was decreased by the augmentation (i.e. δ_{f′}(s, v) < δ_{f}(s, v)). Let u be the vertex just before v in the shortest path from s to v in G_{f′}, i.e. (u, v) ∈ E_{f′}. Then δ_{f′}(s, u) = δ_{f′}(s, v) − 1. Moreover, δ_{f′}(s, u) ≥ δ_{f}(s, u) (by choice of v).

Now we will show that (u, v) ∉ E_{f}, and as a consequence of that we will arrive to contradiction (somehow!).

First why (u, v) ∉ E_{f}? Suppose (u, v) ∈ E_{f}, then δ_{f}(s, v) ≤ δ_{f}(s, u) + 1 (triangle inequality - sum of two sides of the triangle is at least as big as the third side). But, δ_{f}(s, u)+1 ≤ δ_{f′}(s, u)+1 = δ_{f′}(s, v). This implies that δ_{f}(s, v) ≤ δ_{f′}(s, v), contradicts our assumption!

Now consider the scenario that (u, v) ∉ E_{f} and (u, v) ∈ E_{f′}. The flow from v to u must have been increased in Edmonds - Karp algorithm and this edge must be on a shortest path. This implies that δ_{f}(s, v) = δ_{f}(s, u) − 1 ≤ δ_{f′}(s, u) − 1 = δ_{f′}(s, v) − 2, and this contradicts the assumption that δ_{f′}(s, v) < δ_{f}(s, v).

In each iteration of the augmenting path algorithm, at least one edge becomes critical, i.e. flow value becomes equal to its capacity. The critical edge disappears from the residual network. Of course the flow along this edge may be decreased in the future, and this edge may reappear again in the residual network, but this cannot happen more than |V |/2 times. Why?. Say (u, v) became critical, then δ_{f}(s, v) = δ_{f}(s, u) + 1. Flow along (u, v) is decreased only if (v, u) appears on an augmenting path, let f′ be the flow then, δ_{f′}(u) = δ_{f′}(v) + 1. Since the shortest path distances are monotone, this implies that

δ_{f′}(s, u) = δ_{f′}(v) + 1 ≥ δ_{f}(s, v) + 1 = δ_{f}(s, u) + 2.

Therefore the distance to u from the source has increased by at least 2 between two consecutive times that (u, v) became critical. The maximum distance is at most |V | and hence an edge can become critical about |V |/2 times. There are O(|E|) edges in all in the residual graph, and hence the number of augmentations (or iterations) are bounded by O(|V ||E|) times. Each augmentation can be implemented in O(|E|) time, and hence flow between s and t in the graph G = (V, E) can be computed in O(|V ||E|^{2}) time.

4.4 Applications of Network Flow

We can use the flow networks to compute Maximum Matching in a Bipartite Graphs. Recall that a Graph G = (V = A ∪ B, E) is bipartite, if the set of vertex V is partitioned into two sets A and B, such that all the edges in the graph are between vertices of A to vertices in B. A matching in a graph is a collection of edges such that no two edges in the matching are incident to the same vertex. A matching in G is called a maximum matching if the cardinality of the number of edges in it is maximum among all matchings in G. Note that there can be a number of maximum matching in a graph. Using flow networks we can compute easily maximum matching in G. Here is the simple method. We add two vertices, namely s and t, to the set of vertices in G. Vertex s is connected to all the vertices in the set A, and the capacity of all these edges is set to 1. The capacity of all the edges in the set E, i.e., the edges joining vertices in the set A to vertices in the set B, is set to 1. Lastly vertices in B are joined to t, and the capacity of these edges is set to 1. Let G′ be the resulting graph. Compute the maximum s − t flow in G′. Observe that the value of the flow is the size of the maximum matching. Why ?

Note that value of flow in each of the edge will be an integral value, since all capacities are integers (this is one of the exercises in [12]). Since the capacity of all the edges between vertices in A and B is 1, the value of flow on these edges will either be 0 or 1. This implies that no two edges in E incident on the same vertex will ever have nonzero flow. In other words the edges in E which have nonzero flow are the edges in a matching. Also maximum matching corresponds to a largest set of independent edges in G and each of these edges can admit a flow of value 1 and at the same time satisfy all the three conditions required for a flow network. Hence maximum matching in G corresponds to a valid flow in G′.

4.5 Exercises

4.1 Construct a network flow example with 7 vertices and 11 directed edges, where each edge has a positive capacity and compute the maximum flow and minimum cut in this graph. You should show some of the steps in the algorithm. (Follow Edmonds-Karp shortest path heuristic).

4.2 Assume that we have a network flow graph G = (V, E) with positive capacities on each of the edges and two specified vertices s and t. Suggest an efficient algorithm to find an edge in E, such that setting its capacity to zero (i.e. deleting this edge) will result in the largest decrease in the maximum flow in the resulting graph.

4.3 What is the total number of flow augmentations that are performed when in place of taking an arbitrary path in the residual network, we take the shortest path (i.e., the path with minimum number of links)? Please do not provide proof for this - just a couple of line of reasoning is sufficient.

4.4 Suppose we are given a flow network G, where edges have positive integer capacities, and C = ∑_{u,v∈V} c(u, v), where c(u, v) is the capacity of the edge e = (u, v) ∈ E. Show the following

1. The value of the max flow is an integer.

2. There is an assignment of non-negative integer flow values on each edge of G, satisfying all the flow conservation conditions, so that G achieves max flow.

3. Show that the number of iterations required in the Ford-Fulkerson’s algorithm (Residual network, find an augmenting path, augment the flow, repeat) is O(C).

4. Show that in the worst case, Ford-Fulkerson’s algorithm, as stated in Part 3 runs in exponential time.

5. Construct an example, where one can realize the worst case as stated in Part 4.

4.5 Let G = (V, E) be the flow network. Let C = max_{(u,v)∈E} c(u, v) be the maximum capacity. Show the following:

1. Minimum cut of G has a capacity of at most C|E|.

2. For a given number K > 0, show how to find an augmenting path of capacity at least K in O(|E|) time, provided that such a path exists.

3. Execute the following algorithm:

(a) Initialize the flow f = 0;

(b) K = 2^{⌊log C⌋};

(c) While K ≥ 1 do

i. While there exists an augmenting path p of capacity at least K then augment flow f

along p. ii. K := K/2;

(d) Return f.

Show that the above algorithm computes Max-Flow.

4. Show that the loop in Step 3c(i) is executed at most O(|E|) times for each value of K.

5. Show that the algorithm runs in O(|E|^{2} log C) time.

4.6 Let G = (V, E) be a flow network. Recall that G is a complete graph, where some of the edges may have a capacity of zero. Suppose your task in the max flow problem is to increase the flow of a network as much as possible, but you are only allowed to increase the capacity of only one edge, whose capacity is strictly larger than zero. First show that there are networks where such an edge may not exist, i.e. increasing the capacity of a single edge (> 0 capacity) will not alter the value of the max-flow. Show that there are networks, where such an edge may exist. Try to design an algorithm which can detect whether flow can be increased.

4.7 A simple undirected graph G = (V, E) is called k-edge connected if removal of any set of k-edges keeps G still connected. (e.g. cycles are 1-edge connected.) Show how to compute edge connectivity of G by invoking at most |V | network flow computations.

Separators in a Planar Graph

This chapter is based on Kozen [46] and the famous paper of Lipton and Tarjan [48] on the planar separator theorem. Earlier we have seen that in a binary tree on n-nodes, there exists a vertex such that whose removal leaves no component having more than 2(n + 1)/3 nodes. This can be extended quite easily to outerplanar graphs, where we can remove a pair of vertices such that none of the components have more that 2(n + 1)/3 nodes. Usually this phenomenon is referred to as a balanced decomposition using small size separators. This is a ‘key idea’ in most of the divide and conquer type algorithms on these graphs. As can be seen that the depth of recursion will be O(log n) and since the size of the separator is small, the “merge” step will be economical as well. First we start with some preliminaries and then we will prove that in a planar graph there exists a separator of size O(\sqrt{n}).

5.1 Preliminaries

Definition 5.1.1 A graph is called planar if the vertices and edges can be laid out (embedded) in the plane so that no two edges intersect except at their end points.

Definition 5.1.2 In an embedded planar graph, we have vertices, edges and faces. The dual of a planar graph G is the planar graph G^{∗} whose vertices correspond to faces of G and two vertices in G^{∗} are joined together if the corresponding faces in G share an edge.

Definition 5.1.3 A planar graph G is triangulated if each of its face is a triangle, i.e., it is bounded be three edges. In other words, in the dual each vertex has degree three.

Definition 5.1.4 A set S ⊆ V for a graph G = (V, E) is called a separator, if removal of vertices (and incident edges on these vertices) from G results in two disjoint sets of vertices A, B ⊆ V with no edges between them. If the sizes of the sets A and B are a constant fraction of that of the size of V, then S is called as a balanced separator.

Definition 5.1.5 A planar graph G = (V, E) consists of at most |E| = 3|V |−6 edges. This follows from Euler’s relation, i.e. |V | − |E| + |F | = 2. You may like to check the proof at

http:\\www.ics.uci.edu/~eppstein/junkyard/euler/

Definition 5.1.6 A outerplanar graph is a planar graph such that all vertices lie on a single face (usually referred to as the outerface).

Definition 5.1.7 The dual of a triangulated outerplanar graph is a binary tree.

We have seen that complete graph on five vertices, K_{5}, and complete bipartite graph on six vertices, K_{3,3}, is nonplanar. It is easy to see that a tree is planar, and as well as outerplanar graphs are planar. Both of these graphs admit small size separators to get a balanced decomposition. What we will prove in this chapter is that in fact planar graphs have the similar property. This is the theorem we will like to prove:

Theorem 5.1.8 [Lipton and Tarjan [48]] Let G = (V, E) be an embedded undirected triangulated planar graph, where n = |V |. There exists a partition of V into disjoint sets A, B, and S, such that

1. |A|, |B| ≤ \frac{2n}{3}

2. |S| ≤ 4\sqrt{n}

3. There is no edge in E that joins a vertex in A with a vertex in B.

4. Such a set S can be found in linear time.

It will turn out that the way we prove this theorem, it will lead to a linear time algorithm for finding such a separator. Note that if the given graph is not embedded in the plane, then there is a linear time algorithm by Hopcroft and Tarjan that embeds it. In fact that algorithm also figure out in linear time whether the given graph is planar or not, and if it is planar it finds an embedding of it in the plane. Also if the planar graph is not triangulated, then it can be triangulated in linear time, by inserting required number of edges. Other than this essentially we will use breadth first and the concept of fundamental cycles to prove this theorem.

5.2 Proof of the Planar Separator Theorem

Assume that the graph G = (V, E) is undirected, connected, planar, triangulated and embedded. The first step in the proof/algorithm is to do a breadth-first search starting at an arbitrary vertex, say s, in G, and assign levels to vertices. Vertex s is at level 0, vertices adjacent to s are at level 1, vertices adjacent to level 1 vertices that have not been assigned any level are level 2 vertices, and so on. Let l be the last level, and pretend that there is a level l + 1 which consists of no vertex (this is just required for the proof!). Let L(t) denote the set of vertices that are in level t, 0 ≤ t ≤ l. Recall that in BFS, no edge can cross over two or more levels. All edges must connect vertices in the same level or consecutive level. Observe that each of the level, L(t), for 0 < t < l, is a separator in its own right, although may not be of small size and may not lead to a balanced decomposition!

Let t_{1} be the middle level, that is the one which contains the vertex number n/2 in the BFS numbering. Consider the set L(t_{1}). Note that | ∪_{t<t_{1}} L(t)| < n/2 and | ∪_{t≤t_{1}} L(t)| ≥ n/2. If |L(t_{1})| ≤ 4\sqrt{n}, then S = L(t_{1}) and we are done. It is not necessary that L(t_{1}) may satisfy the requirements on the size of the separator. Here is the lemma which will be very handy in that case.

Lemma 5.2.1 There exists levels t_{0} ≤ t_{1} and t_{2} > t_{1} such that, t_{2} − t_{0} ≤ \sqrt{n}, |L(t_{0})| ≤ \sqrt{n} and |L(t_{2})| ≤ \sqrt{n}.

Proof: Note that |L(0)| = 1 and |L(l + 1)| = 0. Let t_{0} ≤ t_{1} be the largest number such that |L(t_{0})| ≤ \sqrt{n}. Let t_{2} > t_{1} be the smallest number such that |L(t_{2})| ≤ \sqrt{n}. Note that every level between t_{0} and t_{2} contains more than \sqrt{n} vertices, therefore by pigeon hole principle they must be fewer than \sqrt{n} levels between t_{0} and t_{2}, otherwise G will have more than n vertices! Therefore t_{2} − t_{0} ≤ \sqrt{n} □

Define three sets C, D and E as follows: C = ∪_{t<t_{0}}L(t), D = ∪_{t_{0}<t<t_{2}}L(t) and E = ∪_{t>t_{2}}L(t). If |D| ≤ 2/3n, then we have the required separator, by setting S = L(t_{0}) ∪ L(t_{2}), A the largest of C, D or E and B the union of the other two.

What if |D| > 2/3n? Then both the sets C and E are small, have less than 1/3n vertices. We will find a 1/3 − 2/3 separator of S_{D}, of D, of size at most 2\sqrt{n}. Let D be split into D′ and D′′ by S_{D}. Then S will include the vertices in L(t_{0}), L(t_{2}), and the separator vertices S_{D}. Set A = max(C, E)∪min(D′, D′′) and B = min(C, E)∪max(D′, D′′). Observe that S, A, and B satisfy the required size criteria.

Next we will present some ideas regarding finding the separator S_{D} of D. First we remove all the vertices that are not in D, except the start vertex s. We connect s to all the vertices in level t_{0} + 1. This can be done still preserving the planarity of D, since the original graph is planar. Now we construct a spanning tree T in D, such that its diameter is at most 2\sqrt{n}. Start with vertices in level L(t_{2} −1). For each vertex in this level, choose one of the vertex in the previous level L(t_{2} −2), adjacent to it as its parent. Continue this process with vertices in levels t_{2} −2, t_{2} −3, ⋯, to obtain the tree T. Next we state two lemmas, that are easy to prove, that will show the critical property relating the tree T, the planar graph D, its dual D^{∗}, and the dual tree T′.

Lemma 5.2.2 Let G = (V, E) be a connected planar graph and G^{∗} be its dual. For any E′ ⊆ E, the subgraph (V, E′) has a cycle if and only if the subgraph (V^{∗}, E − E′) of G^{∗} is disconnected.

Lemma 5.2.3 Let G = (V, E) be a connected planar graph with dual G^{∗} = (V^{∗}, E) and let E′ ⊆ E. Then (V, E′) is a spanning tree of G if and only if (V^{∗}, E − E′) is a spanning tree in G^{∗} (see Figure 5.1 for an illustration).

Let E_{T} be the edges in the spanning tree T in D. Recall that the diameter of T is at most 2\sqrt{n}. Also D is triangulated. Consider the dual D^{∗} of D, and consider the edges in E − E_{T}. They define a spanning tree T′ in D^{∗}. Also we can orient each edge in T′ away from the root. Pick a face of D (say its outer face) and choose this as the root T′. It will turn out that the required separator S_{D} will be defined by an edge, e = (u, v) in e ∈ E − E_{T}, and the path in the tree T between u and v. In other words, e defines a unique cycle, c(e), in T. To compute/define c(e) appropriately we first perform a DFS of T′ and compute the following three quantities.

1. I(e)= number of vertices which are in the interior of the cycle c(e).

2. |c(e)|= number of vertices on the cycle c(e).

3. Linked list representation of c(e).

For each step of DFS, one of the following four cases will occur (see Figure 5.2)

Case 1: DFS visits a leaf of T′ (i.e. a triangular face of D). Then I(e) = 0, |c(e)| = 3, and c(e) = {x, u, v}.

Figure 5.1: The edges of (V, E′) are in bold-solid. The edges of (V^{∗}, E − E′) are dashed-bold and the planar graph edges are in solid.

Figure 5.2: Bold edges represent the DFS traversal of faces of D corresponding to the tree T′. Dashed edges corresponds to edges in E − E_{T} and solid edges corresponds to edges in E_{T}, i.e. a spanning tree of D.

Case 2: DFS visits a triangle corresponding to an edge e = (u, v) ∈ E − E_{T}, its degree is two and the other edge of the triangle is e′ = (u′, v) ∈ E − E_{T} which was visited in the previous step. Moreover u′ ∈ c(e). Then I(e) = I(e′), |c(e)| = |c(e′)| + 1, and c(e) = uc(e′).

Case 3: DFS visits a triangle corresponding to an edge e = (u, v) ∈ E − E_{T}, its degree is two and the other edge of the triangle is e′ = (u′, v) ∈ E − E_{T} which was visited in the previous step. Moreover u′ ∉ c(e). Then I(e) = I(e′) + 1, |c(e)| = |c(e′)| − 1, and c(e) equals to c(e′) except that u′ is removed from the front of the list.

Case 4: DFS visits a triangle corresponding to edge e = (u, v) ∈ E − E_{T} and its degree is three. The other two edges e′ = (u, y) ∈ E − E_{T} and e′′ = (vy) ∈ E − E_{T} have been already visited by the DFS. Let p be the common path between the cycles c(e′) and c(e′′). One of the end points of p is x, and the other end point is y. Then I(e) = I(e′) + I(e′′) + |p| − 1, |c(e)| = |c(e′)| + |C(e′′)| − 2|p| + 1, and c(e) consists of c′xc′′, where c′ is the cycle c(e′) with path p removed, and similarly c′′ is the cycle c(e′′) with path p removed.

Lemma 5.2.4 In the above setting of the graph D, there exists an edge e ∈ E − E_{T} such that I(e) ≤ 2/3n and n − (I(e) + |c(e)|) ≤ 2/3n.

Proof. Let e ∈ E − E_{T} be the first edge in the leaf to root path in T′ such that I(e) + |c(e)| ≥ n/3. Then n − (I(e) + |c(e)|) ≤ 2/3n. We will prove that I(e) ≤ 2/3n. The edge e corresponds to one of the four cases encountered in the DFS.

1. In Case 1, I(e) = 0 ≤ 2/3n.

2. In Case 2, I(e) + |c(e)| = I(e′) + |c(e′)| + 1, and I(e′) + |c(e′)| < n/3, and hence I(e) + |c(e)| ≤ 2/3n.

3. In Case 3, since I(e) + |c(e)| = I(e′) + |c(e′)|, e cannot be the first edge with this property.

4. In Case 4, I(e′) + |c(e′)| < n/3 and so is I(e′′) + |c(e′′)| < n/3. I(e) + |c(e)| = I(e′) + I(e′′) + |p| − 1 + |c(e′)| + |c(e′′)| − 2|p| + 1 ≤ 2/3n − |p| ≤ 2/3n.

5.3 Generalizations of the Planar Separator Theorem

1

In the previous section we saw that the planar separator theorem provides us with a procedure to separate the vertices of a planar graph G = (V, E), |V | = n, into three sets A, B, C, where |A|, |B| ≤ 2n/3, |C| ≤ 4\sqrt{n}, and there exists no edge between A and B. In this section we will consider several generalizations of this theorem.

1This section is contributed by Alexis Beingessner.

5.3.1 Weighted Separators

In our version of the planar separator theorem we considered all vertices to be equal. However, a common variant permits vertices to be weighted such that the sum of all weights is equal to 1 (any other set of non-negative weights can be trivially mapped to one like this). The only difference is that instead of bounding the sizes of the the sets A and B to be ≤ 2n/3, we bound their weights to be ≤ 2/3. The separating set C however is still bounded in terms of the number of vertices it contains, and may therefore have arbitrary weight.

To prove the weighted planar separator theorem, our proof from the previous section is sufficient. We need only change certain references to the sizes of sets to refer to the weight of the sets. The rest of the analysis largely follows unchanged.

From now on, we will assume that the planar separator theorem refers to the weighted variant of the planar separator theorem. If no weights are specified, we will assume that all vertices have equal weight, which coincides with our original definition.

5.3.2 r-Divisions

In this section we will use the planar separator theorem to construct a more general graph partitioning. The contents of this section are based on a paper by Frederickson[21].

We define a region to be a subset of the vertices of a graph G = (V, E). An interior vertex of a region R is contained only in R, and adjacent only to other vertices in R. A boundary vertex is one that is shared between at least two regions. All vertices will be either boundary or interior. Given a parameter r, we will divide the graph into Θ(n/r) regions with O(r) vertices each, and O(\sqrt{r}) boundary vertices each. Such a division will be called an r-division. Remark that the planar separator theorem provides an n-division by taking the two sets A ∪ C and B ∪ C.

We begin with a potential naive algorithm. Start with a single region containing all of V. While any region R contains more than r vertices, apply the planar separator theorem on R to produce A, B, C. Now replace R with R′ = A ∪ C and R′′ = B ∪ C.

Clearly this procedure produces regions with no more than r vertices, and since it reduces the size of a region by at most 2/3 until this bound is satisfied, it follows that each region contains Θ(r) vertices. Consequently, there must be Θ(n/r) regions. However the number of boundary vertices is more complicated. Remark that initially we have a single region that is all interior vertices. Further, A and B consist entirely of interior vertices after constructing R′ and R′′, and C consists entirely of boundary vertices. Therefore we introduce at most 4\sqrt{n} boundary vertices at each recursive step.

To determine the number of boundary vertices, we define b(v) for some vertex v to be one less than the number of regions it is contained in, and B(n, r) to be the sum of b(v) for all v ∈ V. Remark that B(n, r) is strictly greater than the number of boundary vertices, as b(v) ≥ 1 for all boundary vertices, by definition. Our algorithm gives us the following recurrence:

\begin{aligned}B(n, r) & ≤ & 4\sqrt{n} + B(αn + O(\sqrt{n}), r) + B((1 − α)n + O(\sqrt{n}), r) & for n > r \\ B(n, r) & = & 0 & for n ≤ r \end{aligned}

where 1/3 ≤ α ≤ 2/3. This recurrence can be solved for B(n, r) ≤ 4n/\sqrt{r} − O(\sqrt{n}). Therefore, the number of boundary vertices produced by this algorithm is O(n/\sqrt{r}). However this tells us nothing about the number of boundary vertices per region. Indeed, some regions may have many boundary vertices. To resolve this, we perform further processing on regions with more than c\sqrt{r} boundary vertices, for some constant c. Given such a region R, we set all k boundary vertices of R to have weight 1/k, and all interior vertices of R to have weight 0. We then apply the planar separator theorem to R and replace R as before. Since only the boundary vertices have weights, the planar separator theorem will split up the boundary vertices among the two resultant regions. Therefore, after enough iterations all regions will have few enough boundary vertices. Further, since the regions are still strictly shrinking, we cannot have violated the bounds on the size of the region. It remains to be proven that we have not violated the constraint on the maximum number of regions.

If a region has i > c\sqrt{r} boundary vertices, then at most di/(c\sqrt{r}) splits will be performed, for some constants c and d. This will result in at most di/(c\sqrt{r}) new regions. If t_{i} is the number of regions with i boundary vertices, then the number of new regions will be at most

∑_{i}(di/c\sqrt{r})t_{i} = O(n/r)

Therefore, our modified algorithm produces an r-division.

Remark that in our construction, our recursion tree has a depth of O(log(n/r)). Further, each level has us do O(n) work. Therefore, this algorithm runs in O(n log(n/r)) time.

Further processing on the graph can provide our r-division with additional properties such as regions having a constant number of neighbours, and boundary vertices being shared between at most a constant number of regions.

5.3.3 Edge Separators

Up until now, we have only considered vertex separators. Edge separators are exactly the same as vertex separators, except that instead of removing vertices, we wish to remove edges. Specifically, given a graph G = (V, E) we wish to find a cut-set S ⊆ E that separates V into two disjoint subsets A, B. Every edge in S has one endpoint in A and one endpoint in B, and every edge in E \ S has both of its endpoints in only A or B. In general we would like to ensure that A and B are approximately the same size, and S is small.

For graphs with low (e.g. constant) maximum degree, the results for edge separators are generally very similar to those for vertex separators. However on arbitrary planar graphs, edge separators perform much worse.

For instance, consider a graph G = (V, E) in which every vertex has degree 1, except for some vertex v with degree n−1. This graph is a tree, and therefore planar. An excellent vertex separator for G would be {v}, as it would disconnect the entire graph, allowing us to pick any subsets of V \ {v} we want for our separated sets. However an edge separator would have to remove a linear number of edges to get balanced sets.

Figure 5.3: The optimal vertex separator (blue) for this graph is much smaller than a similarly balanced edge separator (red)

From this example it is clear that not all results for vertex separators hold for edge-separators. In general, vertex separators are more powerful, as for every edge an edge-separator would need to remove, a vertex separator would need to remove at most one vertex, but potentially far fewer. Equivalently, if a vertex separator includes some vertex v, an edge separator would need to include every edge of v to achieve the same result. Consequently, results on edge separators often include factors based on the maximum or average degree of the graph[22]. One such measure, Δ(G) = ∑_{v∈V} deg(v)^{2}, is used in the next section.

We conclude our look at the planar separator theorem and its generalizations with a table of separator results. There are far too many results on separators with special requirements and for special classes of graphs to adequately report here. As a result, this table is by no means comprehensive.

Separator  Graph # of Sets Set Sizes Separator Size  Time
Vertex or Edge[53] Tree  2  ≤ 2n/3  1  O(n)
Vertex[48]  Planar  2  ≤ 2n/3  O(\sqrt{n})  O(n)
Vertex[21]  Planar Θ(n/r)  O(r)  O(n/\sqrt{r})  O(n log(n/r))
Vertex[2]  Genus g  -  ≤ ϵn O(\sqrt{(g + 1/ϵ)n})  O(n + g)
Vertex[3]  Planar  -  tw(G)  ≤ 4\sqrt{2σ(G)/t} O(n + T_{SSSP} (G))
Edge[3]  Planar  -  tw(G)  ≤ 4\sqrt{2Δ(G)/t} O(n + T_{SSSP} (G))

5.4 Exercises

5.1 Let T=(V,E) be a connected undirected tree such that each vertex has degree at most 3. Let n=|V |. Show that T has an edge whose removal disconnects T into two disjoint subtrees with no more than (2n+1)/3 vertices each. Give a linear time algorithm to find such and edge; prove its correctness.

5.2 Provide an algorithm running in O(n log k) time to partition the binary tree on n vertices into k (k ≤ n) subtrees, so that each of the subtree is of size at most (2/3)^{k}n. Try to see whether you can improve the running time of this algorithm (this is not easy!).

5.3 Prove the weighted version of the planar-separator theorem. Let G = (V, E) be an embedded undirected triangulated planar graph, where n = |V |. Each vertex v ∈ V has a positive weight w(v) ≥ 0 and ∑_{v∈V} w(v) = 1. There exists a partition of V into disjoint sets A, B, and S, such that

1. w(A), w(B) ≤ \frac{2}{3}, where w(A) is the sum total of weights of all the vertices in set A

2. |S| ≤ 4\sqrt{n}

3. There is no edge in E that joins a vertex in A with a vertex in B.

4. Such a set S can be found in linear time.

5.4 Prove Lemma 5.2.2. Let G = (V, E) be a connected planar graph and G^{∗} be its dual. For any E′ ⊆ E, the subgraph (V, E′) has a cycle if and only if the subgraph (V^{∗}, E − E′) of G^{∗} is disconnected.

5.5 Let T = (V, E) be a connected undirected tree such that all of its vertices have degree at most 3. Let n = |V |. Show that T has an edge whose removal disconnects T into two disjoint subtrees with no more than \frac{2n+1}{3} vertices each.

5.6 Consider the following version of the (weighted) planar-separator theorem. Let G = (V, E) be an embedded undirected triangulated planar graph, where n = |V |. Each vertex v ∈ V has a positive weight w(v) ≥ 0 and ∑_{v∈V} w(v) = 1. There exists a partition of V into disjoint sets A, B, and S, such that

1. w(A), w(B) ≤ \frac{2}{3}, where w(A) is the sum total of weights of all the vertices in set A

2. |S| ≤ 4\sqrt{n}

3. There is no edge in E that joins a vertex in A with a vertex in B.

4. Such a set S can be found in linear time.

Show what changes you need to make in the proof of (unweighted) Planar Separator Theorem to prove the above theorem.

5.7 Prove the following: Let G = (V, E) be a connected planar graph and G^{∗} be its dual. For any E′ ⊆ E, the subgraph (V, E′) has a cycle if and only if the subgraph (V^{∗}, E − E′) of G^{∗} is disconnected.

5.8 Prove the following theorem on Geometric Separators. In 2-dimensions assume that you have n squares of arbitrary sizes. Squares are axis aligned. Moreover none of the points in the plane is inside more than k-squares. Prove that there exists either a vertical or a horizontal line which partitions the set of squares in such a way that at least ⌊\frac{n+1−k}{4}⌋ of squares interiors lie to each side of the line. How fast you can find such a line?

70

Lowest Common Ancestor

Given a rooted binary tree T on n nodes, we are asked to preprocess it in O(n) time so that the following type of queries can be answered in O(1) time. Given any two nodes u and v of T, report their Lowest Common Ancestor LCA(a, b), i.e., among all the common ancestors of nodes a and b, find the one which is furthest from the root of T. This subproblem arises in many graph applications. Orginal algorithm is due to Harel and Tarjan [1984]. Many years later, Schieber and Vishkin [1993] proposed a new algorithm for the same problem while studying parallel algorithms. Both of these algorithms are fairly complex and are considered to be far from being implementable. Recently, Bender and Farach-Colton [2000] proposed a fairly simple algorithm for the LCA problem, and thats what we present in this chapter.

It is well known that the following Range Minima Problem (RMQ) is related to the LCA problem. Given an array A[1...n] consisting of n numbers, preprocess it so that given any two indices i and j, where 1 ≤ i ≤ j ≤ n, report the minimum element (or its index in A) in the subarray A[i...j]. Next we will show the reduction of the LCA problem to RMQ problem, and then provide a solution for the RMQ problem.

6.1 LCA → RMQ

Let T be the given binary rooted tree. Consider the depth first search traversal of T. Observe that the shallowest node encountered in the depth first traversal of T between u and v is the node corresponding to LCA(u, v). (Recall that the main property of dfs traversal is that once it enters a subtree, then it completely visits all the nodes in the subtree - this sort of corresponds to a nice bracketing sequence.) Our aim is to find this node using the RMQs.

Corresponding to the dfs traversal of T, let E be the Euler tour. Recall that E stores the nodes of T in the same order as they are visited during the dfs traversal. The tour E consists of 2n − 1 entries. Let the level of a node in T be its distance from the root. Corresponding to E, define a level array L[1...2n − 1] which stores the level of the node E[i] in L[i]. Furthermore, observe that a node may appear several times in Euler tour. For each node x ∈ T, we maintain an index R(x) that stores the index of the first appearance of x in E. Given our notation, the nodes between E[R(u), ..., R(v)] are nodes in Euler tour between the first visits of u and v. What is the shallowest node among the nodes in E[R(u), ..., R(v)]? For this we will look at the corresponding entries in the level array L. More precisely, we need to report what is the minimum element in the subarray L[R(u)...R(v)]; this returns us the index of the shallowest node (one with the smallest level) and denote this by RMQ_{L}[R(u)...R(v)]. Hence, LCA(u, v) = E[RMQ_{L}[R(u)...R(v)]].

Lemma 6.1.1 LCA problem on a rooted binary tree T of n nodes can be converted to the range minima query problem on an array L of size 2n − 1 elements. The reduction takes O(n) time. Moreover, LCA queries can be answered within O(1) time in addition to the time required to answer the range minima queries on L.

Proof. Notice that the depth first traversal and the construction of Euler tour of T can be done in O(n) time. Within the same time bounds we can maintain the level array as well as keep track of the first appearance of each node in Euler tour. Hence the conversion can be done in linear time. Given the query, LCA(u, v), we need to find the representatives R(u) and R(v) in E, then need to answer the query RMQ_{L}[R(u)...R(v)] followed by one more look up in the array E to report the node corresponding to LCA(u, v). This computation only requires a few pointer manipulation and hence requires O(1) time in addition to answering the range minima query.

6.2 Range Minima Queries

Let A be the array of length n consisting of numbers. Our task is to preprocess A so that the range minima queries RMQ(i, j), 1 ≤ i ≤ j ≤ n, can be answered in O(1) time.

6.2.1 A naive O(n^{2}) algorithm

A simple way to achieve a constant query time is to precompute and store minima for each possible query. In all there are O(n^{2}) possible queries of type RMQ(i, j), where 1 ≤ i ≤ j ≤ n, and for each of them we can compute and store the minima in the range A[i, ..., j]. It is easy to see that this computation can be done in O(n^{2}) time and then given a query it can be answered in O(1) time.

6.2.2 An O(n log n) algorithm

In place of precomputing minima for each possible query, now we precompute minima’s for only O(n log n) selected types of queries. For every i between 1 and n and for every j between 1 and log n, we find minimum element in the subarray A[i, ..., i + 2^{j}] (we are sloppy with boundary conditions here to keep it simple) and store it in a table in location M[i, j]. Next we show that using dynamic programming the table M can be computed in O(n log n) time. Minima in a subarray of size 2^{j} is computed by looking at the minima of two constituent blocks of size 2^{j−1}. Either M[i, j] = M[i, j − 1] or M[i, j] = M[i + 2^{j−1} − 1, j − 1].

How do we answer a range minimum query in O(1) time? Let the query be RMQ(i.j), where 1 ≤ i ≤ j ≤ n. First compute k = ⌊log_{2} j − i⌋. Now observe that 2^{k} is the largest interval, that is a power of 2, that fits in the range from i to j. Compute RMQ(i, j) be finding out the minimum of two entries in the table, namely M[i, k] and M[j − 2^{k} + 1, k]. Notice that these two table values have been precomputed and hence query can be answered in O(1) time.

Lemma 6.2.1 An array A consiting of n numbers can be preprocessed in O(n log n) time so that the range minima queries can be answered in O(1) time.

6.2.3 An O(n) algorithm with ±1 property

Consider the following special case of the array A where each element differs from its previous element either by a +1 or a −1 (this is especially true for the LCA problem as levels of consecutive nodes in Euler tour differs by 1). We will show that in this case A can be preprocessed in O(n) time and RMQs can be answered in O(1) time.

The strategy is pretty simple. First we partition array A into subarrays, where each subarray is of size \frac{log n}{2} (we are assuming that n is a nice power of 2, otherwise we have to use floors and ceilings and that will not add anything more in terms of understanding.) Within each subarray we find the minimum value and then store all these minimas in an array A′. Notice that the size of the array A′ is \frac{2n}{log n} and hence it can be preprocessed in O(n) time by using Lemma 6.2.1.

Consider a range minima query RMQ(i, j) in array A, where i ≤ j. It is answered as follows: Indices i and j may fall within the same subarray, therefore we need to preprocess each subarray for answering RMQs. If i and j fall in different subarrays then we compute the following three quantities:

1. Minimum value starting at index i up to the end of the subarray containing i.

2. Minimum value among the subarrays between the subarray containing i and j. This is computed using the preprocessing done for A′ in constant time.

3. Minimum value from the beginning up to the index j within the subarry containing j.

Now our subproblem is reduced to solving the RMQ problem in subarrays of size \frac{log n}{2} with ±1 property. The key observation here is that we do not have too many different kinds of these subarrays.

Claim 6.2.2 Given two arrays of same size where each element in the first array is constant value more than the corresponding element in the second array, then the answer to RMQ queries (i.e. the index) is identical in both the arrays.

Essentially the preprocessing and the RMQ queries work with relative order of elements in these arrays, and they do not need actual values of the elements. Hence for the two subarrays within the above claim, same preprocessing is sufficient to answer RMQ queries. We normalize each of the subarrays by first subtracting the initial value from each of the elements. Next we show that there are only O(\sqrt{n}) normal subarrays.

Claim 6.2.3 There are at most O(\sqrt{n}) normalized subarrays. Each subarrays has length \frac{log n}{2}, where the first element is a 0, and the elements in the array satisfy ±1 property.

Proof. Each normalized subarray can be specified by a ±1 vector. Therefore, there are only 2^{\frac{1}{2} log n−1} = O(\sqrt{n}) different types of subarrays of length \frac{1}{2} log n. We preprocess each of these

subarrays in O(log^{2} n) time to answer RMQ queries in O(1) time using the naive algorithm. The preprocessing requires in all O(\sqrt{n} log^{2} n) time. We summarize the results in the following.

Lemma 6.2.4 An array A consisting of n-numbers satisfying the ±1 property can be preprocessed in O(n) time so that the range minima queries can be answered in O(1) time.

Corollary 6.2.5 A binary tree on n-nodes can be preprocessed in O(n) time so that the lowest common ancestor queries can be answered in O(1) time.

6.3 RMQ → LCA

Next we show that an instance of the RMQ problem can be converted to an instance of the LCA problem. For a linear array A of size n, the tree T for the LCA problem consists of n nodes and given a RMQ query, we perform an equivalent LCA query on T, and whose answer in turn provides the answer for the original range minima query. This will imply that the general RMQ problem (i.e., even without the ±1 property) can be answered in O(1) time by performing an O(n) time preprocessing. The key to this conversion is the concept of Cartesian Tree.

Let A[1...n] be the input array on which we need to perform RMQ queries. Cartesian tree T for A is defined as follows. It is a rooted binary tree and the root of T stores the index of the smallest element in A. Deleting the minimum element from A splits it into two subarrays. Left and right children of the root are recursively defined Cartesion trees for left and right subarrays of A, respectively.

Claim 6.3.1 Cartesian tree T for an array A of n-numbers can be constructed in O(n) time.

Proof. We scan the array A from left to right and incrementally build the Cartesian tree T = T_{n} as follows. Suppose so far we have built the tree T_{i} with respect to elements A[1..i] and we want to extend it for A[1...i + 1] to obtain T_{i+1}, where i < n. Main observation is that the node storing the index i + 1 in T_{i+1} is on the rightmost path of T_{i+1}. We start at the rightmost node of T_{i} and follow the parent pointers till we find the location to insert i + 1 in Cartesian tree. Note that each comparison will either add a node or removes one from the rightmost path. Since each node can only join the rightmost path once (if it leaves it then it can’t be back to the rightmost path), therefore the total time in constructing T is O(n).

Claim 6.3.2 Let A be the array on n-numbers and T be the corresponding Cartesian tree storing the indices of elements in A in its node. Then RMQ(i, j) = LCA(i, j).

Proof. This follows from the recursive definition of Cartesian tree T. Let k = LCA(i, j) in T. Observe that the node labeled k is the first node that separates i with j. In other words, the element A[k] is the smallest element between A[i] and A[j], i.e. RMQ[i, j] = k.

6.4 Summary

In this chapter, we have shown that the lowest common ancestor query in a rooted binary tree on n-nodes can be answered by solving the range minima query in an array consisting of 2n−1 numbers satisfying the ±1 property. Moreover, the general RMQ problem in an array can be reduced to solving LCA queries on the corresponding Cartesian tree. All our preprocessing algorithms require linear time and the queries can be answered in constant time.

Locality-Sensitive Hashing

Locality-Sensitive Hashing (LSH) is a method which is used for determining which items in a given set are similar. Rather than using the naive approach of comparing all pairs of items within a set, items are hashed into buckets, such that similar items will be more likely to hash into the same buckets. As a result, the number of comparisons needed will be reduced; only the items within any one bucket will be compared. Locality-sensitive hashing is often used when there exist an extremely large amount of data items that must be compared. In these cases, it may also be that the data items themselves will be too large, and as such will have their dimensionality reduced by a feature extraction technique beforehand.

The main application of LSH is to provide a method for efficient approximate nearest neighbor search through probabilistic dimension reduction of high-dimensional data. This dimensional reduction is done through feature extraction realized through hashing (eg. minhash signatures), for which different schemes are used depending upon the data. LSH is used in fields such as data mining, pattern recognition, computer vision, computational geometry, and data compression. It also has direct applications in spell checking, plagiarism detection, and chemical similarity.

In this chapter, we will give an illustration of the technique using the problem of finding similar documents, while concurrently showing supporting theory. Following this will be core theory including the definition of locality-sensitive hash families, and examples of families for different distance measures. Throughout the chapter, examples are also given for the application of families to illustrate their usage.

7.1 Similarity of Documents

We’ll begin to introduce LSH by an illustration of the technique using the problem of finding similar documents. This problem appears, for example, on the web when attempting to find similar, or even duplicate web pages. A search engine would use this technique to allow these similar documents to either be grouped or only shown once on a results page, so that other possible search matches could also be prominently displayed.

Problem 7.1.1 Given a collection of web-pages, find the near duplicate web-pages.

Clearly the content of the page is what matters, so for a preprocessing step any HTML tags are stripped away and main textual content is kept. Typically multiple white spaces are also replaced by a singe space during this preprocessing. As a result we’re left with a document containing a string of text characters. Keeping in mind that we want to compare similarity of documents opposed to exact equality, the text is split up into a set of smaller strings by a process called shingling. This allows the documents to be represented as sets, where fragments of documents can match others. Shingle length is an important factor in this, though will not be explained in depth. Generally though, the shingle length k should be large enough so that the probability of any given k-shingle appearing in any given document is relatively low. For our purposes, it is sufficient to know that choosing k = 5 works well for electronic mail, and k = 9 is suitable for large text documents.

Definition 7.1.2 k-shingle: A k-shingle of a text document is defined to be any substring of length k which appears in the document.

Example 7.1.3 Let the document D = {adbdabadbcdab}, and k = 2. Then the possible k-shingles for D are: {ad, db, bd, da, ab, ba, bc, cd}.

Using shingling, we have seen that a document can be represented as a set of k-shingles. However we can work instead with integers by using a hash function to map each k-shingle to an integer. This is also useful to reduce space requirements, as a regular 9-byte shingle can be converted to a 4-byte integer. Note however that even if we represent shingles this way, we still may be using 4 times more space than the original document. A solution to this is shown in the next section.

Example 7.1.4 Let k = 9, and let H be a hash function that maps the set of characters to integers; H : |C|^{9} → ℤ. Let |ℤ| ≤ 2^{32} − 1.

Now that we have documents mapped to sets of k-shingles, we can use a similarity measure called Jaccard Similarity to compare the two sets. The Jaccard Similarity is defined with respect to two sets S and T, and is the ratio of the size of the intersection of S and T to the size of their union. As a result of this, we can also redefine our problem statement.

Problem 7.1.5 Given a collection of sets, find the pairs which have large intersections.

Definition 7.1.6 Jaccard Similarity: Let S and T be two sets. Define the Jaccard Similarity of the sets S and T as \frac{|S∩T |}{|S∪T |}. See Figure 7.1 for an illustration.

7.2 Similarity-Preserving Summaries of Sets

As mentioned in the previous section, we’ll now present a solution for the storage of sets of shingles using smaller representations called signatures. These signatures will also have the property that the similarity measure between any two will be approximately the same as the similarity between the two sets of shingles which they are derived from.

First, let’s consider the natural representation of a set. We have a universe U from which elements of the set are drawn, which we’ll assume are arbitrarily ordered. A set S with elements from U can be represented by a 0-1 vector of length |U|, where a 1 represents that the corresponding element from the universe is present in the set, and a 0 represents that element’s absence from the set. Similarly for a collection of sets over the universe U, we can associate a characteristic matrix, where each column represents the vector corresponding to a set and each row corresponds to an element of U.

Figure 7.1: Jaccard similarity of two sets. The intersection |S ∩ T | = 3 and the union |S ∪ T | = 10. Therefore their Jaccard similarity is 3/10.

An example is given below in Table 7.1, where we have four sets (representing households in a neighborhood) and a universe U consisting of five elements (possible vacationing destinations). From this characteristic matrix, we can ascertain that set S_{1} prefers cruise and safari, while S_{2} just loves to visit resorts.

S_{1} S_{2} S_{3} S_{4}
Cruise  1 0 0 1
Ski  0 0 1 0
Resorts  0 1 0 1
Safari  1 0 1 1
Stay at Home 0 0 1 0

Table 7.1: A characteristic matrix for 4 sets with a universe consisting of 5 elements.

One effective way to compute the signature for a collection of sets is to use minhashing. For minhashing, the rows of the characteristic matrix are first randomly permuted. Then for each set (column in the characteristic matrix), its minhash value h is the number of the first row which is a 1. Using the previous example, suppose the permutation of rows results in Table 7.2 as shown below. The minhash values are then: h(S_{1}) = 2, h(S_{2}) = 4, h(S_{3}) = 1, and h(S_{4}) = 2.

In the following lemma we establish an important connection between the Jaccard similarity of two sets and the probability that their minhash values are the same after a random permutation of the rows of the characteristic matrix. In fact, we’ll be able to shown that the Jaccard similarity is equal to the probability that these minhash values are the same.

Lemma 7.2.1 For any two sets S_{i} and S_{j}, the probability that h(S_{i}) = h(S_{j}) is equal to the Jaccard similarity of S_{i} and S_{j}.

Proof. Focus on the columns representing the sets S_{i} and S_{j} in the characteristic matrix before the random permutation. For any row, the entries corresponding to these columns are either; (a)

S_{1} S_{2} S_{3} S_{4}
Ski  0 0 1 0
Safari  1 0 1 1
Stay at Home 0 0 1 0
Resorts  0 1 0 1
Cruise  1 0 0 1

Table 7.2: Characteristic matrix after the permutation of rows of Table 7.1.

both 0, (b) both 1, or (c) one is 0 and the other is 1. Let X be the number of rows of type (b) and let Y be the number of rows of type (c). Observe that the Jaccard similarity of S_{i} and S_{j} is \frac{|X|}{|X|+|Y |}.

Now, what is the probability that when we scan the rows from top to bottom, after the random permutation, we meet a type (b) row before a type (c) row? This is exactly \frac{|X|}{|X|+|Y |}, which is also precisely when h(S_{i}) = h(S_{j}).

Next, we’ll turn our attention to minhash signatures, which are smaller matrices created from repeatedly minhashing a characteristic matrix. Let M denote the characteristic matrix for a set system S with universe U. Pick a set of n random permutations of rows of the characteristic matrix. For each set in S compute its h-value with respect to each of the n-permutations. This results in a signature matrix — it consists of |S| columns and n-rows, and the (i, j)-th entry corresponds to the signature of the j-th set with respect to the i-th hash function. Typically the signature matrix should be significantly smaller than M in size, as the size of universe U will be larger than n.

Example 7.2.2 Shown below is the minhash signature created from Table 7.1 using n = 2 permutations (using functions h_{1} = x + 1 mod 5 & h_{2} = 3x + 1 mod 5).

S_{1} S_{2} S_{3} S_{4}
h_{1} 1 3 0 1
h_{2} 0 2 0 0

Table 7.3: A signature matrix for 4 sets.

Computation of this signature matrix should be done in an efficient way, though following the explanation given above we’d be calculating n random permutations of a large characteristic matrix M; a very time-intensive task. Clearly we should attempt to avoid these permutations. To do this, we can use n random hash functions h to ‘permute’ the rows of M. We’ll assume the mapping of |U| elements onto themselves won’t cause many collisions, however as there are certain to be some collisions h(r) is not a ‘perfect permutation’. So, given a row r of M, we can say that h(r) denotes the new row number after a permutation. Shown below is an outline of the required steps to compute this signature matrix.

Step 1: Pick n random hash functions h_{1}, …, h_{n}.

Step 2: Execute the following three steps for each row r of M.

1. Set Signature(r, c) := ∞, for c = 1, …, |S|.

2. Apply each h-function to r for the new row numbers h_{1}(r), …, h_{n}(r) of r.

3. For c = 1, …, |S|; if M[r, c] = 1 then Signature(r, c) := min(h_{i}(r), Signature(r, c)).

Observe that for each non-zero entry of M, we perform O(n) computations, and we do not need any additional memory except to store the description of n hash functions and the characteristic matrix M.

Although the signature matrix is fairly small compared to the characteristic matrix, its size could still be large. Let’s look at the example from [58, Example 3.9], where there are signatures of length 250 for a set of one million documents. The total size of the signature matrix will be 1GB. Moreover, if documents are compared pairwise, the computation will be of the order of comparing a trillion documents!

The above example suggests that we should avoid the computation of pairwise similarity of documents as this requires \binom{|S|}{2} comparison computations, which for a large number of documents any type of signatures will be prohibitively expensive. As an answer to this, we’ll use Locality Sensitive Hashing (LSH) to quickly find any sets of documents which are likely to be similar, without using direct comparisons.

7.3 LSH for Minhash Signatures

The general idea, as expressed at the beginning of this chapter, is to hash documents in such a way that similar documents are more likely to be hashed into the same bucket as opposed to dissimilar documents. The documents will each be hashed multiple times, with the intent of altering the probability that similar items will become candidates while dissimilar items do not. The key point is that only the documents which fall into the same bucket are considered as potentially similar. If two documents do not map to the same bucket in any of the hashings then they are never considered as similar in this technique. Obviously, we want to minimize the number of dissimilar documents falling into the same bucket. At the same time, we hope that similar documents at least hash once into the same bucket. The good thing is that we have a methodology to achieve these objectives.

Let us assume that we have computed the minhash signature matrix for the set of documents of the previous section. We are going to partition the rows of this matrix into b = n/r bands, where each band is made of r-consecutive rows. (See Figure 7.2. We’ll assume that r divides n. For each band we define a hash function h : ℝ^{r} → ℤ, which takes a column vector of length r and maps it to an integer (i.e. a bucket). If we want we can even choose the same hash function for all the bands, but the buckets are kept distinct for each band. Now if two vectors of length r in any one of the bands hash to the same bucket, we declare that the corresponding documents are potentially similar. That’s it — that’s the technique. Let’s do some analysis.

Lemma 7.3.1 Let s be the Jaccard similarity of two documents. The probability that the minhash signature matrix agrees in all the rows of at least one band for these two documents is 1 − (1 − s^{r})^{b}. Note that this is also the probability that they will become a similar pair after the hashing and banding.

Proof. Recall from Lemma 7.2.1, the probability that the minhash signature for these two documents are the same in any particular row of the signature matrix is s. Therefore, the probability

Figure 7.2: A signature matrix which has been separated into four bands of three rows each.

that the signatures agree in all the rows in one particular band is s^{r}. The probability that the signature will not agree in at least one of the rows in this band is 1 − s^{r}.

Next let us calculate the probability that each band of the signature will not agree in at least one of its rows. This will be (1 − s^{r})^{b} as there are b bands. Therefore, the probability that the signatures will agree in at least one of the bands is 1 − (1 − s^{r})^{b}.

The function 1 − (1 − s^{r})^{b} can be plotted with the x-axis representing values of s and the y-axis representing the output values of the function, with the values of b & r fixed. The curve created by this function has the form of an S-shaped curve. See Figure 7.3. In fact, the curve will have this shape regardless of the values of b and r. From inspecting this curve, we see that as the similarity s of documents approaches 1, the value of this function — the probability of the input elements being mapped to the same bucket — also increases towards 1. One important aspect of this curve however, is that the steepest slope occurs at the value of s which is around t = (1/b)^{(1/r)}. In other words, if the Jaccard similarity of the two documents is above the threshold t, then the probability that they will be found potentially similar using LSH is very high. What this technique has done is to give us some idea in terms of which documents are very likely to be similar. If required, we can now actually compare the documents (or their minhash signatures) to find out whether they are actually similar.

At an abstract level what we have done here is to use a family of functions (the minhash functions) and the banding technique to distinguish between pairs which are at a low distance (similar) to the pairs which are at a large distance (dissimilar). The steepness of the S curve suggests a threshold where this technique can be effective. In the next section, we will see that there are other families of functions that can be considered to separate the pairs which are at a low distance from the pairs which are at a higher distance.

7.4 Theory of Locality Sensitive Functions

In this section we will consider a family of functions F. The families are typically hash functions, and we say that f(x) = f(y) if the items x and y are hashed to the same bucket for a function

Figure 7.3: The S-curve.

f ∈ F. For example, the minhash functions seen previously form a family of functions.

Recall that a distance measure satisfies certain natural properties: non-negativity, symmetry, and the triangle inequality. For example, the Euclidean distance between points in a space, edit distance between strings, Jaccard distance, and hamming distance all satisfy these properties.

Definition 7.4.1 Let d be a distance measure, and let d_{1} < d_{2} be two distances in this measure. A family of functions F is said to be (d_{1}, d_{2}, p_{1}, p_{2})-sensitive if for every f ∈ F the following two conditions hold;

1. If d(x, y) ≤ d_{1} then Probability[f(x) = f(y)] ≥ p_{1}

2. If d(x, y) ≥ d_{2} then Probability[f(x) = f(y)] ≤ p_{2}

Figure 7.4: A smaller distance between items corresponds to a higher probability of similarity.

Basically, if the two items are close to each other with respect to the distance function, then the probability that they hash to the same bucket will be high. Conversely, if the two items are far from each other, then the probability that they hash to the same bucket will be low. See Figure 7.4 for an illustration.

For example, consider the Jaccard distance, which is defined as 1− Jaccard similarity of two sets. Let 0 ≤ d_{1} < d_{2} ≤ 1. Note that the family of minhash-signatures is (d_{1}, d_{2}, 1 − d_{1}, 1 − d_{2})- sensitive. Suppose that the Jaccard Similarity between two documents is at least s. Then their distance is at most d_{1} = 1 − s. By Lemma 7.2.1 the probability that they will be hashed to the same value by minhash signatures is s = 1 − d_{1}. Suppose that the distance between two documents is ≥ d_{2}. Hence their Jaccard similarity is at most s = 1 − d_{2} and the probability that the minhash signatures map them to the same value is at most s = 1 − d_{2}.

Suppose we have a (d_{1}, d_{2}, p_{1}, p_{2})-sensitive family F. We can construct a new family G as follows. Each function g ∈ G is formed from a set of r independently chosen functions of F, say f_{1}, f_{2}, …, f_{r} for some fixed value of r. Now, g(x) = g(y) if and only if for all i = 1, …, r, f_{i}(x) = f_{i}(y). This is referred to as an AND-family.

Claim 7.4.2 G is an (d_{1}, d_{2}, p_{1}^{r}, p_{2}^{r}) family.

Proof. Each function f_{i} is chosen independently, and this is the probability of all the r-events to occur simultaneously.

We can also construct G as an OR-family. Each member g in G is constructed by taking b independently chosen members f_{1}, f_{2}, …, f_{b} from F. We say that g(x) = g(y) if and only if f_{i}(x) = f_{i}(y) for at least one of the members in {f_{1}, f_{2}, …, f_{b}}.

Claim 7.4.3 G is an (d_{1}, d_{2}, 1 − (1 − p_{1})^{b}, 1 − (1 − p_{2})^{r}) family.

Proof. Each function f_{i} is chosen independently, and this is the probability of at least one of the b-events to occur. We can compute this by first finding the probability that none of the b events occur.

Let us play with some values of b and r to see the effect of ANDs and ORs. We will also combine ORs and ANDs through function composition. We first construct an AND-family for a certain value of r, and then construct an OR-family on top of it for a certain value of b. Similarly we do this for an OR-family followed by an AND-family. Let us look at the amplification of the probabilities in Table 7.4 for different values of p.

\begin{matrix}p & p^{5} (F_{1}) & 1 − (1 − p)^{5} (F_{2}) & 1 − (1 − p^{5})^{5} (F_{3}) & (1 − (1 − p^{5}))^{5} (F_{4}) \\ 0.2 & 0.00032 & 0.67232 & 0.00159 & 0.13703 \\ 0.4 & 0.01024 & 0.92224 & 0.05016 & 0.66714 \\ 0.6 & 0.07776 & 0.98976 & 0.33285 & 0.94983 \\ 0.8 & 0.32768 & 0.99968 & 0.86263 & 0.99840 \\ 0.9 & 0.59049 & 0.99999 & 0.98848 & 0.99995 \end{matrix}

Table 7.4: Illustration of four families obtained for different values of p. F_{1} is the AND family for r = 5. F_{2} is OR family for b = 5. F_{3} is the AND-OR family for r = 5 and b = 5. F_{4} is the OR-AND family for r = 5 and b = 5.

Let’s try to understand the columns of Table 7.4. Let F be a (0.2, 0.6, 0.8, 0.4)-sensitive minhash function family. This means if the distance between two documents x and y is ≤ 0.2, it is likely with probability ≥ 0.8 that they will hash to the same bucket. Similarly if the distance between x and y is ≥ 0.6, it is unlikely that they will hash to the same bucket (probability of hashing to the same bucket is ≤ 0.4). Let us focus our attention on the two rows corresponding to p_{2} = 0.4 and p_{1} = 0.8. For the AND-family we see that p^{5} is substantially lower than p, but still p_{2}^{5} → 0 and p_{1}^{5} is away from 0. For the OR-family 1 − (1 − p)^{5} ≥ p, but still the value corresponding to p_{1} tends towards 1 and the value corresponding to p_{2} is away from 1. More interesting are the last two columns. Notice that the AND-OR family (F_{3}) corresponds to the LSH for minhash signature family of Section 7.3. In this case the value corresponding to p_{1} tends towards 1 and the value corresponding to p_{2} is closer to 0. Notice that the value of this function with respect to the values of p forms an S-curve. Its fixed-point p = 1 − (1 − p^{5})^{5} is around p ≈ 0.7548 ^{1}. This implies that for values of p significantly less than 0.75, AND-OR family amplifies it towards 0. Similarly, values of p larger than 0.75 are amplified to 1. Notice that this technique amplifies the probabilities in the right direction, provided we can apply the function several times (e.g. 25 times in our example), and are away from the fixed-point.

7.5 LSH Families

In the previous section we saw LSH families for the Jaccard distance measure. In this section we will construct LSH-families for various other distance measures, including Hamming and Euclidean distances. Note that not every distance measure may have a corresponding LSH-family.

7.5.1 Hamming Distance

Consider two d-dimensional vectors x and y. The Hamming distance h(x, y) is defined as the number of coordinates in which x and y differ. Without loss of generality, we can assume that they are Boolean vectors — that is, each of their coordinates are either 1 or 0.

Example 7.5.1 Let x = 110011 and y = 100111. Then h(x, y) = 2, as they differ in exactly two coordinates.

Applying the requirements for a distance measure, we see that distances are non-negative, symmetric, and that the distance between two identical vectors is 0. They are also transitive, as the Hamming distance between any three vectors x, y, and z, satisfy the triangle inequality; h(x, y) + h(y, z) ≥ h(x, z). Therefore, we can use Hamming distance as a metric over the ddimensional vectors.

Next, we construct a locality-sensitive family for the d-dimensional Boolean vectors. Let f_{i}(x) denote the i-th coordinate (bit) of x. Then the probability that f_{i}(x) = f_{i}(y) for a randomly chosen i will equal the number of coordinate agreements out of the total number of coordinates. Since the vectors x and y disagree in h(x, y) positions out of d positions, then they agree in d − h(x, y) positions. Hence P r[f_{i}(x) = f_{i}(y)] = 1 − h(x, y)/d.

Claim 7.5.2 For any d_{1} < d_{2}, F = {f_{1}, f_{2}, …, f_{d}} is a (d_{1}, d_{2}, 1 − d_{1}/d, 1 − d_{2}/d)-sensitive family of hash functions.

1Thanks to WolframAlpha

Proof. Recall Definition 7.4.1. Let p_{1} = 1 − d_{1}/d, and p_{2} = 1 − d_{2}/d. A family of functions F is said to be (d_{1}, d_{2}, p_{1}, p_{2})-sensitive if for every f_{i} ∈ F the following two conditions hold;

1. If h(x, y) ≤ d_{1} then Probability[f_{i}(x) = f_{i}(y)] ≥ p_{1}

2. If h(x, y) ≥ d_{2} then Probability[f_{i}(x) = f_{i}(y)] ≤ p_{2}

Note that the family F consists of only d-functions. This is in contrast to the minhash function family, where the size is not bounded.

7.5.2 Cosine Distance

As an alternative to Hamming distance, we can use Cosine distance to construct a locality-sensitive family of functions for d-dimensional vectors in a space. Consider two vectors in this space, x and y, along with a hyperplane through the origin with a normal vector v. These two vectors x and y have a cosine distance which is equal to the angle θ between them, and taken alone, this angle can be measured in the plane defined by the vectors. The hyperplane is randomly chosen via its normal vector v, and will then determine the sign of the dot products v.x and v.y.

Figure 7.5: Two vectors x and y shown in a plane; two possible hyperplanes are also shown.

In Figure 7.5, the two vectors x and y are shown along with two possible hyperplanes. If the hyperplane is similar to the dashed line, then the dot products v.x and v.y will have different signs. However, if the hyperplane is similar to the dotted line, then v.x and v.y will have the same sign.

The probability that v will produce a hyperplane similar to the dotted line is θ/180, as v would have to lie between x and y on the right (or on the left; 2θ/360). So, we create each hash function f ∈ F using a randomly chosen vector v, and can then say that f(x) = f(y) if the dot products v.x and v.y have the same sign. Our resultant family F is a (d_{1}, d_{2}, (180 − d_{1})/180, (180 − d_{2})/180)- sensitive family of functions. Similar to other families, we can amplify probabilities by taking any combination of ANDs and ORs as before.

7.5.3 Euclidean Distance

To start with, let’s consider points in a 2-dimensional space. Each hash function h will be represented by a line with random orientation in this space. We subdivide the line into intervals of an equal size a, with points landing in the same bucket if their orthogonal projection lies on the same interval. Using this method, if two points are close to each other then there are high chances that they will map to the same interval. Conversely, if the points are far away, it is unlikely they will fall into the same bucket (unless they are almost vertically aligned!).

Without loss of generality, we’ll assume that the line is horizontal. Let p_{i} and p_{j} be two points in the plane. Let θ be the angle of the line passing through p_{i} and p_{j} with respect to the horizontal line. Observe that if the distance between the points is at most a/2 then the probability that the points map to the same interval is at least 1/2. Moreover, if the distance between them is more than 2a, then they will fall in the same interval with a probability of at most 1/3. This is due to the fact that they have to be almost vertical; |p_{i}p_{j}| cos θ ≤ a. For this to happen cos θ ≤ 1/2, or the angle of the line p_{i}p_{j} forms with the horizontal needs to be between 60^{∘} and 90^{∘}. Observe that there is only 1/3rd chance that the angle between the horizontal and p_{i}p_{j} is in that range. See Figure 7.6 for an illustration.

Hence, the family with respect to the projection on a random line with intervals of size a is a (a/2, 2a, 1/2, 1/3)-sensitive family of hash functions. Again, we can easily amplify these probabilities by taking any combination of ANDs and ORs as before.

7.5.4 Fingerprint Matching

Fingerprint matching typically requires comparison of several features of the fingerprint pattern. These include ridge lines which form arches, loops, or circular patterns, along with minutia points & patterns which form ridges, and bifurcations. We can think of a fingerprint as a set of grid points, where matching any two fingerprints amounts to matching elements in corresponding grid cells.

Fingerprint matching falls into an application of Locality-Sensitive Hashing, and as a result of the data representation we are able to use its methods; mainly the family of minhash functions for set comparison. However in this section we will approach the problem in another manner, as the sets produced from fingerprints form a small grid of around 1000 points, and don’t need to have more succinct signatures created.

Consider the probabilities of minutia appearances and matches between fingerprints. For this, let’s say that the probability of a minutia existing in a random grid cell of any given fingerprint is 0.1. Also, assuming that we have two fingerprints of the same finger, let the probability of a minutia appearing in the same grid cell of both fingerprints given that one of them does have a minutia there be 0.9.

We’ll define a locality-sensitive family of functions F as follows. Each function f ∈ F sends two fingerprints to the same bucket if they have minutia in each of three specific grid cells. Let us estimate what the chance is of ending up with two matching fingerprints. First, the probability that two arbitrary fingerprints will map to the same bucket with respect to function f is 0.1^{6} = 0.000001, which is a one in a million chance ^{2}. Assuming that we have two fingerprints which are from the same finger though, we can see that they are mapped to the same bucket by f with a probability of 0.1^{3} × 0.9^{3} = 0.00729. This is only about a 1 in 138 chance, though it can be amplified by using several hash functions (OR-family) and still not produce many false positives.

2Wonder why you are always the unlucky one :(

As an example, suppose we use 1000-way OR-functions. Then two fingerprints from different fingers will map to the same bucket with a probability of 1 − (1 − 0.000001)^{1000}) ≈ 0.0001. Along with this, two fingerprints from the same finger will map to the same bucket with a probability of 1−(1−0.00729)^{1000} ≈ 0.9993. This is significant, and hence finding the match in this case is highly likely. Also note that using this scheme, the chances of false negatives are less than 1 in 1000, and the chances of false positives are negligible.

For another example, let’s now use 2000 functions which are partitioned into two groups of 1000 functions each. Assume also that we have constructed buckets for each of the two composite functions. Given a query fingerprint, we’ll find the fingerprints which are potential matches using the above scheme in each group independently. Then we’ll select only those fingerprints which are potential matches in both the groups (in the intersection). This produces a set of fingerprints which we’ll actually compare to the query fingerprint. Note that in this scheme, actual comparisons of fingerprints happen with only a few fingerprints (those in the intersection). Here, the probability that we will detect a matching fingerprint is (0.9993)^{2} ≈ 0.9986. The number of false negatives are now about 1.4 in a 1000. The probability of false positives will be 0.0001^{2}, which is insignificant. Hence, we have to examine only 0.000001% of the database compared to 0.01% in the previous scheme.

7.5.5 Image Similarity

3 Image similarity can also be found using locality-sensitive hashing, as exemplified in earlier research on LSH. Often, images are processed using global descriptors such as color histograms to produce feature vectors which are then compared using different distance measures. One of the obvious applications of image similarity is in web search, as seen in use by TinEye, Google, and Bing. However most search engines do not use image similarity in search; instead they rely on information gathered from surrounding text on the web page along with any image meta-data. In this section, we will show how locality-sensitive hashing is used by Google in their VisualRank algorithm for visual image search, along with several locality-sensitive families which have been proposed for image similarity measures.

VisualRank

In the VisualRank algorithm, computer vision techniques are used along with locality-sensitive hashing. Consider a regular image search by a text query. An existing search technique may be relied on to retrieve the initial result candidates, which along with other images in the index are clustered according to their similarity (which may be precomputed). Centrality is then measured on the clustering, which will return the most canonical image(s) with respect to the query. The idea here is that agreement between users of the web about the image and its related concepts will result in those images being deemed more similar. VisualRank is defined iteratively by V R = S^{∗} × V R, where S^{∗} is the image similarity matrix. As matrices are used, eigenvector centrality will be the measure applied, with repeated multiplication of V R and S^{∗} producing the eigenvector we’re looking for. Clearly, the image similarity measure is crucial to the performance of VisualRank since it determines the underlying graph structure. See Figure 7.7 for an illustration.

The main VisualRank system begins with local feature vectors being extracted from images using scale-invariant feature transform (SIFT). Local feature descriptors are used instead of the previously mentioned color histograms as they allow similarity to be considered between images with potential rotation, scale, and perspective transformations. Locality-sensitive hashing is then applied to these feature vectors using the p-stable distribution scheme. As seen in previous sections, amplification using AND/OR constructions are applied. As part of the applied scheme, a Gaussian distribution is used under the l_{2} norm.

3This section is contributed by Andrew Wylie.

p-Stable Distributions

This locality-sensitive hashing scheme uses stable distributions to apply a sketching technique to the high dimensional feature vectors. However, instead of using the sketch to estimate the vector norm, it is used to assign hash values to each vector. These values map each vector to a point on the real line, which is the split into equal width segments to represent buckets. If two vectors v and u are close, they will have a small difference between l_{p} norms ||v − u||_{p}, and they should collide with a high probability.

Definition 7.5.3 A distribution D over R is called p-stable, if there exists p ≥ 0 such that for any n real numbers v_{1}, …, v_{n} and independent identically distributed variables X_{1}, …, X_{n} with distribution D, the random variable ∑_{i} v_{i}X_{i} has the same distribution as the variable (∑_{i} |v_{i}|^{p})^{1/p}X, where X is a random variable with distribution D.

Example 7.5.4 Cauchy Distributions are 1-stable, and Gaussian Distributions are 2-stable.

Note how the sum ∑_{i} v_{i}X_{i} appears if we consider that v_{i}’s are taken from a vector v. This operation then corresponds exactly to the dot product between vectors v and a, where a has entries taken from the stable distribution. From our definition then, we see that this random variable has the same distribution as the l_{p} norm when the latter is multiplied by a random variable from the same distribution. This is exactly the method of the sketching technique, where multiple v.a’s are computed to estimate ||v||_{p}.

However as mentioned beforehand, in this case we’ll use the v.a’s to assign a hash value to each v instead of estimating the l_{p} norm. Also, as the dot product v.a projects each vector onto the real line, we’ll separate the line into equal width segments of size r. We define each hash function of our family by

h_{a,b}(v) = ⌊\frac{v.a + b}{r}⌋,

where a is a d-dimensional random vector with entries chosen independently from our stable distribution, and b is a real number chosen uniformly from the range [0, r]. Let f_{p}(t) denote the probability density function of absolute value of our stable distribution, and let c = ||v − u||_{p} for two vectors v, u. Since we have a random vector a from our stable distribution, v.a − u.a is distributed as cX where X is a random variable from our stable distribution. Therefore our probability function is;

p(c) = P r_{a,b}[h_{a,b}(v) = h_{a,b}(u)] = ∫_{0}^{r} \frac{1}{c}f_{p}(\frac{t}{c})(1 − \frac{t}{r})dt.

For a fixed r the probability of collision decreases monotonically with c = ||v − u||_{p}. Thus, our family of hash functions are (r_{1}, r_{2}, p_{1}, p_{2})-sensitive for p_{1} = p(1), p_{2} = p(c), and r_{2}/r_{1} = c. Note that for each c we attempt to choose a finite r to make p as small as possible. χ^{2} Distance

In the context of image retrieval & similarity search, the Euclidean metric (l_{2}) does not always provide as accurate or relevant results when compared to χ^{2} distance. This is especially so when using global descriptors such as color histograms. In this section, we will show an adaptation of the p-stable distribution scheme for χ^{2} distance.

Definition 7.5.5 Given any two vectors v and u with d positive components, the χ^{2} distance between them is

χ^{2}(v, u) = \sqrt{∑_{i=1}^{d} \frac{(v_{i} − u_{i})^{2}}{v_{i} + u_{i}}} .

To begin with, all points are mapped into a space of smaller dimension where they will be clustered in a way which preserves local sensitivity. This is done by projecting all points onto a line l_{a} using a random vector a whose entries are chosen independently from a Gaussian distribution. The line is then uniformly partitioned into intervals of size W with respect to χ^{2} distance. Our hash functions are then defined in a similar manner to the previous section — as the dot product of the feature vector v with a random vector a with entries independently chosen from a Gaussian distribution;

h_{a,b}(v) = ⌊yw(v.a) + b⌋.

In this case, b is a real number taken randomly from the uniform distribution U([0, 1]). The function yw is defined as

yw(x) = \frac{\sqrt{\frac{8x}{W^{2}} + 1} − 1}{2}

as a result of the non-uniform mapping from l_{2} to χ^{2} distance; points mapping to the same bucket in the χ^{2} scheme may be mapped to different buckets in the l_{2} scheme.

Following from the reasoning in p-stable distributions, we know that v.a − u.a is distributed as cX, where c = ||v − u||_{p}, and X is a random variable drawn from the distribution. Also similar to the previous section, f_{p}(t) denotes the probability density function of the absolute value of the 2-stable Gaussian distribution. Therefore our probability function is;

p(c) = P r_{a,b}[h_{a,b}(v) = h_{a,b}(u)] = ∫_{0}^{(n+1)W^{2}} \frac{1}{c}f_{p}(\frac{t}{c}) (1 − \frac{t}{(n + 1)W^{2}} ) dt.

We then define c′ = X^{2}(v, u). Since v, u have positive entries, the two distances will vary similarly when taking the dot product. Therefore, p decreases monotonically with respect to c, and p also decreases monotonically with respect to c′. If we set p_{1} = p(r_{1}) and p_{2} = p(r_{2}) (with r_{1} < r_{2}), then this family of functions is (r_{1}, r_{2}, p_{1}, p_{2})-sensitive.

7.6 Bibliographic Notes

4 Concepts relating to locality-sensitive hashing first started to form around 1994, when a method for finding similar files in a large file system was proposed [51]. This research introduced the idea behind shingling through ‘fingerprinting’, where several anchor strings were chosen to begin a substring used in the similarity search. Along with this, checksums were also used to represent these fingerprints. Shingling itself was then formalized a few years later, along with k-shingles of a document (using words as tokens), and resemblance (similarity) of two documents was defined [10]. The authors also introduced a method for computing a sketch (signature) of a document which could be used in comparisons; hence, preserving similarity.

4This section is contributed by Andrew Wylie.

After this research came a couple of papers which proposed similarity search through the use of hashing [24, 32]. Approximate nearest-neighbor search was first reduced to the problem of point location in equal balls, where locality-sensitive hashing was introduced for sublinear time queries. This algorithm was then improved in [32] with respect to query time. Along with this, the distance measures for hamming distance and the resemblance measure given in [10] were used as schemes. It’s worth noting that the approximate search is deemed accurate enough for practical purposes, where the actual closest neighbor can still be found by checking all approximate near-neighbors [4].

Concurrent with this research, shingling was formally introduced along with min-wise independent families of permutations [9]. These concepts were developed as a result of the authors’ work on the AltaVista web index algorithm which was used for detecting similar documents. Minwiseindependent families also underpin the theory behind minhashing, as the resemblance of documentsets is shown to be equal to the probability that the min-wise permutation of two sets using random permutation functions are equal.

Next in the development of locality-sensitive hashing was a scheme based on p-stable distributions (p ∈ (0, 2]) under the l_{p} norm [15]. This scheme both improved on the l_{2} norm distance running time found in [32], and works in Euclidean space without embeddings. These findings are then improved upon by [4], resulting in a near-optimal algorithm.

Locality-sensitive hashing is used in for video identification and retrieval. In this case, feature vectors are usually constructed from video frames using certain color histograms. In [37], the authors address two weaknesses of using LSH for this purpose. Responding to a non-uniform distribution of points in the space, they focus on using a hierarchical hash table to produce a more uniform distribution. In addition to this, they also attempt to partition dimensions more carefully in order to produce a more even hashing. A new scheme for video retrieval is then proposed in [31], where a color histogram is used which better handles the adverse effects of brightness & color variations. This is used in conjunction with an additional uniform distance shrinking projection which is applied to the produced feature vectors.

Locality-sensitive hashing is also used in image search. Similar to applications in video (for a single frame), images are often processed using color histograms to produce feature vectors which can be compared. This method was used in [24], with the histograms compared using the l_{1} norm. Following this, techniques using chi^{2} distance [25], p-stable distributions [35], and kernelized locality-sensitive hashing [47] have been proposed. Kernelized locality-sensitive hashing has also been used as a basis for search on text images in [54].

Along with these applications, LSH has recently been proposed for use in a wide range of other areas. One of these is the creation of hash values in P2P networks [16]. This would allow for a conceptual search, as data with similar ontologies would be located near each other. Here, data is defined by its extracted concept vectors, which are then hashed into buckets based on the cosine distance measure. Another, in [34], kernelized LSH is applied to an utterance model in order to identify speakers. In this case the hamming distance metric is used. Other areas include use for species diversity estimation by allowing ease of grouping similar DNA sequences [59], and incremental clustering for affinity groups in Hadoop in order to store related data closer together [36].

Recently, [66] also proposed a new scheme based on entropy for LSH. The argument is that an improvement can be made for [15] such that the distribution of mapped values will be approximately uniform. Note how this approach attempts the same result as [37] using different methods.

7.7 Exercises

7.1 Given two documents D_{1} = {“His brow trembled”} and D_{2} = {“The brown emblem”}, compute all k-shingles for each document with k = 4.

7.2 Compute the Jaccard Similarity of the two sets of k-shingles for D_{1} and D_{2}.

7.3 Compute the signature matrix (minhash signature) of a given permutation of characteristic matrix in Table 7.5, using n = 3 different hash functions.

\begin{matrix}Element & S_{1} & S_{2} & S_{3} & S_{4} \\ a & 1 & 0 & 1 & 1 \\ b & 1 & 0 & 1 & 0 \\ c & 0 & 1 & 0 & 1 \\ d & 1 & 0 & 1 & 0 \\ e & 0 & 0 & 1 & 0 \end{matrix}

Table 7.5: A characteristic matrix.

7.4 Explain the reasoning behind the proof of Lemma 7.2.1.

7.5 Explain the overall procedure used to calculate document similarity, where locality-sensitive hashing is used in the process, and why locality-sensitive hashing would be favorable to use.

7.6 Show that the minhash function family is (d_{1}, d_{2}, 1 − d_{1}, 1 − d_{2})-sensitive.

7.7 Calculate hamming distance for the vectors v = [5, 1, 3, 2, 4] and u = [1, 1, 2, 2, 4].

7.8 Discuss the similarity between Jaccard distance and Hamming distance.

7.9 Show that the AND amplification construction is (d_{1}, d_{2}, p_{1}^{r}, p_{2}^{r})-sensitive.

7.10 When applying amplification constructions to a locality-sensitive family of functions, which order of composition is ‘better’, and why? Explain when you would want to use different orders of construction.

Figure 7.6: With a distance d > 2a, points x and z are in separate intervals unless θ is large enough.

Figure 7.7: Images are considered similar if their local features are hashed into the same bin in multiple hash tables. In this example, the blue and green circles are buckets where features are hashed; B is similar to C, and C is similar to D.

92

Dimensionality Reduction - Under Construction

The material of this chapter is from Dasgupta and Gupta [14], Dubhashi & Panconesi [19] and Matousek [].

8.1 Johnson and Lindenstrauss Theorem

In this section we prove a celebrated theorem of Johnson and Lindenstrauss from 1984.

Theorem 8.1.1 Let V be a set of n points in d-dimensions. A mapping f : ℝ^{d} → ℝ^{k} can be computed, in randomized polynomial time, so that for all pairs of points u, v ∈ V,

(1 − ϵ)||u − v||^{2} ≤ ||f(u) − f(v)||^{2} ≤ (1 + ϵ)||u − v||^{2},

where 0 < ϵ < 1 and n, d, and k ≥ 4(\frac{ϵ^{2}}{2} − \frac{ϵ^{3}}{3} )^{−1} ln n are positive integers.

Intuitively, the theorem says that points in d-dimensional space can be mapped to a k dimensional space, where k is significantly less than d, and the interpoint Euclidean distance between a pair of points in V is preserved up to a factor of (1 ± ϵ). The proof is based on showing that the squared length of a random vector is concentrated around its mean when the vector is projected onto a random k-dimensional subspace, i.e., with probability O(1/n^{2}), the length is within (1 ± ϵ) of the mean. In place of estimating the length of a unit vector after projecting it onto a random k-dimensional space, we will estimate the length of a random unit vector when we project it to a fixed k-dimensional space (e.g. in the space spanned by the first k-coordinates).

Recall Standardized Normal Distributions N (0, 1).

Lemma 8.1.2 Let X_{1}, …, X_{d} be d-independent standardized normal N (0, 1) random variables. Let X = (X_{1}, X_{2}, …, X_{d}) and define Y = \frac{1}{||X||}X. Then, Y is a random point drawn uniformly from the surface of the d-dimensional unit sphere.

Proof. Since the distribution for each X_{i} is given by f(x_{i}) = \frac{1}{\sqrt{2π}} e^{−x_{i}^{2}/2}, the distribution of X,

due to Independence, is Π_{i=1}^{d}[\frac{1}{\sqrt{2π}} e^{−x_{i}^{2}/2}] = \frac{1}{2π^{d/2}} e^{− \frac{1}{2} ∑_{i=1}^{d} x_{i}^{2}}. Observe that the density function of X only depends upon the distance of (X_{1}, …, X_{d}) from the origin, and thus it is spherically symmetric. Since Y is obtained by dividing this by ||X||, Y is a random point on the unit sphere.

From now on let Y = (Y_{1}, …, Y_{d}) be a random unit vector as stated in Lemma 8.1.2. Define a vector Z ∈ ℝ^{K}, which is obtained by projecting Y onto its first k-coordinates (i.e. ignoring all other coordinates, except the first k). Let L represent the squared length of Z (L = ||Z||^{2}).

Lemma 8.1.3 E[L] = k/d.

Proof.

E[L] = E[||Z^{2}||] = \frac{1}{||X||^{2}} E[X_{1}^{2} + ⋯ + X_{k}^{2}] = \frac{1}{||X||^{2}} ∑_{1=1}^{k} E[X_{i}^{2}] = k \frac{E[X_{1}^{2}]}{||X||^{2}}.

Observe that

||Y ||^{2} = 1 = \frac{1}{||X||^{2}} ∑_{i=1}^{d} X_{i}^{2}.

This implies that

d\frac{E[X_{1}^{2}]}{||X||^{2}} = 1.

Hence

E[L] = k \frac{E[X_{1}^{2}]}{||X||^{2}} = \frac{k}{d}.

Next we state the main lemma, first without proving it, and show how Theorem 8.1.1 follows from it.

Lemma 8.1.4 Let k < d. Then

1. If β < 1, then P (L ≤ β \frac{k}{d}) ≤ e^{\frac{k}{2} (1−β+ln β)}.

2. If β > 1, then P (L ≥ β \frac{k}{d}) ≤ e^{\frac{k}{2} (1−β+ln β)}.

Proof. (Proof of Theorem 8.1.1) The case of k ≥ d is trivial. Therefore, we assume that k < d. Let S be a random k-dimensional subspace of ℝ^{d} (for example we choose randomly k coordinates). Project each point v ∈ V and let its projection in ℝ^{k} be v′. Let Y := \frac{v−w}{||v−w||} be a unit vector, and let Z := \frac{v′−w′}{||v−w||} be its random projection. Let L = ||Z||^{2}. Note that Z behaves as required in Lemma 8.1.3. Hence E[L] = k/d. For 0 < ϵ < 1, set β = 1 − ϵ. Note that

P (||v′ − w′||^{2} ≤ (1 − ϵ)||v − w||^{2}k/d) = P (L ≤ (1 − ϵ)k/d)  (8.1)
≤ e^{\frac{k}{2} (1−(1−ϵ)+ln(1−ϵ))}  (8.2)
≤ e^{\frac{k}{2} (ϵ−(ϵ+ \frac{ϵ^{2}}{2} ))}  (8.3)
= e^{− \frac{kϵ^{2}}{4}}  (8.4)
≤ e^{−2 ln n}  (8.5)
= \frac{1}{n^{2}}  (8.6)

Note that for Equation 8.3 we used that ln(1 − ϵ) ≤ −ϵ − ϵ^{2}/2, for 0 < ϵ < 1. For Equation 8.5 (and 8.11), we use the fact that k ≥ 4(\frac{ϵ^{2}}{2} − \frac{ϵ^{3}}{3} )^{−1} ln n.

P (||v′ − w′||^{2} ≥ (1 + ϵ)||v − w||^{2}k/d) = P (L ≥ (1 + ϵ)k/d)  (8.7)
≤ e^{\frac{k}{2} (1−(1+ϵ)+ln(1+ϵ))}  (8.8)
≤ e^{\frac{k}{2} (−ϵ+(ϵ− \frac{ϵ^{2}}{2} + \frac{ϵ^{3}}{3} ))}  (8.9)
= e^{− \frac{k(ϵ^{2}/2−ϵ^{3}/3)}{2}}  (8.10)
≤ e^{−2 ln n}  (8.11)
= \frac{1}{n^{2}}  (8.12)

Set the map of each v ∈ V as f(v) = \sqrt{\frac{d}{k}}v′. Consider a pair of points u, v ∈ V, and by above calculations, the probability that (1 − ϵ)||u − v||^{2} ≤ ||f(u) − f(v)||^{2} ≤ (1 + ϵ)||u − v||^{2} does not hold is at most 2/n^{2}. Thus, for any pair of points, the probability this condition is not true is at most \binom{n}{2} \frac{2}{n} = 1 − \frac{1}{n}. Therefore, the mapping f has the desired distortion property. We can boost the success probability further by repeating the experiment several times. Observe that as the mapping is fairly straighforward and can be easily seen to be obtained in polynomial time.

Before we prove Lemma 8.1.4, we need the following.

Lemma 8.1.5 Let X be N (0, 1). Then E[e^{sX^{2}}] = \frac{1}{\sqrt{1−2s}}, for s < 1/2.

Proof. Note that X is given by the probability distribution f(x) = \frac{1}{\sqrt{2π}} e^{− \frac{x^{2}}{2}} . Moreover, expected

value of a function, say H(X), of a random variable X is given by E(H(X)) = ∫_{−∞}^{+∞} H(x)f(x)dx. Thus,

E[e^{sX^{2}}] = \frac{1}{\sqrt{2π}} ∫_{−∞}^{+∞} e^{sx^{2}}e^{− \frac{x^{2}}{2}} dx  (8.13)
= \frac{1}{\sqrt{2π}} ∫_{−∞}^{+∞} e^{−(1−2s)x^{2}/2}dx  (8.14)
set y = x\sqrt{1 − 2s}  (8.15)
= \frac{1}{\sqrt{1 − 2s}} \frac{1}{\sqrt{2π}} ∫_{−∞}^{+∞} e^{−y^{2}/2}dy  (8.16)
= \frac{1}{\sqrt{1 − 2s}} (since \frac{1}{\sqrt{2π}} ∫_{−∞}^{+∞} e^{−y^{2}/2}dy = 1)  (8.17)

Proof.(Proof of Lemma 8.1.4) First, we want to prove that for β < 1, P (L ≤ β \frac{k}{d}) ≤ e^{\frac{k}{2} (1−β+ln β)}. Equivalently, we want to show that

P (d(X_{1}^{2} + ⋯ + X_{k}^{2}) ≤ kβ(X_{1}^{2} + ⋯ + X_{d}^{2})) ≤ e^{\frac{k}{2} (1−β+ln β)}.

Note that P (d(X_{1}^{2} + ⋯ + X_{k}^{2}) ≤ kβ(X_{1}^{2} + ⋯ + X_{d}^{2}))

= P (kβ(X_{1}^{2} + ⋯ + X_{d}^{2}) − d(X_{1}^{2} + ⋯ + X_{k}^{2}) ≥ 0)  (8.18)
= P (e^{t(kβ(X_{1}^{2}+···+X_{d}^{2})−d(X_{1}^{2}+···+X_{k}^{2})} ≥ e^{t.0} = 1) (for t > 0)  (8.19)
≤ E[e^{t(kβ(X_{1}^{2}+···+X_{d}^{2})−d(X_{1}^{2}+···+X_{k}^{2})}] (Using Markov’s inequality)  (8.20)

Recall that if A and B are independent r.v. than E[AB] = E[A]E[B]. Moreover, E[A^{2}] = E[A]^{2}. Also, each of these X_{i}’s have identical distribution, and they are independent r.v.’s. After rearranging and replacing all X_{i}’s by X_{1}, we obtain

= E[e^{tkβX_{1}^{2}}]^{d−k}E[e^{t(kβ−d)X_{1}^{2}}]^{k}  (8.21)
= (\frac{1}{\sqrt{1 − 2tkβ}} )^{d−k}(\frac{1}{\sqrt{1 − 2t(kβ − d)}})^{k} (Using Lemma 8.1.5)  (8.22)

The above inequality holds, provided 0 < t < 1/2kβ. Now we want to minimize the quantity in Inequality 8.22. This is equivalent to maximizing

q(t) = (1 − 2tkβ)^{d−k}(1 − 2t(kβ − d))^{k}.

We obtain, using derivatives of a product etc., that maximum occurs when t = \frac{1−β}{2β(d−kβ)}. The value

of q(t = \frac{1−β}{2β(d−kβ)}) = (\frac{d−k}{d−kβ} )^{d−k}(\frac{1}{β} )^{k}. Substituting this back in Inequality 8.22, as 1/\sqrt{q(t)}, we

obtain that P (L ≤ β \frac{k}{d}) ≤ β^{\frac{k}{2}} (1 + \frac{(1−β)k}{(d−k)})^{(d−k)/2}. With simple manipulation and using the fact

that ln(1 + x) ≤ x, we obtain that β^{\frac{k}{2}} (1 + \frac{(1−β)k}{(d−k)})^{(d−k)/2} ≤ e^{\frac{k}{2} (1−β+ln β)}, as desired. One can show this by taking natural log on both the sides.

Next, using the identical method as above, we show that for β > 1, P (L ≥ β \frac{k}{d}) ≤ e^{\frac{k}{2} (1−β+ln β)}. This amounts to showing that

P (d(X_{1}^{2} + ⋯ + X_{k}^{2}) ≥ kβ(X_{1}^{2} + ⋯ + X_{d}^{2})) ≤ (\frac{1}{\sqrt{1 + 2tkβ}} )^{d−k}(\frac{1}{\sqrt{1 + 2t(kβ − d)}})^{k} (8.23)
= \frac{1}{\sqrt{q(−t)}}  (8.24)

Now the rest of the proof requires finding the value that maximizes q(−t). This is maximized at t = −\frac{1−β}{2β(d−kβ)}. Rest of the proof is same.

8.2 Applications

8.2.1 Sort of the Plan

We will first show that any n-point metric space (X, d) can be embedded isometrically into ndimensional L_{∞}. For this we will first define a metric space, then show that it cannot be embedded isometrically always in the plane endowed with Euclidean distance. Why should we even worry about these kinds of embedding? Then highlight the significance of JL-Theorem where embedding is done with some (not so significant) distortion by significantly reducing the number of dimensions. We will also breeze through some of the results by Matousek and Bourgain in the process. This material is derived from many sources, but the main link is the Lecture Notes of Harold Racke on a course on metric embeddings offered at IIIT Chicago a while ago.

8.3 Problems

8.1 Show that for any embedding f of a metric space (X, d) to another metric space (X′, d′), the distortion of f is at least 1.

98

Stable Matchings

This entire chapter is contributed by Michael St-Jules.

9.1 Stable Marriages

The stable marriage problem is the following: given n men and n women, each man with a list ranking the women in order of preference and each woman with a preference list of the men, find a stable matching of the men and women, i.e. n man-woman pairs that is stable.

Definition 9.1.1 An instance of the stable marriage problem is a set of M of men and a set W of women, each of size n ∈ ℕ and for each m ∈ M, an ordered list L(m) that is a permutation of the women in W and for each w ∈ W, an ordered list L(w) that is a permutation of the men in M.

We will use m, m′, m′′, m_{1}, m_{2}, m_{3}, …, m_{n} to denote men, and similarly for women with w. The lists L(m) and L(w) are m’s and w’s preference lists (or simply lists), respectively. We say m prefers w to w′, or w is a better partner for m than w′ or w′ is a worse partner for m than w, if w appears before w′ in L(m). We do the same for w, m and m′.

The following is an instance:

m_{1} [w_{1} w_{3} w_{2}] w_{1} [m_{2} m_{3} m_{1}]
m_{2} [w_{3} w_{1} w_{2}] w_{2} [m_{3} m_{1} m_{2}]
m_{3} [w_{3} w_{1} w_{2}] w_{3} [m_{1} m_{2} m_{3}]

In practice, the instance would be represented as two n × n matrices, one with the men’s lists filling out the rows and the other with the women’s lists filling out the rows.

Definition 9.1.2 A matching for an instance of the stable marriage problem is a set M ⊆ M ×W such that each m ∈ M and each w ∈ W appear in exactly one ordered pair in the set M.

Hence each man is paired with exactly one woman and each woman, with exactly one man. In fact, M : M → W defines a bijection, with M(m) as the partner of m ∈ M in the matching and M^{−1}(w) being that of w ∈ W.

Definition 9.1.3 A pair (m, w) ∈ M × W is a blocking pair for the matching M if m and w both prefer each other to their own partners in M.

Definition 9.1.4 A matching M is stable or a solution, if there is no blocking pair in M × W for M.

The motivation for such a definition is that given a blocking pair (m, w), m and w would leave their partners for each other.

Definition 9.1.5 (m, w) ∈ M ×W is a possible pairing if (m, w) ∈ M for some stable matching M. Furthermore, m is called a possible partner for w, and similarly w for m.

Definition 9.1.6 A stable matching M is man-optimal (man-pessimal) if each m ∈ M is paired with the w ∈ W he prefers most (least, respectively) of all possible partners. We let best(m) = w (worst(m) = w, resp.), to define a function best : M → W (worst : M → W, resp.).

Define woman-optimal, woman-pessimal, best : W → M and worst : W → M similarly. We will prove that the following algorithm, originally from Gale and Shapley [23], always

computes a stable matching (so that one always exists), and in O(n^{2}) time. We shall also see that the stable matching returned is man-optimal and woman-pessimal, so that it does not depend on the order in which men are chosen to propose. The following more general version of the algorithm is adapted from Kleinberg and Tardos [42], and its analysis is from Gale and Shapley’s original paper and some proofs adapted from Irving [33].

Stable marriage algorithm

While there remains a man m ∈ M who has no proposal to another held, do w := the next women on m’s list to whom m has not proposed m proposes to w if w holds no proposal, then w holds m’s proposal

else m′ := the man from whom w holds a proposal if w prefers m′ to m, then w rejects m else

w rejects m′

w holds m’s proposal

return the set M := {(m, w) ∈ M × W | w holds a proposal from m}

Let’s consider running the algorithm on the following instance:

m_{1} [w_{1} w_{3} w_{2}] w_{1} [m_{2} m_{3} m_{1}]
m_{2} [w_{3} w_{1} w_{2}] w_{2} [m_{3} m_{1} m_{2}]
m_{3} [w_{3} w_{1} w_{2}] w_{3} [m_{1} m_{2} m_{3}]

m_{1} proposes to w_{1}, who accepts: (m_{1}, w_{1}) m_{2} proposes to w_{3}, who accepts: (m_{1}, w_{1}), (m_{2}, w_{3}) m_{3} proposes to w_{3}, who rejects m_{3}: (m_{1}, w_{1}), (m_{2}, w_{3}) m_{3} proposes to w_{1}, who accepts m_{3} and rejects m_{1}: (m_{3}, w_{1}), (m_{2}, w_{3}) m_{1} proposes to w_{3}, who rejects m_{1}: (m_{3}, w_{1}), (m_{2}, w_{3}) m_{1} proposes to w_{2}, who accepts: (m_{3}, w_{1}), (m_{2}, w_{3}), (m_{1}, w_{2}) Then M := {(m_{3}, w_{1}), (m_{2}, w_{3}), (m_{1}, w_{2})} is a stable matching.

Lemma 9.1.7 The stable marriage algorithm runs in O(n^{2})-time.

Proof. We assume that preference checking by women and finding the next person on a preference list by men are O(1), and so given the women’s preference lists, we can scan them and form a ranking matrix for the women, where the entry ij corresponds to w_{i}’s ranking of m_{j}, i.e. m_{j}’s position on L(w_{i}). Similarly, given a ranking matrix for men, we can construct the men’s preference lists by copying the ranking matrix, pairing each entry with the corresponding woman ranked, and then sorting the women on each row with respect to rank. This preprocessing takes O(n^{2}) time.

Observe that each man proposes at most once to each woman, so at most n^{2} proposals are made, and each iteration of the while loop corresponds to a single proposal and takes time O(1). Hence O(n^{2}) time in all.

Note also that O(n^{2}) is linear in the input size.

Lemma 9.1.8 After the moment w ∈ W is first proposed to, w always holds a proposal.

Proof. Once w holds a proposal, she only rejects it for a better one, and there is no ’unproposing’.

In fact, each man does worse with each proposal made, while women are never worse off after a proposal than before.

Lemma 9.1.9 The set M returned at the end of the stable marriage algorithm is a matching.

Proof. Suppose, by way of contradiction, that M is not a matching. First, since women can hold only a single proposal, each m ∈ M and w ∈ W appears at most once in M, so it follows that some m ∈ M or some w ∈ W is unpaired. These two conditions are equivalent, again since women can only hold a single proposal, and also because |M| = |W | = n. Hence, we have both an unpaired man m ∈ M and an unpaired woman w ∈ W. Since m is unpaired, he had no proposal held at the end of the procedure and so must have proposed to w before termination. Then, by Lemma 9.1.8, w could not have been unpaired.

Lemma 9.1.10 If w ∈ W rejects m ∈ M in the stable marriage algorithm, then m and w cannot be partners in any stable matching.

Proof. Suppose m’s rejection by w is the first to occur such that (m, w) is a pairing in a stable matching M. w rejected m because she already held or later received a proposal from a better partner, say m′. Then, so that (m′, w) is not a blocking pair for M, contradicting stability, it must be that m′ prefers his own partner, w′, in M to w. However, since m′ prefers w′ to w, m′ must have proposed to and been rejected by w′ before w, contrary to m’s rejection by w being the first such.

Corollary 9.1.11 If in the stable marriage algorithm, m ∈ M proposed to w ∈ W, then, in any stable matching

(i) m cannot have a better partner than w;

(ii) w cannot have a worse partner than m.

Proof. m proposed to w since he was rejected by all the women he preferred to w and hence could not be paired with (by Lemma 9.1.10), so (i) follows.

Suppose, by way of contradiction, that (m′, w) ∈ M × W is a pair in some stable matching M, where w prefers m to m′. Then m is paired with some w′ ∈ W and by (i), m prefers w to w′. Hence (m, w) is a blocking pair for M, contradicting stability.

Theorem 9.1.12 The stable marriage algorithm computes a man-optimal and woman-pessimal stable matching in O(n^{2})-time.

Proof. The O(n^{2}) upper bound follows from Lemma 9.1.7. The set M returned is a matching by Lemma 9.1.9. M is stable, since if (m, w) ∈ M × W were a blocking pair, then m prefers w to M(m), contradicting Corollary 9.1.11(i). That M is man-optimal and woman-pessimal also follows from Corollary 9.1.11.

Corollary 9.1.13 A stable matching M is man-optimal if and only if it is woman-pessimal.

Corollary 9.1.14 There always exists a stable matching, i.e. M := {(m, best(m)) ∈ M × W |m ∈ M} = {(worst(w), w) ∈ M × W |w ∈ W }.

9.2 Hospitals/Residents (College Admissions)

The hospitals/residents problem is a generalization of the stable marriage problem in which residents and hospitals are considered instead of men and women, allowing multiple residents to be matched to a hospital up to its quota. The problem was presented as the college admissions problem and solved by Gale and Shapley in the same paper as the stable marriages problem, with essentially the same algorithm, the chief difference being that colleges (hospitals) can hold multiple applications, up to their quota. In this setting, we do not require that every applicant is assigned to a college or that every college meets its quota, and we also allow the possibility that a college may prefer not to accept a particular applicant even when its quota is not met or that an applicant may prefer not to apply to a particular college even if it would mean not being accepted anywhere.

In the applicant-optimal procedure, when a college receives a new application while its currently held applicants would meet its quota, it rejects the worst application from the new one and the ones it already held.

In the college-optimal procedure, colleges make offers to applicants which have not rejected them up until they have enough offers held to fill their quotas.

Definition 9.2.1 An instance of the college admission problem consists of a set A of n applicants, a set C of m colleges with a quota q_{c} ≥ 1 for each c ∈ C and the (possibly incomplete) preference lists for all a ∈ A, c ∈ C.

Definition 9.2.2 An assignment is a set A ⊆ A × C such that each applicant a ∈ A appears in at most one pair, and each college c ∈ C appears in at most q_{c} pairs.

Definition 9.2.3 A blocking pair (a, c) ∈ A × C for the assignment A is a pair such that a and c are in each others’ preference lists and one of the following holds:

(i) c’s quota is met, a prefers c to his current college and c prefers a to at least one of its applicants;

(ii) c’s quota is not met and a prefers c to his current college.

Definition 9.2.4 An assignment A is stable if there is no blocking pair for it in A × C.

It’s clear that if we had a blocking pair (a, c), then in case (i), c would like to reject its worst student in order to accept a, and in case (ii), c would simply accept a because it has the room to do so.

Lemma 9.1.10 generalizes to both the applicant- and college-optimal algorithms, and it follows from it that the assignment is stable and, in fact, applicant- or college-optimal, respectively.

For the applicant-optimal procedure, we can represent the list of applicants to a college using a heap data structure with keys as the applicant ranking for that college, since we only need access to the worst applicant on the list when a college must reject one. Then this lookup is O(1) and insertion and deletion are O(log(q_{c})). For simplicity’s sake, we assume that ∀a ∈ A, c ∈ C, a ∈ L(c) ⇒ c ∈ L(a) to avoid having each college rank every applicant, even those which it would never accept. Otherwise, when a ∈ A applies to c ∈ C, we must check if a ∈ L(c) and we’d like to do this in constant time (in particular, without having to traverse L(c)). For q := the largest quota greater than or equal to n (or 2 if none, since we only need to reject applications if a college’s quota is met), the running time will be O(log(q)(n + Σ_{a∈A}|L(a)|)) ⊆ O(log(q)nm).

For the college-optimal procedure, there is no need to store each college’s list of applicants; we need only know how many offers each college has made. Here, we assume that ∀a ∈ A, c ∈ C, c ∈ L(a) ⇒ a ∈ L(c) to avoid having each applicant rank every college, even those to which they would never apply. The college-optimal algorithm’s running time will be O(m+Σ_{c∈C}|L(c)|) ⊆ O(n + m + Σ_{a∈A}|L(a)| + Σ_{c∈C}|L(c)|) ⊆ O(nm), where the second of these three is linear.

9.3 Stable Roommates

The stable roommates problem also generalizes stable marriages: instead of separate sets of men and women, there is only a single set of n individuals, each with a list the other n − 1 individuals sorted by preference. Irving [33] gave an O(n^{2})-time algorithm to find a stable matching if one exists, and reporting that none exists, otherwise.

Definition 9.3.1 An instance of the stable roommates problem is a set of X of individuals, with size n ∈ ℕ, and for each x ∈ X, a preference list L(x), that is a permutation of the remaining individuals in X. We may assume X = {1, 2, …, n}. Again, we say x prefers y to z, if y appears before z in L(x). Preference lists (not necessarily the original ones) together are called a (preference) table, and a table T′ is a subtable of a table T, if x ∈ X is in y ∈ X’s list in T′ implies the same in T and x comes before z on y’s list in T′ implies the same in T. A table for the instance is a subtable of the original table. Since the instance is understood to be fixed, we will usually just refer to a table for the instance just as table.

Definition 9.3.2 A matching for an instance of the stable roommates problem is a bijection M : X → X such that M(x) ≠ x for all x ∈ X and M(x) = y iff M(y) = x (or M(M(x)) = x for all x ∈ X, i.e. M is its own inverse).

Note that M = {(x, M(x)) ∈ X × X|x ∈ X}.

It follows from this definition that n must be even for a matching to exist, since we do not allow x ∈ X to be alone (M’s domain is X) or matched with himself (M(x) ≠ x).

Equivalently, we can define a matching as a set of unordered pairs in P(X) (the power set of X) where each individual appears in exactly one pair. In this case, any matching must have size n/2.

Blocking pair and stable matching/solution and possible pairing (possible partner) are defined again as in Definitions 9.1.3, 9.1.4 and 9.1.5.

Any stable matching for a stable marriage instance will be a solution to the stable roommates instance constructed by adding to the end of each man’s preference list the remaining men and doing the same for the women. Like the stable marriage problem, then, a stable roommates instance may have multiple solutions. Unlike the stable marriage problem, however, not every instance of the stable roommates problem has a solution, e.g.

1 [2 3 4] 2 [3 1 4] 3 [1 2 4] 4 [1 2 3]

There are three matchings, but each has a blocking pair:

{(1,2), (3,4)} is blocked by (2,3); {(1,3), (2,4)}, by (1,2); and {(1,4), (2,3)}, by (1,3).

Note that by the symmetry of the instance, 4’s list can actually be an arbitrary permutation of {1, 2, 3}, and the same will still hold.

Irving’s algorithm is split up into two phases, the first of which is similar to the stable marriage algorithm, but we now allow anyone to propose or hold proposals (and even both). In pseudocode:

Phase 1

while there remains x ∈ X who has no proposal held and has not proposed to everyone, do next := the next on x’s list to whom x has not yet proposed x proposes to next if next holds no proposal, then next holds x’s proposal

else y := the individual from whom next holds a proposal if next prefers y to x, then next rejects proposer else

next rejects y

next holds x’s proposal

In words, we let those who have no proposal held to propose to the individual they prefer most of those to whom they’ve not already proposed, if one exists, but if there is no such individual left, then we stop the procedure.

We assume again that we have a ranking matrix for the individuals as we described in Lemma 9.1.7, and if not, we can construct one in O(n) time. Then, we can use a linked list to implement the set of individuals who have no proposal held, so that insertion and deletion are O(1), resulting in O(n) preprocessing time to copy X. Alternatively, we can traverse X letting proposer be each individual in turn with a nested while-loop inside that updates proposer to y if y is rejected by next, so that rejected individuals have priority in proposing. In either case, the operations inside the loop(s) are O(1) all together and each iteration corresponds to a single proposal (except possibly the last, if there is no stable marriage), of which most (n − 1)^{2}, so the procedure takes O(n^{2}) time.

Results corresponding to Lemma 9.1.8, Lemma 9.1.10, Corollary 9.1.11, which were adapted from Irving’s original paper, hold again in this setting with nearly identical proofs (so they are omitted here):

Lemma 9.3.3 After the moment x ∈ X is first proposed to, x always holds a proposal.

Lemma 9.3.4 If y ∈ X rejects x ∈ X in Phase 1, then x and y cannot be partners in any stable matching.

Corollary 9.3.5 If during Phase 1, x ∈ X proposed to y ∈ X, then, in any stable matching

(i) x cannot have a better partner than y;

(ii) y cannot have a worse partner than x.

Corollary 9.3.6 If after Phase 1, x ∈ X has no proposal held and has proposed to everyone on his list, then no stable matching exists.

Proof. x was therefore rejected by everyone, so by the Lemma 9.3.4, cannot be partners with anyone in a stable matching.

Then we would exit reporting so, as we should, but after we find that an individual’s list becomes empty after the following reductions to the original table:

Corollary 9.3.7 If, after Phase 1, y ∈ X holds a proposal from x ∈ X, then y’s preference list can be reduced (without eliminating possible partners) by deleting from it

(i) all those to whom y prefers x (i.e. all those after x);

(ii) all those who hold a proposal from a person whom they prefer to y (including those who have rejected y, i.e. all those before x);

In the resulting table,

(iii) y is first on x’s list; and x, last on y’s;

(iv) in general, b appears on a’s list if and only if a appears on b’s.

Proof.(i) and (ii) follow from Corollary 3.4.1.(ii); and (iii), from (i) and the parenthesized part of (ii). For (iv), suppose a holds a proposal from c; and b, from d. Then a is on b’s list ⟺ b does not prefer d to a (by (i)), and a does not prefer c to b (by (ii)) ⟺ a is on b’s list (by (ii) and (i)).

We will refer to this table as the Phase 1 table. In fact, for a particular instance, the Phase 1 table is always the same. When no list is empty, the resulting table is an example of a stable table:

Definition 9.3.8 A table T for an instance is stable if, in T:

(i) x is first in y’s list, abbreviated f_{T} (x) = y, iff y is last in x’s list, abbreviated l_{T} (y) = x;

(ii) for x, y ∈ X, x is not in y’s list iff y is not in x’s iff x prefers l_{T} (x) to y or y prefers l_{T} (y) to x (in the original table);

(iii) no list is empty.

(i) and (iii) imply that f_{T} and l_{T} are inverse functions X → X, and so must both be bijective, i.e. injective and surjective. Note that once all of the f_{T} (x) are determined (or all of the l_{T} (x) are), then the whole table is determined by (ii). It should be clear that any table can be made so that (i) and (ii) hold by reducing the lists, although a list may become empty in doing so. In fact, any table for which (i) and (ii) hold is a subtable of the Phase 1 table, and the reductions to obtain the Phase 1 table are exactly those necessary to establish (i) and (ii):

Lemma 9.3.9 Definition 9.3.8 (i) and (ii) hold for the Phase 1 table, and so if no list is empty, the table is stable.

Furthermore,

Lemma 9.3.10 If each list in a stable table T contains exactly one person, then they specify a stable matching.

Proof. By Definition 9.3.8(i), the lists specify a matching. Suppose that (x, y) ∈ X × X but x and y are not paired in the matching. Then x and y are not in each others’ lists in T. Then, by Definition 9.3.8(ii), x prefers the last (i.e. sole) person on his list to y, or y, the last (i.e. sole) person on his list, to x, so that (x, y) cannot be a blocking pair. The matching is therefore stable.

Then, using Lemma 9.3.9, the following holds:

Corollary 9.3.11 If each list in the Phase 1 table contains exactly one person, then they specify a stable matching.

Before we include Phase 2 of the algorithm in pseudocode, we describe and justify it.

We prove that at each step until termination, Definitions 9.3.8(i) and (ii) hold. The base case is established by Lemma 9.3.9. Hence, if the reduced lists contain exactly one person each, then we are done by Lemma 9.3.10 (Corollary 9.3.11). Otherwise, at least one is empty, in which case, we will see that there is no stable matching so that we end the procedure, or at least one has more than one person in it. In this last case, we will search for a rotation (an ”all-or-nothing cycle” in Irving’s original paper) and eliminate it: we reduce the preference lists again with respect to this rotation in such a way that Definitions 9.3.8(i) and (ii) continue to hold. If any list becomes empty, we exit reporting that no stable matching exists. Otherwise, the inductive step holds.

Definition 9.3.12 A rotation in a stable table is a cyclic sequence (a_{1}, b_{1}), (a_{2}.b_{2}), …, (a_{r}, b_{r}) ∈ X × X of pairs of individuals such that, where a_{r+1} = a_{1} and b_{r+1} = b_{1}:

(i) the a_{i}’s are distinct;

(ii) b_{i} is first on a_{i}’s list (f_{T} (a_{i}) = b_{i}) for each i; equivalently, a_{i} is last on b_{i}’s list for each i (l_{T} (b_{i}) = a_{i});

(iii) b_{i+1} is the second on a_{i}’s list for each i.

First, note that a_{i} is in b_{i+1}’s list for each i.

Then, a rotation can be depicted in the following way with respect to the reduced preference lists in the stable table:

a_{1} [ b_{1} b_{2} …] b_{1}[ … a_{r} … a_{1}]
a_{2} [ b_{2} b_{3} …] b_{2}[ … a_{1} … a_{2}]
a_{3} [ b_{3} b_{4} …] b_{3}[ … a_{2} … a_{3}]
⋮  ⋮
a_{i} [ b_{i} b_{i+1} …]  b_{i}[ …a_{i−1} … a_{i}]
⋮  ⋮
a_{r−1} [ b_{r−1} b_{r} …] b_{r−1}[ …a_{r−2} …a_{r−1}]
a_{r} [ b_{r} b_{1} …]  b_{i}[ …a_{r−1} … a_{r}]

We observe that a rotation is completely determined once one a_{i} or b_{i} is known: once a_{i} is known, (iii) gives us b_{i+1} and (ii), a_{i+1}; once b_{i} is known, (ii) gives us a_{i} and we repeat the previous argument. It then follows that since the a_{i} are all distinct, so must be the b_{i}.

Also, note that the length of the rotation is r ≥ 2, since otherwise b_{1} = b_{2}, so that b_{1} = b_{2} is both first and second on a_{1}’s list, a contradiction.

Lemma 9.3.13 A stable table in which a’s list has length at least 2 contains a rotation.

Proof. To find a rotation, we let p_{1} ∈ X be an individual whose list has length at least 2, and define, inductively:

q_{i+1} = the second in p_{i}’s reduced list

p_{i+1} = the last in q_{i+1}’s reduced list (so q_{i+1} is first in p_{i+1}’s)

until this sequence repeats some p_{s}, so that p_{s+r} = p_{s}. It follows that each p_{i} will have at least 2 individuals on his reduced list, for if p_{i} has only q_{i} on his list and i is the least such that this occurs, then q_{i} must have only p_{i} on his list by Definition 9.3.8(i), because q_{i} is both the first and the last on p_{i}’s list. By uniqueness, it follows that p_{i} = p_{i−1}, contradicting i being the least. Hence, we can, in fact, choose q_{i+1} at each step.

Then, we let a_{i} = p_{s+i−1} for i = 1, …, r and b_{i} = q_{s+i−1}, for i = 1, …, r to give us a rotation, and we call p_{1}, …, p_{s−1} the tail for the rotation.

Tails are not generally unique for a rotation, since instead of starting our search from p_{1}, starting from any other p_{j} would have also lead to the same rotation. Also, tails may be empty, as is the case when p_{1} = a_{1}.

Then, having found a rotation, we eliminate the rotation: we force each b_{i} to reject a_{i} and have each a_{i} propose to b_{i+1}, reducing the preference lists again to satisfy Definition 9.3.8 (i) and (ii) and proving the inductive step in our reductions. Hence x ∈ X and y ∈ X are removed from each other’s lists in the rotation elimination if (and only if) one of them is b_{i} for some i and the other succeeds a_{i−1} in b_{i}’s list. The result for the cycle is that, for each i:

a_{i} [ b_{i} b_{i+1} …] becomes a_{i} [ b_{i+1} …], and

b_{i} [ … a_{i−1} … a_{i}] becomes b_{i} [ … a_{i−1}].

Additionally, we remove b_{i} from the lists of the successors of a_{i−1} in b_{i}’s list, so that the removals remain symmetric. The proof for the following lemma is incomplete.

Lemma 9.3.14 If there exists a rotation (a_{1}, b_{1}), …, (a_{r}, b_{r}) in a stable table T and the table resulting from the elimination of this rotation, T′, has no empty list, then T′ is a stable subtable of T′:

(i) f_{T′}(a_{i}) = b_{i+1} for each i;

(ii) l_{T′}(b_{i}) = a_{i−1} for each i;

(iii) f_{T′}(x) = f_{T} (x) for x ∈ X, x ≠ a_{i} for each i, and l_{T′}(x) = l_{T} (x) for x ∈ X, x ≠ b_{i} for each i;

(iv) if p_{1}, …, p_{s−1} is a tail (as described in Lemma 9.3.13 for the rotation and the 2nd on p_{j}’s list changes after the rotation elimination, then either p_{j} is matched in T′ or j = s − 1 (i.e. the last on the tail), but not both.

Proof. Stability of T′ and (i)-(iii) follow from the way we eliminate rotations.

For (iv), suppose p_{j}’s 2nd value changed from q_{j+1} ∈ X after the rotation elimination. Then this occurred either because f_{T} (p_{j}) was removed from p_{j}’s list and vice-versa (so f_{T′}(p_{j}) = q_{j+1} or q_{j+1} was also removed) or p_{j} and q_{j+1} were removed from each other’s lists, so that one of them is b_{i} for some i and the other succeeded a_{i−1} in b_{i}’s list before the rotation was eliminated.

If f_{T} (p_{j}) was removed from p_{j}’s, then by (i) and (iii), it follows that p_{j} = a_{i} for some i, contrary to p_{j} being in the tail. Hence, p_{j} = b_{i} or q_{j+1} = b_{i} for some i.

If p_{j} = b_{i}, then p_{j} prefers a_{i−1} to q_{j+1}, where both were on his list but q_{j+1} was 2nd, so that a_{i−1} was first, i.e. f_{T} (p_{j}) = a_{i−1}, and since l_{T′}(p_{j}) = l_{T′}(b_{i}) = a_{i−1}, a_{i−1} is the only individual remaining on p_{j}’s list and vice-versa by Definition 9.3.8(i), so that p_{j} = b_{i} and a_{i−1} are matched.

Otherwise q_{j+1} = b_{i}, i.e. the second on p_{j}’s list was b_{i} and so p_{j+1} was last on b_{i}’s list, i.e. p_{j+1} = a_{i}. Then p_{j} was not among the a_{k} but p_{j+1} was, so that p_{j+1} must have been the first, i.e. p_{j+1} = a_{1} and so j = s − 1 (and i = 1).

Why not both? This is unclear!

The following lemma, whose proof we leave as an exercise, justifies rotation elimination.

Lemma 9.3.15 Let (a_{1}, b_{1}), …, (a_{r}, b_{r}) be a rotation in a stable table T. Then,

(i) in any stable matching contained in T, either a_{i} and b_{i} are partners for all values of i or for no value of i;

(ii) if there is such a stable matching in T in which a_{j} and b_{j} are partners, then modifying it so that a_{i} and b_{i+1} are instead partners for each i also gives a stable matching.

Then, the following two results follow immediately, by induction on the Phase 2 steps outlined earlier:

Corollary 9.3.16 If the original problem instance admits a stable matching, then there is a stable matching contained in any of the tables in the sequence of reductions.

Corollary 9.3.17 If one or more among the lists in one of the tables in the sequence is empty, then the original problem instance admits no stable matching.

With that above two corollaries, we’ve proven the correctness of the procedure, whose remainder we now include:

Phase 2

reduce the preference lists as described in Corollary 9.3.7 if any list becomes empty during the reductions, then

exit the procedure reporting that there is no stable matching if, after the reductions, each list has a unique individual in it, then

return M := {(x, y) ∈ X × X | y is in x’s reduced list} while there remains an individual in X with at least 2 in his reduced list, do

find a rotation using the tail of the previous one (if any)

eliminate the rotation

if any list becomes empty during the reductions, then

exit the procedure reporting that there is no stable matching return M := {(x, y) ∈ X × X | y is in x’s reduced list}

The only issue that remains to implement Phase 2 efficiently.

In the first reduction after Phase 1, we keep track of the first individual in X whose reduced list has size at least 2, called first unmatched.

When Phase 2 starts, to find rotations, we first store an array of length n with entry i indicating whether we’ve ’visited’ i ∈ X in our search for a rotation. We of course initialize each entry to ’unvisited’. Then, at each step in which we search for a rotation, we store a linked list of visited individuals, the p_{i}, for i = 1, …, s + r − 1, and mark them as visited in the array until we find one that’s already been visited, starting with p_{1} = first unmatched for the first rotation. We eliminate the rotation we’ve found and then ’unvisit’ all of the a_{i} and remove them from our list.

If there is no tail from the previous rotation, we start at first unmatched (which may have to be updated beforehand).

If there is a tail, we start our search from p_{s−1}, the last individual in the tail, to avoid repeating the sequence p_{1}, … p_{s−1}. This is justified by Lemma 9.3.14, since we have two possibilities. The first is that our new search starting from p_{1} will lead back to p_{s−1} as the 2nd individuals on the p_{j}’s lists up to but not necessarily including j = s − 1 (by (iv)) as well as the last individuals on the q_{j}s’ lists (up until the rotation) remained the same (by (iii)). The second possibility is that once p_{j} is repeated for some j > s − 1 in our new search, it will follow that each p_{k}, for k ≥ j, will have at least 2 on his list by induction (in the same way we guaranteed each p_{i} had at least two on his list for each i to prove Lemma 9.3.13, so that those p_{k} are still unmatched and hence the 2nd on their lists remained the same after eliminating the rotation (by (iv)).

Furthermore, reducing the preference lists can be done implicitly by storing the indices of the first, second and last person on an individual’s list that should remain in the reduced list and updating these as necessary. We store arrays for these, where f[x], s[x] and l[x] are the corresponding values for x. Then, with the ranking matrix, each removal can then be done in constant time, since it just means comparing the rank of an individual to be removed with the indices of the first and last and updating these appropriately. We only update the value of s[x] during the search for a cycle, and the new value can be found by traversing x’s list from s[x] until we find some p who prefers x to l[p], i.e. p and x are in each others’ reduced lists, and we update s[x] := p.

Theorem 9.3.18 The running time of Irving’s algorithm is O(n^{2}).

Proof. Phase 1 takes O(n^{2}) time. Since each removal takes constant time and there are at most n(n−2)+1 removals (n for each individual, n−2 to reduce an individual’s list to a single individual and the +1 in the case that one list becomes empty and we stop there), so all removals together in all reductions require O(n^{2}) time. Checking whether a stable matching is found and constructing it also takes O(n^{2}) time, as we only scan the list X and pair each x ∈ X with f[x] = l[x].

first unmatched is updated at most n times and s[x], at most n − 1 times for each x, corresponding to traversing the lists from left to right at most once, so together, this is again O(n^{2}).

All that remains is to show that the time spent finding rotations is also O(n^{2}):

If m rotations are found and eliminated during the entire procedure, we let t_{i} and r_{i} denote the lengths of the tail and the rotation for the ith rotation. For i = 1, … m, finding the ith rotation took at most c + d(r_{i} + t_{i} − t_{i−1}) + er_{i} operations, for some constants, c, d, e > 0, since r_{i} + t_{i} is the number of individuals in the list of p_{j}, but the first t_{i−1} were already visited in the search for the i − 1th, so we do not revisit them, and we only visit r_{i} + t_{i} − t_{i−1} new individuals at this step (where t_{0} = 0, hence d(r_{i} + t_{i} − t_{i−1}). Also, after the search, we unvisit the a_{j} of the cycle, hence er_{i}.

Note also that since the elimination of the ith rotation means at least 2r_{i} removals from the preference lists, and r_{i} ≥ 2 for each i, so that

4m ≤ 2Σ_{i=1}^{m}r_{i} ≤ n^{2}

Then, taking the sum over all the rotations, we get at most

Σ_{i=1}^{m}[c + d(r_{i} + t_{i} − t_{i−1}) + er_{i}] = cm + (d + e)Σ_{i=1}^{m}r_{i} + d(t_{m} − t_{0})

≤ \frac{c}{2}n^{2}/4 + \frac{d + e}{2}n^{2} + dn,

which is O(n^{2}).

9.4 Exercises

9.1 Prove Lemma 9.3.14. Hint: for (ii), first consider the case where a_{i} = b_{j} for some i, j and use (i).

9.2 Geometric Stable Roommates. Consider the stable roommate problem where the individuals are elements of the real numbers, ℝ and x prefers y to z iff |x − y| < |x − z|, i.e. the norm replaces the preference lists, and assume there are no ties. Describe an algorithm to find a stable matching in Θ(nlog(n))-time. (Hint: use dynamic binary search trees.) What about the vector space ℝ^{d} for d ∈ ℤ^{+}, where x = (x_{1}, x_{2}, …, x_{d}) and |x| = \sqrt{(x_{1})^{2} + ⋯ + (x_{d})^{2}}?

9.3 Median Matching. Given a stable roommates instance with a solution, let S be a subset of all the stable matchings such that |S| is odd. Show that M, defined by M(x) := the x’s median partner in all matchings in S, is a stable matching. Hint: first consider the case where S contains only 3 stable matchings.

More Exercises

1. Construct the string matching automaton for the pattern P = aabab and illustrate its operation on the text string T = aaababaabaab.

2. Assume that the decision version of the 3-SAT, Vertex Cover and Clique problems are NPcomplete. An independent set of an undirected graph G = (V, E) is a subset I of V such that no two vertices in I are connected by an edge in E. The decision version of the independent set problem (G, k), k ≥ 0, is to determine whether G has an independent set of size at least k. Prove that the Independent Set problem is NP-Complete.

3. Let G = (V, E) be an undirected connected simple graph. A matching is a set of edges of G such that no two edges in the set are incident on the same vertex. A matching is maximal if it is not a proper subset of any other matching. A matching is maximum if the number of edges in the matching is largest. A vertex cover of G = (V, E) is a set of vertices V′ ⊆ C such that if (u, v) ∈ E, then either u ∈ V′ or v ∈ V′ or both in V′. The size of the vertex cover is the cardinality of the set V′.

(a) Show that the size of a maximum matching in G is a lower bound on the size of any vertex cover of G.

(b) Consider a maximal matching M in G = (V, E). Let

T = {v ∈ V : some edge in M is incident on v}.

What can you say about the subgraph of G induced by the vertices of G that are not in T. Conclude that 2|M| is the size of a vertex cover for G.

(c) Present an O(|E|) time (greedy) algorithm to compute maximal matching.

(d) Conclude that the above algorithm for computing maximal matching is a 2approximation algorithm for maximum matching.

Consider the Proposal Algorithm that we discussed in the class. Its outlined below: Input: A list of n men and n women, where each man has an ordered list of n women (a permutation) whom he will like to marry. Similarly, each woman has an ordered list (a permutation) of n men whom she will like to marry. Output: A set of n pairs forming a stable perfect matching. Each pair consists of a man and a woman.

Proposal-Algorithm (M,W)
1. While there exists an unmarried man m do
2.     m proposes to the most preferred woman w on his list
         whom he has not proposed so far.
3.     if w is not married then w marries m (end if)
4.     if w is married to m' but prefers m over m' then
5.        w divorces m' and marries m
       (end if)
   (endwhile)

(a) Prove or Disprove: The output of this algorithm is always the same and is independent of in which order the men are picked up in Step 1. In other words the above algorithm produces the same set of stable matchings and is independent of the choice of men m in Line 1.

(b) Show that this algorithm is favorable to men compared to women. One possible way to show this is to construct example(s) where women do much better than men when the same algorithm is run by interchanging men with women.

(c) List all the data structures (including the operations) that you will use to implement the Proposal Algorithm. (e.g. Stacks/Arrays/...).

(d) State the complexity of the Proposal Algorithm using O() notation (Remember to use your data structures and their operations from Problem 3). (Recall that there are at most n ∗ n iterations in this algorithm, since a man never proposes to the same woman twice and in each iteration there is a proposal made)

(e) Can you devise a proposal algorithm that is unbiased?

4. Given a binary tree with all the relevant pointers (child/parent), describe a simple algorithm, running in linear time, that can compute and store the size of the subtrees at each node in the tree. (Size of a subtree at a node v is the total number of nodes, including itself, in the subtree rooted at v.) Justify why your algorithm is correct and analyze it and show that it runs in linear time.

5. Outline a search algorithm, running in O(log n) time, to report the i-th smallest number in a set consisting of n elements. The set is represented as a red-black tree where in addition to the usual information that we store at a node (pointer to left child, right child, parent, key, colour) we also store the size of the subtree at that node. The parameter i is supplied to the search algorithm at the run-time and assume that the red-black tree with the additional information about the size of the subtrees has been precomputed.

6. Assume that we have n-integers in the range 1000 to 9999. In the radix-sort method we sorted them by first sorting them using the (stable) counting sort by the Least-Significant digit and then the next least significant digit and so on. Why does this algorithm works? Sketch the main idea in the proof. Why does this algorithm fails when we first sort them using most-significant bit and then the second most significant bit and so on?

7. Assume that we are given n intervals (possibly overlapping) on a line. Each interval is specified by its left and right end points. Devise an efficient algorithm that can find a point on the line which is contained in the maximum number of intervals. Your algorithm should run in O(n^{2}) time. (Bonus Problem: Devise an algorithm that runs in O(n log n) time? Another Bonus Problem: Show that this problem has Ω(n log n) as its lower bound.)

8. Devise an O(n log k) time algorithm to merge k-sorted lists into a single sorted list, where n is the total number of elements in all input lists.

9. Suppose you are given n-points in the plane. We can define a complete graph on these points, where the weight of an edge e = (u, v), is Euclidean distance between u and v. We need to partition these points into k non-empty clusters, for some n > k > 0. The property that this clustering should satisfy is that the minimum distance between any two clusters is maximized. (The distance between two clusters A and B is defined to be the minimum among the distances between pair of points, where one point is from cluster A and the other from cluster B.) Show that the connected components obtained after running Kruskal’s algorithm till it finds all but the last k−1 (most expensive) edges of MST produces an optimal clustering.

10. Although the 3CNF-SAT is NP-Complete, show that in polynomial time we can determine whether a boolean formula given in disjunctive normal form is satisfiable; the formula consists of n variables and k clauses. A formula is in Disjunctive normal form, if clauses are joined by ORs and literals within a clause are joined by ANDs (DNF is OR of ANDs and CNF is AND of ORs!). You need to provide an algorithm and show that its correct and its running time is polynomial in n and k.

11. Although the 3CNF-SAT is NP-Complete, show that 2CNF-SAT is solvable in polynomial time. (This is same as exercise 34.4-7 in Book and contains Hints!).

12. Given an integer m × n matrix A and an integer m-vector b, the 0-1 integer programming problem asks whether there is an integer n-vector x with elements in the set {0,1} such that Ax ≤ b. Prove that 0-1 integer programming problem is NP-Complete by providing a reduction from 3CNF-SAT or Subset-Sum problem.

13. Construct an instance of the subset-sum problem corresponding to the following 3CNF-SAT:

(x_{1} ∨ ¬x_{2} ∨ ¬x_{3}) ∧ (¬x_{1} ∨ ¬x_{2} ∨ ¬x_{3}) ∧ (¬x_{1} ∨ ¬x_{2} ∨ x_{3}) ∧ (x_{1} ∨ x_{2} ∨ x_{3})

Provide a satisfying assignment to 3CNF-SAT and show that it provides a valid solution for the subset-sum problem.

14. For the same 3CNF-SAT in Problem 4, illustrate the reduction to the clique problem. Construct an equivalent graph and show that for a satisfying assignment you have an appropriate size clique and vice verse (Use colours to illustrate the clique!)

15. Consider the problem of CNF-Satisfiability (we are not putting any restriction on number of literals in each clause), where each variable only occurs at most twice (positive and negative form of a variable are not counted separately). Show that satisfiability can be determined in polynomial time in this case.

Hint: If a variable only occurs in its positive (or negative) form, then we can just satisfy those clauses and remove them from consideration. The problem starts when we have both version of the variable (one clause containing it in positive form and the other in the negative form - but then the following resolution rule applies: if we have two clauses of the form (x ∨ w ∨ y ∨ z) ∧ (¬x ∨ u ∨ y ∨ z), then this is satisfiable if and only if the clause (y ∨ z ∨ u ∨ w) is satisfiable. In other words we can remove the variable x from consideration!)

16. You have invited 500 guests for your graduation party in a huge hall in Chateau Laurier. The guests needs to be seated, where each table has 20 seats. Unfortunately, the guests are not all friendly with each other; for sure you do not want two of your guests to sit on the same table if they are not friendly. Suppose you know the complete friendship matrix F (a 0-1 matrix indicating whether a pair of guests (i, j) are friendly or not). Can you devise a decision algorithm to decide whether it is possible to hold this party with the restriction of 20 per table and no two enemies land up on the same table! What about in general, where n is the number of guests and k is the number of guests per table and we can assume k divides n? Is this problem NP-Complete?

17. Consider two sets A and B, each having n integers in the range from 0 to cn, where c > 1 is a constant. Define the Cartesian sum of A and B as the set C given by C = {x + y : x ∈ A and y ∈ B}. Note that the integers in C are in the range 0 to 2cn. We want to find all the elements of C and the number of times each element of C is realized as a sum of elements in A and B. Provide an O(n log n) algorithm for this problem.

18. Given an undirected graph G = (V, E) in which each vertex v ∈ V has an associated positive weight w(v). For any vertex cover V′ ⊆ V, define the weight of the vertex cover w(V′) = ∑_{v∈V′} w(v). The goal is to find a vertex cover of minimum weight. Provide an Integer Linear Programming formulation for this problem. Then provide a relaxation of the integer program. Show that using the rounding techniques we can obtain a 2-approximation algorithm for this problem. Provide a formal proof that your solution is a 2-approximation.

19. Given a simple graph G = (V, E), we define a cut to be a partition of the vertex set V into two non-empty sets A and B, where A ∪ B = V and A ∩ B = ∅. An edge (a, b) ∈ E is said to cross the cut if a ∈ A and b ∈ B. The size of the cut corresponding to the partition (A, B) is defined to be the number of edges crossing the cut. The maximum cut problem is to find a partition of V such that the size of the cut is maximized. Consider the following algorithm: Step 1: Find any partition of V. Step 2: For every vertex v ∈ V, if v would have more edges crossing the cut if placed in the opposite partition, then move v to the opposite partition. Prove the following

(a) Prove that the above algorithm runs in polynomial time. What is the running time?

(b) Prove that the size of the cut produced by the above algorithm is at least half of the size of the maximum cut. (In other words its an 1/2-approximation algorithm.)

20. Although the 3CNF-SAT is N P-Complete, show that in polynomial time we can determine whether a boolean formula given in disjunctive normal form is satisfiable. The formula consists of n variables and k clauses. (A formula is in Disjunctive normal form, if clauses are joined by ORs and literals within a clause are joined by ANDs). You need to provide an algorithm whose running time is polynomial in n and k.

Show what is wrong with the following argument: Given a 3CNF-SAT, we can use distributive law to construct an equivalent formula in Disjunctive Normal Form. Here is an example: (x_{1} ∨ x_{2} ∨ x̄_{3}) ∧ (x̄_{1} ∨ x̄_{2}) = (x_{1} ∧ x̄_{1}) ∨ (x_{1} ∧ x̄_{2}) ∨ (x_{2} ∧ x̄_{1}) ∨ (x_{2} ∧ x̄_{2}) ∨ (x̄_{3} ∧ x̄_{1}) ∨ (x̄_{3} ∧ x̄_{2}). We have just now shown that DNF is in P. This implies that 3CNF-SAT is in P and P = N P.

116

Bibliography

[1] A. V. Aho, J. E. Hopcroft, and J. D. Ullman. The Design and Analysis of Computer Algorithms. Addison-Wesley Publishing Company, 1974.

[2] L. Aleksandrov and H. Djidjev. Linear algorithms for partitioning embedded graphs of bounded genus. SIAM J. Discret. Math., 9(1):129–150, February 1996.

[3] Lyudmil Aleksandrov, Hristo Djidjev, Hua Guo, and Anil Maheshwari. Partitioning planar graphs with costs and weights. J. Exp. Algorithmics, 11, February 2007.

[4] A. Andoni and P. Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In Foundations of Computer Science, 2006. FOCS ’06. 47th Annual IEEE Symposium on, pages 459–468, 2006.

[5] Bla Bollobs. Modern Graph Theory. Graduate texts in mathematics. Springer, Heidelberg, corrected edition, 1998.

[6] J. A. Bondy and U. S. R. Murty. Graph Theory with Applications. Elsevier, New York, 1976.

[7] Otakar Borvka. O jistém problému minimálním. Prce mor. prodovd. spol. v Brn III, 3:37–58, 1926.

[8] Otakar Borvka. Příspěvek k řešení otázky ekonomické stavby elektrovodních sítí. Elektrotechnickỳ obzor, 15:153–154, 1926.

[9] Andrei Z. Broder, Moses Charikar, Alan M. Frieze, and Michael Mitzenmacher. Min-wise independent permutations. Journal of Computer and System Sciences, 60:327–336, 1998.

[10] A.Z. Broder. On the resemblance and containment of documents. In Compression and Complexity of Sequences 1997. Proceedings, pages 21–29, 1997.

[11] Timothy M. Chan. Backwards analysis of the Karger-Klein-Tarjan algorithm for minimum spanning. Inf. Process. Lett., 67(6):303–304, 1998.

[12] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to Algorithms. The MIT Press, 3rd edition, 2009.

[13] Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to Algorithms, Third Edition. The MIT Press, 3rd edition, 2009.

[14] Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of Johnson and Lindenstrauss. Random Struct. Algorithms, 22(1):60–65, 2003.

[15] Mayur Datar and Piotr Indyk. Locality-sensitive hashing scheme based on p-stable distributions. In In SCG 04: Proceedings of the twentieth annual symposium on Computational geometry, pages 253–262. ACM Press, 2004.

[16] L.B. de Paula, R.S. Villaca, and M.F. Magalhaes. A locality sensitive hashing approach for conceptual classification. In Semantic Computing (ICSC), 2010 IEEE Fourth International Conference on, pages 408–413, 2010.

[17] E.W. Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik, 1(1):269–271, 1959.

[18] B. Dixon, M. Rauch, and R. Tarjan. Verification and sensitivity analysis of minimum spanning trees in linear time. SIAM Journal on Computing, 21(6):1184–1192, 1992.

[19] D.P. Dubhashi and A. Panconesi. Concentration of measure for the analysis of randomized algorithms. Cambridge University Press, 2009.

[20] Shimon Even. Graph Algorithms. W. H. Freeman & Co., New York, NY, USA, 1979.

[21] Greg N. Frederickson. Fast algorithms for shortest paths in planar graphs, with applications. SIAM J. Comput., 16(6):1004–1022, 1987.

[22] Greg N. Frederickson. Optimal algorithms for tree partitioning. In Proceedings of the Second Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’91, pages 168–177, Philadelphia, PA, USA, 1991. Society for Industrial and Applied Mathematics.

[23] D. Gale and L. S. Shapley. College admissions and the stability of marriage. The American Mathematical Monthly, 69(1):9–15, 1972.

[24] Aristides Gionis, Piotr Indyk, and Rajeev Motwani. Similarity search in high dimensions via hashing. pages 518–529, 1997.

[25] D. Gorisse, M. Cord, and F. Precioso. Locality-sensitive hashing for chi2 distance. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(2):402–409, 2012.

[26] Torben Hagerup. An even simpler linear-time algorithm for verifying minimum spanning trees. In Christophe Paul and Michel Habib, editors, Graph-Theoretic Concepts in Computer Science, volume 5911 of Lecture Notes in Computer Science, pages 178–189. Springer Berlin Heidelberg, 2010.

[27] Torben Hagerup and Christine Rüb. A guided tour of Chernoff bounds. Inf. Process. Lett., 33(6):305–308, 1990.

[28] Frank Harary. Graph theory. Addison-Wesley, 1991.

[29] D. Harel and R. Tarjan. Fast algorithms for finding nearest common ancestors. SIAM Journal on Computing, 13(2):338–355, 1984.

[30] Dov Harel. A linear time algorithm for the lowest common ancestors problem. In Foundations of Computer Science, 1980., 21st Annual Symposium on, pages 308–319, 1980.

[31] S. Hu. Efficient video retrieval by locality sensitive hashing. In Acoustics, Speech, and Signal Processing, 2005. Proceedings. (ICASSP ’05). IEEE International Conference on, volume 2, pages 449–452, 2005.

[32] Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing, STOC ’98, pages 604–613, New York, NY, USA, 1998. ACM.

[33] Robert W. Irving. An efficient algorithm for the ”stable roommates” problem. J. Algorithms, 6(4):577–595, 1985.

[34] Woojay Jeon and Yan-Ming Cheng. Efficient speaker search over large populations using kernelized locality-sensitive hashing. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, pages 4261–4264, 2012.

[35] Yushi Jing and S. Baluja. Visualrank: Applying pagerank to large-scale image search. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30(11):1877–1890, 2008.

[36] K.A. Kala and K. Chitharanjan. Locality sensitive hashing based incremental clustering for creating affinity groups in hadoop; hdfs - an infrastructure extension. In Circuits, Power and Computing Technologies (ICCPCT), 2013 International Conference on, pages 1243–1249, 2013.

[37] Zixiang Kang, Wei Tsang Ooi, and Qibin Sun. Hierarchical, non-uniform locality sensitive hashing and its application to video identification. In Multimedia and Expo, 2004. ICME ’04. 2004 IEEE International Conference on, volume 1, pages 743–746 Vol.1, 2004.

[38] David R. Karger, Philip N. Klein, and Robert E. Tarjan. A randomized linear-time algorithm to find minimum spanning trees. J. ACM, 42(2):321–328, March 1995.

[39] V. King. A simpler minimum spanning tree verification algorithm. Algorithmica, 18(2):263– 270, 1997.

[40] Valerie King. A simpler minimum spanning tree verification algorithm. In SelimG. Akl, Frank Dehne, Jrg-Rdiger Sack, and Nicola Santoro, editors, Algorithms and Data Structures, volume 955 of Lecture Notes in Computer Science, pages 440–448. Springer Berlin Heidelberg, 1995.

[41] Philip N. Klein and Robert E. Tarjan. A randomized linear-time algorithm for finding minimum spanning trees. In Proceedings of the twenty-sixth annual ACM symposium on Theory of computing, STOC ’94, pages 9–15, New York, NY, USA, 1994. ACM.

[42] Jon Kleinberg and Eva Tardos. Algorithm Design. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2005.

[43] Donald E. Knuth. The art of computer programming, volume 1-3. Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, USA, 1998.

[44] J. Komlos. Linear verification for spanning trees. In Foundations of Computer Science, 1984. 25th Annual Symposium on, pages 201–206, 1984.

[45] J. Komls. Linear verification for spanning trees. Combinatorica, 5(1):57–65, 1985.

[46] D. Kozen. The design and analysis of algorithms. Springer, 1992.

[47] B. Kulis and K. Grauman. Kernelized locality-sensitive hashing. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(6):1092–1104, 2012.

[48] Richard J. Lipton and Robert E. Tarjan. A separator theorem for planar graphs. SIAM Journal on Applied Mathematics, 36(2):177–189, 1979.

[49] L. Lovász and M.D. Plummer. Matching Theory. Akadémiai Kiadó, Budapest, 1986. Also published as Vol. 121 of the North-Holland Mathematics Studies, North-Holland Publishing, Amsterdam.

[50] A. Maheshwari and M. Smid. Introduction to Theory of Computation. Free Online, 2012.

[51] Udi Manber. Finding similar files in a large file system. In in Proceedings of the USENIX Winter 1994 Technical Conference, pages 1–10, 1994.

[52] P.L. Meyer. Introductory probability and statistical applications. Addison-Wesley, Boston, MA, USA, 1970.

[53] Gary L. Miller, Shang-Hua Teng, William Thurston, and Stephen A. Vavasis. Separators for sphere-packings and nearest neighbor graphs. J. ACM, 44(1):1–29, January 1997.

[54] Tanmoy Mondal, Nicolas Ragot, Jean-Yves Ramel, and Umapada Pal. A fast word retrieval technique based on kernelized locality sensitive hashing. In Document Analysis and Recognition (ICDAR), 2013 12th International Conference on, pages 1195–1199, 2013.

[55] Rajeev Motwani and Prabhakar Raghavan. Randomized algorithms. Cambridge University Press, New York, NY, USA, 1995.

[56] Rajeev Motwani and Prabhakar Raghavan. Randomized algorithms. Cambridge University Press, New York, NY, USA, 1995.

[57] Jaroslav Neetil, Eva Milkov, and Helena Neetilov. Otakar borvka on minimum spanning tree problem translation of both the 1926 papers, comments, history. Discrete Mathematics, 233(13):3 – 36, 2001. ¡ce:title¿Czech and Slovak 2¡/ce:title¿.

[58] Anand Rajaraman and Jeffrey David Ullman. Mining of Massive Datasets. Cambridge University Press, New York, NY, USA, 2011.

[59] Z. Rasheed, H. Rangwala, and D. Barbara. Lsh-div: Species diversity estimation using locality sensitive hashing. In Bioinformatics and Biomedicine (BIBM), 2012 IEEE International Conference on, pages 1–6, 2012.

[60] Kenneth H. Rosen. Discrete Mathematics and Its Applications. McGraw-Hill Higher Education, 5th edition, 2002.

[61] Baruch Schieber and Uzi Vishkin. On finding lowest common ancestors: Simplification and parallelization. In JohnH. Reif, editor, VLSI Algorithms and Architectures, volume 319 of Lecture Notes in Computer Science, pages 111–123. Springer New York, 1988.

[62] V. Strassen. Gaussian elimination is not optimal. Numerische Mathematik, 13:354–356, 1969.

[63] Robert Tarjan. Data Structures and Network Algorithms, volume 44, chapter 6. Minimum Spanning Trees, pages 71–83. SIAM, 1983.

[64] Robert Endre Tarjan. Efficiency of a good but not linear set union algorithm. J. ACM, 22(2):215–225, April 1975.

[65] Robert Endre Tarjan. Applications of path compression on balanced trees. J. ACM, 26(4):690– 715, October 1979.

[66] Qiang Wang, Zhiyuan Guo, Gang Liu, and Jun Guo. Entropy based locality sensitive hashing. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, pages 1045–1048, 2012.